Calibre behind NGINX causes BAD REQUEST during sync setup

[MasterEmit]

After struggling around I finally was able to get calibre running at an internet server (with PLESK). So calibre was installed with docker and the port for the standard GUI bound via NGINX proxy to an address (https://calibre.somedomainname.net/) and after setting up the calibre network server with mandatory username and password and bound this port to another address with NGINX proxy I was able to get a JSON file from https://calibrerest.somedomainname.net/ajax/library-info. Of course before the standard auth dialog of the browser appeared, where I needed to enter the credentials entered before.

Now I tried to setup the calibre sync in the app, but I always get a BAD REQUEST.

c7.k: Bad Request
	at c7.h.b(r8-map-id-07fbb00b34f24ab1f070b9a277f5ffdf1afe2b069e1895198727d74fc16daeba:147)
	at c7.h.d(r8-map-id-07fbb00b34f24ab1f070b9a277f5ffdf1afe2b069e1895198727d74fc16daeba:355)
	at c7.e.call(r8-map-id-07fbb00b34f24ab1f070b9a277f5ffdf1afe2b069e1895198727d74fc16daeba:44)
	at java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:651)
	at java.lang.Thread.run(Thread.java:1119)```

Preferences:

```Bookshelf.CurrentBookshelf=Alle Bücher
bookList.style.current=995c246e-4356-41c1-aa98-5e3a4f855834
booklist.context.menu=0
booklist.fastscroller.overlay=3
booklist.rebuild.state=0
bookshelf.default.id=3
calibre.enabled=true
calibre.fields.update.Book:fileSpec:0=1
calibre.fields.update.author_list=2
calibre.fields.update.clb_book_id=1
calibre.fields.update.clb_book_main_format=1
calibre.fields.update.clb_book_uuid=1
calibre.fields.update.date_published=1
calibre.fields.update.description=1
calibre.fields.update.format=1
calibre.fields.update.identifier_list=2
calibre.fields.update.language=1
calibre.fields.update.notes=1
calibre.fields.update.publisher_list=2
calibre.fields.update.read=1
calibre.fields.update.read_end=1
calibre.fields.update.read_start=1
calibre.fields.update.series_list=2
calibre.fields.update.tag_list=2
calibre.fields.update.thumbnail.0=1
calibre.fields.update.title=1
calibre.folder=content://com.android.externalstorage.documents/tree/primary%3Acalibre
calibre.host.password=********
calibre.host.url=https://calibrerest.somedomain.net
calibre.host.user=********
calibre.timeout.connect=10
calibre.timeout.read=10
camera.torch.status=false
camera.zoom.control.value=0.0
edit.book.isbn.checks=1
edit.book.tab.externalId=false
edit.capitalize.title=1
fields.update.usage.Book:fileSpec:0=1
fields.update.usage.Book:fileSpec:1=1
fields.update.usage.Book:fileSpec:2=1
fields.update.usage.Book:fileSpec:3=1
fields.update.usage.author_list=2
fields.update.usage.color=1
fields.update.usage.date_published=1
fields.update.usage.description=1
fields.update.usage.first_publication=1
fields.update.usage.format=1
fields.update.usage.identifier_list=2
fields.update.usage.isbn=1
fields.update.usage.language=1
fields.update.usage.list_price=1
fields.update.usage.list_price_currency=1
fields.update.usage.pages=1
fields.update.usage.print_run=1
fields.update.usage.publisher_list=2
fields.update.usage.series_list=2
fields.update.usage.tag_list=2
fields.update.usage.thumbnail.0=1
fields.update.usage.thumbnail.1=1
fields.update.usage.thumbnail.2=1
fields.update.usage.thumbnail.3=1
fields.update.usage.title=1
fields.update.usage.title_original_lang=1
fields.update.usage.toc_list=2
fields.update.usage.translation_orig_lang=1
fields.visibility=9223372036854775807
fields.visibility.loaned_to=true
language.mapping.cached.deu=true
language.mapping.cached.eng=true
network.allow.metered=true
scan.queue=
search.reformat.color=true
search.reformat.format=true
sort.title.reordered=true
startup.backupCountdown=1
startup.startCount=4
startup.startCountdown=1
storage.volume.index=0
ui.dialog.mode=2
ui.fastscroller.draghandle=0
ui.menu.context.mode=2
ui.theme=0
ui.theme.color=0

So from the browser the url "https://calibrerest.somedomain.net/ajax/library-info" shows as expected:

{"library_map": {"books": "books"}, "default_library": "books"}

[tfonteyn]

I currently have no access to my calibre server, so my apologies but I will only be able to look into the issue maybe just before Christmas, or somewhat later.

Just a quick tip: be aware that only "basic" authentication is supported right now. My reasoning at the time was it would be usually on home networks and/or using ssl.

[tfonteyn]

I double checked my source code. It's the login which fails.
The connection validation does a simple connection to https://calibrerest.somedomain.net
while setting the basic authentication header with your username and password. Nothing more.

I've tested (quite some time back now) direct with calibre and with an Apache in front, but never with nginx. I'll have a go with that setup by/after xmas.

One more thing to check: have you added the CA certificate to NTMB?

[MasterEmit]

Hi @tfonteyn - the address is signed with LetsEncrypt and I would guess, that if there is no CA cert uploaded to the Android app, the default Android TrustStore is used. But I did not checked on that at the sourcecode. I will try to debug the issue today :-)

[MasterEmit]

So. The problem is really the usage of digest-authentication. The bigger problem is, I could not find in my solution a way, to hand-over any argument during the docker startup to the calibre server. Of course I could fork the docker project and make changes, but this is really to much for "I just want a possibility to sync different devices". Setting up calibre was already "ugly". I made some small, fix changes to see, if using OkHttp with https://github.com/rburgst/okhttp-digest could solve the issue and yes, it would. The changed version of CalibreContentServer.validateConnection was successful after that (and I could see the response JSON in the debugger).

    private OkHttpClient createHttpClient() {
        final OkHttpClient.Builder builder = ServiceLocator
                .getInstance()
                .getOkHttpClient()
                .newBuilder()
                .connectTimeout(connectTimeoutInMs, TimeUnit.MILLISECONDS)
                .readTimeout(readTimeoutInMs, TimeUnit.MILLISECONDS);

        if (sslContext != null) {
            builder.setSocketFactory$okhttp(sslContext.getSocketFactory());
            if (hostnameVerifier != null) {
                builder.setHostnameVerifier$okhttp(hostnameVerifier);
            }
        }

        final Map<String, CachingAuthenticator> authCache = new ConcurrentHashMap<>();

        final Credentials credentials = new Credentials(this.userName, this.passWord);
        final BasicAuthenticator basicAuthenticator = new BasicAuthenticator(credentials);
        final DigestAuthenticator digestAuthenticator = new DigestAuthenticator(credentials);

        DispatchingAuthenticator authenticator = new DispatchingAuthenticator.Builder()
                .with("basic", basicAuthenticator)
                .with("digest", digestAuthenticator)
                .build();

        builder.authenticator(new CachingAuthenticatorDecorator(authenticator, authCache));
        builder.addInterceptor(new AuthenticationCacheInterceptor(authCache, new DefaultRequestCacheKeyProvider()));

        return builder.build();
    }

and then

    public boolean validateConnection(@NonNull final Context context)
            throws StorageException,
                   IOException {
        final String url = String.format(GET_LIBRARY_INFO, serverUri);

        final Request.Builder builder = new Request.Builder()
                .url(url)
                .header(HttpConstants.HOST, new URL(url).getHost())
                .header(HttpConstants.USER_AGENT,
                        HttpConstants.BROWSER_USER_AGENT)

                .header(HttpConstants.ACCEPT_ENCODING,
                        HttpConstants.ACCEPT_ENCODING_GZIP)

                .header(HttpConstants.CONNECTION,
                        HttpConstants.CONNECTION_KEEP_ALIVE);
        final Request request = builder.build();
        final OkHttpClient client = createHttpClient();
        final Call call = client.newCall(request);

        final Response response = call.execute();
        final String string = response.body().string();
        return !string.isEmpty();
    }

But of course no error handling and anything else around that. Just a short dirty test. If I understood it right, basic and digest auth would work with the same setup.

Would switching completely to OkHttp an option?

[tfonteyn]

Would switching completely to OkHttp an option?

During 7.7.0 devlopment, I have in fact already done that in a local branch due to OpenLibrary having problems with my 'old' method.

But OkHttp turned out to be strangely problematic with several websites which were not behaving well. I ended up using OkHttp for images only.

Let me rebase that branch, and I'll check what I did with calibre - in theory, there is no such issue as above with Calibre of course.

[tfonteyn]

if there is no CA cert uploaded to the Android app, the default Android TrustStore is used

correct :)

ok - I rebased the old code, and split of the Calibre changes and the underlying changes, and uploaded that branch:

https://github.com/tfonteyn/NeverTooManyBooks/tree/OkHttp

I comes with HttpCall which replaces the Future-based client + the changes in the CalibreContentServer (and testcase).

If you check that branch out, building should work fine.
Then try a connection to your Calibre server. If that works, then sync should work as well.

If not, I'll need to integrate your changes of course.

Please keep in mind:

• this branch has NOT been tested on my personal network/data. Only on a test-setup.
• do NOT run this branch as a permanent solution, this is based of the 'dev' branch after all.

But if you/me can get this branch to work properly with you Calibre setup, then I will be very happy to merge it into 'dev' and get it into the next release. Obviously I will want to test if it still works fine on my direct and on an Apache fronted setup as well.

Timeframe:

• I can't run major tests right now (and none with Calibre) but should be able to start in a week, probably after xmas.
• release: providing no major issues with the above and/or with the other stuff scheduled, this should happen before New Year.
  As always... no promises, but I'll try :)

[MasterEmit]

We are using OkHttp a lot in big projects. Maybe let's have a chat where the problems were located. I can help with debugging or asking some of my team mates.

[MasterEmit]

I will check the branch and no hurry :)

[tfonteyn]

appreciate the offer :) that's something for next year then. In a nutshell the issues I had was that some of the search-sites needed special handling when using okhttp, while my future-client 'just worked' (with the exception of the dnb cert issue... a story for another day)

[MasterEmit]

So. It worked. Attached find a patch file which I used. At least from a very rough test it seems to work. Sadly basic and digest auth did not worked with the setup together, as I thought. Anyway, this would be easy to fix with a checkbox in the settings or so. I just commented out the basic auth parts.

okhttptest.patch

It is not really the topic here, but what is the purpose of the virtual libraries at Calibre? Is this just some additional field at Calibre and I would not need to sync shelf for shelf? And is syncing really syncing or more export and import? At the moment I have no real data at Calibre. Just the original Android app from you with some mass ISBN scanning tests - which were quite satisfying :-)

[tfonteyn]

brilliant - I'll try to check it as soon as possible.

basic and digest

no big deal, I can add that to the settings

virtual libraries

This is a calibre feature. It's basically a labelled search-expression. In Calibre they show up as tabs.
I've got for example on defined with:

• tags:"=Non-Fiction"

so all my non-fiction shows up under that tab.

If my little extension code is installed, you can map a virtual library to a Bookshelf.
Without them the whole calibre library is mapped to a single Bookshelf. Of course you can still use NTMB filters on tags to get the same result. It's a bit historical now... I implemented Calibre sync long before I implemented tags of my own.

[tfonteyn]

I missed this question:

And is syncing really syncing or more export and import?

It's a real sync based on last-modification date on a book-by-book basis.
It is (on purpose) done using two actions which are a bit misleadingly named "export" (ntmb->Calibre) and "import" (Calibre->ntmb).

The reason for the latter is both logical and dumb 😇
The app uses identical logic and similar/same code for any type of export/import; including some of the UI/options.

So, in short:

• 'export' pushes changes made in ntmb to Calibre
• 'import' pulls changes made in Calibre into ntmb
• in both cases only changes are propagated so it's a lot faster after the initial sync.