Booklore Installer

[Pinewood538]

Booklore only provides a docker image for installation, so this was a bit tricky.
The basic steps are:

1. Install the dependencies: Java, MariaDB, Caddy, crane (for help extracting the docker image)
2. Setup MariaDB
3. Extract the backend .jar and frontend from the docker image using crane (to avoid having to build Booklore locally)
4. Initialize the backend and serve the frontend with Caddy

-Unfortunately MariaDB specifically is required, so the installation directory is bulky (about 1GB, of which MariaDB is ~730MB). I've tried to slim down MariaDB after the download by removing some usused plugins and binaries, which did help.

-It is possible to extract the necessary booklore files from their docker image without using crane, but the logic was tricky and the download links seem fragile. crane makes the logic a lot simpler and future-proof if booklore decides to change something on their end.

-I don't know much about setting up Caddy. Did my best with mostly trial and error on how to set up the config and it seems to work. AI wasn't much help. It's a very small block of code so please adjust it if necessary.

-considering all the dependencies, I kept everything in the Booklore installation folder for easy cleanup and to avoid cluttering other directories

Thanks for helping in the discord with how to get started. I did my best to follow the advice given (not binding to localhost, etc). I also tried to follow along the basic format of the other scripts here and stole some code from there (like the little function to generate an open port). I looked at the ports other apps were using here and tried to pick a range outside of them.

I'm a novice at this stuff, so please give feedback or adjust it however you like before merging.

booklore.sh

#!/bin/bash

mkdir -p "$HOME/.logs/"
export log="$HOME/.logs/booklore.log"
touch "$log"
SUBNET_IP=$(cat "$HOME/.install/subnet.lock")
BASE="$HOME/booklore"

function java_install() {
    echo "Downloading Java 21..."
    curl -sL "https://api.adoptium.net/v3/binary/latest/21/ga/linux/x64/jre/hotspot/normal/eclipse" -o /tmp/jre21.tar.gz
    tar -xzf /tmp/jre21.tar.gz --strip-components=1 -C "$BASE/java"
    rm /tmp/jre21.tar.gz
}

function maria_install() {
    echo "Downloading MariaDB..."
    curl -sL "https://archive.mariadb.org/mariadb-12.1.2/bintar-linux-systemd-x86_64/mariadb-12.1.2-linux-systemd-x86_64.tar.gz" -o /tmp/mariadb.tar.gz
    tar -xzf /tmp/mariadb.tar.gz --strip-components=1 -C "$BASE/mariadb"
    rm /tmp/mariadb.tar.gz

    # remove useless directories (tests and such)
    rm -rf "$BASE/mariadb"/{mariadb-test,sql-bench,include,man,docs}

    # remove massive storage engines not used by Booklore (MyRocks, Mroonga, etc.)
    rm -f "$BASE/mariadb/lib/plugin"/{ha_rocksdb.so,ha_mroonga.so,ha_spider.so,ha_connect.so,ha_oqgraph.so,libgalera_smm.so}

    # remove some unused binaries (mariadb-dump is kept instead of mariadb-backup)
    rm -f "$BASE/mariadb/bin/"{mariadb-backup,garbd,mariadb-ldb,sst_dump,mariadb-slap,mariadb-test,mariadb-client-test,aria_s3_copy,mbstream}
}

function caddy_install() {
    echo "Downloading Caddy..."
    curl -sL "https://caddyserver.com/api/download?os=linux&arch=amd64" -o "$BASE/bin/caddy"
    chmod +x "$BASE/bin/caddy"
}

# crane is helpful to extract the necessary files from the Booklore docker image.
# it's possible without it, but the hardcoded links seemed fragile, and crane makes
# the logic MUCH more straightforward anyway.
function crane_install() {
    echo "Downloading crane (for Booklore docker extraction)..."
    curl -sL "https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz" -o /tmp/crane.tar.gz
    tar -xzf /tmp/crane.tar.gz -C "$BASE/bin" crane
    rm /tmp/crane.tar.gz
}

function port() {
    LOW_BOUND=$1
    UPPER_BOUND=$2
    comm -23 <(seq ${LOW_BOUND} ${UPPER_BOUND} | sort) <(ss -Htan | awk '{print $4}' | cut -d':' -f2 | sort -u) | shuf | head -n 1
}

function _docker_extract() {
    local tmp=$(mktemp -d)
    trap 'rm -rf "$tmp"' EXIT
    echo "Extracting Booklore via crane..."

    # extract the necessary files from the docker image using crane
    "$BASE/bin/crane" export ghcr.io/booklore-app/booklore:latest - | \
    tar -xf - -C "$tmp" \
        --wildcards \
        "app/*.jar" \
        "usr/share/nginx/html/*" \
        >> "$log" 2>&1

    # verify and move the files to the install directory. crane gives full paths so the jar is likely at /app/app.jar
    if [[ -f "$tmp/app/app.jar" && -d "$tmp/usr/share/nginx/html" ]]; then
        echo "Files verified. Installing to $BASE..."
        mkdir -p "$BASE/app"

        # Backup old jar
        [[ -f "$BASE/app/app.jar" ]] && mv "$BASE/app/app.jar" "$BASE/app/app.jar.bak"

        # Move new jar
        mv "$tmp/app/app.jar" "$BASE/app/app.jar"

        # Move new frontend
        rm -rf "$BASE/public"
        mv "$tmp/usr/share/nginx/html" "$BASE/public"
    else
        echo "ERROR: crane export failed or file structure is different than expected. script needs to be updated."
        rm -rf "$tmp"
        exit 1
    fi

    rm -rf "$tmp"
}

function _install() {
    mkdir -p "$BASE/"{app,bin,bookdrop,books,config,java,public,tmp,mariadb,mariadb/data,mariadb/tmp}

    # dependency installs
    java_install
    caddy_install
    crane_install
    maria_install


    DB_PORT=$(port 17000 17200)
    JAVA_PORT=$(port 17201 17400)
    PUBLIC_PORT=$(port 17401 17999)
    DB_PASS=$(openssl rand -hex 16)

    # configure MariaDB
    echo "Setting up MariaDB on $SUBNET_IP:$DB_PORT..."
    cat <<EOF > "$BASE/mariadb/my.cnf"
[mysqld]
port = $DB_PORT
bind-address = $SUBNET_IP
socket = $BASE/mariadb/mysql.sock
datadir = $BASE/mariadb/data
tmpdir = $BASE/mariadb/tmp
skip-log-bin
innodb_log_file_size = 16M
innodb_buffer_pool_size = 128M
pid-file = $BASE/mariadb/pid
log-error = $BASE/mariadb/error.log
EOF

    echo "Initializing database..."
    "$BASE/mariadb/scripts/mariadb-install-db" --defaults-file="$BASE/mariadb/my.cnf" --basedir="$BASE/mariadb" >> "$log" 2>&1

    cat <<EOF > "$HOME/.config/systemd/user/booklore-db.service"
[Unit]
Description=Booklore MariaDB
[Service]
ExecStart=$BASE/mariadb/bin/mariadbd --defaults-file=$BASE/mariadb/my.cnf
Restart=on-failure
[Install]
WantedBy=default.target
EOF

    systemctl --user daemon-reload
    systemctl --user enable --now booklore-db

    # Wait for the DB to be ready
    if ! "$BASE/mariadb/bin/mariadb-admin" --socket="$BASE/mariadb/mysql.sock" --wait=30 ping --silent; then
        echo "ERROR: DB failed to start."; tail -n 20 "$BASE/mariadb/error.log"; exit 1
    fi

    # Initialize and try root user first, then current user if that fails
    SQL="CREATE DATABASE IF NOT EXISTS booklore; GRANT ALL PRIVILEGES ON booklore.* TO 'booklore'@'%' IDENTIFIED BY '$DB_PASS'; FLUSH PRIVILEGES;"
    "$BASE/mariadb/bin/mariadb" --socket="$BASE/mariadb/mysql.sock" -u root -e "$SQL" 2>/dev/null || \
    "$BASE/mariadb/bin/mariadb" --socket="$BASE/mariadb/mysql.sock" -u "$(whoami)" -e "$SQL"

    # Extract content from Booklore docker via crane
    _docker_extract

    # Booklore backend configuration. use the proper subnet IP instead of localhost
    cat <<EOF > "$BASE/application.yml"
server:
  port: $JAVA_PORT
  address: $SUBNET_IP
spring:
  datasource:
    url: jdbc:mariadb://$SUBNET_IP:$DB_PORT/booklore
    username: booklore
    password: $DB_PASS
app:
  path-config: $BASE/config
  bookdrop-folder: $BASE/bookdrop
EOF

    # Caddy config for frontend
    cat <<EOF > "$BASE/Caddyfile"
:$PUBLIC_PORT {
    encode gzip
    root * $BASE/public

    @backend path /api/* /oauth2/* /ws* /swagger-ui/* /v3/api-docs/*

    handle @backend {
        reverse_proxy $SUBNET_IP:$JAVA_PORT {
            header_up Host {host}
            header_up X-Real-IP {remote_host}
            header_up X-Forwarded-For {remote_host}
            header_up X-Forwarded-Host {host}
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-Port {http.request.port}
        }
    }

    handle {
        try_files {path} /index.html
        file_server
    }
}
EOF

    # systemd services
    cat <<EOF > "$HOME/.config/systemd/user/booklore.service"
[Unit]
Description=Booklore Backend (Java)
After=booklore-db.service
[Service]
WorkingDirectory=$BASE
ExecStart=$BASE/java/bin/java -Xmx1g -Djava.io.tmpdir=$BASE/tmp -jar $BASE/app/app.jar --spring.config.additional-location=file:$BASE/application.yml
Restart=on-failure
[Install]
WantedBy=default.target
EOF

    cat <<EOF > "$HOME/.config/systemd/user/booklore-web.service"
[Unit]
Description=Booklore Frontend (Caddy)
After=booklore.service
[Service]
ExecStart=$BASE/bin/caddy run --config $BASE/Caddyfile --adapter caddyfile
Restart=on-failure
[Install]
WantedBy=default.target
EOF

    systemctl --user enable --now booklore
    systemctl --user enable --now booklore-web
    touch "$HOME/.install/.booklore.lock"
    echo "Booklore has been installed and should be running at http://$(hostname -f):$PUBLIC_PORT (give it a minute or so to initialize)"
}

function _remove() {
    systemctl --user stop booklore booklore-web booklore-db
    systemctl --user disable booklore booklore-web booklore-db

    rm -f "$HOME/.config/systemd/user/booklore"*.service
    rm -f "$HOME/.install/.booklore.lock"
    rm -rf "$BASE/"

    systemctl --user daemon-reload
    echo "Uninstall Complete."
}

function _upgrade() {
    if [ ! -f "$HOME/.install/.booklore.lock" ]; then
        echo "Booklore is not installed."
        exit 1
    fi

    echo "Upgrading Booklore..."

    systemctl --user stop booklore booklore-web
    _docker_extract
    systemctl --user start booklore booklore-web

    echo "Upgrade complete. Services restarted."
}

echo 'This is unsupported software. You will not get help with this, please answer `yes` if you understand and wish to proceed'
if [[ -z ${eula} ]]; then
    read -r eula
fi

if ! [[ $eula =~ yes ]]; then
  echo "You did not accept the above. Exiting..."
  exit 1
else
  echo "Proceeding with installation"
fi

echo "Welcome to the Booklore Installer"
echo ""
echo "What do you like to do?"
echo "Logs are stored at ${log}"
echo "install = Install Booklore"
echo "upgrade = Upgrade Booklore to the latest version"
echo "uninstall = Completely removes Booklore (including the database! backup yourself if needed)"
echo "exit = Exit"

while true; do
    read -r -p "Enter choice: " choice
    case $choice in
        "install")
            _install
            break
            ;;
        "upgrade")
            _upgrade
            break
            ;;
        "uninstall")
            _remove
            break
            ;;
        "exit")
            break
            ;;
        *)
            echo "Unknown Option."
            ;;
    esac
done
exit

[Pinewood538]

Looks like I forgot some headers (fixed in the above script now). Still going through the rest of the docs.

EDIT: after adding the headers from that link, I'm still struggling to sync to my kobo. my kobo sees the book it needs to sync, but the download fails. need to debug this some more.

EDIT2: the download link generated by booklore appears to remove the port from the URL. setting forward-headers-strategy=framework adds a port to the url, but it's the wrong port (the backend one, not the one for caddy). need to learn how this stuff works

EDIT3: have to give up for tonight. every permutation of the caddyfile and application.yml i've tried either sends the wrong port OR no port in the download link. will try again tomorrow.

EDIT4: the KoboUrlBuilder.java in Booklore's repo has this statement:

if (IP_ADDRESS_PATTERN.matcher(host).matches() || "localhost".equals(host)) {
	builder.port(port);
}

which wants either an IP address or localhost.

So I'm assuming that statement will never evaluate to true given the URL sent by me (XX.whatever.itbsy.design:PORT, etc...)

If I hardcode the IP and port into the caddyfile, it syncs fine, but that's obviously not a good long-term solution.

EDIT5: there's a PR that will hopefully simplify some of this script by merging the frontend into the jar.
The above issues with KoboUrlBuilder have already been reported on Booklore's github, so hopefully that is fixed sometime soon.

There is another issue with Kobo sending invalid headers (komga docs) that do not include the port:

The Kobo eReader does not send valid HTTP Host header, which could break the covers and file download in certain circumstances. Komga will try its best to automatically fix the incorrect Kobo headers, but in some specific cases it's not possible.
If you experience some issues downloading covers or files from the Kobo, you will need to configure the Kobo Sync external port (in Server Settings) to the port that your Kobo is accessing. Normally it should be what you have configured in api_endpoint.

This can be resolved by hardcoding the port into the caddyfile (same fix as above), but I've also asked for this option to be included in booklore as well.