deploy.sh: hardcoded paths and silent failures break sensor deployment on non-default installs

[zadzagy]

Pre-submission checklist

• Searched existing issues — found related reports in Challenges with Deploying Sensor to the Hive after upgrading from version 22.04.0 to 24.04.0 #1544 and Sensors not sending data to the hive #1543 (401 errors after sensor deployment)
• Searched discussions — found related path discussion in T-Pot preview dies and throws errors at restart - /opt/tpot/etc/tpot.yml: No such file or directory - Unit tpot.service could not be found. #1413 and credential confusion in check web user credentials? #1529
• Reviewed T-Pot README and Troubleshoot section
• Reviewed Docker and Elastic documentation

Basic Support Information

• Hive OS: Ubuntu 24.04 LTS
  • uname -a:
    • Linux TPotServer 6.17.0-14-generic include ews.cfg  #14~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 15 15:52:10 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
  • lsb_release -a:
    • No LSB modules are available.
    • Distributor ID: Ubuntu
    • Description: Ubuntu 24.04.3 LTS
    • Release: 24.04
    • Codename: noble
• Sensor OS: Debian 13 (Trixie)
• T-Pot version: 24.04.x
• Architecture: VMware vSphere VMs (on-premises)
• Hive install path: /opt/tpot/ (installer placed files here, not ~/tpotce/)
• Sensor install path: ~/tpotce/
• Installation age: Fresh install, issue encountered during first deploy.sh run
• Modifications: No modifications to scripts or configs prior to encountering the issue. Symlink (ln -s /opt/tpot ~/tpotce) was created as a workaround after the initial failure.
• Container status: All containers running normally on both Hive and Sensor after manual credential fix (see Workaround below)

Summary

deploy.sh and deploy.yml hardcode ~/tpotce/ for all file paths. When the Hive is installed to a different location (e.g., /opt/tpot/), multiple operations fail — some visibly, some silently. The end result is that deploy.sh appears to complete successfully, but the sensor cannot authenticate to the Hive (continuous 401 errors) because the credential write to lswebpasswd on the Hive failed without any error handling.

This may be related to the 401 authentication issues reported in #1544 and #1543.

Root Cause

All paths in both deploy.sh and deploy.yml are hardcoded to ~/tpotce/:

deploy.sh line 7:

myENV_FILE="$HOME/tpotce/.env"

deploy.yml lines 7-11:

vars:
  local_nginx_cert_path: "~/tpotce/data/nginx/cert/nginx.crt"
  remote_cert_path: "~/tpotce/data/hive.crt"
  remote_sensor_yml_path: "~/tpotce/compose/sensor.yml"
  env_file_path: "~/tpotce/.env"

When the Hive is installed to /opt/tpot/ (which can happen depending on the installation method), every path reference in these files is wrong.

This is related to the path issues discussed in #1413.

What Fails

1. HIVE type check fails (deploy.sh line 23):

if ! grep -q 'TPOT_TYPE=HIVE' "$HOME/tpotce/.env";

The script immediately exits with "This script is only supported on HIVE installations" even though the Hive is correctly configured — just at a different path.

2. Ansible certificate copy fails (deploy.yml lines 18-20):

- name: Copy nginx.crt from local to remote host
  ansible.builtin.copy:
    src: "{{ local_nginx_cert_path }}"   # ~/tpotce/data/nginx/cert/nginx.crt
    dest: "{{ remote_cert_path }}"

Could not find or access '~/tpotce/data/nginx/cert/nginx.crt' on the Ansible Controller.

3. Credential write to lswebpasswd fails silently (deploy.sh lines 136-147):

This is the most damaging failure. After Ansible completes successfully, deploy.sh writes sensor credentials to lswebpasswd:

: > "${HOME}"/tpotce/data/nginx/conf/lswebpasswd
for i in $myENV_LS_WEB_USER;
  do
    if [[ -n $i ]];
      then
        echo -n "$i" | base64 -d -w0
        echo
        echo -n "$i" | base64 -d -w0 | tr -d '\n' >> ${HOME}/tpotce/data/nginx/conf/lswebpasswd
        echo >> ${HOME}/tpotce/data/nginx/conf/lswebpasswd
      fi
done

If ${HOME}/tpotce/data/nginx/conf/lswebpasswd doesn't exist, this fails with errors that are easy to miss in the output:

./deploy.sh: line 138: /home/user/tpotce/data/nginx/conf/lswebpasswd: No such file or directory
./deploy.sh: line 146: /home/user/tpotce/data/nginx/conf/lswebpasswd: No such file or directory
./deploy.sh: line 147: /home/user/tpotce/data/nginx/conf/lswebpasswd: No such file or directory

The script exits with code 0 (success) despite these failures. The user sees the Ansible portion complete (ok=10, failed=0) and assumes everything worked.

The sensor reboots and starts forwarding logs, but the Hive's nginx has no credentials to authenticate the sensor. The result is continuous 401 errors in the sensor's Logstash logs with no obvious connection back to the failed credential write during deployment.

4. .env update also fails silently (deploy.sh line 135):

sed -i "/^LS_WEB_USER=/c\LS_WEB_USER=$myENV_LS_WEB_USER" "${myENV_FILE}"

Since myENV_FILE points to the wrong path, LS_WEB_USER is never updated in the Hive's actual .env. This means tpotinit will regenerate an empty lswebpasswd on the next restart, even if the user manually fixes the file.

Steps to Reproduce

1. Install T-Pot as HIVE — installer places files in /opt/tpot/ (not ~/tpotce/)
2. Install T-Pot as SENSOR on another host
3. Run deploy.sh from the Hive
4. Script fails at HIVE type check (workaround: create symlink ln -s /opt/tpot ~/tpotce)
5. With symlink, Ansible portion completes successfully (ok=10, failed=0)
6. deploy.sh outputs credential write errors that are easy to miss
7. Script exits 0 (appears successful)
8. Sensor Logstash logs show continuous 401 errors:

   [ERROR] http - Encountered non-2xx HTTP code {:code=>401, ...url=>"https://hive-ip:64294"}
   

9. Hive's lswebpasswd is empty:

   cat ~/tpotce/data/nginx/conf/lswebpasswd
   # (empty)

Cascading Impact

Because the credential write fails silently, the user must manually create new credentials. This leads to additional confusion because the Hive and Sensor use different credential variables with different encodings:

Variable	Location	Format	Generate with
LS_WEB_USER	Hive .env	base64 of user:$apr1$hash	htpasswd -n -b 'user' 'pass' | base64 -w0
TPOT_HIVE_USER	Sensor .env	base64 of user:plaintext	echo -n 'user:pass' | base64 -w0

Without clear documentation of this difference (the .env comments don't explain it), users frequently put the wrong encoding in the wrong variable, prolonging the 401 errors. See also Discussion #1529 for related credential confusion.

Additionally, docker compose restart does not re-read .env changes — users must use docker compose down + docker compose up -d to pick up new credential values. This is another undocumented gotcha that compounds the troubleshooting difficulty.

Suggested Fixes

1. Use dynamic path detection instead of hardcoded ~/tpotce/ (deploy.sh):

# Replace line 7:
# myENV_FILE="$HOME/tpotce/.env"
# With:
TPOT_PATH="$(dirname "$(readlink -f "$0")")"
myENV_FILE="${TPOT_PATH}/.env"

2. Pass the path to Ansible as a variable (deploy.sh → deploy.yml):

# Add to the ansible-playbook command:
ansible-playbook ${myANSIBLE_TPOT_PLAYBOOK} \
  -i ${mySENSOR_IP}, -c ssh -u ${mySSHUSER} \
  --ask-become-pass \
  -e "ansible_port=${myANSIBLE_PORT}" \
  -e "tpot_local_path=${TPOT_PATH}"

# In deploy.yml, replace hardcoded paths with variables:
vars:
  local_nginx_cert_path: "{{ tpot_local_path }}/data/nginx/cert/nginx.crt"

3. Add error handling for the lswebpasswd write (deploy.sh):

myLSWEBPASSWD="${TPOT_PATH}/data/nginx/conf/lswebpasswd"
if [ ! -f "$myLSWEBPASSWD" ]; then
    echo ""
    echo "# ERROR: $myLSWEBPASSWD not found."
    echo "# Cannot write sensor credentials to Hive."
    echo "# Manually add sensor credentials to LS_WEB_USER in your .env"
    echo "# and restart tpotinit + nginx."
    exit 1
fi

4. Improve .env comments to document the credential format difference:

# LS_WEB_USER (HIVE only): base64 of htpasswd-hashed credentials
# Generate: htpasswd -n -b 'user' 'pass' | base64 -w0
#
# TPOT_HIVE_USER (SENSOR only): base64 of PLAINTEXT credentials
# Generate: echo -n 'user:pass' | base64 -w0
# WARNING: These are different encodings of the same username/password

Current Workaround

1. Create a symlink: ln -s /opt/tpot ~/tpotce
2. Run deploy.sh — Ansible portion will succeed
3. After deploy.sh completes, manually add sensor credentials to the Hive:
   • Generate htpasswd base64: htpasswd -n -b 'username' 'password' | base64 -w0
   • Add to LS_WEB_USER in the Hive's .env
   • Restart tpotinit and nginx: docker compose stop tpotinit nginx && docker compose up -d tpotinit && sleep 5 && docker compose up -d nginx
4. On the sensor, verify TPOT_HIVE_USER was set by Ansible:
   • grep "^TPOT_HIVE_USER" ~/tpotce/.env | cut -d= -f2 | base64 -d
   • Should show username:password in plaintext
5. If TPOT_HIVE_USER is empty, set it manually:
   • Generate plaintext base64: echo -n 'username:password' | base64 -w0
   • Set TPOT_HIVE_USER=<value> in the sensor's .env
   • Recreate (not restart) Logstash: docker compose down logstash && docker compose up -d logstash

[t3chn0m4g3]

We are aware and the situation is well documented in the Readme and has been discussed accordingly.
Currently we have no plans to support T-Pot installs outside $HOME.