Portainer with reverse proxy (caddy) can't connect to edge agent but gets heartbeats

[BitWuehler]

Since a while I try now to connect my portainer instance at home to the home server at my mothers house. For that I red, the safest way to use is edge agent.

At home I run portainer behind a caddy reverse proxy on a raspberry pi 4.
Caddy is configured to route https://portainer.mydomain.de:443 to 192.168.178.3:9233 and tcp://portainer.mydomain.de:8000 to port 192.168.178.3:8111. In Docker I configured, 8111:8000 and 9233:9000 in the portainer compose file on my server.
I opened up port 8000 and 443 in my router (tcp and udp). Also in ufw I allowed port 443 and 8000.

Portainer is working well so far.

On my mothers server I opened up port 9001 in the router. Ufw is also configured so far.

Now I tried to set up edge agent. I used:

sudo docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  -v /:/host \
  -v portainer_agent_data:/data \
  --restart always \
  -e EDGE=1 \
  -e EDGE_ID=----------------------------------- \
  -e EDGE_KEY=-------------------------------------------------------------- \
  -e EDGE_INSECURE_POLL=1 \
  --name portainer_edge_agent \
  portainer/agent:2.15.0

I now can see a heartbeat under Environments but if I try to connect it says Failed loading environment Environment is unreachable.

The portainer logs say:

time="2022-09-15T22:18:25+02:00" level=info msg="2022/09/15 22:18:25 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 7.157875] [message: environment tunnel monitoring]"
time="2022-09-15T22:18:28+02:00" level=info msg="2022/09/15 22:18:28 http: proxy error: dial tcp 127.0.0.1:51018: connect: connection refused"
time="2022-09-15T22:23:30+02:00" level=info msg="2022/09/15 22:23:30 http error: Unable to find the container (err=Error: No such container: 3bfdd889277c8539ed7f13f4df61339c6821c53ad3a5a404730793545eab88c6) (code=404)"
time="2022-09-15T22:23:30+02:00" level=info msg="2022/09/15 22:23:30 http error: Unable to find the container (err=Error: No such container: dae984b1b0af5e2ab7d8a7d4a8f4d04f8d278091412641c87250d3700a5d10dd) (code=404)"
time="2022-09-15T22:34:45+02:00" level=info msg="2022/09/15 22:34:45 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 3.817940] [message: environment tunnel monitoring]"
time="2022-09-15T22:34:50+02:00" level=info msg="2022/09/15 22:34:50 http: proxy error: dial tcp 127.0.0.1:64692: connect: connection refused"
time="2022-09-15T22:37:04+02:00" level=info msg="2022/09/15 22:37:04 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 8.822090] [message: environment tunnel monitoring]"
time="2022-09-15T22:37:05+02:00" level=info msg="2022/09/15 22:37:05 http: proxy error: dial tcp 127.0.0.1:55147: connect: connection refused"
time="2022-09-15T22:41:24+02:00" level=info msg="2022/09/15 22:41:24 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: REQUIRED] [status_time_seconds: 0.182232] [message: environment tunnel monitoring]"
time="2022-09-15T22:41:34+02:00" level=info msg="2022/09/15 22:41:34 http: proxy error: dial tcp 127.0.0.1:65013: connect: connection refused"
time="2022-09-15T23:12:44+02:00" level=info msg="2022/09/15 23:12:44 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: REQUIRED] [status_time_seconds: 1.361693] [message: environment tunnel monitoring]"
time="2022-09-15T23:12:53+02:00" level=info msg="2022/09/15 23:12:53 http: proxy error: dial tcp 127.0.0.1:60140: connect: connection refused"
time="2022-09-15T23:13:34+02:00" level=info msg="2022/09/15 23:13:34 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 2.269864] [message: environment tunnel monitoring]"
time="2022-09-15T23:13:41+02:00" level=info msg="2022/09/15 23:13:41 http: proxy error: dial tcp 127.0.0.1:61949: connect: connection refused"
time="2022-09-15T23:14:34+02:00" level=info msg="2022/09/15 23:14:34 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 2.271884] [message: environment tunnel monitoring]"
time="2022-09-15T23:14:41+02:00" level=info msg="2022/09/15 23:14:41 http: proxy error: dial tcp 127.0.0.1:60159: connect: connection refused"

The Agent logs:

2022/09/16 08:59:53 [INFO] [main] [message: Agent running on Docker platform]
2022/09/16 08:59:53 [INFO] [edge] [message: Edge key loaded from options]
2022/09/16 08:59:53 [INFO] [edge,registry] [message: Starting registry credential server]
2022/09/16 08:59:53 [INFO] [http] [server_addr: 172.01.02.03] [server_port: 9001] [use_tls: false] [api_version: 2.15.0] [message: Starting Agent API server]
2022/09/16 09:00:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:00:38 client: Connection error: websocket: bad handshake
2022/09/16 09:00:38 client: Give up
2022/09/16 09:01:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:01:38 client: Connection error: websocket: bad handshake
2022/09/16 09:01:38 client: Give up
2022/09/16 09:02:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:02:38 client: Connection error: websocket: bad handshake
2022/09/16 09:02:38 client: Give up
2022/09/16 09:03:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:03:38 client: Connection error: websocket: bad handshake

I googled a lot, tried a lot but nothing changes something in a better way.
Maybe it could be a problem with caddy? Also here I tried a lot. That's my config at the moment:

portainer.{$DOMAIN}:443 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:9233
}

tcp://portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:8111
}

And for the sake of completeness here also my portainer docker-compose.yml:

version: '3'

networks:
  caddy:
    external: true

services:
  portainer:
    image: portainer/portainer-ce:latest
    command: -H unix:///var/run/docker.sock
    container_name: portainer
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer-data:/data
    ports:
      - 9233:9000
      - 8111:8000
    networks:
      caddy:
        ipv4_address: 192.168.112.8
        ipv6_address: 2001:ab12::8

Im not sure if it is a problem with the agent, portainer or caddy but I hope some of you has an idea!

[deviantony]

Hey @BitWuehler

Do you mind sharing the agent logs with us?

Also have you tried to update the Caddy config to remove the tcp protocol from the Edge specific proxy?

portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:8111
}

I don't think the tcp bit is actually required as the agent will initiate the communications over web socket.

[BitWuehler]

@deviantony

Do you mind sharing the agent logs with us?

Sure! I added it above.

Yes, I tried it without the tcp://. Was just the last state, after I tried a lot.

[BitWuehler]

Nobody an Idea? I will add to main as portainer issue too...

[Shurelol]

portainer/portainer-compose#24 (comment)

you should look at this

[atol71]

It seems agents use ipv6. Fresh install and:

netstat -atn | grep 9001
tcp6       0      0 :::9001                 :::*                    LISTEN