[BUG] Prometheus Metrics Not Exposed in Hauler Registry Serve

[hdsakatti]

Issue Summary

Prometheus metrics are not accessible despite being configured in the hauler registry serve configuration file. The debug endpoint configuration appears to be partially working but not serving metrics correctly.

Environment Information

• Hauler Version: 1.1.1
• Git Commit: 090f4dc
• Build Date: 2024-12-09T13:36:45Z
• Go Version: go1.23.4 X:boringcrypto

Expected Behavior

Based on the documentation at https://docs.hauler.dev/docs/1.1.x/hauler-usage/store/serve/registry, Prometheus metrics should be accessible when configured with:

http:
  debug:
    prometheus:
      enabled: true
      path: /metrics

Metrics should be available at either:

• http://localhost:5001/metrics (debug port)
• http://localhost:5000/metrics (main port)

Actual Behavior

• Main port metrics endpoint returns 404 page not found
• Debug server on port 5001 accepts connections but times out without response
• Registry starts and functions correctly for container operations on port 5000
• No errors in logs about prometheus configuration
• Storage directory is correctly set to local path

Configuration File Used

version: 0.1
log:
  level: debug
  formatter: text

storage:
  filesystem:
    rootdirectory: /var/log/registry

http:
  addr: ":5000"
  debug:
    addr: ":5001"
    prometheus:
      enabled: true
      path: /metrics

Commands Executed

1. Start Registry:

   hauler store serve registry --config hauler-config.yaml

2. Test Metrics Endpoints:

   curl -s http://localhost:5000/metrics  # Returns: 404 page not found
   curl -s http://localhost:5001/metrics --max-time 10  # Connection timeout

Detailed Curl Debug Output (-vvv)

Main Port (5000) - Returns 404

$ curl -vvv http://localhost:5000/metrics --max-time 10

* Host localhost:5000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
*   Trying [::1]:5000...
* Connected to localhost (::1) port 5000
> GET /metrics HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Wed, 24 Sep 2025 07:37:08 GMT
< Content-Length: 19
<
{ [19 bytes data]
100    19  100    19    0     0  16407      0 --:--:-- --:--:-- --:--:-- 19000
* Connection #0 to host localhost left intact

404 page not found

Debug Port (5001) - Connection Timeout

$ curl -vvv http://localhost:5001/metrics --max-time 10

* Host localhost:5001 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
*   Trying [::1]:5001...
* connect to ::1 port 5001 from ::1 port 64576 failed: Connection refused
*   Trying 127.0.0.1:5001...
* Connected to localhost (127.0.0.1) port 5001
> GET /metrics HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
* Operation timed out after 10006 milliseconds with 0 bytes received
  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
* Closing connection
curl: (28) Operation timed out after 10006 milliseconds with 0 bytes received

3. Verify Registry Function:

   curl -s http://localhost:5000/v2/_catalog
   {"repositories":["rancher/mirrored-pause","rancher/shell"]}

Registry Logs

Stdout:

[90m2025-09-24 08:32:20[0m [32mINF[0m rancher/shell:v0.3.0
[90m2025-09-24 08:32:20[0m [32mINF[0m 2025/09/24 08:32:20 existing manifest: v0.3.0@sha256:b739e4a2c71dcd973d0e871dda255dfa5d254e378544d16145c4389fa124bdb2
[90m2025-09-24 08:32:20[0m [32mINF[0m rancher/mirrored-pause:3.7
[90m2025-09-24 08:32:20[0m [32mINF[0m 2025/09/24 08:32:20 existing manifest: 3.7@sha256:f81611a21cf91214c1ea751c5b525931a0e2ebabe62b3937b6158039ff6f922d
[90m2025-09-24 08:32:20[0m [32mINF[0m copied artifacts to [127.0.0.1:64321]
[90m2025-09-24 08:32:20[0m [32mINF[0m starting registry on port [5000]

time="2025-09-24T08:32:20.640903+01:00" level=debug msg="using \"text\" logging formatter"
time="2025-09-24T08:32:20.641345+01:00" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version="go1.23.4 X:boringcrypto" version=v3.0.0+unknown
time="2025-09-24T08:32:20.64136+01:00" level=info msg="redis not configured" go.version="go1.23.4 X:boringcrypto" version=v3.0.0+unknown
time="2025-09-24T08:32:20.641405+01:00" level=info msg="Starting upload purge in 31m0s" go.version="go1.23.4 X:boringcrypto" version=v3.0.0+unknown
time="2025-09-24T08:32:20.641536+01:00" level=info msg="listening on [::]:5000" go.version="go1.23.4 X:boringcrypto" version=v3.0.0+unknown

Network Port Analysis

netstat -an | grep 500
tcp46      0      0  *.5000                 *.*                    LISTEN
tcp4       0      0  127.0.0.1.5001         *.*                    LISTEN
tcp4       0      0  127.0.0.1.5000         *.*                    LISTEN

• Port 5000 is listening (main registry)
• Port 5001 is listening (debug server) but times out on HTTP requests
• Connection attempts to 5001 show CLOSE_WAIT/FIN_WAIT_2 states indicating accepted but incomplete connections

Hauler Store Information

hauler store info
+-----------------------------------------------+-------+-------------+----------+----------+
| REFERENCE                                     | TYPE  | PLATFORM    | # LAYERS | SIZE     |
+-----------------------------------------------+-------+-------------+----------+----------+
| XXX.XXX.X.XXX:XXXX/rancher/mirrored-pause:3.7 | image | linux/amd64 |        1 | 307.0 kB |
| XXX.XXX.X.XXX:XXXX/rancher/shell:v0.3.0       | image | linux/amd64 |        6 | 99.7 MB  |
+-----------------------------------------------+-------+-------------+----------+----------+
|                                                                        TOTAL   | 100.0 MB |
+-----------------------------------------------+-------+-------------+----------+----------+

References

• Hauler Registry Serve Documentation
• Distribution Registry Configuration