Webserver and fileserver parametric ports

[lorenzospano]

Motivation

This PR aims at making configurable all the ports that clearml-server uses.
Right now, some ports are hardcoded and not parametric ie webserver on port 80 and fileserver on port 8081.
Issue descrition - #293

Fileserver port parameterization

The listening port for the fileserver is hardcoded in fileserver/fileserver.py (when launched via flask) and in docker/build/internal_files/entrypoint.sh (when launched via gunicorn).

Flask setup

The code portion in fileserver/fileserver.py is reported below for simplicity.

    parser.add_argument(
        "--port", "-p", type=int, default=8081, help="Port (default %(default)d)"
   
    parser.add_argument(
        "--ip", "-i", type=str, default="0.0.0.0", help="Address (default %(default)s)"

    parser.add_argument("--debug", action="store_true", default=False)

I modified the fileserver to read the default values for ip, port and debug from the config and added the configurations inside of fileserver/config/default/fileserver.conf.
This behavior was guided by the existing mechanism in apiserver.

Gunicorn setup

Similarly, I replicated the existing behavior of the apiserver to also make gunicorn configurable via environment variables.
The default port is preserved, while now it is also possible to override it with FILESERVER_GUNICORN_BIND.
The intended behavior is the same as the one used for the apiserver.

Webserver port parameterization

The webserver port is hardcoded in docker/build/internal_files/clearml.conf.template.

    listen       80 default_server;
    ${COMMENT_IPV6_LISTEN}listen       [::]:80 default_server;

Per my understanding this is the conf that nginx uses to create the webserver UI, and it is not configurable via environment variables.
For this reason I modified the template to accepts an additional environment variable that allows to compile the webserver port via envsubst.
I also modified the envsubst command that is execute inside of docker/build/internal_files/entrypoint.sh to include such a value when the template is compiled.

Testing

I tested the startup and basic interactions among those services.
I could not test (yet) more advanced features such as agents and pipeline runs.

If there is any known component that features hardcoded ports please highlight it so that I can improve my contribution

[ainoam]

Thanks for this @lorenzospano - We'll try to review it as soon as we can.

[shyallegro]

Hi @lorenzospano
can you please explain why you need to change the internal docker webserver port?
I don't see any reason to do that, since you can expose it to any port you like through docker configuration

[lorenzospano]

Hi @shyallegro,
In some environments, such as HPC where security is a concern, docker might not be available.

In those cases, where network isolation and port mapping are not viable options, it would be useful to have a mechanism that allows to fully configure all the servers that clearml needs to run.

[dadastory]

Hi @lorenzospano can you please explain why you need to change the internal docker webserver port? I don't see any reason to do that, since you can expose it to any port you like through docker configuration

@shyallegro I deployed ClearML using docker-compose according to the official instructions, and I modified the mapped ports. However, when I created new credentials in the UI's workspace, I found that the ports and addresses that could be copied were incorrect, and the address for files_server was not filled in. Are there any steps or tutorials for changing the port? It seems that the related configurations are not modified together.

[shyallegro]

Hi @dadastory
UI can "guess" the right address in some cases, but in this case you need to configure the address manually by passing the data to displayedServerUrls this can be done through docker environment variables as follows:

webserver:
    environment:
        - WEBSERVER__displayedServerUrls={"apiServer":"","filesServer":""}

[dadastory]

@shyallegro thanks , it finally works!!! but it seem do not support env inject like this：WEBSERVER__CLEARML_API_HOST: "http://${CLEARML_HOST_IP}:18008"
WEBSERVER__CLEARML_FILES_HOST: "http://${CLEARML_HOST_IP}:18081"

[shyallegro]

@dadastory if you wish to configure webserve nginx reveres proxies, you can use these 2 variables
NGINX_APISERVER_ADDRESS
NGINX_FILESERVER_ADDRESS

[dadastory]

@shyallegro thanks, i will try again

[dadastory]

@shyallegro Hi

I followed your instructions and configured my ClearML docker-compose like this:

webserver:
    command:
      - webserver
    container_name: clearml-webserver
    environment:
      # - WEBSERVER__displayedServerUrls={"apiServer":"http://172.0.0.22:18008","filesServer":"http://172.0.0.22:18081"}
      - WEBSERVER__fileBaseUrl="https://files.clearml.mydomain.com"
      - WEBSERVER__useFilesProxy=true
      - NGINX_APISERVER_ADDRESS="https://api.clearml.mydomain.com"
      - NGINX_FILESERVER_ADDRESS="https://files.clearml.mydomain.com"

However, when I access the services through Nginx reverse proxy, I always get 404 errors.

I have a few questions:

1. How should I properly configure Nginx to work with ClearML in this setup?
2. Is it possible to access all ClearML services (webserver, API server, file server) using a single domain instead of separate domains?
3. I have tried following the official documentation and using separate domains for each service, but it still doesn’t work. Could you provide a detailed Nginx configuration example that works with ClearML?

Thanks in advance!

[shyallegro]

@dadastory you only need to specify the external addresses if the servers are not on the same docker network.
by default NGINX_APISERVER_ADDRESS points to apiserver which is available through the frontend network, fileserver works in a similar way.
if they are not on the same network, make sure the webserver container has access to api and file hosts.
for display in credentials popup, the UI will assume the URLs are app., api. and files.* otherwise specify the correct external address with WEBSERVER__displayedServerUrls and in WEBSERVER__fileBaseUrl so we can rewrite the URL in case of plotly with background images.

[dadastory]

@lorenzospano Thank you very much for the clarification. It turns out that no extra configuration is required, and a simple reverse proxy setup is sufficient.