City Infrastructure Platform

City Infrastructure Platform REST-API backend application.

Setting up the environment for development

There are multiple ways to prepare the project for development. Described here is a method to run the auxiliary services using docker containers and a method to run these services directly using system packages. For both paths you should go through the common dependencies and setting up the python environment first.

Cloning the repository

git clone git@github.com:City-of-Helsinki/city-infrastructure-platform.git

Setup git hooks

cd city-infrastructure-platform
./setup-git-hooks.sh

Common dependencies

Install required system packages, some of these are needed to build some of the python project dependencies and some of these are used in runtime.

sudo apt install binutils gdal-bin libpq-dev libproj-dev pipx

Install Poetry for managing python dependencies.

# Let pipx handle it (without messing with the project's environment, the x at the end of pipx is important!)
pipx install poetry

# Or use the system package (may be less bleeding-edge but good enough for this project)
sudo apt install python3-poetry

Common dependencies - python environment

Check whether you have the minimum required version of python (3.11.9)

python --version

or

python3 --version

Common dependencies - python environment - use correct python version (optional)

If your version of python is equal to or newer than 3.11.9, you may skip this section.

If your python version is older, we recommend you install pyenv to manage the installation of a newer version, follow the instructions to set up the suggested build environment for your system, and then finally install and activate python 3.11.9.

# This command will build python version 3.11.9, it is a lengthy process
# so make sure you have followed the steps for setting up the suggested
# build environment linked above.
pyenv install 3.11.9

If you encounter trouble building python 3.11.9 even after setting up the suggested build environment, you should consult the pyenv common problems FAQ to proceed.

# This command will activate python 3.11.9
pyenv global 3.11.9

Common dependencies - python environment - virtual environment

Ensure you are using python 3.11.9 or newer and at the root of this project run

# Create the environment
python -m venv venv

# Activate the environment
source venv/bin/activate

# Install project dependencies (development)
poetry install --no-root

# Install project dependencies (production)
poetry install --no-dev

(optional) You can also use poetry to update the project's python dependencies:

# Update python dependencies / lock file
poetry update

Development environment with docker

Install docker, their provided packages are preferred since system packages are likely to be lagging behind the official distribution.

# Add docker's official repository as described in the link above
# ...
# Install required docker packages
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Development environment with docker - rootless mode (optional)

By default, docker needs to be executed with root privileges, which may put your system at risk of malicious docker images, so you may consider setting rootless mode. If you installed docker using the official packages and the commands suggested in the install instructions on a fresh Ubuntu system it will be as easy as running the following command as a regular user:

dockerd-rootless-setuptool.sh install

The script will be available in your PATH, and will let you know if you are missing any other system packages and give you a command to install them. Follow those steps and re-run the script, and it's done.

Development environment without docker

Note: These instructions are currently outdated, and could use updating to a newer LTS version of Ubuntu

Development environment without docker - PostgreSQL and PostGIS

Install PostgreSQL and PostGIS.

# Ubuntu 18.04
sudo apt-get install python3-dev libpq-dev postgresql postgis

Development environment without docker - GeoDjango extra packages

# Ubuntu 18.04
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal

Development environment without docker - Prepare the database

Enable PostGIS extension for the default template

sudo -u postgres psql -h localhost -d template1 -c "CREATE EXTENSION IF NOT EXISTS postgis;"

Create user and database

sudo -u postgres createuser -h localhost -P -R -S city-infrastructure-platform  # use password `city-infrastructure-platform`
sudo -u postgres createdb -h localhost -O city-infrastructure-platform city-infrastructure-platform

Allow user to create test database

sudo -u postgres psql -h localhost -c "ALTER USER city-infrastructure-platform CREATEDB;"

Running the project

Running the project - Django

Environment variables are used to customize configuration in cityinfra/settings.py. If you wish to override any settings, you can place them in a local .env file which will automatically be sourced when Django imports the settings file. Another way, easiest with Azurite emulation is to copy cityinfra/local_settings_example.py to cityinfra/local_settings.py and modify the settings there. Example already sets AZURE_BLOBSTORAGE settings to use Azurite emulation.

# Use example configuration
cp .env.example .env


# Enable debug (optional)
echo 'DEBUG=True' >> .env

# Run database migrations
# (Requires that the database is up and running)
python manage.py migrate

# Run server (host defaults to 127.0.0.1 and port defaults to 8000)
# (Requires that the database is up and running)
python manage.py runserver
# or with the local_settings
python manage.py runserver --settings=cityinfra.local_settings

Running the project - Docker services (without compose)

Build Docker image:

docker build -t city-infrastructure-platform .

Run container:

docker run -d -p 8000:8000 -e DEBUG=1 city-infrastructure-platform

Available configs (environment variables):

To set any of the settings below, use the -e <ENV_VAR>=<VALUE> flag when running the Docker container.

• DATABASE_HOST: Set to the host address of the PostgreSQL (with PostGIS) database server, default is empty value.
• DATABASE_PORT: Set to port of the database server, default is 5432.
• DEV_SERVER: Set to 1 to run manage.py runserver instead of uwsgi, default is empty value.
• COLLECT_STATIC: Set to 1 to collect static files on startup, default is empty value.
• APPLY_MIGRATIONS: Set to 1 to run manage.py migrate on startup, default is empty value.

Running the project - Docker services (with compose)

The file docker-compose.yml contains the environment configuration for the different services that can be executed.

Run the entire application, with all its auxiliary services

docker-compose up

Running the project - Docker services - individual services

# Run just the DB
docker-compose up db

# Run ClamAV locally (initialize clamd and then launch clamv-api)
docker compose up clamd
docker compose up clamv-api

# Run azurite (local emulation of azure storage)
# NOTE: To you use azurite you must also set the EMULATE_AZURE_BLOBSTORAGE variable on your .env file to True
docker compose -f docker-compose.azurite.yml up azurite

# Create the media, upload storage containers in azurite and grant it public read permissions for serving static files
docker compose -f docker-compose.azurite.yml up azurite-init

# Wipe the media, upload storage containers on azurite (you will need to re-run azurite-init after this)
docker compose -f docker-compose.azurite.yml up azurite-delete-storage-containers

Translations (fi)

# Ensure gettext is installed
sudo apt install gettext

# Run the script
./makemessages.sh

Traffic sign icons

Traffic sign icons are from Finnish Transport Infrastructure Agency which has released these icons in public domain under Creative Commons 1.0 universal (CC0 1.0) license. Original icons can be found here.