Skip to content

chore: Update & restructure instructions for setting up dev environment #637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shundread merged 1 commit into from
Aug 20, 2025
Merged

chore: Update & restructure instructions for setting up dev environment #637

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
223 changes: 173 additions & 50 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,81 +7,191 @@

City Infrastructure Platform REST-API backend application.

## Development
## Setting up the environment for development

### Install required system packages
There are multiple ways to prepare the project for development. Described here is a method to run the auxiliary services using docker containers and a method to run these services directly using system packages. For both paths you should go through the common dependencies and setting up the python environment first.

#### PostgreSQL and PostGIS
### Common dependencies

Install PostgreSQL and PostGIS.
Install required system packages, some of these are needed to build some of the python project dependencies and some of these are used in runtime.


```bash
sudo apt install binutils gdal-bin libpq-dev libproj-dev pipx
```

Install [Poetry](https://github.com/python-poetry/poetry#installation) for managing python dependencies.

```bash
# Let pipx handle it (without messing with the project's environment, the x at the end of pipx is important!)
pipx install poetry

# Or use the system package (may be less bleeding-edge but good enough for this project)
sudo apt install python3-poetry
```

#### Common dependencies - python environment

Check whether you have the minimum required version of python (3.11.9)

```bash
python --version
```

or

```
python3 --version
```

##### Common dependencies - python environment - use correct python version (optional)

If your version of python is equal to or newer than 3.11.9, you may skip this section.

If your python version is older, we recommend you install [pyenv](https://github.com/pyenv/pyenv) to manage the installation of a newer version, follow the instructions to set up the [suggested build environment](https://github.com/pyenv/pyenv/wiki#suggested-build-environment) for your system, and then finally install and activate python 3.11.9.


```bash
# This command will build python version 3.11.9, it is a lengthy process
# so make sure you have followed the steps for setting up the suggested
# build environment linked above.
pyenv install 3.11.9
```

If you encounter trouble building python 3.11.9 even after setting up the suggested build environment, you should consult the [pyenv common problems FAQ](https://github.com/pyenv/pyenv/wiki/Common-build-problems) to proceed.
Copy link
Contributor

@jforsman jforsman Aug 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe even add the specific 3.11.9 installation problem with ubuntu here? I do not remember which libraries needed to be installed.

Copy link
Contributor Author

@shundread shundread Aug 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually for Ubuntu 24.04 the problem is irrelevant since it's got a newer python than 3.11.9. But the suggested build environment on the previous paragraph (should) cover all build dependencies for 3.11.9

From the VM:

thiago@ubuntu-sandbox:~$ pyenv global system
thiago@ubuntu-sandbox:~$ python3 --version
Python 3.12.3
thiago@ubuntu-sandbox:~$ head -1 /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"


```bash
# This command will activate python 3.11.9
pyenv global 3.11.9
```

##### Common dependencies - python environment - virtual environment

# Ubuntu 18.04
sudo apt-get install python3-dev libpq-dev postgresql postgis
Ensure you are using python 3.11.9 or newer and at the root of this project run

#### GeoDjango extra packages

# Ubuntu 18.04
sudo apt-get install binutils libproj-dev gdal-bin
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
```bash
# Create the environment
python -m venv venv

### Creating a Python virtualenv
# Activate the environment
source venv/bin/activate

Create a Python 3.x virtualenv either using the [`venv`](https://docs.python.org/3/library/venv.html) tool or using
the great [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/) toolset. Assuming the latter,
once installed, simply do:
# Install project dependencies (development)
poetry install --no-root

mkvirtualenv -p /usr/bin/python3 city-infrastructure-platform
# Install project dependencies (production)
poetry install --no-dev
```

The virtualenv will automatically activate. To activate it in the future, just do:
(optional) You can also use poetry to update the project's python dependencies:

workon city-infrastructure-platform
```bash
# Update python dependencies / lock file
poetry update
```

Install [Poetry](https://github.com/python-poetry/poetry#installation) for installing requirements
### Development environment with docker

### Creating and updating requirements
Install [docker](https://docs.docker.com/engine/install/ubuntu/), their provided packages are preferred since system packages are likely to be lagging behind the official distribution.

* Run `poetry update`
```bash
# Add docker's official repository as described in the link above
# ...
# Install required docker packages
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

### Installing Python requirements
#### Development environment with docker - rootless mode (optional)

* For development run `poetry install`
* For production run `poetry install --no-dev`
By default, docker needs to be executed with root privileges, which may put your system at risk of malicious docker images, so you may consider setting [rootless mode](https://docs.docker.com/engine/security/rootless/). If you installed docker using the official packages and the commands suggested in the [install instructions](https://docs.docker.com/engine/install/ubuntu/) on a fresh Ubuntu system it will be as easy as running the following command as a regular user:

### Prepare the database
```bash
dockerd-rootless-setuptool.sh install
```

The script will be available in your PATH, and will let you know if you are missing any other system packages and give you a command to install them. Follow those steps and re-run the script, and it's done.

### Development environment without docker

**Note:** These instructions are currently outdated, and could use updating to a newer LTS version of Ubuntu

#### Development environment without docker - PostgreSQL and PostGIS

Install PostgreSQL and PostGIS.

```bash
# Ubuntu 18.04
sudo apt-get install python3-dev libpq-dev postgresql postgis
```

#### Development environment without docker - GeoDjango extra packages

```bash
# Ubuntu 18.04
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
```

#### Development environment without docker - Prepare the database

Enable PostGIS extension for the default template

sudo -u postgres psql -d template1 -c "CREATE EXTENSION IF NOT EXISTS postgis;"
```bash
sudo -u postgres psql -h localhost -d template1 -c "CREATE EXTENSION IF NOT EXISTS postgis;"
```

Create user and database

sudo -u postgres createuser -P -R -S city-infrastructure-platform # use password `city-infrastructure-platform`
sudo -u postgres createdb -O city-infrastructure-platform city-infrastructure-platform
```bash
sudo -u postgres createuser -h localhost -P -R -S city-infrastructure-platform # use password `city-infrastructure-platform`
sudo -u postgres createdb -h localhost -O city-infrastructure-platform city-infrastructure-platform
```

Allow user to create test database

sudo -u postgres psql -c "ALTER USER city-infrastructure-platform CREATEDB;"
```bash
sudo -u postgres psql -h localhost -c "ALTER USER city-infrastructure-platform CREATEDB;"
```

## Running the project

### Django configuration
### Running the project - Django

Environment variables are used to customize configuration in `cityinfra/settings.py`. If you wish to override any
settings, you can place them in a local `.env` file which will automatically be sourced when Django imports
the settings file.

Copy .env.example file as .env: `cp .env.example .env`
```
# Use example configuration
cp .env.example .env


# Enable debug (optional)
echo 'DEBUG=True' >> .env

# Run database migrations
# (Requires that the database is up and running)
python manage.py migrate

### Running development environment
# Run server (host defaults to 127.0.0.1 and port defaults to 8000)
# (Requires that the database is up and running)
python manage.py runserver
```

* Enable debug `echo 'DEBUG=True' >> .env`
* Run `python manage.py migrate`
* Run `python manage.py runserver 0.0.0.0:8000`
#### Running the project - Docker services (without compose)

### Docker
Build Docker image:

Build Docker image: `docker build -t city-infrastructure-platform .`
```bash
docker build -t city-infrastructure-platform .
```

Run container: `docker run -d -p 8000:8000 -e DEBUG=1 city-infrastructure-platform`
Run container:

```bash
docker run -d -p 8000:8000 -e DEBUG=1 city-infrastructure-platform
```

**Available configs (environment variables):**

Expand All @@ -93,25 +203,38 @@ To set any of the settings below, use the `-e <ENV_VAR>=<VALUE>` flag when runni
* COLLECT_STATIC: Set to `1` to collect static files on startup, default is empty value.
* APPLY_MIGRATIONS: Set to `1` to run `manage.py migrate` on startup, default is empty value.

### Docker Compose
#### Running the project - Docker services (with compose)

The file `docker-compose.yml` contains the environment configuration for the different services that can be executed.

Run the entire application, with all its auxiliary services

```bash
docker-compose up
```

Run the application `docker-compose up`
##### Running the project - Docker services - individual services

#### To run just database
`docker-compose up db`
```bash
# Run just the DB
docker-compose up db

#### To run local ClamAV instances
first do `docker-compose up clamd`
after it is started `docker-compose up clamv-api`
# Run ClamAV locally (initialize clamd and then launch clamv-api)
docker compose up clamd
docker compose up clamv-api
```

### Translations (fi)
## Translations (fi)

Run script `./makemessages.sh`
```
# Ensure gettext is installed
sudo apt install gettext

Running this script locally needs gettext to be installed:
`apt-get install gettext`
# Run the script
./makemessages.sh
```

### Traffic sign icons
## Traffic sign icons

Traffic sign icons are from Finnish Transport Infrastructure Agency which has released these icons in public
domain under Creative Commons 1.0 universal (CC0 1.0) license. Original icons can be found
Expand Down
Loading