Installation Instructions - Manual

1. Go to https://master.garching.cluster.campar.in.tum.de/home.html
2. Click submit job, choose "Config Editor"
3. Copy and paste the contents of tum-cluster-config.yaml into the text box
4. Click submit
5. Wait for the job to start
6. Once the job is running, click on the job name to open the job details
7. Click "SSH Info" to get the ssh command to connect to the cluster
8. Connect to the cluster with VSCode
9. Once connected, install git and clone this repo: git clone https://github.com/anilzeybek/tum-p3dcv.git
10. cd into the repo: cd tum-p3dcv, and run the following command to download kitti data:

    ./quicksetup.sh
    

11. Run the following command to convert the kitti data to the format expected by the nerfstudio (transforms.json):

    python convertor.py
    

12. Last command will print the command to start the training with 3d-gs. Run it.

Ensure dataset is structured in the following manner:

tum-p3dcv/ ├── kitti-data/ │ ├── calibration/ │ │ ├── calib_cam_to_pose.txt │ │ ├── calib_cam_to_velo.txt │ │ ├── calib_sick_to_velo.txt │ │ ├── image_02.yaml │ │ ├── image_03.yaml │ │ └── perspective.txt │ ├── data_2d_raw/ │ │ └── 2013_05_28_drive_0009_sync/ │ ├── data_3d_semantics/ │ │ └── train/ │ └── data_poses/ │ └── 2013_05_28_drive_0009_sync/ ├── nerfstudio/ etc ├── ... etc

Installation Instructions - Docker for Custom Nerfstudio

This guide explains how to set up and run our custom Nerfstudio inside a Docker container. It includes building the environment, mounting the necessary directories, and ensuring everything runs smoothly.

1. Install & Verify Docker

Ensure Docker is installed and running.

1.1 Install Docker (if not installed)

Run the following commands:

sudo apt update
sudo apt install -y docker.io

1.2 Verify Docker Installation

docker --version

Example output:

Docker version 20.10.14, build a224086

1.3 Check if Docker Daemon is Running

systemctl status docker

If not running, start it:

sudo systemctl start docker

1.4 Ensure Your User Has Docker Permissions

If docker ps gives a permission error:

sudo usermod -aG docker $USER
newgrp docker

Log out and log back in to apply changes.

2. Verify or Build the Nerfstudio Docker Image

2.1 Check if the Nerfstudio Image Exists

docker images

If you see nerfstudio listed, proceed to Step 3. If not, continue below.

2.2 Pull the Official Nerfstudio Image (If Needed)

docker pull nerfstudio

2.3 Build the Nerfstudio Docker Image (If Custom Build is Required)

If you have a Dockerfile, navigate to its directory and run:

docker build -t nerfstudio .

3. Create a Docker Run Script

To simplify launching Docker, create a script:

3.1 Navigate to the Project Directory

Replace with your path:

cd /path/to/tum-p3dcv

3.2 Create the runDocker.sh Script

nano runDocker.sh

3.3 Paste the Following Content

#!/bin/bash
docker run --gpus all \
  --shm-size=10gb \
  -it \
  -v /path/to/tum-p3dcv:/workspace \
  --expose=7007 \
  --hostname="docker-trainer" \
  nerfstudio bash

3.4 Save and Make the Script Executable

Save and exit (CTRL + X, then Y, then ENTER). Then run:

chmod +x runDocker.sh

4. Run the Docker Container

Launch the container using the script:

./runDocker.sh

You should now be inside the Docker container.

Either run the quicksetup.sh for an automatic dowload of the dataset + an installation of the custom nerfstudio and skip to step 8, ...

or, ...

...simpy install the nerfstudio manually if you already have the dataset downloaded correctly. Continue with step 5:

5. Remove Preinstalled Nerfstudio (If Necessary)

Inside the container, remove the preinstalled version:

rm -rf /usr/local/lib/python3.10/dist-packages/nerfstudio

6. Install Required Dependencies

apt update && apt install -y python3-pip unzip htop tmux git

7. Install Custom Nerfstudio

Navigate to your repo inside the container:

cd /workspace/nerstudio

Install it in editable mode:

pip install -e .

8. Running Code & Editing Files

Since /workspace is mounted to your host directory, you can:

• Edit code in VSCode or any editor on the host.
• Pull updates from Git inside the container:

  cd /workspace
  git pull

9. Running Training or Scripts

Example:

python3 launch_calibration.py

10. Stopping & Restarting the Container

10.1 Stop the Container

To exit the running container safely:

exit

10.2 Restart the Same Container Later

Find the container ID:

docker ps -a

Restart it:

docker start -ai <container_id>

11. Accessing the Nerfstudio Viewer in a Docker Container

When running Nerfstudio inside a Docker container, the viewer is not accessible via localhost:7007. Instead, you need to find the correct Docker container IP address.

11.1 Find Your Docker Container's IP Address

Run the following command inside your terminal (on the host machine):

ip a | grep inet

This will output a list of IP addresses. Look for an entry under scope global docker0, which typically looks like:

inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

However, this might still not be the correct address to use. Instead, try replacing the last digit (1) with 2, making it:

http://172.17.0.2:7007/

This works in most cases but may vary for different users.

11.2 Verify the Correct Container IP

If 172.17.0.2 does not work, find the exact container IP by running inside the container:

ip a | grep inet

Look for an entry under eth0, which typically shows something like:

inet 172.17.0.2/16 scope global eth0

Use that IP with port 7007 in your browser:

http://:7007/

11.3 Alternative: Using Host Machine’s IP

If accessing from another device on your local network, use the host machine’s IP instead. Find it with:

ip a | grep inet

Look for the IP associated with your Wi-Fi or Ethernet interface (e.g., wlo1 or eth0). Example:

inet 192.168.2.147/24 brd 192.168.2.255 scope global dynamic noprefixroute wlo1

Then access the viewer from another device using:

http://192.168.2.147:7007/

(Replace 192.168.2.147 with your actual host IP.)

Running Instructions

1. Select your calibration configuration in the config.json file
2. Ensure you have your nerfstudio conda environment activated
3. Simply run the launch_calibration.py file
4. While training, you can check the training at the viser link printed in the terminal.

Configuration Parameters

Algorithm

• The algorithm parameter specifies which algorithm to use during training.
• Acceptable values are:
  • splatfacto: Uses 3D Gaussian Splatting.
  • nerfacto: Uses NeRF.

Please ensure that the value is set correctly in the config.json file. The default is splatfacto due to higher reconstruction accuracy.

(let's find out if one reconstruction algorithm is better than the other in terms of calibration accuracy!)

Additional Parameters

• The rotation_noise_std parameter is in radians, where 0.01 ≈ 0.57°.
• The translation_noise_std is in meters, where 0.01 = 1 cm.
• 0.01 and 0.01 represent realistic miscalibration errors.
• That level of error is unnoticable to the naked eye however, so set both values to 0.1 to see the noise working.

Explanations

Injecting noise

For now, we consider the LIDAR and left camera to be perfect and ground truth and only inject noise to the right camera. That noise is random, but consistent among all image pairs (i.e. the rotation and translation noise between two pairs of images - left and right - is consistent among all pairs of frames)

(there is potentially a bug where the error is indeed the same for all frames, but relative to world reference, instead of relative to its left image counterpart, which would explain the drift during the right hand turn of the car that we were observing yesterday. However, today, no drift is noticeable and the noise seems to be applied correctly relative to the left image)

The strength of the noise can be changed in the config file.