Skip to content

Latest commit

 

History

History
203 lines (134 loc) · 5.81 KB

README.md

File metadata and controls

203 lines (134 loc) · 5.81 KB

ReMoSPLAT: Reactive Mobile Manipulation Control on a Gaussian Splat

Nicolas Marticorena · Tobias Fischer · Niko Suenderhauf .

Queensland University of Technology

arXiv | Dataset

Dependencies

Most dependencies can be installed via either pip or conda. However, we instead follow a git submodule approach, which offers the following advantages:

  • Our forks of key libraries include small additional features and updated build backends.
  • By changing the build backend, we can install the libraries in editable mode using pixi (as setuptools is incompatible), allowing easier tweaking of underlying dependencies.

If you need to update any dependency to retrigger compilation of the C/C++ extensions, run:

pixi reinstall <library-name>

For example, if you modify gsplat:

pixi reinstall gsplat

Included dependencies

Name Description
gsplat Gaussian Splatting implementation from NeRFStudio. We apply a patch for 2D depth rendering (nerfstudio-project/gsplat#477).
neural_robot Wrapper around the Robotics Toolbox for Python, exposing functionality in PyTorch.
nerf_tools Scripts for managing NeRF-style datasets and recordings.
safer-splats Utilities for sphere-to-ellipsoid distance computation.

Getting Started

To ensure reproducible installation, we use the package manager pixi.
If you haven’t installed it yet, run:

curl -fsSL https://pixi.sh/install.sh | bash

After installation, restart your terminal or reload your shell configuration so the changes take effect.
See the pixi documentation for more details.

Clone the repository and install dependencies:

git clone --recursive git@github.com:nmarticorena/RemoSplat.git
cd RemoSplat
pixi install

Once installation is complete, activate the environment with:

pixi shell

Download Scenes

The synthetic scenes described in the paper are available on Hugging Face:

👉 https://huggingface.co/datasets/nmarticorena/ReMoSplat-scenes

These datasets follow the standard NeRF-style format, where camera poses are stored in a transforms.json file.

For quicker testing, we also provide a smaller sample dataset:

👉 https://huggingface.co/datasets/nmarticorena/ReMoSplat-scenes-sample

Download it using the Hugging Face CLI:

hf download nmarticorena/ReMoSplat-scenes-sample --repo-type dataset --local-dir data/scenes

To download the full dataset:

hf download nmarticorena/ReMoSplat-scenes --repo-type dataset --local-dir data/scenes

Train Splats

We follow the training workflow from gsplat.
If your dataset consists of posed depth images, our compute-aabb script is useful. It preprocesses the dataset to compute bounding boxes, estimate normals, and subsample points from dense point clouds.

View available options with:

compute-aabb --help

An example training pipeline for our synthetic scenes is provided in:

scripts/training/rmmi/train_scene.sh

Run:

bash scripts/training/rmmi/train_scene.sh table N
bash scripts/training/rmmi/train_scene.sh bookshelf N

Where N is the number of scenes (out of 500) to train.
If using the sample dataset, set N=10.

Run the Controller

The controller implementation is located in:

remo_splat/controller.py

Multiple distance metrics are implemented in:

remo_splat/lidar.py

Control hyperparameters are defined in:

remo_splat/configs/controllers.py

To run the simulated benchmark:

python3 scripts/benchmark/rmmi.py --n-scenes N

Where N is the number of scenes to evaluate.

Visualize Behaviours

Controller outputs are stored in:

logs/{exp_name}/{sim_frequency}

You can visualise them with:

python remo_splat/experiment_visualizer.py --folder-name {exp_name}/{frequency} --env-names [scene_names ...] --robot-name curobo

This will open:

  • one 3D visualisation window
  • one depth rendering window showing the depth rendering from one of the spheres that represent the robot geometry (determined by the robot_sphere_id)

Diagram of visualizer

Acknowledgements

Our approach builds upon several excellent open-source projects.
We thank the authors of gsplat for their implementation of both 3D and 2D Gaussian Splatting, and the authors of safer-splat for the PyTorch implementation of the sphere-to-ellipsoid distance described in Geometric Tools.

Please consider citing these works accordingly.

Citation

If you use ReMoSPLAT in your research, please cite:

@misc{marticorena2025remosplat,
      title={ReMoSPLAT: Reactive Mobile Manipulation Control on a Gaussian Splat},
      author={Nicolas Marticorena and Tobias Fischer and Niko Suenderhauf},
      year={2025},
      eprint={2512.09656},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2512.09656},
}

Pending

  • Add cyclonedds as a conda package to simplify installation of the NeRFCapture scripts