VARS Quickstart @ PUBLIC

Docker-based quickstart/orchestrator for MBARI's VARS (Video Annotation and Reference System) microservices stack. This stack is for public use.

This repo is based off an internal project used for running the VARS microservices on a single server on MBARI's ships.

For Users

What is VARS?

VARS (Video Annotation and Reference System) is a comprehensive suite of microservices for managing and annotating underwater video data. This quickstart repository provides a streamlined way to deploy the full VARS stack using Docker Compose on the following server environments:

1. localhost - Local testing server against a testing database
2. namedhost - Production server on an intranet

Prerequisites

• Docker Engine 20.10+ with Docker Compose V2
• Bash shell
• Approximately 8GB of available RAM
• Network access to pull Docker images from Docker Hub
• SSL certificates (can be generated with the built-in mkcert command)
• Python for running support utilities (optional)

Quick Start

1. Clone the repository

   git clone https://github.com/mbari-org/vars-quickstart-mbari.git
   cd vars-quickstart-mbari

2. View available environment targets

   ./varsq targets

3. Configure your environment

   ./varsq configure etc/env/localhost.env

4. Generate SSL certificates for local development

   ./varsq mkcert

5. Start the services

   ./varsq start

6. Check service status

   ./varsq status

Common Commands

Environment Management

# List available environment configurations
./varsq targets

# Configure environment (merges source + core configuration)
./varsq configure etc/env/localhost.env

# View merged environment file (with variable references)
./varsq env

# View resolved environment (all variables evaluated to final values)
./varsq resolved

Service Operations

# Start all services
./varsq start

# Stop all services
./varsq stop

# Check service status
./varsq status

# View service logs
./varsq docker logs <service-name>
./varsq docker logs annosaurus

# Follow logs for a service
./varsq docker logs -f annosaurus

# Build containers (if needed)
./varsq build

# Update to latest service images
./varsq update

SSL Certificate Management

# Generate self-signed SSL certificate for local testing
# Uses mkcert if available, falls back to openssl
./varsq mkcert

Advanced Docker Compose Operations

The varsq docker command passes through any Docker Compose command:

# Execute a command in a running container
./varsq docker exec -it annosaurus bash

# Restart a specific service
./varsq docker restart annosaurus

# View container processes
./varsq docker ps

# Remove stopped containers
./varsq docker rm <service-name>

Python Scripts

The repository includes Python utility scripts for common VARS operations. These scripts are located in etc/python/.

Setting Up Python Environment

1. Create a virtual environment (recommended)

   python3 -m venv venv
   source venv/bin/activate

2. Install dependencies

   pip install -r etc/python/requirements.txt

Running Scripts

# List available scripts
./varsq scripts

# Run a script
./varsq run <script-name.py> [arguments]

# Example: Register media files
./varsq run vars_register_media.py --help

Services Included

The VARS stack includes the following microservices:

• annosaurus: Annotation service for managing video annotations
• vampire-squid: Video asset management service
• oni: Knowledge base service for taxonomic and descriptive data
• panoptes: Image and framegrab management service
• raziel: API gateway and service aggregator
• charybdis: Query service for complex data retrieval
• beholder: Image cache service
• skimmer: Image processing service
• nginx: Reverse proxy with SSL termination

All services are accessible through the configured VARS_WEB_SERVER hostname via HTTPS (default port 443) or HTTP (default port 80).

Troubleshooting

Services won't start

• Ensure SSL certificates exist: ./varsq mkcert
• Check that required environment variables are set: ./varsq resolved
• Verify Docker is running: docker ps
• Check logs: ./varsq docker logs <service-name>

Port conflicts

• Verify no other services are using ports 80, 443, or the configured service ports
• Check current port bindings: ./varsq resolved | grep PORT

Configuration issues

• Ensure you've run ./varsq configure <target> before starting services
• Verify the environment file exists: ls docker/env.sh
• Check for typos in environment variable names
• Not all services run as root, so allow docker to read all subdirectories in ./temp: sudo chgrp -R docker ./temp

---

VARS Applications

vars-quickstart-public provides the infrastructure to run various annotation applications. Below is a list of publicly available VARS apps.

VARS Annotation

Used to create annotations on video. Runs on Windows, macOS, and Linux. If you need a build for your OS, request one here.

Requires a VARS-compatible video player:

• Sharktopoda
• jsharktopoda

Important

Videos must be registered in VARS before they can be annotated. Use: varsq run vars_register_media.py --help for more information.

VARS Query

A web application for searching and retrieving annotations, videos, and images. A link is available on the Welcome to the MBARI Media Management (M3) Server page (http://localhost).

VARS Knowledgebase Editor

A web application for editing the knowledge base (lexicon and phylogenetic tree of annotation terms). A link is available on the Welcome to the MBARI Media Management (M3) Server page (http://localhost).

VARS Gridview

VARS Gridview is a bulk editing tool for reviewing and correcting bounding box annotations, such as machine learning generated annotations.

Mondrian (alpha)

Mondrian is an image annotation application.

Important

Images must be registered in VARS before they can be annotated. Use: varsq run vars_register_images.py --help for more information.

---

For Developers

Architecture Overview

The VARS quickstart uses a two-tier configuration system that merges environment-specific settings with core configuration values. The varsq Bash script serves as the primary orchestrator, managing environment setup and proxying Docker Compose operations.

Service Communication

Services communicate internally via Docker hostnames (e.g., http://annosaurus:8080) and externally via the configured VARS_WEB_SERVER hostname. The Raziel service coordinates internal service URLs and acts as an API gateway.

Project Structure

vars-quickstart-mbari/
├── varsq                    # Main orchestrator script
├── docker/
│   ├── compose.yml         # Docker Compose service definitions
│   ├── env.sh              # Generated merged environment file
│   └── nginx/              # Nginx reverse proxy configuration
├── etc/
│   └── env/
│       ├── core.env.sh     # Core environment variables (overrides)
│       ├── localhost.env   # Docker testing configuration for localhost
│       └── *.env           # Additional environment targets
└── temp/                   # Temporary files, SSL certificates

Environment Configuration System

The configuration system uses a two-tier approach:

1. Source files (etc/env/*.env): Environment-specific base configuration
2. Core file (etc/env/core.env.sh): Canonical variables that override source files

Critical Configuration Precedence

When ./varsq configure <source> runs, it:

1. Writes the source file FIRST to docker/env.sh
2. Appends etc/env/core.env.sh to the same file

In Bash, later export statements override earlier ones, so variables in core.env.sh take precedence. This ordering must not be changed.

See varsq:79-84 for the implementation.

Required Environment Variables

Source files must define these prerequisite variables:

• VARS_LOG_LEVEL - Logging level (TRACE, DEBUG, INFO, WARN, ERROR)
• VARS_WEB_SERVER - Web server hostname
• VARS_DATABASE_SERVER - Database server hostname
• POSTGRES_PWD / POSTGRES_USER - Database credentials
• BEHOLDER_CACHE_DIR, FRAMEGRABS_DIR, M3_TEMP_DIR - File system paths
• SSL_CERT_FILE, SSL_KEY_FILE - SSL certificate paths

Adding New Services

To add a new service to the VARS stack:

1. Add environment variables to etc/env/core.env.sh

   export NEWSERVICE_PUBLIC_PORT="9000"
   export NEWSERVICE_DOCKER_URL="http://newservice:8080"
   export NEWSERVICE_PUBLIC_URL="https://${VARS_WEB_SERVER}/newservice"

2. Update docker/compose.yml with the service definition

   newservice:
     container_name: newservice
     image: mbari/newservice
     restart: always
     ports:
       - "${NEWSERVICE_PUBLIC_PORT}:8080"
     environment:
       - LOGBACK_LEVEL=${LOGBACK_LEVEL}
     networks:
       - m3

3. Update the _update() function in varsq:211-223 to include image pulls

   docker pull mbari/newservice

4. Update Nginx configuration (if the service needs external access) Add proxy configuration in docker/nginx/conf/ directory

Service URL Patterns

Follow the existing pattern of dual URL variables:

• *_DOCKER_URL: Internal Docker network URL (e.g., http://service:8080)
• *_PUBLIC_URL: External URL via web server (e.g., https://vars.example.com/service)

Development Workflow

1. Create a new environment target

   cp etc/env/docker-dp.env etc/env/myenv.env
   # Edit myenv.env with your settings

2. Configure your environment

   ./varsq configure etc/env/myenv.env

3. Verify resolved configuration

   ./varsq resolved | grep MYSERVICE_

4. Start services

   ./varsq start

5. Test your changes

   ./varsq docker logs myservice
   ./varsq docker exec -it myservice bash

Working with the varsq Script

The varsq script is the main entry point (varsq:1-277). Key implementation details:

• Working directory: All Docker Compose operations execute in the docker/ directory
• Environment sourcing: The merged docker/env.sh is sourced at startup (varsq:45-49)
• Pass-through commands: The docker subcommand forwards all arguments to Docker Compose (varsq:94-105)

Environment File Conventions

• Use bash-style export VAR=value syntax
• Follow quoting conventions in etc/env/core.env.sh
• Never commit real secrets (use placeholders only)
• Use descriptive variable names with service prefixes (e.g., ANNOSAURUS_*, PANOPTES_*)

Nginx Configuration

Nginx serves as the SSL-terminating reverse proxy. Configuration files are in docker/nginx/conf/.

Key volume mounts:

• ${FRAMEGRABS_DIR} → /usr/local/nginx/html/framegrabs
• ${MEDIA_DIR}/media → /usr/local/nginx/html/media
• ${SSL_CERT_FILE} and ${SSL_KEY_FILE} for SSL termination

Testing Changes

After modifying configuration:

1. Reconfigure

   ./varsq configure etc/env/your-env.env

2. Rebuild if needed

   ./varsq build

3. Restart services

   ./varsq stop
   ./varsq start

4. Verify

   ./varsq status
   ./varsq docker logs <service-name>

Contributing

When contributing to this repository:

1. Test your changes with multiple environment targets
2. Ensure the configuration precedence system remains intact
3. Document any new environment variables in this README
4. Follow the existing patterns for service URL variables
5. Update the _update() function for any new Docker images
6. Keep environment files free of real secrets

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

Copyright 2025 Monterey Bay Aquarium Research Institute (MBARI)

Support

For issues and questions:

• File an issue on the GitHub repository
• Contact the MBARI VARS development team