Skip to content

Victor-Zarzar/api-cost-map

BranchesTags

Repository files navigation

API Cost Map - Python FastAPI Application

Python FastAPI MySQL Nginx Docker Grafana Pytest Alembic

Professional cost mapping and management API built with FastAPI, featuring MySQL database, Nginx reverse proxy, Grafana/Loki logging stack, database migrations with Alembic, and production-ready containerized deployment.

Table of Contents

Tech Stack

Core Technologies

  • Python 3.12+ - Modern Python with latest features
  • FastAPI - High-performance async web framework
  • MySQL 9.6 - Robust relational database management system
  • Nginx - High-performance reverse proxy and load balancer
  • Docker - Containerized deployment
  • Alembic - Database migration management
  • Pytest - Comprehensive testing framework

Infrastructure & Monitoring

  • Nginx - Reverse proxy with custom error pages and logging
  • Grafana - Metrics visualization and monitoring dashboards
  • Loki - Log aggregation and querying system
  • Promtail - Log collector and shipper to Loki
  • Docker Compose - Multi-container orchestration

Features & Integrations

  • Database Migrations - Automated schema version control with Alembic
  • MySQL Integration - Production-ready database with persistent storage
  • Reverse Proxy - Nginx with environment-based configuration
  • Centralized Logging - Grafana/Loki stack for log aggregation and visualization
  • RESTful API - Clean and scalable API architecture
  • Environment-based Configuration - Separate dev/prod settings
  • Custom Error Pages - Branded error handling through Nginx
  • Log Management - Nginx access and error logs with Promtail collection
  • Hot Reload - Development mode with automatic code reloading
  • Database Shell Access - Direct MySQL command-line interface
  • Code Quality Tools - Automated formatting and linting

Prerequisites

Before starting, ensure you have the following installed:

Optional: Python 3.12+ if you prefer running the app without Docker.

Installation & Setup

1. Clone the Repository

git clone https://github.com/your-username/api-cost-map
cd api-cost-map

2. Open in your editor

code .   # VS Code
# or
zed .    # Zed Editor

3. Environment Configuration

Copy the example environment file and configure your credentials:

cp .env-example .env.dev

Key configurations needed:

  • Database: MySQL credentials (user, password, database name)
  • API Settings: Host, port, and other application settings
  • Environment: Development or production mode

Important: Never commit your .env.dev or .env.prod files to version control. They should be in .gitignore.

4. Build Development Environment

make build-dev

This will:

  • Build the Docker image with tag api-cost-map-web-api:1.0.0
  • Set up the MySQL database container
  • Configure networking between services

Usage

Available Commands

View all available Make commands:

make help

Local Development

Start the development server (port 8000):

make up-dev

Access the API at:

  • API: http://localhost:8000
  • API Documentation: http://localhost:8000/docs
  • Alternative Docs: http://localhost:8000/redoc

Docker Deployment

Development Commands

make build-dev    # Build development Docker image
make up-dev       # Start development server with hot reload
make down-dev     # Stop development server
make logs-dev     # View development logs in real-time
make test         # Run tests with pytest
make format       # Format code with Ruff
make lint         # Lint code with pylint
make shell        # Access container bash shell
make migrate      # Run database migrations

Database Management

make access-db-local  # Access MySQL shell directly
make migrate          # Apply pending database migrations

Access database with credentials from your .env.dev:

  • Host: localhost
  • Port: 3306
  • Database: costdb
  • User: admin
  • Password: pass

Cleanup Commands

make clean        # Clean local environment and containers
make clean-all    # Remove all containers, volumes, and images

View Logs

make logs-dev     # Development logs
make logs-prod    # Production logs

Or directly with Docker:

docker logs -f api-cost-map
docker logs -f mysql-server

Makefile Commands Reference

Command Description
make build-dev Build development Docker image
make up-dev Start development server with hot reload
make down-dev Stop and remove development containers
make logs-dev Display development logs in real-time
make build-prod Build production Docker image
make up-prod Start production server (detached mode)
make down-prod Stop and remove production containers
make logs-prod Display production logs in real-time
make test Run automated tests with pytest
make format Format code with Ruff
make lint Lint code with pylint
make shell Access container bash shell
make migrate Run database migrations with Alembic
make access-db-local Access MySQL shell directly
make clean Clean local environment and containers
make clean-all Remove all containers, volumes, and images
make help Show all available commands with descriptions

Running Tests

make test

Or manually with Docker:

docker compose -f docker-compose.dev.yaml exec web pytest

API Documentation

FastAPI provides automatic interactive API documentation:

  • Swagger UI: http://localhost:8000/docs
  • ReDoc: http://localhost:8000/redoc

Database Management

Migrations with Alembic

This project uses Alembic for database schema version control.

Create a New Migration

# Inside the container
docker exec -it api-cost-map alembic revision --autogenerate -m "Description of changes"

Apply Migrations

make migrate
# or
docker exec -it api-cost-map python -m alembic upgrade head

Rollback Migration

docker exec -it api-cost-map python -m alembic downgrade -1

Direct Database Access

Access the MySQL shell:

make access-db-local

This opens an interactive MySQL session where you can run SQL queries directly.

Example queries:

-- Show all tables
SHOW TABLES;

-- Describe a table structure
DESCRIBE your_table_name;

-- Query data
SELECT * FROM your_table_name LIMIT 10;

Project Structure 

api-cost-map/
├── app/                        # Application code
│   ├── main.py                 # FastAPI application entry point
│   ├── routers/                # API route handlers
│   ├── models/                 # Database models (SQLAlchemy)
│   ├── schemas/                # Pydantic schemas
│   ├── services/               # Business logic
│   ├── database.py             # Database connection and session
│   └── config.py               # Configuration management
├── alembic/                    # Database migrations
│   ├── versions/               # Migration files
│   └── env.py                  # Alembic configuration
├── nginx/                      # Nginx configuration
│   ├── nginx.conf.template     # Nginx config with env vars
│   └── errors/                 # Custom error pages
├── grafana/                    # Grafana configuration
│   └── provisioning/           # Auto-provisioned datasources/dashboards
├── logs/                       # Application logs
│   └── nginx/                  # Nginx access and error logs
├── tests/                      # Test files
│   ├── test_api.py             # API endpoint tests
│   └── test_services.py        # Service layer tests
├── docker-compose.dev.yaml     # Development configuration
├── docker-compose.prod.yaml    # Production configuration
├── Dockerfile                  # Development Docker image
├── Dockerfile.prod             # Production Docker image
├── entrypoint.sh               # Container startup script
├── loki-config.yml             # Loki log aggregation config
├── promtail-config.yml         # Promtail log collection config
├── requirements.txt            # Python dependencies
├── .env.example                # Environment variables template
├── .env.dev                    # Development environment (not in git)
├── .env.prod                   # Production environment (not in git)
├── alembic.ini                 # Alembic configuration file
├── pyproject.toml              # Python project configuration
├── Makefile                    # Build automation
└── README.md                   # This file

Code Quality & Formatting

Ruff Configuration

This project uses Ruff for fast Python linting and code formatting. The configuration is defined in pyproject.toml:

[tool.ruff]
line-length = 88
target-version = "py312"

[tool.ruff.format]
quote-style = "double"
indent-style = "space"

[tool.ruff.lint]
select = ["E", "W", "F", "I", "B", "UP"]
ignore = ["E501"]
fixable = ["ALL"]
unfixable = []

Configuration details:

  • Line length: 88 characters (Black compatible)
  • Target version: Python 3.12
  • Quote style: Double quotes for strings
  • Linting rules:
    • E, W - pycodestyle errors and warnings
    • F - Pyflakes
    • I - isort (import sorting)
    • B - flake8-bugbear
    • UP - pyupgrade (modern Python syntax)
  • Ignored rules: E501 (line too long)

Running Code Quality Tools

All code quality tools run inside the Docker container:

# Format code with Ruff
make format

# Lint code with pylint
make lint

Or directly with Docker Compose:

# Format code
docker compose -f docker-compose.dev.yaml exec web ruff format app

# Lint code
docker compose -f docker-compose.dev.yaml exec web pylint app

API Endpoints

Core Endpoints

The API provides RESTful endpoints for cost management:

  • GET /api/v1/costs - List all costs
  • POST /api/v1/costs - Create new cost
  • GET /api/v1/costs/{id} - Get cost by ID
  • PUT /api/v1/costs/{id} - Update cost
  • DELETE /api/v1/costs/{id} - Delete cost

Authentication

Depending on your configuration, endpoints may require authentication. Check the API documentation for details:

  • Development: http://localhost:8000/docs
  • Production (via Nginx): http://localhost/docs

Deployment

Docker Production

Build and run the production container:

# Start production environment
make up-prod

# Check logs
make logs-prod

# Stop when needed
make down-prod

Production Architecture

The production environment includes:

  • FastAPI Application - Main API service (port 8006 internal)
  • MySQL Database - Persistent data storage (port 3306)
  • Nginx - Reverse proxy (port 80)
  • Grafana - Monitoring dashboard (port 3000)
  • Loki - Log aggregation (port 3100)
  • Promtail - Log collection from Nginx

Accessing Services

Once deployed, access the services at:

  • API: http://localhost (via Nginx reverse proxy)
  • API Docs: http://localhost/docs
  • Grafana Dashboard: http://localhost:3000
  • MySQL: localhost:3306
  • Loki API: http://localhost:3100

Production Considerations

  • Use strong database credentials
  • Configure proper backup strategies for MySQL
  • Set up monitoring alerts in Grafana
  • Use environment-specific configuration files
  • Consider using Docker secrets for sensitive data
  • Implement proper SSL/TLS certificates for Nginx
  • Configure log retention policies in Loki
  • Set up automated database backups

Monitoring & Logging

Grafana Dashboard

Access Grafana at http://localhost:3000 to visualize metrics and logs.

Default credentials (configure in .env.prod):

  • Check your environment file for credentials

Features:

  • Pre-configured Loki datasource
  • Nginx access and error log visualization
  • Real-time log streaming
  • Log filtering and searching
  • Custom dashboard creation

Loki Log Aggregation

Loki aggregates logs from multiple sources:

Log Sources:

  • Nginx access logs (/var/log/nginx/access.log)
  • Nginx error logs (/var/log/nginx/error.log)
  • Application logs (via Promtail configuration)

Querying Logs:

Access Loki directly at http://localhost:3100 or through Grafana's Explore interface.

Example LogQL queries:

# All nginx logs
{job="nginx"}

# Error logs only
{job="nginx"} |= "error"

# Logs from specific time range
{job="nginx"} | json | line_format "{{.timestamp}} {{.message}}"

Promtail Configuration

Promtail collects logs and ships them to Loki. Configuration location:

./promtail-config.yml

Monitored paths:

  • /var/log/nginx - Nginx logs mounted from host

Nginx Logging

Nginx logs are stored in:

./logs/nginx/access.log  # HTTP access logs
./logs/nginx/error.log   # Nginx error logs

These logs are automatically collected by Promtail and sent to Loki.

Custom Error Pages

Nginx serves custom error pages from:

./nginx/errors/

Configure error pages in nginx.conf.template to maintain branding during errors.

Configuration Files

Nginx Configuration

Location: ./nginx/nginx.conf.template

Environment variables used:

  • ${BACKEND_HOST} - FastAPI service hostname (default: web)
  • ${BACKEND_PORT} - FastAPI service port (default: 8006)

Example configuration:

upstream backend {
    server ${BACKEND_HOST}:${BACKEND_PORT};
}

server {
    listen 80;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Loki Configuration

Location: ./loki-config.yml

Key settings:

  • Log retention period
  • Storage configuration
  • Ingestion limits

Promtail Configuration

Location: ./promtail-config.yml

Configure:

  • Log file paths
  • Labels and metadata
  • Loki server endpoint

Grafana Provisioning

Location: ./grafana/provisioning/

Auto-provisioned:

  • Datasources (Loki)
  • Dashboards (if configured)
  • Alert rules (if configured)

Troubleshooting

Database Connection Issues

Database not accessible:

  • Ensure MySQL container is running: docker ps
  • Check database credentials in .env.prod
  • Verify port 3306 is not in use by another service
  • Check Docker network connectivity

Migration errors:

  • Ensure database is initialized: make access-db-local
  • Check Alembic configuration in alembic.ini
  • Review migration files in alembic/versions/

Nginx Issues

502 Bad Gateway:

  • Verify FastAPI container is running: docker ps | grep web
  • Check backend configuration in nginx.conf.template
  • Ensure BACKEND_HOST and BACKEND_PORT are correct
  • View Nginx error logs: cat logs/nginx/error.log

Custom error pages not showing:

  • Verify error page files exist in ./nginx/errors/
  • Check Nginx configuration for error_page directives
  • Restart Nginx: docker restart api-cost-map-nginx

Logging & Monitoring Issues

Grafana not accessible:

  • Ensure Grafana container is running: docker ps | grep grafana
  • Check port 3000 availability: lsof -i :3000
  • Verify credentials in .env.prod

Logs not appearing in Grafana:

  • Check Promtail is running: docker ps | grep promtail
  • Verify Loki datasource is configured in Grafana
  • Check Promtail configuration: cat promtail-config.yml
  • Ensure log files exist: ls -la logs/nginx/
  • Test Loki directly: curl http://localhost:3100/ready

Loki storage issues:

  • Check disk space: df -h
  • Review Loki configuration for retention settings
  • Check Loki logs: docker logs loki

Container Issues

Port already in use:

# Check what's using port 80
lsof -i :80

# Or port 3306 for MySQL
lsof -i :3306

# Or port 3000 for Grafana
lsof -i :3000

Clean restart:

make clean
make build-dev
make up-dev

Network Issues

Containers can't communicate:

  • Verify all containers are on app-network: docker network inspect app-network
  • Check Docker Compose network configuration
  • Restart Docker Compose: make down-prod && make up-prod

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the project
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Development Guidelines

  • Write tests for new features
  • Follow PEP 8 style guide
  • Update documentation as needed
  • Ensure all tests pass before submitting PR
  • Create migrations for database schema changes
  • Run make format and make lint before committing

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

Victor Zarzar - @Victor-Zarzar

Project Link: https://github.com/Victor-Zarzar/api-cost-map

Made with by Victor Zarzar

About

No description or website provided.

Topics

mysql python docker grafana fastapi

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages