Vault Project Structure

This document provides a comprehensive overview of the Vault repository structure, explaining the purpose and organization of each component.

Repository Overview

Vault is a TypeScript/Express.js implementation of the Outbox Pattern using PostgreSQL WAL (Write-Ahead Log) for reliable event streaming. The repository contains:

• TypeScript implementation - Core library and consumer service
• Express.js integration - HTTP server and middleware
• Infrastructure - Docker, scripts, and configuration files

Directory Structure

vault/
├── src/                    # Source code
│   ├── server/             # Express.js server
│   ├── lib/                # Core library packages
│   ├── consumers/          # Event consumers
│   └── types/              # TypeScript type definitions
├── scripts/                # Utility scripts
├── docker-compose.yaml     # Docker Compose configuration
├── Dockerfile              # Dockerfile for Vault service
├── package.json            # Node.js dependencies
├── tsconfig.json           # TypeScript configuration
├── justfile                # Task runner configuration
├── README.md               # Main project documentation
└── structure.md            # This file

Detailed Structure

/src/server - Express.js Server

HTTP server and API endpoints.

src/server/
├── index.ts                # Main Express.js server entry point
├── routes/                 # API routes
│   ├── health/            # Health check endpoints
│   │   └── index.ts       # Health check route handlers
│   └── metrics/            # Metrics endpoints
│       └── index.ts       # Prometheus metrics route handlers
├── middleware/             # Express middleware
│   ├── errorHandler.ts    # Error handling middleware
│   └── logger.ts          # Request logging middleware
└── config/                 # Server configuration
    ├── config.ts          # Configuration management
    └── logger.ts          # Logger initialization

Purpose:

• src/server/ - Express.js HTTP server with health checks and metrics
• src/server/routes/health/ - Health check endpoints for monitoring
• src/server/routes/metrics/ - Prometheus metrics endpoint
• src/server/middleware/ - Express middleware for error handling and logging

/src/lib - Core Library Packages

Reusable TypeScript packages that form the core of Vault.

src/lib/
├── common/                 # Shared types and utilities
│   ├── fact.ts            # Fact interface and validation
│   └── types.ts           # Common interfaces and type aliases
│
├── logger/                 # Logging abstraction
│   ├── logger.ts          # Logger interface
│   └── default.ts         # Default logger implementation
│
├── metrics/                # Prometheus metrics
│   └── metrics.ts         # Metric definitions and registration
│
├── outbox/                 # Outbox pattern implementation
│   ├── consumer/          # Event consumer
│   │   ├── consumer.ts    # OutboxConsumer - main consumer logic
│   │   ├── consumer.test.ts # Consumer tests
│   │   └── kafka.ts       # Kafka adapter for publishing events
│   └── producer/          # Event producer
│       └── producer.ts    # PostgresAdapter - emits events to WAL
│
├── postgres/               # PostgreSQL integration
│   ├── wal.ts             # WALSubscriber - logical replication consumer
│   ├── wal.integration.test.ts # Integration tests for WAL
│   ├── health.ts          # Health check utilities
│   ├── pool.ts            # Connection pool management
│   └── types.ts           # PostgreSQL-specific types and interfaces
│
└── proto/                  # Protocol Buffer definitions
    ├── outbox.proto       # Protobuf schema for OutboxEvent
    └── outbox.ts          # Generated TypeScript types from proto

Key Components:

src/lib/common/

• fact.ts: Defines the Fact interface representing an event
• types.ts: Common interfaces (OutboxClient, OutboxWorker, EventHandler)

src/lib/outbox/producer/

• producer.ts: PostgresAdapter - Emits events using pg_logical_emit_message()
  • Works within database transactions
  • Serializes events as Protocol Buffers
  • Supports prefix-based routing

src/lib/outbox/consumer/

• consumer.ts: OutboxConsumer - Orchestrates event consumption
  • Subscribes to WAL events
  • Routes events to registered handlers
  • Manages acknowledgment flow
• kafka.ts: KafkaAdapter - Kafka producer implementation
  • Publishes events to Kafka topics
  • Handles acknowledgments
  • Supports SSL/SASL authentication

src/lib/postgres/

• wal.ts: WALSubscriber - PostgreSQL logical replication subscriber
  • Manages replication slots and publications
  • Consumes logical replication stream
  • Filters events by prefix
  • Tracks LSN (Log Sequence Number) for resumability
• pool.ts: Connection pool management using pg or pgx libraries
• health.ts: Health check utilities for monitoring

src/lib/proto/

• outbox.proto: Protocol Buffer schema defining OutboxEvent message
• outbox.ts: Generated TypeScript types (do not edit manually)

/src/consumers - Consumer Services

Consumer service implementations.

src/consumers/
├── vault-consumer/         # Main Vault consumer service
│   ├── index.ts           # Consumer service entry point
│   └── handlers/          # Event handlers
│       └── default.ts     # Default event handler
│
└── e2e/                    # End-to-end testing consumer
    └── index.ts           # E2E test runner demonstrating full pipeline

Purpose:

• src/consumers/vault-consumer/ - Production-ready consumer service that bridges PostgreSQL WAL to Kafka
• src/consumers/e2e/ - Demonstrates complete workflow: producer → WAL → consumer → Kafka

/scripts - Utility Scripts

Helper scripts for setup and development.

scripts/
├── init-postgres.sh        # PostgreSQL initialization script
│                          # Creates publication and replication slot
│
└── console-config.yml      # Kafka console configuration

Usage:

• init-postgres.sh: Run once to set up PostgreSQL for logical replication
• console-config.yml: Configuration for Kafka console tools

Root Files

Configuration Files

• docker-compose.yaml: Docker Compose setup for development

  • PostgreSQL with logical replication enabled
  • Kafka broker
  • Zookeeper (for Kafka)
  • Vault service

• Dockerfile: Docker image for Vault consumer service

• justfile: Task runner configuration (alternative to Make)

  • Common tasks: build, test, lint, run

• package.json: Node.js dependencies and scripts

• tsconfig.json: TypeScript compiler configuration

Documentation

• README.md: Main project documentation

  • Problem statement
  • Architecture overview
  • Setup instructions
  • Usage examples
  • Reliability guarantees

• structure.md: This file - repository structure documentation

Key Design Patterns

1. Outbox Pattern

• Events are emitted atomically within database transactions
• Separate consumer process reads and publishes events
• Ensures consistency between database and message broker

2. Adapter Pattern

• PostgresAdapter abstracts event emission
• KafkaAdapter abstracts Kafka publishing
• Supports multiple database drivers (pg, pgx)

3. Interface Segregation

• SQLExecutor interface for database operations
• OutboxProducer interface for event emission
• EventHandler interface for event processing

4. Protocol Buffers

• Events serialized as Protocol Buffers
• Language-agnostic event format
• Supports schema evolution

Testing Structure

src/lib/
├── outbox/
│   └── consumer/
│       └── consumer.test.ts    # Unit tests for consumer
│
└── postgres/
    └── wal.integration.test.ts  # Integration tests for WAL

Test Types:

• Unit Tests: Test individual components in isolation
• Integration Tests: Test WAL subscription with real PostgreSQL
• E2E Tests: Full pipeline test in src/consumers/e2e/

Monitoring and Observability

Metrics (src/lib/metrics/metrics.ts)

Prometheus metrics exposed at /metrics:

• vault_events_emitted_total - Total events emitted
• vault_events_consumed_total - Total events consumed
• vault_event_processing_seconds - Event processing latency
• vault_kafka_produce_success_total - Successful Kafka publishes
• vault_kafka_produce_failures_total - Failed Kafka publishes
• vault_wal_replication_errors_total - WAL replication errors
• vault_event_handler_success_total - Successful handler executions
• vault_event_handler_failures_total - Failed handler executions

Health Checks (src/server/routes/health/)

HTTP endpoint: GET /health

• Returns 200 OK if WAL consumer is healthy
• Returns 503 Service Unavailable if unhealthy
• Monitors replication slot status and connection health

Development Workflow

1. Setup PostgreSQL: Run scripts/init-postgres.sh
2. Start Infrastructure: docker-compose up -d postgres kafka
3. Install Dependencies: npm install or yarn install
4. Build: npm run build or just build
5. Run Consumer: npm start or docker-compose up vault
6. Emit Events: Use producer from your application
7. Monitor: Check /health and /metrics endpoints

Language Support

Language	Status	Location	Purpose
TypeScript	✅ Production	src/	Core library and consumer service
Express.js	✅ Production	src/server/	HTTP server and API

Dependencies

Node.js Dependencies (package.json)

• pg or pgx - PostgreSQL driver
• pg-logical-replication - Logical replication library
• kafkajs or node-rdkafka - Kafka client
• protobufjs - Protocol Buffer support
• prom-client - Prometheus metrics
• express - HTTP server framework
• typescript - TypeScript compiler

Contributing

When adding new features:

1. Core Logic: Add to appropriate src/lib/ package
2. Server Routes: Add to src/server/routes/ directory
3. Tests: Add alongside code with .test.ts suffix
4. Documentation: Update README.md and structure.md
5. Examples: Add to src/consumers/e2e/ for demonstration

File Naming Conventions

• TypeScript files: camelCase.ts or kebab-case.ts
• Test files: *.test.ts or *.spec.ts
• Proto files: *.proto
• Config files: *.yaml, *.yml, *.json
• Scripts: *.sh

Important Notes

1. Never edit generated files: outbox.ts (from proto) is generated
2. Transaction safety: Always emit facts within database transactions
3. LSN tracking: Consumer tracks LSN positions for resumability
4. Prefix filtering: Events are filtered by prefix in WAL subscriber
5. Health monitoring: Consumer exposes health checks for monitoring
6. TypeScript strict mode: Project uses strict TypeScript configuration
7. Async/await: Prefer async/await over callbacks for better error handling

---

Last updated: Generated from current repository structure