Titan Orchestrator is a distributed job orchestration system built with .NET 8, gRPC, and Redis. It demonstrates a high-performance, cloud-native architecture using the "Master-Worker" pattern, deployed via AWS Fargate.
The system is architected for security and scalability using AWS best practices:
- Public Subnet: Hosts the Application Load Balancer (ALB), which serves as the secure entry point for external HTTP traffic.
- Private Subnet (The "Vault"):
- Master Node: An ASP.NET Core API that manages the job queue and orchestrates workers via a persistent, bi-directional gRPC stream.
- Worker Nodes: Auto-scaling console applications that run in isolation. They have no public IP and reach the internet (for ECR pulls) only via a NAT Gateway.
- Redis: Provides distributed state management, accessible only by the Master node.
You can run the entire system locally using Docker Compose to simulate the distributed environment.
- Prerequisites: Ensure you have Docker and Docker Compose installed.
- Run:
docker-compose up --build
- Access:
- Master HTTP API:
http://localhost:5050 - Master gRPC:
http://localhost:5001 - Redis:
localhost:6379
- Master HTTP API:
Submit a job to the Master via curl:
curl -X POST http://localhost:5050/api/jobs \
-H "Content-Type: application/json" \
-d '{"payload": "Process Image A"}'This project is a Proof of Concept (PoC) designed to demonstrate distributed system patterns in .NET 8. To graduate this to a production environment, the following improvements would be prioritized:
- Durability: Replace the in-memory
ConcurrentQueuewith Amazon SQS or Redis Streams to ensure jobs survive a Master node restart. - Security: Implement mTLS (Mutual TLS) for the gRPC channel to encrypt traffic between Master and Workers.
- Observability: Integrate OpenTelemetry to trace requests from the Load Balancer -> Master -> Redis -> Worker.
- Identity: Add OIDC Authentication (e.g., Auth0 or AWS Cognito) to the REST API.