Chrome Extension Micro Service

A Node.js microservice designed to process webhook requests from Chrome extensions for YouTube and Instagram platform data. The service uses BullMQ for sequential job processing and PostgreSQL for data storage.

🏗️ Architecture Overview

This microservice processes webhook requests through a series of workflows:

1. Webhook Reception: Receives requests from Chrome extensions
2. Queue Management: Uses BullMQ for sequential processing
3. Multi-Workflow Processing: Processes data through 6 interconnected workflows
4. Database Operations: Stores and retrieves data from PostgreSQL databases
5. External API Integration: Fetches data from YouTube APIs

🔄 Complete System Flow

graph TD
    A["Chrome Extension"] --> B["POST /api/chrome-extension"]
    B --> C["WORKFLOW 1: Webhook Processing"]
    
    C --> D["Rate Limiting & Validation"]
    D --> E["BullMQ Queue"]
    E --> F["Sequential Job Processor"]
    F --> G{Platform Type?}
    
    G -->|YouTube| H["WORKFLOW 2: Channel Validation"]
    G -->|Instagram| I["Instagram User Processing"]
    
    H --> J["Check Channel in Database"]
    J --> K{Channel Exists?}
    K -->|Yes| L["Return Channel Exists"]
    K -->|No| M["Fetch from YouTube API"]
    
    M --> N["🚫 VEVO Channel Check"]
    N --> O{Contains 'VEVO'?}
    O -->|Yes| P["⏭️ Skip Processing"]
    O -->|No| Q{Channel Valid?}
    Q -->|No| R["Mark as Deleted"]
    Q -->|Yes| S["Update Channel Data"]
    
    S --> T["WORKFLOW 3: Playlist Generation"]
    T --> U["Generate Playlist IDs"]
    U --> V["Update Channel with Playlists"]
    
    V --> W["WORKFLOW 4: Video Processing"]
    W --> X["Fetch Latest Videos"]
    X --> Y{New Videos?}
    Y -->|No| Z["Update Channel Status"]
    Y -->|Yes| AA["Insert New Videos"]
    
    AA --> BB["WORKFLOW 5: Video Details Processing"]
    BB --> CC["Fetch Video Details API"]
    CC --> DD["Validate Video Data"]
    DD --> EE["Distribute to Workers"]
    
    EE --> FF["WORKFLOW 6.0: Worker Group 1"]
    EE --> GG["WORKFLOW 6.1: Worker Group 2"]
    EE --> HH["WORKFLOW 6.2: Worker Group 3"]
    EE --> II["WORKFLOW 6.3: Worker Group 4"]
    EE --> JJ["WORKFLOW 6.4: Worker Group 5"]
    
    FF --> KK["Update Video Details"]
    GG --> KK
    HH --> KK
    II --> KK
    JJ --> KK
    
    KK --> LL["Update Video Statistics"]
    LL --> MM["Processing Complete"]
    
    I --> NN["Check Instagram User"]
    NN --> OO{User Exists?}
    OO -->|Yes| PP["Return User Exists"]
    OO -->|No| QQ["Create New User"]

Loading

📋 Table of Contents

• Features
• Tech Stack
• Project Structure
• Installation
• Configuration
• API Endpoints
• Workflow Documentation
• Docker Deployment
• Environment Variables
• Database Schema
• Queue Management
• Logging
• Development
• Testing
• Contributing

✨ Features

• Multi-Platform Support: Handles YouTube and Instagram data processing
• Sequential Processing: Uses BullMQ for reliable job queue management
• 🚫 VEVO Channel Filtering: Automatically filters out VEVO channels from processing
• Scalable Architecture: Microservice-based design with Docker support
• Rate Limiting: Built-in protection against abuse (1000 requests per minute)
• Comprehensive Logging: Structured logging with Winston
• Health Monitoring: Health check endpoints for service monitoring
• Error Handling: Robust error handling with retry mechanisms
• Database Pooling: Efficient PostgreSQL connection management

🛠️ Tech Stack

• Runtime: Node.js 18+
• Framework: Express.js
• Language: TypeScript
• Queue: BullMQ with Redis
• Database: PostgreSQL
• ORM: Native PostgreSQL queries with connection pooling
• Logging: Winston
• Containerization: Docker & Docker Compose
• Process Management: PM2 (via npm scripts)

📁 Project Structure

Chrome-Extension-Micro-Service/
├── src/
│   ├── app.ts                 # Main application entry point
│   ├── config/
│   │   ├── config.ts          # Configuration management
│   │   └── database.ts        # Database connection setup
│   ├── controllers/
│   │   ├── WebhookController.ts    # Main webhook handler
│   │   ├── WorkerController.ts     # Worker job controller
│   │   └── YouTubeController.ts    # YouTube-specific controller
│   ├── interfaces/
│   │   └── WebhookPayload.ts       # Type definitions
│   ├── models/
│   │   └── Video.ts               # Video data models
│   ├── repositories/
│   │   ├── VideoDetailsRepository.ts   # Video details data access
│   │   └── VideoStatsRepository.ts     # Video stats data access
│   ├── routes/
│   │   ├── Webhook.routes.ts      # Webhook routing
│   │   └── youtube.routes.ts      # YouTube routing
│   ├── services/
│   │   ├── DatabaseService.ts          # Database operations
│   │   ├── LoggerService.ts            # Logging service
│   │   ├── QueueService.ts             # BullMQ queue management
│   │   ├── YouTubeValidatorService.ts  # YouTube data validation
│   │   ├── PlaylistGeneratorService.ts # Playlist ID generation
│   │   ├── VideoProcessorService.ts    # Video processing logic
│   │   ├── VideoDetailsProcessorService.ts # Video details processing
│   │   └── VideoWorkerService.ts       # Individual video processing
│   └── types/
│       └── youtube.ts             # YouTube type definitions
├── docs/                          # Documentation files
├── logs/                          # Application logs
├── tests/                         # Test files
├── docker-compose.yml             # Docker Compose configuration
├── Dockerfile                     # Docker image configuration
├── package.json                   # Node.js dependencies
└── tsconfig.json                  # TypeScript configuration

🚀 Installation

Prerequisites

• Node.js 18 or higher
• PostgreSQL databases (YouTube and Instagram)
• Redis server
• Docker (optional, for containerized deployment)

Local Development Setup

1. Clone the repository

   git clone <repository-url>
   cd Chrome-Extension-Micro-Service

2. Install dependencies

   npm install

3. Set up environment variables

   cp .env.example .env
   # Edit .env with your configuration

4. Build the project

   npm run build

5. Start the development server

   npm run dev

⚙️ Configuration

Environment Variables

Create a .env file in the root directory:

# Server Configuration
PORT=4000
NODE_ENV=development

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=your_redis_password

# YouTube API Configuration
YOUTUBE_FETCH_URL=https://your-youtube-api.com
YOUTUBE_API_KEY=your_youtube_api_key

# YouTube Database Configuration
YOUTUBE_DB_HOST=your_youtube_db_host
YOUTUBE_DB_PORT=5432
YOUTUBE_DB_USER=your_youtube_db_user
YOUTUBE_DB_PASSWORD=your_youtube_db_password
YOUTUBE_DB_NAME=your_youtube_db_name
YOUTUBE_DB_SSL=require

# Instagram Database Configuration
INSTAGRAM_DB_HOST=your_instagram_db_host
INSTAGRAM_DB_PORT=5432
INSTAGRAM_DB_USER=your_instagram_db_user
INSTAGRAM_DB_PASSWORD=your_instagram_db_password
INSTAGRAM_DB_NAME=your_instagram_db_name
INSTAGRAM_DB_SSL=require

🔌 API Endpoints

Webhook Endpoints

POST /api/chrome-extension

Processes webhook requests from Chrome extensions.

Request Body:

{
  "platform": "youtube|instagram",
  "username": "channel_username",
  "channel_id": "UC1234567890",
  "channel_username": "@channel_name",
  "yt_channel_id": "UC1234567890"
}

Response:

{
  "success": true,
  "message": "Request queued for processing",
  "requestId": "req_1234567890_abc123",
  "status": "queued"
}

Monitoring Endpoints

GET /health

Health check endpoint for service monitoring.

Response:

{
  "status": "ok",
  "timestamp": "2024-01-15T10:30:00.000Z"
}

GET /api/queue/status

Returns current queue statistics and job status.

Response:

{
  "success": true,
  "data": {
    "waiting": 5,
    "active": 1,
    "completed": 100,
    "failed": 2,
    "delayed": 0,
    "total": 108,
    "recentJobs": [...],
    "currentJob": {...}
  }
}

📊 Workflow Documentation

The service processes data through 6 interconnected workflows:

Workflow 1: Webhook Processing

• Receives and validates webhook requests
• Queues requests for sequential processing
• Returns immediate response to client

Workflow 2: Channel Validation

• Validates YouTube channel existence
• Fetches channel data from YouTube API
• Updates channel details in database

Workflow 3: Playlist Generation

• Generates playlist IDs for different content types
• Updates channel with generated playlist IDs

Workflow 4: Video Processing

• Fetches latest videos from playlists
• Identifies new videos for processing
• Inserts new video records

Workflow 5: Video Details Processing

• Fetches detailed video information
• Distributes videos across worker groups
• Coordinates parallel video processing

Workflow 6: Video Worker Processing

• Updates individual video details
• Processes video statistics
• Handles video-specific data operations

For detailed workflow diagrams, see the docs/workflows/ directory.

🐳 Docker Deployment

Using Docker Compose

1. Start all services

   docker-compose up -d

2. View logs

   docker-compose logs -f

3. Stop services

   docker-compose down

Individual Docker Commands

1. Build the image

   docker build -t chrome-extension-service .

2. Run the container

   docker run -d \
     --name webhook-service \
     -p 4000:4000 \
     --env-file .env \
     chrome-extension-service

🗄️ Database Schema

YouTube Database Tables

• channel_details: Channel information and metadata
• channel_stats: Channel statistics and metrics
• video_details: Video information and metadata
• video_stats: Video statistics and engagement data

Instagram Database Tables

• ig_users: Instagram user profiles and data

📋 Queue Management

The service uses BullMQ for reliable job processing:

• Sequential Processing: Jobs are processed one at a time
• Retry Logic: Failed jobs are automatically retried with exponential backoff
• Job Persistence: Jobs are persisted in Redis for reliability
• Progress Tracking: Real-time job progress and status monitoring

Queue Configuration

{
  concurrency: 1,              // Sequential processing
  attempts: 3,                 // Retry failed jobs 3 times
  backoff: {
    type: 'exponential',
    delay: 2000                // Start with 2-second delay
  },
  removeOnComplete: 100,       // Keep last 100 completed jobs
  removeOnFail: 50            // Keep last 50 failed jobs
}

📝 Logging

The service uses Winston for structured logging:

• Error Logs: Stored in logs/error.log
• Workflow Logs: Stored in logs/workflow.log
• Console Output: Colored console logging for development

Log Levels

• error: Error conditions and exceptions
• warn: Warning conditions
• info: General information and workflow steps
• debug: Detailed debugging information

🛠️ Development

Available Scripts

npm run dev          # Start development server with nodemon
npm run build        # Build TypeScript to JavaScript
npm start            # Start production server
npm test             # Run test suite
npm run lint         # Run ESLint code analysis

Development Workflow

1. Make changes to TypeScript files in src/
2. The development server will automatically restart on changes
3. Test your changes using the provided endpoints
4. Run tests to ensure functionality
5. Build and test the production version

🧪 Testing

Run the test suite:

npm test

The project uses Jest for testing with TypeScript support.

📈 Monitoring and Observability

Health Checks

• Service health: GET /health
• Queue status: GET /api/queue/status
• Docker health checks included in container configuration

Logging

• Structured JSON logging for production
• Colored console logs for development
• Separate log files for errors and workflows
• Request tracking with unique request IDs

Metrics

Monitor these key metrics:

• Queue processing rate
• Job success/failure rates
• Database connection health
• API response times
• Memory and CPU usage

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Code Style

• Use TypeScript for all new code
• Follow existing naming conventions
• Add appropriate error handling
• Include logging for important operations
• Write tests for new functionality

🆘 Support

For support and questions:

1. Check the documentation in the docs/ directory
2. Review the workflow diagrams for understanding the data flow
3. Check the logs for detailed error information
4. Ensure all environment variables are properly configured

🔧 Troubleshooting

Common Issues

1. Database Connection Errors

   • Verify database credentials in .env
   • Check database server accessibility
   • Ensure SSL configuration is correct

2. Redis Connection Issues

   • Verify Redis server is running
   • Check Redis connection parameters
   • Ensure Redis is accessible from the application

3. Queue Processing Issues

   • Check Redis connectivity
   • Monitor queue status endpoint
   • Review worker logs for processing errors

4. API Integration Problems

   • Verify external API endpoints are accessible
   • Check API key validity
   • Monitor rate limiting and quotas