feat(pipeline): introduce advanced infrastructure for resilience and observability

[dev-mirzabicer]

This commit introduces a comprehensive suite of advanced systems to enhance the HiRAG pipeline's robustness, observability, and operational control. These systems are integrated into the core HiRAG class and are configurable via new parameters.

The new infrastructure includes:

• Checkpointing System (_checkpointing.py): Enables resumable ingestion operations. If the pipeline fails, it can be restarted from the last successful stage, saving progress and resources.

• Retry Manager (_retry_manager.py): Implements intelligent retry logic for LLM calls with exponential backoff, jitter, and a circuit breaker pattern to prevent cascading failures.

• Rate Limiter (_rate_limiting.py): Provides sophisticated, per-model rate limiting using a token bucket algorithm, adaptive adjustments, and backpressure to manage API usage effectively.

• Progress Tracker (_progress_tracking.py): Adds real-time progress monitoring, ETA calculations, and a rich terminal-based dashboard for detailed operational insight.

• Token Estimator (_token_estimation.py): A full framework to estimate token usage and cost for the entire pipeline, enabling better planning and resource management.

• Estimation Database (_estimation_db.py): A learning component that stores actual token usage to continuously improve the accuracy of the Token Estimator over time.

@claude Please review the changes in an architectural / logical standpoint. By that I mean don't focus on the "testing" or "security" aspects, as I will implement them in a later step. Focus on the core logic, the pipeline, flow of information, the soundness, the architectural robustness, and such.

[claude]

Claude finished @dev-mirzabicer's task —— View job

---

Architectural Review Complete ✅

Overall Assessment: Excellent Architecture with Minor Recommendations

This PR introduces a sophisticated, enterprise-grade infrastructure that significantly enhances HiRAG's robustness, observability, and operational control. The architecture demonstrates excellent design principles with thoughtful component separation and integration.

✅ Architectural Strengths

1. Component Cohesion & Separation of Concerns

• Each module has a clear, single responsibility with well-defined interfaces
• Clean separation between infrastructure (reliability) and business logic (extraction/reasoning)
• Excellent use of dependency injection pattern for component integration

2. Information Flow Architecture

TokenEstimator → PipelineEstimate → ProgressTracker → Dashboard
     ↓                ↓                    ↓
EstimationDB ← RateLimiter ←─── RetryManager ←─── CheckpointManager

• Data flows logically from estimation → execution → learning
• Each component enriches the data without tight coupling
• Proper event-driven architecture with async patterns

3. Resilience & Error Handling

• Circuit Breaker Pattern: Prevents cascading failures across the system
• Graceful Degradation: Components disable themselves on initialization failure rather than crashing
• State Preservation: Checkpointing ensures no work is lost during failures
• Intelligent Retry Logic: Sophisticated failure classification with appropriate retry strategies

4. Learning & Adaptability

• Feedback Loop: Actual usage data flows back to improve future estimates
• Dynamic Parameters: Rate limits and estimates adjust based on real-world performance
• Statistical Analysis: EstimationDatabase provides insights for continuous improvement

✅ Technical Excellence

1. Async Architecture

• Proper use of asyncio throughout with appropriate locking mechanisms
• Non-blocking operations with intelligent waiting strategies
• Well-structured concurrent execution patterns

2. Configuration Management

• Feature flags enable/disable each component independently
• Sensible defaults with environment-specific overrides
• Clear configuration dependency chains

3. Observability

• Comprehensive metrics collection at multiple levels
• Rich terminal UI with real-time progress tracking
• Export capabilities for external monitoring integration

🔧 Architectural Robustness Analysis

Pipeline Integration Points (hirag.py:480-500):

def _wrap_llm_functions(self):
    # Excellent layered approach - each wrapper adds capability
    original_func → retry_wrapper → rate_limit_wrapper → estimation_wrapper

This demonstrates excellent decorator pattern usage for cross-cutting concerns.

Inter-Component Communication:

• ProgressTracker properly aggregates data from RateLimiter and RetryManager
• TokenEstimator feeds data to EstimationDatabase for learning
• CheckpointManager preserves state across all components
• No circular dependencies or tight coupling detected

Data Consistency:

• Atomic operations in CheckpointManager prevent partial state
• Lock-based coordination prevents race conditions
• Proper error boundaries prevent corruption propagation

💡 Minor Recommendations for Enhancement

1. Token Bucket Refinement (_rate_limiting.py:136-142)

def time_until_available(self, tokens_needed: int) -> float:
    # Current implementation is good, but could benefit from:
    # - Burst allowance consideration in timing calculations
    # - Priority queue integration for fairer resource allocation

2. Checkpoint Granularity (_checkpointing.py:569-596)

• Consider adding sub-stage checkpoints for long-running operations (entity extraction loops)
• Potential for parallel stage execution where dependencies allow

3. Cost Prediction Accuracy (_token_estimation.py:686-702)

• Current model pricing is static; consider integrating real-time pricing APIs
• Add confidence intervals to cost estimates based on historical variance

4. Memory Management

• Consider implementing memory-efficient streaming for large datasets
• Add memory usage tracking to progress metrics

🏗️ Architectural Patterns Analysis

Factory Pattern: Excellent use throughout (create_* functions) providing consistent initialization
Observer Pattern: Progress tracking and statistics collection follow this well
Strategy Pattern: Different retry strategies and rate limiting approaches are well-abstracted
Command Pattern: Checkpoint operations could benefit from this for undo/redo capabilities

📊 Information Flow Soundness

The data flow architecture is exceptionally well-designed:

1. Estimation Phase: TokenEstimator analyzes input → produces PipelineEstimate
2. Execution Phase: Components coordinate through shared state and events
3. Learning Phase: Actual usage data flows back to EstimationDatabase
4. Monitoring Phase: All components contribute metrics to ProgressTracker

Each phase has proper error boundaries and rollback mechanisms.

🎯 Performance Considerations

Strengths:

• Token bucket algorithm provides smooth rate limiting without bursts
• Exponential backoff prevents overwhelming failing services
• Connection pooling and async operations minimize blocking
• Efficient checkpoint serialization with compression potential

Potential Optimizations:

• Consider batch processing for EstimationDatabase writes
• Implement predictive scaling based on pipeline estimation
• Add query optimization for large checkpoint histories

Final Verdict

This is an architecturally sound and well-engineered addition to HiRAG. The infrastructure demonstrates:

• Excellent separation of concerns with clean interfaces
• Robust error handling with multiple recovery mechanisms
• Intelligent resource management with adaptive behaviors
• Comprehensive observability for operational excellence
• Learning capability for continuous improvement

The code quality is production-ready with proper async patterns, error handling, and configuration management. The architecture will scale well and provides the foundation for enterprise deployment.

Recommendation: ✅ Approve with confidence

This infrastructure significantly enhances HiRAG's operational capabilities while maintaining clean architecture principles. The thoughtful design will pay dividends in production reliability and maintainability.

---