YT Framework

PyPI | Documentation | Examples

---

Overview

A powerful Python framework for building and executing data processing pipelines on YTsaurus (YT) clusters. YT Framework simplifies pipeline development with automatic stage discovery, seamless dev/prod mode switching, and comprehensive support for YT operations.

Architecture

YT Framework follows a pipeline-based architecture where pipelines consist of stages, and stages execute operations.

Key Components:

• Pipeline: Orchestrates stages, their execution order, and configuration management
• Stages: Reusable units of work that execute operations
• Operations: Specific tasks (Map, Vanilla, YQL, S3, Table operations)
• Configuration: YAML-based configuration system for flexible pipeline setup

Key Features

• Pipeline & Stage Architecture: Organize complex workflows into reusable stages
• Automatic Stage Discovery: No manual registration needed - just create stages and run
• Dev/Prod Modes: Develop locally with file system simulation, deploy to YT cluster seamlessly
• Multiple Operation Types: Support for Map, Vanilla, YQL, and S3 operations
• Code Upload: Automatic code packaging and deployment to YT cluster
• Docker Support: Custom Docker images for special dependencies
• Checkpoint Management: Built-in support for ML model checkpoints
• Configuration Management: Flexible YAML-based configuration with multiple config support

Installation

For Users

Install from PyPI:

pip install yt-framework

For Developers and Contributors

Install in editable mode from source:

git clone https://github.com/GregoryKogan/yt-framework.git
cd yt-framework
pip install -e .

For development with testing tools:

pip install -e ".[dev]"

See Installation Guide for prerequisites and detailed setup instructions.

Quick Start

Create your first pipeline in 3 steps:

What you'll build: A simple pipeline that creates a stage, logs a message, and demonstrates the basic framework structure.

1. Create pipeline structure:

   mkdir my_pipeline && cd my_pipeline
   mkdir -p stages/my_stage configs

2. Create pipeline.py:

   from yt_framework.core.pipeline import DefaultPipeline
   
   if __name__ == "__main__":
       DefaultPipeline.main()

3. Create stage and config:

   # stages/my_stage/stage.py
   from yt_framework.core.stage import BaseStage
   
   class MyStage(BaseStage):
       def run(self, debug):
           self.logger.info("Hello from YT Framework!")
           return debug

   # configs/config.yaml
   stages:
     enabled_stages:
       - my_stage
   
   pipeline:
     mode: "dev"  # Use "dev" for local development

Run your pipeline:

python pipeline.py

Next Steps:

• See the Quick Start Guide for a complete example with table operations
• Explore Examples to see more complex use cases
• Read about Pipelines and Stages in the documentation

Examples

The examples/ directory contains comprehensive examples demonstrating most framework features. Each example includes a README explaining what it demonstrates and how to run it.

Requirements

Prerequisites Checklist

• Python 3.11+ installed
• YT cluster access and credentials (for production mode)

YT Cluster Requirements

When running pipelines in production mode, code from ytjobs executes on YT cluster nodes. The cluster's Docker image (default or custom) must include:

• Python 3.11+
• ytsaurus-client >= 0.13.0 (for checkpoint operations)
• boto3 == 1.35.99 (for S3 operations)
• botocore == 1.35.99 (auto-installed with boto3)

Important: Ensure your cluster's default Docker image satisfies these dependencies, or always use custom Docker images for your pipelines. See Cluster Requirements and Custom Docker Images for details.

Documentation

Full documentation available at: yt-framework.readthedocs.io

For local development, source documentation is available in the docs/ directory.

Examples - Complete working examples for most features

Getting Help

• Documentation: Check the full documentation for detailed guides
• Troubleshooting: See the Troubleshooting Guide for common issues
• Examples: Browse working examples to see how features are used
• GitHub Issues: Report bugs or request features on GitHub Issues
• Questions: Open a GitHub issue with the question label

Contributing

We welcome contributions! Whether it's bug fixes, new features, documentation improvements, or examples, your help makes YT Framework better.
See CONTRIBUTING.md for detailed guidelines.