Petros Raptopoulos, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou
Making Contract Review Accessible to Everyone Through AI
Try PAKTON | View Evaluation and Experiments | Read Paper | View Poster | View Recording | Underline
Reviewing contracts is often slow, complex, and requires expert legal knowledge. Legal language can be vague and open to interpretation, making it hard for non-experts to understand. On top of that, contracts are usually private, which limits the use of proprietary AI tools and calls for open-source solutions.
PAKTON solves these problems with an open-source, end-to-end framework for automated contract review. It uses a team of LLM agents working together, along with smart retrieval tools (RAG), to make legal document analysis easier, more private, and customizable.
PAKTON user flow: legal query submission followed by comprehensive report generation
PAKTON was published at the Main Conference of EMNLP 2025 and presented orally by Petros Raptopoulos.
🚀 Live Deployed Version at pakton.site
PAKTON Login/Signup Page
Contract upload and chat interface
⚠️ Important Note: The deployed version and the code currently in the repository are missing a few components that will be added shortly. These updates are being organized to ensure a clean and robust push.
PAKTON employs a sophisticated multi-agent architecture that orchestrates specialized AI agents to handle different aspects of contract analysis. The framework leverages collaborative agent workflows combined with advanced retrieval-augmented generation (RAG) to provide comprehensive, accurate, and explainable contract review.
Detailed PAKTON architecture showing the multi-agent workflow and RAG component integration
We evaluated PAKTON using both qualitative and quantitative methods to ensure its effectiveness in real-world legal tasks. You can explore all experiment results and details at: 🔗 https://pakton.site/evaluation
- Experiments Overview - Complete evaluation framework and methodology
- Human Evaluation - Human assessment methodology and results
- GEVAL Assessment - Automated qualitative evaluation using LLM-as-a-judge
- Statistical Agreement - Statistical validation of alignment between LLM and human evaluations
- ContractNLI Classification - Classification Performance
- LegalBenchRAG Performance - Retrieval ability Performance
- Superior Generation Quality: Outperforms baseline methods on the ContractNLI dataset
- State-of-the-Art Retrieval: RAG component (Researcher) leads performance on LegalBenchRAG benchmark
- Human-Preferred: Chosen by human evaluators over ChatGPT for contract analysis—especially for Explainability and Completeness.
- LLM Validation: GEVAL evaluations show consistent preference for PAKTON over GPT-4o
- Statistical Validation: Strong statistical agreement (cosine similarity 0.88-0.92) between automated and human evaluation methods confirms reliability of assessment results
- Privacy-First: Fully open-source with on-premise deployment capabilities
- Robust: According to our robustness analysis, it bridges performance gaps between small and large LLMs, enabling smaller open-source models to rival larger proprietary ones
- Plug-and-Play: Modular architecture for seamless extension and custom workflow integration
- Transparent Design: Explainable outputs that contrast with typical black-box AI models
PAKTON/
├── LICENSE # License information
├── README.md # This file
├── CONTRIBUTING.md # Contributing Guidelines
├── Docs/ # Documentation and research papers
│ ├── ACL_Anthology_version.pdf # ACL Anthology published version
│ ├── EMNLP 2025_Poster.pdf # Conference poster
│ └── Preprint_May_25.pdf # Research preprint
├── deployment/ # Deployment configurations
│ ├── development/ # Development environment configs
│ ├── production/ # Production environment configs
│ └── nginx/ # Nginx server configurations
├── PAKTON Framework/ # Core framework implementation
│ ├── API/ # Backend API service
│ ├── Archivist/ # Archivist agent implementation
│ ├── Interrogator/ # Interrogator agent implementation
│ ├── Researcher/ # Researcher agent (RAG component)
│ └── Frontend/ # Frontend applications
├── Experiments and Evaluation/ # All experimental work and evaluation
│ ├── Frontend/ # Frontend for experiments visualization
│ ├── Qualitative/ # Qualitative evaluation methods
│ │ ├── Human Evaluation/ # Human assessment results
│ │ ├── LLM as a judge - GEVAL/ # Automated evaluation using GEVAL
│ │ └── Statistical Agreement/ # Statistical validation between LLM and human evaluations
│ └── Quantitative/ # Quantitative performance evaluation
│ ├── Classification Performance - ContractNLI/ # ContractNLI experiments
│ └── RAG Performance - LegalBenchRAG/ # LegalBenchRAG experiments
└── Machine Learning Experimentation/ # Additional ML experiments (not mentioned in the paper)
PAKTON is dedicated to making contractual obligations clearer and more accessible to everyone. We believe in the power of community-driven development and welcome contributors (ideas, code, feedback).
Join our vibrant Discord community where developers, researchers, and legal tech enthusiasts come together to:
- Share ideas and get instant feedback
- Troubleshoot and solve implementation challenges
- Find collaborators for new features and research
- Stay ahead with the latest updates and releases
Whether you're fixing bugs, adding features, improving documentation, or sharing use cases, your contribution matters! To get started, please review our Contributing Guidelines.
Ways to contribute:
- Report bugs and issues
- Suggest new features or improvements
- Improve documentation
- Submit pull requests
- Help with translations and accessibility
- Share PAKTON with others who might benefit
This project is licensed under the terms specified in the LICENSE file.
Democratizing contract analysis
🌟 Star this repository if PAKTON helped you! 🌟