📦 Project Complexity Management - Simplification and Organization

[ootakazuhiko]

📦 Project Complexity Management - Simplification and Organization

Priority: MEDIUM - Maintainability and Developer Experience

Overview

The ae-framework has grown into a comprehensive system with significant complexity that impacts maintainability and onboarding experience. Strategic simplification is needed to improve developer productivity.

Complexity Indicators

1. Package.json Overload

• Current State: 230+ npm scripts across multiple categories
• Categories:
  • Core build/test: ~30 scripts
  • Quality gates: ~40 scripts
  • Agent/MCP servers: ~25 scripts
  • Benchmarking/performance: ~20 scripts
  • Formal verification: ~15 scripts
  • CI/automation: ~35 scripts
  • Specialized tools: ~65 scripts

2. ESLint Configuration Complexity

• Broad Exclusions: 15+ directories excluded for "Verify Lite stability"
• Multiple Rule Sets: Base, type-checked, escalated rules by directory
• Technical Debt: Temporary exclusions becoming permanent

3. Project Structure Scale

• Source Structure: 30+ top-level src directories
• Package Count: 3+ workspace packages
• Configuration Files: 15+ config files in root
• CI Workflows: 35 GitHub Actions workflows

Impact Analysis

Developer Onboarding

• Learning Curve: High cognitive load for new contributors
• Documentation Debt: Config scattered across multiple files
• Tool Conflicts: Multiple overlapping tools for similar purposes

Maintenance Overhead

• Dependency Management: Complex pnpm workspace coordination
• Version Alignment: Multiple TypeScript configs with different targets
• Build Complexity: Multi-stage builds with interdependencies

CI/CD Pipeline

• Workflow Proliferation: 35 workflows with complex triggering logic
• Resource Usage: Heavy CI resource consumption
• Debug Difficulty: Complex failure diagnosis across workflows

Proposed Solutions

Phase 1: Script Consolidation (Weeks 1-2)

1. Package.json Reorganization

   {
     "scripts": {
       // Core (keep as-is)
       "build": "...",
       "test": "...",
       "dev": "...",
       
       // Consolidated categories
       "quality": "node scripts/quality/run-all.mjs",
       "formal": "node scripts/formal/run-suite.mjs", 
       "agents": "node scripts/agents/manage.mjs",
       "ci": "node scripts/ci/orchestrate.mjs"
     }
   }

2. Script Categorization

   • Create scripts/ directory structure by purpose
   • Implement unified CLI for script management
   • Add interactive script discovery: pnpm run help

Phase 2: Configuration Simplification (Weeks 2-4)

1. ESLint Cleanup Strategy

   // Gradual inclusion approach
   export default ts.config(
     {
       ignores: [
         // Temporary exclusions with removal timeline
         'src/ui/**', // TODO: Enable by Q2 2025
         'src/cli/**', // TODO: Enable by Q1 2025
       ]
     }
   );

2. Config File Consolidation

   • Move configs to configs/ directory
   • Create master config with includes
   • Standardize config naming patterns

Phase 3: Architecture Simplification (Month 2)

1. Workflow Consolidation

   # Master workflow with matrix strategy
   name: Unified CI
   strategy:
     matrix:
       suite: [fast, security, formal, ui, benchmarks]
       include:
         - suite: fast
           runs-on: ubuntu-latest
           timeout: 10
         - suite: security  
           runs-on: ubuntu-latest
           timeout: 30

2. Package Architecture Review

   • Evaluate package boundaries
   • Consider monorepo vs. workspace tradeoffs
   • Optimize dependency graphs

Phase 4: Developer Experience (Month 2-3)

1. Unified CLI Interface

   # Single entry point for all operations
   ae-framework --help
   ae-framework quality --run all
   ae-framework ci --local --fast
   ae-framework setup --interactive

2. Interactive Setup and Discovery

   • Guided project setup wizard
   • Interactive script discovery
   • Contextual help system

Implementation Roadmap

Week 1-2: Quick Wins

• Audit and categorize all package.json scripts
• Create scripts/ directory structure
• Implement script consolidation for top 5 categories
• Document script migration plan

Progress (merged PRs):

• PR chore: standardize runner flags (Phase 1) #1752: standardize runner flags (quality/verify), improve help output, refresh script inventory note, add regression tests
• PR chore: standardize script chaining to pnpm run #1753: standardize script chaining in package.json from npm run to pnpm run
• PR chore: consolidate eslint config under configs #1756: ESLint config consolidation (move to configs/eslint.config.js, root shim)
• PR chore: move root tsconfig into configs #1757: tsconfig consolidation (configs/tsconfig/tsconfig.root.json + root extends)

Week 3-4: Configuration Cleanup

• Move configuration files to configs/
• Create ESLint inclusion roadmap with timelines
• Standardize config file naming and structure
• Update documentation references (PR docs: update config reference notes (#1006) #1766)

Month 2: Workflow Optimization

• Analyze workflow overlap and redundancy
• Design unified CI workflow strategy
  • DoD: publish v1 strategy doc mapping current workflows to target set (5-10), required checks impact, migration sequence, and rollback notes.
  • Current: v1 doc in docs/notes/issue-1006-ci-consolidation-draft.md (PR docs: refresh ci consolidation status for issue 1006 #1808)
• Implement workflow consolidation (5-10 workflows target)
  • Current: 44 workflows in .github/workflows (main, 2026-01-26 UTC)
  • Current: entry consolidation PRs chore(ci): move ci-fast entry into ci.yml #1797/chore(ci): route ci-extended entry via ci.yml #1798/chore(ci): route hermetic entry via ci.yml #1799/chore(ci): route parallel tests via ci.yml #1800/chore(ci): route podman smoke via ci.yml #1801/chore(ci): limit ci-fast/extended to workflow_call #1812/chore(ci): route QA bench via ci.yml #1813/docs: note workflow dispatch dependencies #1814/chore(ci): route pr-verify via ci.yml #1815/chore(ci): route verify entry via ci.yml #1816 merged
• Test and validate consolidated approach
  • Current: PR chore(ci): route verify entry via ci.yml #1816 の必須チェック（verify-lite/gate）を含む主要CIがSUCCESS

Month 3: Developer Experience

• Create unified CLI interface (PR feat(cli): add entry runner routing #1802, PR feat(cli): add ae help command #1804)
  • DoD: single entrypoint (ae/ae-framework) routes to category runners (test/quality/verify/flake/security) and exposes --help.
  • Current: ae/ae-framework routes to category runners via ae entry, and ae help is available on main.
• Implement interactive setup wizard
  • Current: setup wizard draft (PR docs: add setup wizard draft for issue 1006 #1806), ae setup command (PR feat: add ae setup command for installer templates #1807), interactive wizard (PR feat: add interactive setup wizard #1809)
• Add contextual help and discovery tools (PR feat(cli): add ae help command #1804)
• Update onboarding documentation (PR docs: add CLI entry migration guide #1803, PR docs(setup): add cli discovery section #1805)

Success Metrics

Quantitative Goals

• Package.json scripts: 230+ → <100
• ESLint exclusions: 15+ directories → <5
• CI workflows: 35 → <15
• Onboarding time: Reduce by 50%

Qualitative Improvements

• Clear command discovery path
• Consistent configuration patterns
• Simplified mental model for contributors
• Reduced maintenance overhead

Risk Mitigation

1. Backward Compatibility

   • Maintain script aliases during transition
   • Gradual deprecation with clear timelines
   • Comprehensive testing of consolidated scripts

2. Feature Preservation

   • Ensure no functionality loss during consolidation
   • Maintain all quality gates and checks
   • Preserve specialized tool capabilities

3. Team Coordination

   • Communicate changes clearly to contributors
   • Provide migration guides for existing workflows
   • Maintain documentation during transition

Related Documentation

• Project Organization Guide
• Development Baseline
• Agent Working Rules

[ootakazuhiko]

PR #1345 で scripts の棚卸し（プレフィックス別件数と総数）をドキュメント化しました。Phase 1 の「監査・分類」着手として記録済みです。

[ootakazuhiko]

Phase 1 の構造棚卸しとして、src/ と packages/ のトップレベル構成をまとめたドキュメントを追加しました: PR #1347

[ootakazuhiko]

Phase 1.5 draft として、src/ の構成を目的別に分類し、重複候補を整理したメモを追加しました: PR #1348

[ootakazuhiko]

Phase 1 の一環として、workflow 数と内訳を棚卸ししたドキュメントを追加しました: PR #1350

[ootakazuhiko]

Phase 1.5 の補助として、ワークフローの重複候補一覧を追加しました: PR #1351

[ootakazuhiko]

Phase 1.5 として workflow のトリガー分布マップを追加しました: PR #1352