Add extreme-scale performance tests with dynamic schema generation

[koxudaxi]

Summary by CodeRabbit

• Tests

  • Added comprehensive performance tests for extreme-scale scenarios, including large schemas, massive file inputs, and duplicate name handling to improve benchmarking coverage.

• Chores

  • Updated CI/CD and test configuration for performance testing optimization.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Warning

Rate limit exceeded

@koxudaxi has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 8 minutes and 35 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 623d500 and 01abb82.

📒 Files selected for processing (2)

• .github/workflows/codspeed.yaml
• tests/main/test_performance.py

📝 Walkthrough

Walkthrough

This PR adds extreme-scale performance testing infrastructure with three new fixtures generating large JSON schemas dynamically (2000+ models, 200 files, duplicate-name scenarios). It includes six new performance test functions and adjusts CI configuration by changing CodSpeed from instrumentation to simulation mode and omitting the performance test file from coverage tracking.

Changes

Cohort / File(s)	Summary
CI and Test Configuration \.github/workflows/codspeed\.yaml, pyproject\.toml	Modified CodSpeed run mode from "instrumentation" to "simulation" in benchmark workflow step; extended run.omit in pyproject to exclude tests/main/test_performance\.py from coverage alongside existing test data exclusions.
Performance Test Suite tests/main/test_performance\.py	Added three new fixtures for dynamic extreme-scale schema generation (extreme_large_schema, massive_files_input, extreme_duplicate_names_schema); introduced six new performance tests covering large single-schema processing, massive multi-file inputs, duplicate-name handling, with and without Pydantic v2 models; includes validation checks for generated class counts (≥1000–2000) and specific naming patterns.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• #2782: Directly overlapping changes adding large-schema performance e2e tests/fixtures and CI workflow adjustments (CodSpeed and pytest configuration).
• #2817: Related CI/test configuration modifications around tests/main/test_performance.py workflow execution and omission patterns.

Poem

🐰 Large schemas hop through performance tests we brew,
Two thousand models dance with fixtures anew,
Massive files tumble, duplicates we tame,
CodSpeed shifts gently—simulation's the game! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main changes: adding extreme-scale performance tests with dynamic schema generation, which aligns with the additions of new performance test fixtures and test cases in tests/main/test_performance.py.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

📚 Docs Preview: https://pr-2818.datamodel-code-generator.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.51%. Comparing base (1e4db90) to head (01abb82).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2818      +/-   ##
==========================================
+ Coverage   98.72%   99.51%   +0.79%     
==========================================
  Files          90       89       -1     
  Lines       14120    13924     -196     
  Branches     1661     1660       -1     
==========================================
- Hits        13940    13857      -83     
+ Misses        149       36     -113     
  Partials       31       31              

Flag	Coverage Δ
unittests	99.51% <ø> (+0.79%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

tests/main/test_performance.py (1)

659-688: Strengthen verification for massive file input tests.

Both test_perf_massive_files_input and test_perf_massive_files_single_output only verify that at least 1 Python file exists, but don't validate that the expected models were actually generated. The fixture creates 4000 models (200 files × 20 models), so the tests should verify a reasonable number of models were produced.

Consider adding model counting similar to test_perf_multiple_files_input (lines 218-222) to ensure the generator successfully processed the massive input.

🔎 Proposed enhancement

 @pytest.mark.perf
 def test_perf_massive_files_input(tmp_path: Path, massive_files_input: Path) -> None:
     """Performance test: Process 200 separate schema files (4000 models total).
 
     Tests directory input handling with a very large number of files.
     """
     output_dir = tmp_path / "output"
     generate(
         input_=massive_files_input,
         input_file_type=InputFileType.JsonSchema,
         output=output_dir,
     )
     assert output_dir.exists()
     py_files = list(output_dir.glob("**/*.py"))
     assert len(py_files) >= 1
+    # Verify a substantial number of models were generated
+    total_models = 0
+    for py_file in py_files:
+        content = py_file.read_text()
+        total_models += content.count("class Module")
+    assert total_models >= 1000  # At least 25% of the 4000 models

Apply similar enhancement to test_perf_massive_files_single_output.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1e4db90 and 623d500.

📒 Files selected for processing (3)

• .github/workflows/codspeed.yaml
• pyproject.toml
• tests/main/test_performance.py

🧰 Additional context used

🧬 Code graph analysis (1)

tests/main/test_performance.py (2)

src/datamodel_code_generator/__init__.py (1)

• generate (450-1016)

src/datamodel_code_generator/enums.py (2)

• InputFileType (35-45)
• DataModelType (48-56)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (14)

• GitHub Check: benchmarks
• GitHub Check: 3.10 on macOS
• GitHub Check: py312-pydantic1 on Ubuntu
• GitHub Check: py312-isort5 on Ubuntu
• GitHub Check: py312-isort6 on Ubuntu
• GitHub Check: py312-black23 on Ubuntu
• GitHub Check: 3.10 on Windows
• GitHub Check: 3.11 on Windows
• GitHub Check: 3.11 on Ubuntu
• GitHub Check: 3.12 on Windows
• GitHub Check: 3.13 on Ubuntu
• GitHub Check: 3.13 on Windows
• GitHub Check: 3.14 on Windows
• GitHub Check: Analyze (python)

🔇 Additional comments (2)

pyproject.toml (1)

246-246: LGTM! Appropriate exclusion of performance tests from coverage.

Excluding performance tests from coverage tracking is the correct approach, as these tests focus on timing and scalability rather than code path coverage.

.github/workflows/codspeed.yaml (1)

35-35: Verify the mode change from "instrumentation" to "simulation" is intentional.

The CodSpeed mode has been changed from "instrumentation" to "simulation." This is a significant change that affects how performance is measured:

• Instrumentation mode: Profiles actual execution with precise measurements but adds overhead
• Simulation mode: Uses heuristics/sampling, faster but potentially less accurate

Given that this PR adds extreme-scale tests (2000+ models, 200 files), simulation mode may be necessary to avoid excessive overhead. However, please confirm:

1. Is this change intentional for the new extreme-scale tests?
2. Are there known compatibility or performance issues with instrumentation mode for these tests?
3. Does simulation mode provide adequate metrics for your performance tracking needs?

If the mode change is specifically for the extreme tests, consider whether separate workflows (one for standard benchmarks in instrumentation mode, one for extreme tests in simulation mode) would provide better measurement fidelity.

[codspeed-hq]

CodSpeed Performance Report

Merging #2818 will create unknown performance changes

_{Comparing perf/add-extreme-scale-tests (01abb82) with main (269b6b4)¹}

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

🆕 26 new
⏩ 83 skipped²

Benchmarks breakdown

	Mode	Benchmark	BASE	HEAD	Efficiency
🆕	WallTime	test_perf_large_models	N/A	2.9 s	N/A
🆕	WallTime	test_perf_combined_large_models_with_formatting	N/A	3.1 s	N/A
🆕	WallTime	test_perf_large_models_pydantic_v2	N/A	3 s	N/A
🆕	WallTime	test_perf_graphql_style_pydantic_v2	N/A	712.2 ms	N/A
🆕	WallTime	test_perf_duplicate_names	N/A	852 ms	N/A
🆕	WallTime	test_perf_large_models_dataclass	N/A	2.9 s	N/A
🆕	WallTime	test_perf_kubernetes_style	N/A	2.2 s	N/A
🆕	WallTime	test_perf_openapi_large_field_constraints	N/A	2.6 s	N/A
🆕	WallTime	test_perf_graphql_style_typed_dict	N/A	558.9 ms	N/A
🆕	WallTime	test_perf_all_options_enabled	N/A	5.6 s	N/A
🆕	WallTime	test_perf_graphql_style	N/A	675.9 ms	N/A
🆕	WallTime	test_perf_kubernetes_style_pydantic_v2	N/A	2.3 s	N/A
🆕	WallTime	test_perf_aws_style_openapi_pydantic_v2	N/A	1.7 s	N/A
🆕	WallTime	test_perf_stripe_style	N/A	1.7 s	N/A
🆕	WallTime	test_perf_large_models_typed_dict	N/A	2.6 s	N/A
🆕	WallTime	test_perf_stripe_style_pydantic_v2	N/A	1.8 s	N/A
🆕	WallTime	test_perf_deep_nested	N/A	5.2 s	N/A
🆕	WallTime	test_perf_duplicate_names_multiple_files	N/A	849.9 ms	N/A
🆕	WallTime	test_perf_openapi_large	N/A	2.5 s	N/A
🆕	WallTime	test_perf_complex_refs_collapse_root	N/A	1.8 s	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Footnotes

1. No successful run was found on main (1e4db90) during the generation of this report, so 269b6b4 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

2. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

Breaking Change Analysis

Result: No breaking changes detected

Reasoning: This PR only adds new performance tests and modifies CI/CD configuration. It does not change any production code, APIs, CLI options, templates, code generation logic, default behaviors, Python version support, or error handling. All changes are isolated to test infrastructure: new test functions in tests/main/test_performance.py, coverage configuration in pyproject.toml, and CodSpeed workflow settings in .github/workflows/codspeed.yaml.

---

This analysis was performed by Claude Code Action