test(docs): add end to end evaluation doc tests#2442
Open
BloggerBust wants to merge 9 commits intoconfident-ai:mainfrom
Open
test(docs): add end to end evaluation doc tests#2442BloggerBust wants to merge 9 commits intoconfident-ai:mainfrom
BloggerBust wants to merge 9 commits intoconfident-ai:mainfrom
Conversation
- end to end tests for docs/docs/evaluation-end-to-end-llm-evals.mdx - add deterministic offline E2E tests covering single-turn and multi-turn flows - validate EvaluationResult/TestResult shape and dataset JSON/CSV artifact schemas - add offline fixtures to disable dotenv loading and browser opening - add networked CLI smoke test gated on OPENAI_API_KEY
- add dedicated GitHub Actions workflow to run docs-based tests - run DeepEval end-to-end documentation tests in CI with secrets - support maintainer-only PRs, main branch pushes, and manual dispatch - temporarily disable Confident docs tests pending fixes
Contributor
|
Skipped: This PR was not opened by one of your configured authors: ( |
|
@BloggerBust is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
- add deterministic metrics for missing-param and raising error scenarios - add ErrorConfig tests for skip_on_missing_params and ignore_errors (incl precedence) - add AsyncConfig, CacheConfig, and DisplayConfig behavior/validation coverage
…ites - Extract generic evaluate() e2e flows into dedicated test files - Add cache behavior coverage for write_cache/use_cache and on-disk artifacts - Add evaluate config coverage for AsyncConfig/ErrorConfig/DisplayConfig - Introduce top-level test fixtures for telemetry opt-out, settings reset, and tracing cleanup - Remove the monolithic end-to-end test file and reorganize fixtures between tests/ and tests/test_core/
Confident tests can all go under tests/confident, so we can flatten this test suite
trevor-cai
changed the title
trevor-cai
changed the title
A-Vamshi
force-pushed
the
test/docs-end-to-end-llm-evals
branch
from
760fe18 to
dca7006
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.