Skip to content

feat(DATAGO-121849): Document eval test instruction#954

Open
alimosaed wants to merge 1 commit intomainfrom
ap/DATAGO-121849/fix_eval_test
Open

feat(DATAGO-121849): Document eval test instruction#954
alimosaed wants to merge 1 commit intomainfrom
ap/DATAGO-121849/fix_eval_test

Conversation

@alimosaed
Copy link
Collaborator

What is the purpose of this change?

Running eval tests are confusing because there is no instruction or script for running eval tests.

How was this change implemented?

  • Added evaluation/README.md with step-by-step instructions for running evaluation tests, environment variable requirements, and manual/automated setup options.

  • Introduced new Makefile targets for evaluation tests (test-eval, test-eval-local, test-eval-workflow, test-eval-remote) and updated help text to document these options.

  • Improved configuration setup in evaluation/run.py by ensuring the configs directory exists and automatically copying shared_config.yaml from examples if missing.

Key Design Decisions

How was this change tested?

  • Manual testing: Ran the make test-eval and verified that the virtual environment is created and eval tests are executed.
  • Unit tests
  • Integration tests
  • Known limitations

Is there anything the reviewers should focus on/be aware of?

  • SAM should be completely setup with valid LLM keys and broker configs to run eval tests.
  • SAM is not properly installed via uv package manager in the Make file. Therefore, we had to directly activate virtual environments via source command and run the sam application.

@sonarqube-solacecloud
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant