Custom Terms Test Changes - Create + List Experiment #1701
Conversation
bhanvimenghani
changed the title
| ("", "Empty term or model value is not allowed"), | ||
| (" ", "Empty term or model value is not allowed"), | ||
| ("test@term", "is invalid. Only letters, numbers, underscores (_), and hyphens (-) are allowed"), | ||
| ("test#term", "is invalid. Only letters, numbers, underscores (_), and hyphens (-) are allowed"), |
There was a problem hiding this comment.
You can define these messages in utils and use it here
|
|
||
|
|
||
| @pytest.mark.negative | ||
| @pytest.mark.parametrize("plots_datapoint_value", [0, -1, -10]) |
There was a problem hiding this comment.
Add tests for valid but incorrect plots_datapoint_value along with duration_in_days
There was a problem hiding this comment.
Validate plots data in the recommendations in the above scenarios
There was a problem hiding this comment.
Added the test in create experiment that checks if plots datapoint is positive but greater than the duration in days value, it is creating the experiment, need to make backend change to not allow experiment creation. For now i have added this test but adding skip option so that this does not runs untill i make backend changes
| - Update recommendations with end_time preceding start_time | ||
| - [SANITY] Update recommendations with valid results for MIG accelerator | ||
| - [SANITY] Update recommendations with valid results for Non-MIG Accelerator | ||
| - [SANITY] Update recommendations with valid results for a custom term experiment |
There was a problem hiding this comment.
Add details on what term setting combinations are being tested here
|
|
||
|
|
||
| @pytest.mark.negative | ||
| @pytest.mark.parametrize("plots_datapoint_value", [0, -1, -10]) |
There was a problem hiding this comment.
Validate plots data in the recommendations in the above scenarios
| # Note: JSON parsing will automatically deduplicate keys, keeping only the last occurrence | ||
| # This test validates the behavior - the experiment should be created with only one instance | ||
| # or fail depending on backend validation | ||
| assert response.status_code == SUCCESS_STATUS_CODE or response.status_code == ERROR_STATUS_CODE |
There was a problem hiding this comment.
Update the assertion as per the expected behavior
| response = delete_experiment(tmp_json_file) | ||
| print("delete exp = ", response.status_code) | ||
|
|
||
| # Sanity test for custom term |
There was a problem hiding this comment.
Add a test to validate if all the default terms can be specified while creating the experiment.
| "my_custom_term": { | ||
| "duration_in_days": 5.0, | ||
| "duration_threshold": "3 days", | ||
| "plots_datapoint": 5, |
There was a problem hiding this comment.
Include a test with above settings to generate recommendations and validate the recommendations and the plots data count in the recommendations are as expected.
| def test_list_exp_mixed_terms_and_partial_auto_population(cluster_type): | ||
| """ | ||
| Test Description: This comprehensive test validates auto-population behavior for: | ||
| 1. Default terms (weekly, monthly) are auto-populated with system-defined values |
There was a problem hiding this comment.
Include a parameterized test to validate auto-population for all default terms
| # Verify default terms are auto-populated with system-defined values | ||
| assert "weekly" in terms | ||
| assert "duration_in_days" in terms["weekly"] | ||
| assert terms["weekly"]["duration_in_days"] == 7.0 |
There was a problem hiding this comment.
Isn't duration_threshold auto-populated, don't see an assertion for this?
|
i have addressed the review comments and stashed the changes locally. Since there were changes in the json design the dev prs needs to be updated to support latest json structure and also incremental prs for it raised post that I will be changing the tests as required and updating this pr |
2 participants
Description
This pr includes test changes for the Create Exp + List Exp api in remote monitoring tests for custom terms functionality.
https://docs.google.com/document/d/1yjavz5LnEWQOlGJZTED5YdEiCUNJJUqFr0uhhiJrsqE/edit?usp=sharing
Fixes # (issue)
Type of change
How has this been tested?
Please describe the tests that were run to verify your changes and steps to reproduce. Please specify any test configuration required.
Test Configuration
Checklist 🎯
Additional information
Include any additional information such as links, test results, screenshots here