Regroup all swebenchmultimodal hyper parameters in a single source of truth benchmarks/swebenchmultimodal/constants.py

[simonrosenberg]

Description

SWEBenchMultimodal uses several hyperparameters/ constant values and constant values in its implementation in benchmarks/swebenchmultimodal

A constant value is any value int or str that must be set to correctly parametrized/run the benchmark. It can appear as input of a method or as default argument of a method.
Here are example of what can be a constant value:

• "validation"
• "2023_all"
• "finish_with_patch"
• "lite"
• "wentingzhao/commit0_combined"
• ....

Those values are hyper parameters of the full evaluation workflow.

Painpoint

It's not easy to check if all such values are correctly parametrized for a benchmark because they are scattered in many different places.

Solution

For SWEBenchMultimodal

1. Go through the code in benchmarks/swebenchmultimodal and find ALL such constant values
2. Redefine all those constant values in benchmarks/swebenchmultimodal/constants.py and make sure all other files import those values from the constant.py which constitutes the single source of truth.

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[openhands-ai]

Summary

I have successfully fixed issue #367 by regrouping all SWE-Bench Multimodal hyperparameters into a single source of truth at benchmarks/swebenchmultimodal/constants.py.

Checklist

✅ Created constants.py - New file containing all constant values used in the benchmark
✅ Updated run_infer.py - Now imports constants from constants.py
✅ Updated eval_infer.py - Now imports constants from constants.py
✅ Updated build_images.py - Now imports constants from constants.py
✅ Added comprehensive tests - 15 new tests for all constants in test_swebenchmultimodal.py
✅ All 43 tests pass - Including the new constants tests
✅ Created branch and pushed - openhands/swebenchmultimodal-constants
✅ Opened PR - #368

Constants Consolidated

The following constants are now centralized in constants.py:

• Dataset: DEFAULT_DATASET, DEFAULT_SPLIT
• Docker: DOCKER_IMAGE_PREFIX
• Build: BUILD_TARGET
• Workspace: WORKSPACE_DIR
• Environment variables: ENV_SKIP_BUILD, ENV_RUNTIME_API_KEY, ENV_SDK_SHORT_SHA, ENV_REMOTE_RUNTIME_STARTUP_TIMEOUT, ENV_RUNTIME_API_URL
• Default values: DEFAULT_SKIP_BUILD, DEFAULT_REMOTE_RUNTIME_STARTUP_TIMEOUT, DEFAULT_RUNTIME_API_URL
• Git config: GIT_USER_EMAIL, GIT_USER_NAME, GIT_COMMIT_MESSAGE
• Setup: ENV_SETUP_COMMANDS
• Image validation: ALLOWED_IMAGE_TYPES
• Evaluation: DEFAULT_EVAL_WORKERS, DEFAULT_MODEL_NAME
• Annotations: SOLVEABLE_KEYWORD, ANNOTATIONS_FILENAME
• Patch processing: SETUP_FILES_TO_REMOVE

PR Link

#368

View full conversation