Evaluate

[Su-my]

Thank you for your excellent work! noticing that the results reported in the paper are different from the original LUFFY and SRFT paper, could you please release the evaluation script? Thank you very much!

[telxt]

Sorry for my late reply. We perform the evaluation using the scripts provided by DeepMath, which are widely used for mathematical tasks. You can conduct the evaluation by following this instruction. We use the same script in DeepMath and our hyperparameter for evaluation is below.

chat_template_name="default"
system_prompt_name="simplerl"
data_id=(
    # All supported datasets from DeepMath uni_eval.py
    "math-ai/aime25:test"
    "zwhe99/MATH:math500"
    "zwhe99/aime90:2024"
    "zwhe99/simplerl-minerva-math:test"
    "zwhe99/simplerl-OlympiadBench:test"
    "AI-MO/aimo-validation-amc:train"
)
max_model_len=8192
temperature=0.6
top_p=0.95
n=32 # for AIME24, AIME25, AMC
n=1 # for others

We will soon update a more detailed README.