Generating rollout for Llama-3.2-3b-instruct is buggy, and generates responses in tool format

[3rdAT]

System Info

Problem:

Rollouts generated from Llama-3.2-3b-instruct is in tool format by default.

Example:

For a simple gsm8k question

Response 0: {"name": "twice as many as", "parameters": {"x": "10"}} 
Response 1: "{\"name\": \" Emma \" \"parameters\": {\"from_bank\": 2000, \"bought_furniture\": 400}}"
Response 2: {"name": " Emma's new amount after buying furniture", "parameters": {"initial_amount": "2000", "furniture_cost": "400"}}
Response 3: {"name": "age", "parameters": {"x": 35}} 
Response 4: {"name": "iscalculate", "parameters": {"x": "4", "a": "0", "b": "5"}}

To understand why it happens,

tok = AutoTokenizer.from_pretrained('meta-llama/Llama-3.2-3B-Instruct')

messages = [{'role': 'user', 'content': 'What is 2+2?'}]

# With tools=None (default - no tool calling)
out_none = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, tools=None)

# With tools=[] (empty list)
out_empty = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, tools=[])

print('=== tools=None ===')
print(repr(out_none))
print()
print('=== tools=[] ===')
print(repr(out_empty))
print()
print('=== Are they the same? ===')
print(out_none == out_empty)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Just run training on Llama-3.2 model.

Expected behavior

The rollouts should be actual response rather than a tool style output.

[shirley-wu]

I have encountered the same issue. I dumped the input and output during rollout, and see that the input looks like this:

system

Environment: ipython
Cutting Knowledge Date: December 2023
Today Date: 14 Feb 2026

...my own system prompt... user

Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.Do not use variables.

... some user request... assistant

So the prompt adds some tool calling instructions in addition to my user input. Looking at Llama-3.2-3b-instruct's jinja template, it seems it always inserts the tool-calling prompt when tools is not none - and tools always is not none, even though the task is text-only, potentially because of the way they parse DataProto.

Right now I'm using a hacky fix: I explicitly set config.actor_rollout_ref.model.custom_chat_template as a copy of the original jinja template, except adding

{%- set tools = none %}

after this block of code:

{%- if not tools is defined %}
    {%- set tools = none %}
{%- endif %}

so the tools is always recognized as none. This can work, but I would be happy to see a more elegant solution.