Add stricter basic economy policy restrictions

[chrisgorgo]

Summary

Added explicit policy restrictions for basic economy reservations to prevent agents from using upgrade workarounds to bypass cancellation rules.

Problem

Agents were finding loopholes in basic economy policies by:

1. Upgrading basic economy to economy class first
2. Then cancelling the "economy" reservation (which has fewer restrictions)

This allowed circumventing the stricter basic economy cancellation rules.

Changes

Policy Updates (policy.md)

• Flight modifications: Clarified that reservations originally booked as basic economy cannot have flight segments modified, regardless of subsequent cabin changes
• Cabin changes: Added note that upgrading cabin doesn't remove the underlying flight modification restrictions
• Cancellation rules: Basic economy can only be cancelled if:
  • Booking made within last 24 hours
  • Flight cancelled by airline
  • User has travel insurance AND reason is covered (health/weather)
• Refund restrictions: Basic economy refunds are airline credits only, unless user has travel insurance

Database Updates (db.json)

• Changed 2 reservations from basic_economy to economy to align with updated task expectations
• Added insurance to one reservation for proper test coverage

Task Updates (tasks.json)

• Task 7: Updated to expect agent to correctly explain basic economy cannot be cancelled (instead of using upgrade workaround)
• Task 67: Added health reason instruction for cancellation scenario
• Task 81: Added medical reason and payment method instructions

Impact

Agents will now correctly enforce basic economy restrictions without allowing upgrade-then-cancel workarounds. The evaluation expects agents to explain policy limitations rather than find loopholes.

🤖 Generated with Claude Code

[victorb-sierra]

Thanks for the PR. The fact that it is possible to upgrade and then cancel is not an unknown loophole but a known feature of the policy (as you pointed out, task 7 uses it).
It is strange, yes!, but since it was in the initial version of tau-bench we kept it and tried to make sure that all the tasks were compatible with that fact.

Are you saying that task 7 is incorrect? (and same for the two other tasks? - I think your numbering if off btw).
Or are you saying that the fact that this option exists results in failure in some other tasks (ie legitimate cancellations happening when no cancellations are expected)?

[chrisgorgo]

Sorry I should've added more context. What we observed is that Opus 4.5 started finding creative ways to use the loopholes in questions that did not expect them. For example in questions 36 and 31. So in general there was inconsistency of when the loopholes were expected and when they would not be allowed. The PR makes the policy clear and adjusts the the other questions to not have to rely on the loopholes.

[victorb-sierra]

The fact that you can upgrade to economy and then do the change is not something we were not aware of. The tasks were designed with this in mind.

What would be most helpful would be to be able to see the trajectories where the model is using this rule (upgrade and then cancel) in way that leads to a failure of the benchmark.

What we observed is that Opus 4.5 started finding creative ways to use the loopholes in questions that did not expect them. For example in questions 36 and 31.

Here also, could you share trajectories? Are we talking about the same issue or a different one?

[muotaz]

Hello, and sorry for imposing, but doesn't the policy say clearly that: In other cases, all reservations, including basic economy, can change cabin without changing the flights. So the solution is not-policy compliant because changing the flight details is implicitly excluded.

[karthikncode]

@chrisgorgo @victorb-sierra I'd actually suggest that for tasks 31 and 36, we can add more instructions for the user to refuse cabin upgrades if the agent asks about it. That way, we keep the expected outcomes the same to maintain backward compatibility while dealing with the case of the agent trying out this upgrade strategy. What do you think?

[chrisgorgo]

This could solve the current problem (although I am not sure if it would be truly backward compatible), but I am worried that future models might start using loopholes in other questions. Is keeping an existing loophole in the policy realistic?

Another alternative is to come up with a more robust grading.

[victorb-sierra]

Relevant for RFC #128 (Tau3 domain fixes)