Skip to content

Support anthropic extended thinking#70

Open
ibash wants to merge 7 commits intogsuuon:mainfrom
ibash:thinking
Open

Support anthropic extended thinking#70
ibash wants to merge 7 commits intogsuuon:mainfrom
ibash:thinking

Conversation

@ibash
Copy link
Contributor

@ibash ibash commented Mar 1, 2025

Adds support for anthropic extended thinking.

Messages are now parsed into content objects with a list of blocks, instead of a string for content. This is a common openai based format, so existing providers work as normal.

Thinking / redacted thinking blocks are represented as blocks within markup tags. The signature is a property as a closing tag, which is a bit odd. This was done to make it easy to support streaming (the signature comes at the end of the message).

Redacted thinking is also supported. Redacted thinking doesn't have a signature (because the content is encrypted, so it's already tamper proof).

Here's an mchat with extended thinking for example:

claude-3-7
---
{
  params = {
    max_tokens = 64000,
    model = "claude-3-7-sonnet-latest",
    thinking = {
      budget_tokens = 16000,
      type = "enabled"
    }
  }
}
---
> -


hi

======
<thinking>
The human has sent a simple greeting. I should respond warmly and professionally, and indicate that I'm ready to help them with whatever they need.
</thinking signature="ErUBCkYIARgCIkASEYuO9cW7nl+PmAfeb7Hn4CZwYjHMMxDwQ1tGuClVV33nB2ug8AgQARxq+0UIVXDvIJyCWbeaCwXxl1FQE2nFEgyMKXc/FSZDtyudeTUaDPqUN+rKzMTU5o4jMCIwGTaTRKc6VONBK6i2lcpT/2oJ1hVSFPg74DmoNxtd8gkkjNi2dHoU5/LJaDXTzwMoKh0NpZKJEf0dOBWRUnIMDOL+GvQwg+evDFp1AjPuEA==">

Hello! It's nice to meet you. I'm Claude, an AI assistant created by Anthropic. How can I help you today? I'm here to answer questions, have a conversation, help with creative projects, or assist with information on a wide range of topics. What would you like to discuss?
======

@ibash ibash force-pushed the thinking branch 2 times, most recently from 08f795e to 92894d8 Compare March 1, 2025 11:09
Adds support for anthropic extended thinking.

Messages are now parsed into content objects with a list of blocks,
instead of a string for content. This is a common openai based format,
so existing providers work as normal.

Thinking / redacted thinking blocks are represented as blocks within
markup tags. The signature is a property as a closing tag, which is a
bit odd. This was done to make it easy to support streaming (the
signature comes at the end of the message).

Redacted thinking is also supported. Redacted thinking doesn't have a
signature (because the content is encrypted, so it's already tamper
proof).
@ibash ibash marked this pull request as ready for review March 1, 2025 22:46
@gsuuon
Copy link
Owner

gsuuon commented Mar 3, 2025

Thanks for the PR! Changing message content to blocks is a good direction, I'd avoided it for simplicity at the start - now it'll break in places where the final request is built by just concatenating with msg.content, like here. The associated type is ChatMessage which needs to be updated.

There's a lot of anthropic specific code that made it into core/chat, I think blocks with just data and type instead are a good compromise. Provider specific stuff can go in the provider handler, so only providers/anthropic.lua should know about things like signature.

I've also been avoiding letting anything that might be part of the normal response have special meaning in mchat, though it'll never be fool-proof due to needing to roundtrip to plaintext. XML tags should be avoided though since those are pretty common in responses. I went with a git conflict-marker inspired <<<<<< >>>>>> (but 6 instead of 7 chars), which basically should generalize as

<<<<<< type
data
>>>>>>

for each special block. This also lets the tree-sitter grammar recognize these as foldable and can be styled differently (though I've hardcoded <<<<<< reasoning for now, regrettably).

@gsuuon
Copy link
Owner

gsuuon commented May 19, 2025

Oops! Sorry I added some commits here meant for main, we can just squash when merging. Are you still interested in this PR? If not I can fix up the comments and get this merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants