conversations/responses needs to evolve to handle more complex conversation transcripts

[simonw]

The current conversation/response/storage abstraction has some problems.

Gemini, Anthropic and OpenAI have been getting more sophisticated with how they handle longer conversations and tool calling. In particular some of the more recent models can run tools server-side - like their web search tool - so the model does one or more tool calls before returning from the API.

These tool calls can even take place during the "reasoning" phase. The reasoning stuff is a problem because that can return text that is reasoning text that is not technically part of the final response but still deserves to be stored in SQLite and optionally made available to the Python API users.

Sometimes models return encrypted blocks of additional context - these need to be stored too so they can be round-tripped to the API in future calls. This all adds up to much more complex requirements for how the conversation mechanism should work and how ongoing conversations should be persisted.

I brainstormed with Claude Opus 4.5 a bit, it was very useful: https://gistpreview.github.io/?8e871d0030a7b28fa40616d32611660e

[simonw]

As pointed out in #1160 we have a problem right now where llm logs -n 5 returns 5 most recent responses without any indication as to whether they were part of a conversation or not. That needs to be addressed by this new work as well.

[simonw]

I think this work is also where we'll end up solving for annotations:

• Annotations abstraction for responses that are not just a stream of plain text #716

[simonw]

Next job on this task will be to collect as many real-world JSON examples of complex transcripts that need to be logged from different API providers.

[simonw]

Issues to consider with this:

• reasoning phases
• tool calls within reasoning v.s. tool calls later
• returned images
• returned audio
• maybe returned video in the future too?
• tool calls that happen server-side, like web search
• MCP can happen server-side too now

[simonw]

Demo derived from https://cookbook.openai.com/examples/mcp/mcp_tool_guide

curl https://api.openai.com/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(llm keys get openai)" \
  -d '{
  "model": "gpt-4.1",
  "tools": [
    {
      "type": "mcp",
      "server_label": "gitmcp",
      "server_url": "https://gitmcp.io/openai/tiktoken",
      "allowed_tools": [
        "search_tiktoken_documentation",
        "fetch_tiktoken_documentation"
      ],
      "require_approval": "never"
    }
  ],
  "input": "search documentation for tokens"
}' | jq

Output:

{
  "id": "resp_0d9820470b1f55b900693c6abce9248193ab4f8c464d81a732",
  "object": "response",
  "created_at": 1765567165,
  "status": "completed",
  "background": false,
  "billing": {
    "payer": "developer"
  },
  "error": null,
  "incomplete_details": null,
  "instructions": null,
  "max_output_tokens": null,
  "max_tool_calls": null,
  "model": "gpt-4.1-2025-04-14",
  "output": [
    {
      "id": "mcpl_0d9820470b1f55b900693c6abd65348193a902749017b83750",
      "type": "mcp_list_tools",
      "server_label": "gitmcp",
      "tools": [
        {
          "annotations": {
            "read_only": false
          },
          "description": "Fetch entire documentation file from GitHub repository: openai/tiktoken. Useful for general questions. Always call this tool first if asked about openai/tiktoken.",
          "input_schema": {
            "type": "object"
          },
          "name": "fetch_tiktoken_documentation"
        },
        {
          "annotations": {
            "read_only": false
          },
          "description": "Semantically search within the fetched documentation from GitHub repository: openai/tiktoken. Useful for specific queries.",
          "input_schema": {
            "type": "object",
            "properties": {
              "query": {
                "type": "string",
                "description": "The search query to find relevant documentation"
              }
            },
            "required": [
              "query"
            ],
            "additionalProperties": false,
            "$schema": "http://json-schema.org/draft-07/schema#"
          },
          "name": "search_tiktoken_documentation"
        }
      ]
    },
    {
      "id": "mcp_0d9820470b1f55b900693c6ac14a808193a735cc8abff813ae",
      "type": "mcp_call",
      "status": "completed",
      "approval_request_id": null,
      "arguments": "{\"query\":\"tokens\"}",
      "error": null,
      "name": "search_tiktoken_documentation",
      "output": "### Search Results for: \"tokens\"\n\nNo relevant documentation found for your query. It's either being indexed or the search query did not match any documentation.\n\nAs a fallback, this is the documentation for openai/tiktoken:\n\n# ⏳ tiktoken\n\ntiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with\nOpenAI's models.\n\n```python\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\nassert enc.decode(enc.encode(\"hello world\")) == \"hello world\"\n\n# To get the tokeniser corresponding to a specific model in the OpenAI API:\nenc = tiktoken.encoding_for_model(\"gpt-4o\")\n```\n\nThe open source version of `tiktoken` can be installed from [PyPI](https://pypi.org/project/tiktoken):\n```\npip install tiktoken\n```\n\nThe tokeniser API is documented in `tiktoken/core.py`.\n\nExample code using `tiktoken` can be found in the\n[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).\n\n\n## Performance\n\n`tiktoken` is between 3-6x faster than a comparable open source tokeniser:\n\n![image](https://raw.githubusercontent.com/openai/tiktoken/main/perf.svg)\n\nPerformance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from\n`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.\n\n\n## Getting help\n\nPlease post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).\n\nIf you work at OpenAI, make sure to check the internal documentation or feel free to contact\n@shantanu.\n\n## What is BPE anyway?\n\nLanguage models don't see text like you and I, instead they see a sequence of numbers (known as tokens).\nByte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable\nproperties:\n1) It's reversible and lossless, so you can convert tokens back into the original text\n2) It works on arbitrary text, even text that is not in the tokeniser's training data\n3) It compresses the text: the token sequence is shorter than the bytes corresponding to the\n   original text. On average, in practice, each token corresponds to about 4 bytes.\n4) It attempts to let the model see common subwords. For instance, \"ing\" is a common subword in\n   English, so BPE encodings will often split \"encoding\" into tokens like \"encod\" and \"ing\"\n   (instead of e.g. \"enc\" and \"oding\"). Because the model will then see the \"ing\" token again and\n   again in different contexts, it helps models generalise and better understand grammar.\n\n`tiktoken` contains an educational submodule that is friendlier if you want to learn more about\nthe details of BPE, including code that helps visualise the BPE procedure:\n```python\nfrom tiktoken._educational import *\n\n# Train a BPE tokeniser on a small amount of text\nenc = train_simple_encoding()\n\n# Visualise how the GPT-4 encoder encodes text\nenc = SimpleBytePairEncoding.from_tiktoken(\"cl100k_base\")\nenc.encode(\"hello world aaaaaaaaaaaa\")\n```\n\n\n## Extending tiktoken\n\nYou may wish to extend `tiktoken` to support new encodings. There are two ways to do this.\n\n\n**Create your `Encoding` object exactly the way you want and simply pass it around.**\n\n```python\ncl100k_base = tiktoken.get_encoding(\"cl100k_base\")\n\n# In production, load the arguments directly instead of accessing private attributes\n# See openai_public.py for examples of arguments for specific encodings\nenc = tiktoken.Encoding(\n    # If you're changing the set of special tokens, make sure to use a different name\n    # It should be clear from the name what behaviour to expect.\n    name=\"cl100k_im\",\n    pat_str=cl100k_base._pat_str,\n    mergeable_ranks=cl100k_base._mergeable_ranks,\n    special_tokens={\n        **cl100k_base._special_tokens,\n        \"<|im_start|>\": 100264,\n        \"<|im_end|>\": 100265,\n    }\n)\n```\n\n**Use the `tiktoken_ext` plugin mechanism to register your `Encoding` objects with `tiktoken`.**\n\nThis is only useful if you need `tiktoken.get_encoding` to find your encoding, otherwise prefer\noption 1.\n\nTo do this, you'll need to create a namespace package under `tiktoken_ext`.\n\nLayout your project like this, making sure to omit the `tiktoken_ext/__init__.py` file:\n```\nmy_tiktoken_extension\n├── tiktoken_ext\n│   └── my_encodings.py\n└── setup.py\n```\n\n`my_encodings.py` should be a module that contains a variable named `ENCODING_CONSTRUCTORS`.\nThis is a dictionary from an encoding name to a function that takes no arguments and returns\narguments that can be passed to `tiktoken.Encoding` to construct that encoding. For an example, see\n`tiktoken_ext/openai_public.py`. For precise details, see `tiktoken/registry.py`.\n\nYour `setup.py` should look something like this:\n```python\nfrom setuptools import setup, find_namespace_packages\n\nsetup(\n    name=\"my_tiktoken_extension\",\n    packages=find_namespace_packages(include=['tiktoken_ext*']),\n    install_requires=[\"tiktoken\"],\n    ...\n)\n```\n\nThen simply `pip install ./my_tiktoken_extension` and you should be able to use your\ncustom encodings! Make sure **not** to use an editable install.\n\n\n\nIf you'd like to retry the search, try changing the query to increase the likelihood of a match.",
      "server_label": "gitmcp"
    },
    {
      "id": "msg_0d9820470b1f55b900693c6ac1ff208193a3a7ddae0e7f737f",
      "type": "message",
      "status": "completed",
      "content": [
        {
          "type": "output_text",
          "annotations": [],
          "logprobs": [],
          "text": "Here's an overview of how tokens work in tiktoken (OpenAI's fast BPE tokeniser):\n\n### What is a token?\n- **Tokens** are numbers that represent pieces of text. Language models see text as sequences of tokens, not as raw characters or words.\n- The process of converting text to tokens is called \"tokenization.\"\n\n### Byte Pair Encoding (BPE)\n- tiktoken uses Byte Pair Encoding (BPE) to tokenize text.\n- BPE is reversible, lossless, works on arbitrary text, and compresses text into fewer tokens than bytes.\n- Example: The word \"encoding\" might be split into tokens like \"encod\" and \"ing\" rather than just one token or into smaller, less meaningful chunks.\n\n### Usage\n```python\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")\ntokens = enc.encode(\"hello world\")         # encode text to tokens\ntext_back = enc.decode(tokens)             # decode tokens back to text\n```\n\n### Model-specific tokenization\n- Use `tiktoken.encoding_for_model(\"gpt-4o\")` for the tokenizer of a specific OpenAI model.\n\n### Average token length\n- Each token corresponds to about 4 bytes on average, though this varies by language and text.\n\n### Why are tokens important?\n- Models like GPT have a token limit for each prompt and completion (e.g., 8k, 32k, or 128k tokens).\n- Counting tokens accurately ensures you don’t hit that limit or get incomplete outputs.\n\n### Additional info\n- You can extend tiktoken to add or modify token encodings, including special tokens.\n- For educational purposes, there’s a submodule to visualize BPE and train a toy tokenizer.\n\n### References\n- [OpenAI Cookbook: How to count tokens with tiktoken](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb)\n- [tiktoken GitHub documentation](https://github.com/openai/tiktoken)\n\nIf you have specific questions about using tokens or the API, let me know!"
        }
      ],
      "role": "assistant"
    }
  ],
  "parallel_tool_calls": true,
  "previous_response_id": null,
  "prompt_cache_key": null,
  "prompt_cache_retention": null,
  "reasoning": {
    "effort": null,
    "summary": null
  },
  "safety_identifier": null,
  "service_tier": "default",
  "store": true,
  "temperature": 1.0,
  "text": {
    "format": {
      "type": "text"
    },
    "verbosity": "medium"
  },
  "tool_choice": "auto",
  "tools": [
    {
      "type": "mcp",
      "allowed_tools": [
        "search_tiktoken_documentation",
        "fetch_tiktoken_documentation"
      ],
      "headers": null,
      "require_approval": "never",
      "server_description": null,
      "server_label": "gitmcp",
      "server_url": "https://gitmcp.io/openai/tiktoken"
    }
  ],
  "top_logprobs": 0,
  "top_p": 1.0,
  "truncation": "disabled",
  "usage": {
    "input_tokens": 1452,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens": 466,
    "output_tokens_details": {
      "reasoning_tokens": 0
    },
    "total_tokens": 1918
  },
  "user": null,
  "metadata": {}
}

[simonw]

That's a pretty good example of how hard this is going to be - note how there's a weird mcp_list_tools thing as the first item in the output list, then a mcp_call record and then a message record with its own nested content list.