Add Support for Inserting Multi-Modal Documents via the Helper Pane in the Expression Editor

[dan-niles]

Current Limitation

Currently, users need to manually type out the syntax to add multi-modal inputs - such as file, image, and audio documents - into the expression editor, when writing prompts for model providers:

${<ai:ImageDocument>{content: "https://ballerina.io/img/branding/ballerina_logo_dgrey_png.png"}}
${<ai:FileDocument>{content: "https://ballerina.io/pdf/ballerina_docs.pdf"}}
${<ai:AudioDocument>{content: "https://ballerina.io/mp3/branding/tech_talk.mp3"}}

This approach is not user-friendly for low-code users, as it requires prior knowledge of the syntax and document types.

Suggested Improvement

Enhance the expression helper pane to allow users to easily insert multi-modal document objects into the expression editor.
The helper pane currently supports inserting items like Inputs, Variables, Configurables, and Functions (as shown below).

We can introduce a new section — for example, Documents — that lets users conveniently add ai:ImageDocument, ai:FileDocument, or ai:AudioDocument objects when editing an expression field of type ai:Prompt.

Version

1.5.0

[dan-niles]

Update: Discussed with @shafreenAnfar, @kanushka, and @SasinduDilshara, and created the following wireframe outlining the proposed UI/UX flow for adding documents through the expression helper pane:

Excalidraw Link: https://excalidraw.com/#json=rLN3gDmwt2RDwM1I9imfF,7D0X2SF193Rda8fUEHFEJw

[dan-niles]

To-Do Checklist

• Add a Documents option to the helper pane list.
• Create a Documents page within the helper pane to display the available document types.
• Develop a variable selection page in the helper pane that lists only the following variable types: AI Documents Types, string, ai:Url, byte[]
• Implement the "Add from URL" feature to allow users to use hardcoded URLs.
• Display record-type variables in the document list. (Depends on Include record kind information in the variable completion item #1862)
• Fix the issue where record types are automatically added when selecting nested fields. (See comment below)
• Create a chip UI element for document types.
• Adjust the display of $ + curly braces (${}) in the expression editor (either hide them or make them less visually prominent)
• Limit showing Documents option in helper pane only for fields with type ai:Prompt.
• Hide backticks ("`") from the expression editor when field type is ai:Prompt.
• Introduce a Prompt ↔ Expression mode switcher for ai:Prompt typed fields. (Prompt mode hides backticks and interpolations)
• Add support for inserting variables, inputs functions and configurables inside interpolations when inserting them through helper pane.

[dan-niles]

Noticed an issue in the helper pane behavior, when selecting a variable of object type and clicking the right arrow next to it, the text is automatically inserted into the expression editor with a trailing dot.

For example:

• Clicking the arrow next to order1 adds order1. to the editor.
• If order1 has nested fields and we navigate further (e.g., order1 -> name), the helper pane appends name to form order1.name.
• However, when navigating back, the inserted text (order1.name) is not cleared from the editor.

This breaks the UX for adding record types as document chips since the helper pane keeps appending nested fields breaking the chip rendering.

I’ll be working on fixing this behavior to ensure the editor text stays consistent with the current navigation state in the helper pane. The ideal fix would be to defer inserting text into the editor until the user makes the final selection, rather than updating the text on every arrow click.

Screen.Recording.2025-11-14.at.8.04.18.AM.mov

[dan-niles]

Update: Discussed with @kanushka, and we decided to introduce a mode switcher in the prompt field. This will allow users to toggle between:

• Template mode – where backticks are hidden and the field behaves like a simplified template editor.
• Expression mode – where users can write full Ballerina expressions without any restrictions.

We already use a similar mode switcher for string-typed fields (Text Mode ↔ Expression Mode), so this approach aligns with the existing UX.

[dan-niles]

Closing this issue, as this was completed with wso2/vscode-extensions#893