Skip to content

feat(ai): CodeExecutionTool types for allowing model to run code #8866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
russellwheatley wants to merge 6 commits into
base: main
Choose a base branch
from
+244 −3
Open

feat(ai): CodeExecutionTool types for allowing model to run code #8866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
568199b
feat: CodeExecutionTool types for allowing model to run code
russellwheatley Jan 29, 2026
7ce4d95
test: update AI package tests
russellwheatley Jan 30, 2026
5938ce2
fix: add two new properties
russellwheatley Jan 30, 2026
c233f26
chore: record diff between firebase-js-sdk and react native firebase …
russellwheatley Jan 30, 2026
f186fa5
ci: workaround for ios workflows
russellwheatley Feb 2, 2026
1f96772
Merge branch 'main' into ai-updates
russellwheatley Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .cursor/rules/ai/ai-package.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,16 @@ registerVersion(name, version);

**Reason**: Chrome's on-device AI is browser-specific and not available in React Native. This is a web-only feature.

### Audio Conversation (startAudioConversation, etc.)
**Firebase JS SDK has:**
- `src/methods/live-session-helpers.ts` – `startAudioConversation`, `AudioConversationController`, `StartAudioConversationOptions`
- Uses Web Audio API (`AudioContext`, `AudioWorkletNode`, `AudioWorkletProcessor`), `navigator.mediaDevices.getUserMedia()`, and `MediaStream` for mic capture and playback

**React Native Firebase does NOT have:**
- `live-session-helpers.ts` or any audio-conversation exports

**Reason**: Audio conversation depends on browser-only APIs (Web Audio API, getUserMedia). React Native has no equivalent; mic and playback use different native modules or libraries. Do not port.

### Browser-Specific APIs Not Used
- No `window` object references (except in comments/docs)
- No `document` object usage (except in JSDoc examples)
Expand Down Expand Up @@ -707,6 +717,7 @@ Any difference NOT documented in these cursor rules is a potential feature gap.
| **Chrome Adapter** | ✅ Has | ❌ Doesn't have | Browser-only feature |
| **Hybrid Mode** | ✅ Has | ❌ Doesn't have | Browser-only feature |
| **HybridParams** | ✅ Has | ❌ Doesn't have | Browser-only feature |
| **Audio Conversation** (startAudioConversation, etc.) | ✅ Has | ❌ Doesn't have | Web Audio API / getUserMedia not in RN |
| **Polyfills** | ❌ Doesn't need | ✅ Requires | RN environment |
| **Dependencies** | `@firebase/*` | `@react-native-firebase/*` | Different ecosystem |
| **AIService** | Complex (with _FirebaseService) | Simple | Different architecture |
Expand Down
11 changes: 11 additions & 0 deletions packages/ai/__tests__/generate-content.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ import {
HarmBlockMethod,
HarmBlockThreshold,
HarmCategory,
Language,
Outcome,
// RequestOptions,
} from '../lib/types';
import { ApiSettings } from '../lib/types/internal';
Expand Down Expand Up @@ -172,6 +174,15 @@ describe('generateContent()', () => {
);
});

it('codeExecution', async function () {
const mockResponse = getMockResponse(BackendName.VertexAI, 'unary-success-code-execution.json');
jest.spyOn(request, 'makeRequest').mockResolvedValue(mockResponse as Response);
const result = await generateContent(fakeApiSettings, 'model', fakeRequestParams);
const parts = result.response.candidates?.[0]?.content?.parts;
expect(parts?.some(part => part.codeExecutionResult?.outcome === Outcome.OK)).toBe(true);
expect(parts?.some(part => part.executableCode?.language === Language.PYTHON)).toBe(true);
});

it('blocked prompt', async () => {
const mockResponse = getMockResponse(
BackendName.VertexAI,
Expand Down
82 changes: 82 additions & 0 deletions packages/ai/__tests__/generative-model.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,50 @@ const fakeAI: AI = {
};

describe('GenerativeModel', () => {
it('passes CodeExecutionTool with other tools through to generateContent', async function () {
const genModel = new GenerativeModel(fakeAI, {
model: 'my-model',
tools: [
{
functionDeclarations: [
{
name: 'myfunc',
description: 'mydesc',
},
],
},
{ googleSearch: {} },
{ codeExecution: {} },
],
toolConfig: { functionCallingConfig: { mode: FunctionCallingMode.NONE } },
systemInstruction: { role: 'system', parts: [{ text: 'be friendly' }] },
});
expect(genModel.tools?.length).toBe(3);
expect(genModel.toolConfig?.functionCallingConfig?.mode).toBe(FunctionCallingMode.NONE);
expect(genModel.systemInstruction?.parts[0]!.text).toBe('be friendly');
const mockResponse = getMockResponse(
BackendName.VertexAI,
'unary-success-basic-reply-short.json',
);
const makeRequestStub = jest
.spyOn(request, 'makeRequest')
.mockResolvedValue(mockResponse as Response);
await genModel.generateContent('hello');
expect(makeRequestStub).toHaveBeenCalledWith(
expect.objectContaining({
model: 'publishers/google/models/my-model',
task: request.Task.GENERATE_CONTENT,
apiSettings: expect.anything(),
stream: false,
requestOptions: {},
}),
expect.stringMatching(
new RegExp(`myfunc|googleSearch|codeExecution|${FunctionCallingMode.NONE}|be friendly`),
),
);
makeRequestStub.mockRestore();
});

it('passes params through to generateContent', async () => {
const genModel = new GenerativeModel(fakeAI, {
model: 'my-model',
Expand Down Expand Up @@ -214,6 +258,44 @@ describe('GenerativeModel', () => {
makeRequestStub.mockRestore();
});

it('passes CodeExecutionTool through to chat.sendMessage', async function () {
const genModel = new GenerativeModel(fakeAI, {
model: 'my-model',
tools: [
{ functionDeclarations: [{ name: 'myfunc', description: 'mydesc' }] },
{ googleSearch: {} },
{ codeExecution: {} },
],
toolConfig: { functionCallingConfig: { mode: FunctionCallingMode.NONE } },
systemInstruction: { role: 'system', parts: [{ text: 'be friendly' }] },
generationConfig: { topK: 1 },
});
expect(genModel.tools?.length).toBe(3);
const mockResponse = getMockResponse(
BackendName.VertexAI,
'unary-success-basic-reply-short.json',
);
const makeRequestStub = jest
.spyOn(request, 'makeRequest')
.mockResolvedValue(mockResponse as Response);
await genModel.startChat().sendMessage('hello');
expect(makeRequestStub).toHaveBeenCalledWith(
expect.objectContaining({
model: 'publishers/google/models/my-model',
task: request.Task.GENERATE_CONTENT,
apiSettings: expect.anything(),
stream: false,
requestOptions: {},
}),
expect.stringMatching(
new RegExp(
`myfunc|googleSearch|codeExecution|${FunctionCallingMode.NONE}|be friendly|topK`,
),
),
);
makeRequestStub.mockRestore();
});

it('passes text-only systemInstruction through to chat.sendMessage', async () => {
const genModel = new GenerativeModel(fakeAI, {
model: 'my-model',
Expand Down
2 changes: 2 additions & 0 deletions packages/ai/lib/methods/chat-session-helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ export function validateChatHistory(history: Content[]): void {
functionResponse: 0,
thought: 0,
thoughtSignature: 0,
executableCode: 0,
codeExecutionResult: 0,
};

for (const part of parts) {
Expand Down
89 changes: 87 additions & 2 deletions packages/ai/lib/types/content.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

import { Role } from './enums';
import { Language, Outcome, Role } from './enums';

/**
* Content type for both prompts and response candidates.
Expand All @@ -36,7 +36,9 @@ export type Part =
| InlineDataPart
| FunctionCallPart
| FunctionResponsePart
| FileDataPart;
| FileDataPart
| ExecutableCodePart
| CodeExecutionResultPart;

/**
* Content part interface if the part represents a text string.
Expand All @@ -52,6 +54,8 @@ export interface TextPart {
* @internal
*/
thoughtSignature?: string;
executableCode?: never;
codeExecutionResult?: never;
}

/**
Expand All @@ -72,6 +76,8 @@ export interface InlineDataPart {
* @internal
*/
thoughtSignature?: never;
executableCode?: never;
codeExecutionResult?: never;
}

/**
Expand Down Expand Up @@ -105,6 +111,8 @@ export interface FunctionCallPart {
* @internal
*/
thoughtSignature?: never;
executableCode?: never;
codeExecutionResult?: never;
}

/**
Expand All @@ -121,6 +129,8 @@ export interface FunctionResponsePart {
* @internal
*/
thoughtSignature?: never;
executableCode?: never;
codeExecutionResult?: never;
}

/**
Expand All @@ -138,6 +148,8 @@ export interface FileDataPart {
* @internal
*/
thoughtSignature?: never;
executableCode?: never;
codeExecutionResult?: never;
}

/**
Expand Down Expand Up @@ -202,3 +214,76 @@ export interface FileData {
mimeType: string;
fileUri: string;
}

/**
* Represents the code that is executed by the model.
*
* @beta
*/
export interface ExecutableCodePart {
text?: never;
inlineData?: never;
functionCall?: never;
functionResponse?: never;
fileData: never;
thought?: never;
/**
* @internal
*/
thoughtSignature?: never;
executableCode?: ExecutableCode;
codeExecutionResult?: never;
}

/**
* Represents the code execution result from the model.
*
* @beta
*/
export interface CodeExecutionResultPart {
text?: never;
inlineData?: never;
functionCall?: never;
functionResponse?: never;
fileData: never;
thought?: never;
/**
* @internal
*/
thoughtSignature?: never;
executableCode?: never;
codeExecutionResult?: CodeExecutionResult;
}

/**
* An interface for executable code returned by the model.
*
* @beta
*/
export interface ExecutableCode {
/**
* The programming language of the code.
*/
language?: Language;
/**
* The source code to be executed.
*/
code?: string;
}

/**
* The results of code execution run by the model.
*
* @beta
*/
export interface CodeExecutionResult {
/**
* The result of the code execution.
*/
outcome?: Outcome;
/**
* The output from the code execution, or an error message
* if it failed.
*/
output?: string;
}
36 changes: 36 additions & 0 deletions packages/ai/lib/types/enums.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,3 +304,39 @@ export const InferenceMode = {
* @public
*/
export type InferenceMode = (typeof InferenceMode)[keyof typeof InferenceMode];

/**
* Represents the result of the code execution.
*
* @beta
*/
export const Outcome = {
UNSPECIFIED: 'OUTCOME_UNSPECIFIED',
OK: 'OUTCOME_OK',
FAILED: 'OUTCOME_FAILED',
DEADLINE_EXCEEDED: 'OUTCOME_DEADLINE_EXCEEDED',
};

/**
* Represents the result of the code execution.
*
* @beta
*/
export type Outcome = (typeof Outcome)[keyof typeof Outcome];

/**
* The programming language of the code.
*
* @beta
*/
export const Language = {
UNSPECIFIED: 'LANGUAGE_UNSPECIFIED',
PYTHON: 'PYTHON',
};

/**
* The programming language of the code.
*
* @beta
*/
export type Language = (typeof Language)[keyof typeof Language];
16 changes: 15 additions & 1 deletion packages/ai/lib/types/requests.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ export interface RequestOptions {
* Defines a tool that model can call to access external knowledge.
* @public
*/
export type Tool = FunctionDeclarationsTool | GoogleSearchTool;
export type Tool = FunctionDeclarationsTool | GoogleSearchTool | CodeExecutionTool;

/**
* Structured representation of a function declaration as defined by the
Expand Down Expand Up @@ -316,6 +316,20 @@ export interface GoogleSearchTool {
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
export interface GoogleSearch {}

/**
* A tool that enables the model to use code execution.
*
* @beta
*/
export interface CodeExecutionTool {
/**
* Specifies the code execution configuration.
* Currently, this is an empty object, but it's reserved for future configuration options.
*/
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
codeExecution: {};
}

/**
* A `FunctionDeclarationsTool` is a piece of code that enables the system to
* interact with external systems to perform an action, or set of actions,
Expand Down
Loading