Skip to content

GH-5229: Support Gemini Computer use#5335

Open
yasu89 wants to merge 1 commit intospring-projects:mainfrom
yasu89:GH-5229
Open

GH-5229: Support Gemini Computer use#5335
yasu89 wants to merge 1 commit intospring-projects:mainfrom
yasu89:GH-5229

Conversation

@yasu89
Copy link

@yasu89 yasu89 commented Jan 27, 2026

Prerequisite: the bug fix in #5007 is required.

This PR adds support for Gemini’s Computer Use feature. Close #5229
https://ai.google.dev/gemini-api/docs/computer-use

  • Adds options related to Computer Use.
  • In Computer Use, FunctionResponse parts must include a screenshot and the current URL.
    • To support this, we introduce the ToolCallResult class so tool call results can store new metadata in ToolResponse.
    • Using that metadata, we append the screenshot and URL to FunctionResponse parts.

If there’s a better approach for implementing FunctionResponse parts, I’m happy to adjust.

Signed-off-by: Wada Yasuhiro <19990967+yasu89@users.noreply.github.com>
@ilayaperumalg
Copy link
Member

@ddobrin Could you help with the review of this PR? Thanks!

@yasu89
Copy link
Author

yasu89 commented Jan 29, 2026

I’ve created a demo implementation using this PR to demonstrate the Computer Use feature:
https://github.com/yasu89/spring-ai-computer-use-demo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Gemini Computer Use

2 participants