Feat/private memory/rag integ

[henrypark133]

• Add vector store API surface: new models, routes, and OpenAPI tag/paths for vector store CRUD, search, file attachment, and file batch endpoints.
• Introduce RAG integration: optional RAG service client/config, a thin VectorStoreService proxy with ID prefix translation, and graceful “not configured” handling.
• Persist vector store refs: new vector_stores table + repository for workspace scoping, auth, and pagination.
• Wire services and routing: add vector_store_service to domain/app state and router; preserve file_search_provider when injecting MCP factory.
• Extend file repository interface to support batch lookup by IDs; add new ID prefixes for vector store resources and env example config for RAG.

---

Note

Medium Risk
Introduces new externally-calling RAG proxy logic and new authenticated endpoints plus a DB migration; failures or mismatched ID/pagination handling could impact API behavior, though changes are mostly additive and guarded when RAG is unconfigured.

Overview
Adds a Vector Stores API surface (CRUD, search, file attach/detach, and file-batch operations) and wires it into routing, OpenAPI, and application state via a new vector_store_service.

Introduces an optional VPC RAG integration (RagServiceClient + ApiConfig.rag_service) and a thin VectorStoreServiceImpl proxy that enforces workspace scoping via a new vector_stores ref table, validates referenced files, filters request fields to avoid mass-assignment, and adds/strips new ID prefixes (vs_, vsfb_). When RAG is not configured, endpoints fail gracefully via NotConfiguredRagService.

Extends persistence and tests: adds migration/repository for vector_stores, expands the file repository port to support batch lookup by IDs, and adds end-to-end vector store tests with injectable mock RAG; also fixes MCP-factory test wiring to preserve file_search_provider instead of dropping it.

^{Written by Cursor Bugbot for commit cd66e14. This will update automatically on new commits. Configure here.}

[gemini-code-assist]

Summary of Changes

Hello @henrypark133, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the application's capabilities by integrating a RAG-based private memory system, allowing users to create and manage vector stores for file search. It establishes a new API surface, introduces dedicated services for interacting with an external RAG service, and sets up the necessary database persistence and routing. The changes enable robust management of vector stores and their associated files, ensuring proper workspace isolation and authentication, while also providing a flexible and extensible architecture for future RAG-related features.

Highlights

• Vector Store API Surface: Introduced a comprehensive API surface for managing vector stores, including new models, routes, and OpenAPI documentation for CRUD operations, search functionality, file attachment, and file batch processing.
• RAG Integration: Implemented an optional RAG (Retrieval-Augmented Generation) service client and configuration, featuring a thin VectorStoreService proxy that handles ID prefix translation and provides graceful error handling when the RAG service is not configured.
• Vector Store Persistence: Added a new vector_stores database table and an associated repository (PgVectorStoreRefRepository) to persist references to vector stores, enabling workspace scoping, authentication, and pagination for these resources.
• Service Wiring and Routing: Integrated the new vector_store_service into the application's domain and API state, and extended the main router to include all new vector store-related routes. Also ensured file_search_provider is preserved during MCP factory injection.
• File Repository Enhancements and ID Prefixes: Extended the file repository interface to support batch lookup of files by IDs and introduced new ID prefixes (vs_, vsf_, vsfb_) for vector store resources, along with example RAG service configuration in env.example.

Changelog

• crates/api/src/lib.rs
  • Added vector_store_service to DomainServices struct.
  • Implemented initialization logic for the RAG service client, making it optional based on configuration.
  • Created and initialized VectorStoreServiceImpl, conditionally using a NotConfiguredRagService stub if RAG is not enabled.
  • Added vector_store_service to the DomainServices return struct.
  • Preserved file_search_provider when initializing ResponseService with MCP client factory.
  • Added vector_store_service to AppState during application build.
  • Integrated vector_store_routes into the main application router.
• crates/api/src/models.rs
  • Introduced new data models for vector store expiration (ExpiresAfterAnchor, VectorStoreExpiresAfter).
  • Added models for chunking strategies (StaticChunkingConfig, ChunkingStrategyType, ChunkingStrategy).
  • Defined VectorStoreFileCounts and VectorStoreStatus enums.
  • Created VectorStoreObject to represent a vector store.
  • Added error models for vector store files (VectorStoreFileErrorCode, VectorStoreFileError).
  • Defined VectorStoreFileStatus enum and VectorStoreFileObject for individual files.
  • Introduced VectorStoreFileBatchObject for file batches.
  • Added list response models (VectorStoreListResponse, VectorStoreFileListResponse).
  • Defined delete response models (VectorStoreDeleteResponse, VectorStoreFileDeleteResponse).
  • Created request models for vector store CRUD operations (CreateVectorStoreRequest, ModifyVectorStoreRequest).
  • Added request models for vector store file operations (CreateVectorStoreFileRequest, UpdateVectorStoreFileAttributesRequest).
  • Introduced FileBatchFileSpec and CreateVectorStoreFileBatchRequest for batch file handling.
  • Added models for vector store search functionality (VectorStoreSearchRequest, VectorStoreSearchResponse, VectorStoreSearchResult, SearchResultContent).
• crates/api/src/openapi.rs
  • Added a new OpenAPI tag: 'Vector Stores' for managing file search.
  • Registered all new vector store-related routes for OpenAPI documentation.
  • Included all new vector store and file models in the OpenAPI schema components.
• crates/api/src/routes/api.rs
  • Added vector_store_service to the AppState struct.
• crates/api/src/routes/mod.rs
  • Added vector_stores module to the API routes.
• crates/api/src/routes/vector_stores.rs
  • New file: Implements all API routes for vector store management.
  • Includes helper functions for parsing UUIDs with prefixes and mapping service errors to HTTP responses.
  • Defines ListQueryParams and FileListQueryParams for pagination and filtering.
  • Implements CRUD endpoints for vector stores (create_vector_store, list_vector_stores, get_vector_store, modify_vector_store, delete_vector_store).
  • Implements search endpoint for vector stores (search_vector_store).
  • Implements endpoints for managing files within vector stores (create_vector_store_file, list_vector_store_files, get_vector_store_file, update_vector_store_file, delete_vector_store_file).
  • Implements endpoints for managing file batches within vector stores (create_vector_store_file_batch, get_vector_store_file_batch, cancel_vector_store_file_batch, list_vector_store_file_batch_files).
• crates/api/tests/common/mod.rs
  • Added rag_service: None to the test configuration for ApiConfig.
• crates/config/src/types.rs
  • Added rag_service: Option<RagServiceConfig> to ApiConfig.
  • Introduced RagServiceConfig struct for RAG service configuration (app ID, gateway subdomain, auth token file, timeout).
  • Implemented from_env() for RagServiceConfig to load settings from environment variables.
• crates/database/src/lib.rs
  • Exported PgVectorStoreRefRepository from the database module.
• crates/database/src/migrations/sql/V0044__add_vector_stores.sql
  • New migration file: Creates the vector_stores table.
  • Table includes id (UUID, primary key), workspace_id (UUID, foreign key), created_at, and deleted_at.
  • Added indexes for workspace_id and for pagination (workspace_id, created_at, id).
• crates/database/src/repositories/file.rs
  • Added get_by_ids_and_workspace method to FileRepository to fetch multiple files by IDs within a specific workspace.
  • Implemented verify_workspace_ownership method to check if a list of file IDs all belong to a given workspace.
• crates/database/src/repositories/mod.rs
  • Added vector_store module to the database repositories.
  • Exported PgVectorStoreRefRepository.
• crates/database/src/repositories/vector_store.rs
  • New file: Implements PgVectorStoreRefRepository for managing VectorStoreRef objects.
  • Provides methods for creating, retrieving, listing, and soft-deleting vector store references, ensuring workspace scoping and pagination.
• crates/services/src/files/ports.rs
  • Added get_by_ids_and_workspace to FileRepositoryTrait for batch file retrieval.
  • Added verify_workspace_ownership to FileRepositoryTrait for checking file ownership.
• crates/services/src/id_prefixes.rs
  • Added new ID prefixes: PREFIX_VS (vector stores), PREFIX_VSF (vector store files), PREFIX_VSFB (vector store file batches).
  • Included these new prefixes in the ALL_PREFIXES array.
• crates/services/src/lib.rs
  • Added rag and vector_stores modules to the services library.
• crates/services/src/rag/client.rs
  • New file: Implements RagServiceClient, an HTTP client for communicating with the RAG service over VPC.
  • Handles request building, authentication (bearer token from file), response checking, and JSON serialization/deserialization.
  • Provides methods for all RAG service operations (CRUD for vector stores, files, file batches, and search).
• crates/services/src/rag/mod.rs
  • New file: Exports RagServiceClient and RagServiceTrait.
  • Includes NotConfiguredRagService, a stub implementation that returns NotConfigured errors for all methods when the RAG service is not configured.
• crates/services/src/rag/ports.rs
  • New file: Defines RagError enum for RAG service-specific errors.
  • Defines RagServiceTrait, an asynchronous trait for interacting with the RAG service, using serde_json::Value for flexible data passthrough.
• crates/services/src/vector_stores/mod.rs
  • New file: Implements VectorStoreServiceImpl, which acts as a thin proxy between the API and the RAG service.
  • Includes helper functions (strip_prefix, prefix_field, add_id_prefixes, strip_file_ids_in_body) for translating ID formats between the API and the RAG service.
  • Defines internal request structs (AttachFileRequest, FileSpec, CreateFileBatchRequest, RagFileMetadataEntry).
  • Defines VectorStoreServiceError enum for service-specific errors.
  • Implements VectorStoreServiceTrait with methods for all vector store operations, including workspace verification and file ownership checks before proxying requests to the RAG service.
• crates/services/src/vector_stores/ports.rs
  • New file: Defines VectorStoreRef domain model for local persistence (ID, workspace ID, timestamps).
  • Defines PaginationParams struct for consistent pagination across vector store lists.
  • Defines VectorStoreRefRepository trait for database operations on VectorStoreRef objects.
• env.example
  • Added example environment variables for RAG service configuration, including RAG_SERVICE_APP_ID and RAG_SERVICE_TIMEOUT.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant new feature: vector store integration with a RAG service, adding new API routes, data models, service-layer logic, and database tables. However, a high-severity Broken Access Control (IDOR) vulnerability has been identified in the file batch creation logic, potentially allowing unauthorized file access due to inconsistent input verification. Additionally, several endpoints are susceptible to Mass Assignment. The review also focuses on improving the robustness of request handling in new API endpoints and cleaning up unused code. It is recommended to enforce strict validation and use typed structs for all service-layer interactions.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}