Microwave AI – Contact List Enrichment Agent

Microwave AI is a chat‑based agent that helps users clean, validate, and enrich CSV contact lists.
The assistant uses Gemini 3, Supabase, and a Vercel Sandbox worker to generate and execute AI‑written scraping/enrichment code, while the UI is orchestrated through tool calls rendered inside the chat.

---

High‑Level Architecture

• Frontend (Next.js App Router)

  • Chat page: app/chat/[id]/page.tsx fetches the conversation and messages from Supabase and renders ChatInterface.
  • Tool rendering: components/chat/tool-renderer.tsx maps AI tool calls to concrete React components:
    • UploadComponent – CSV upload
    • DataSelector – enrichment options
    • ProgressTracker – job status
    • EnrichedDataTable – results table for a job
    • DownloadButton – download enriched CSV

• AI Agent (Chat & Tools)

  • System prompt: lib/ai/system-prompt.ts
    • Specializes the agent in contact‑list cleaning and enrichment.
    • Defines strict workflow rules (always call tools, never fabricate CSV paths or configs).
  • Chat endpoint: app/api/chat/route.ts
    • Uses google(gemini-3-*) via the ai SDK.
    • Applies SYSTEM_PROMPT and exposes the tools from lib/ai/tools.ts.
    • Streams responses to the UI and persists messages + conversation metadata in Supabase.

• Enrichment Orchestrator

  • Tool definitions: lib/ai/tools.ts
    • showUploadComponent – ask UI to show CSV upload.
    • showDataSelector – show enrichment choices.
    • startEnrichment – create a job, normalize CSV path, and trigger the enrichment pipeline.
    • showProgressTracker / checkJobProgress – show or fetch job status.
    • showDataTable – tell the UI to display job results.
    • showDownloadButton – show download CTA for a finished job.
  • AI Coding Agent: lib/enrichment/ai-coding-agent.ts
    • Class AiCodingAgent uses google('gemini-3-pro-preview') and a dedicated coding system prompt.
    • For each active enrichment type (linkedin, website, custom):
      • Builds a prompt from csvSample, csvPath, and the enrichment config.
      • Uses tools (getApifyDocs, getFirecrawlDocs, getDownloadLink, getUploadLink) to discover APIs and get signed URLs from Supabase.
      • Generates JavaScript code that:
        • Downloads input CSV from Supabase.
        • Calls external services (Apify, Firecrawl) using provided API keys.
        • Writes an enriched CSV to Supabase (typically *-enriched.csv).
      • Runs the code in a sandbox and retries on runtime errors.
  • Sandbox Worker: lib/sandbox/worker.ts
    • SandboxWorker receives a WorkerContext with:
      • jobId, csvPath, enrichmentType, generatedCode, Supabase credentials, and external API tokens.
    • Flow:
      • Creates a Vercel Sandbox using a Puppeteer‑ready snapshot.
      • Writes the AI‑generated code to /vercel/sandbox/enrichment.js.
      • Starts it in the background and tails /vercel/sandbox/enrichment.log.
      • Parses PROGRESS: {...} lines from logs to update the jobs table (status, processed_rows, progress).
      • Searches the log for a final JSON line with { success: boolean, ... } and returns it to the orchestrator.
    • The worker never marks the job as completed itself; it reports progress and results back to the orchestrator.

• Backend API Routes

  • Start enrichment (fire‑and‑forget): app/api/enrich/route.ts
    • POST /api/enrich accepts { jobId, request }, where request is a StartEnrichmentRequest.
    • Marks the job as processing, calls aiCodingAgent.enrichData(...), and then:
      • Optionally uploads result.enrichedCsv as a new file and links it to the job.
      • Or finds / creates a files record for any outputPath returned by enrichment steps.
      • Updates the jobs row with status='completed', progress=100, and a serialized result.
    • Response is 202 Accepted; the heavy work runs in the background.
  • Job status: app/api/job/[id]/status/route.ts
    • GET /api/job/:id/status (auth‑protected).
    • Returns status, progress, processedRows, totalRows, errorMessage, and result.
    • Used by ProgressTracker and the checkJobProgress tool.
  • Download enriched CSV: app/api/job/[id]/download/route.ts
    • GET /api/job/:id/download (auth + job ownership required).
    • Locates the output via:
      • job.output_file_id → files.storage_path, or
      • job.result.enrichments[*].outputPath/output_path/output.path, or
      • job.result.enrichedCsv (streams inline), or
      • Fallback job.config.csvPath → *-enriched.csv.
    • Returns a short‑lived Supabase signed URL or a streamed CSV download.

---

End‑to‑End Flow

1. User starts a chat

• File: app/chat/[id]/page.tsx
• Flow
  • User navigates to /chat/:id.
  • Page:
    • Verifies the Supabase user (redirects to /login if missing).
    • Loads the conversation and messages from the database.
    • Renders ChatInterface with initialMessages and the tool invocation stream.

2. Agent guides upload & enrichment selection

• System behavior (from lib/ai/system-prompt.ts):
  • When the user mentions uploading / cleaning / enriching a CSV:
    • The agent must call showUploadComponent immediately (no “please upload below” text‑only UX).
  • After file upload:
    • Agent analyzes the sample, explains what it sees, and then calls showDataSelector.
  • After the user confirms their selections:
    • Agent extracts:
      • csvPath and sample rows/columns from the actual upload tool response.
      • The exact config object from the selectEnrichments callback (no modifications).
    • Calls startEnrichment once with these real values.

3. Job creation & background enrichment

• Tool: startEnrichment in lib/ai/tools.ts
  • Validates the logged‑in user via getCurrentUser.
  • Normalizes csvPath:
    • Cross‑checks Supabase files for the user.
    • Verifies the path exists in csv-files.
    • Prefers an existing *-enriched.csv as the new input, if present.
  • Creates a jobs row via createJob('data_enrichment', { ... }) with config that includes the resolved csvPath.
  • Reads FIRECRAWL_API_KEY and APIFY_API_TOKEN from the environment.
  • Calls the fire‑and‑forget /api/enrich endpoint with { jobId, request }.
  • Returns a tool result with:
    • type: 'progress'
    • jobId
    • Initial tasks (LinkedIn, Website, Custom) set to pending.
• UI:
  • ToolRenderer receives the startEnrichment tool result with type: 'progress'.
  • Renders ProgressTracker for the returned jobId.

4. Progress tracking

• Component: components/chat/tools/progress-tracker.tsx
  • Auto‑polls GET /api/job/:id/status every few seconds until the job is completed or failed.
  • Displays:
    • Overall progress bar.
    • Per‑task chips (LinkedIn, Website, Custom) and statuses.
    • Processed vs total rows, lightweight ETA estimate, and error messages.
  • When the backend marks the job as completed:
    • Sends a message back to the agent (via onMessage) instructing it to show the data table and explaining that jobId should be reused for later table/download actions.

5. Viewing results

• Tool: showDataTable in lib/ai/tools.ts
  • The agent calls this tool with the jobId and an optional message.
  • ToolRenderer responds by rendering EnrichedDataTable.
• Data grid UI: components/chat/tools/data-table.tsx
  • Generic, column‑driven @tanstack/react-table implementation.
  • Features:
    • Sortable column headers.
    • Global search across all fields.
    • Pagination with row counts and page navigation.
    • Optional summary stats (cleaned / enriched / failed).
    • Optional onDownload callback for direct CSV download.
  • The backing API that serves actual row data lives elsewhere; this component focuses purely on rendering behavior and controls.

6. Downloading the enriched CSV

• Tool: showDownloadButton in lib/ai/tools.ts
  • The agent calls this when the user is satisfied with the results.
  • ToolRenderer renders DownloadButton, which:
    • Calls GET /api/job/:id/download.
    • Receives { downloadUrl, fileName }.
    • Creates a temporary <a> element and triggers a browser download.
• Additionally, ToolRenderer handles startEnrichment messages of type enrichment_complete by:
  • Showing a compact summary of enrichments and columns added.
  • Rendering another DownloadButton wired to the same download logic.

---

Key Files & Responsibilities

• Agent behavior

  • lib/ai/system-prompt.ts – natural‑language spec for the assistant’s role and tool usage rules.
  • lib/ai/tools.ts – structured tool definitions consumed by the ai SDK and the UI.

• Chat & conversations

  • app/api/chat/route.ts – Gemini 3 streaming chat endpoint with tools and system prompt.
  • app/chat/[id]/page.tsx – server component that gates access and initializes the chat UI.

• Enrichment pipeline

  • lib/enrichment/ai-coding-agent.ts – orchestrates AI code generation + sandbox execution for each enrichment type.
  • lib/sandbox/worker.ts – Vercel Sandbox integration, job log parsing, and progress updates into Supabase.
  • app/api/enrich/route.ts – background job entrypoint and job finalization logic.
  • app/api/job/[id]/status/route.ts – job progress polling API.
  • app/api/job/[id]/download/route.ts – secure download API for the enriched CSV.

• UI tools

  • components/chat/tool-renderer.tsx – maps tool names → UI components and handles tool result lifecycles.
  • components/chat/tools/progress-tracker.tsx – live job progress visualization + callbacks back to the agent.
  • components/chat/tools/data-table.tsx – generic, searchable, paginated data grid for job results.

---

Running the Project

Prerequisites

• Node.js: v18+ recommended.
• Supabase project with:
  • Auth (users)
  • Tables such as conversations, messages, jobs, and files
  • Storage bucket csv-files
• External APIs
  • Apify account and token
  • Firecrawl API key
  • Vercel Sandbox / Puppeteer snapshot configured (for PUPPETEER_SNAPSHOT_ID)

Environment Variables (non‑exhaustive)

Set these in your .env.local (and in Vercel for production):

• Supabase
  • NEXT_PUBLIC_SUPABASE_URL
  • NEXT_PUBLIC_SUPABASE_ANON_KEY
  • SUPABASE_SERVICE_ROLE_KEY
• AI / Models
  • GOOGLE_GENERATIVE_AI_API_KEY (for Gemini 3 via @ai-sdk/google)
• Enrichment / scraping
  • APIFY_API_TOKEN
  • FIRECRAWL_API_KEY
• Sandbox / Vercel
  • PUPPETEER_SNAPSHOT_ID
  • VERCEL_URL (automatically set in production; used by startEnrichment to call /api/enrich)

Development

Install dependencies and run the dev server:

npm install
npm run dev

Then open http://localhost:3000 in your browser and log in via Supabase auth to start a conversation.

---

Notes for Contributors

• Tool discipline: The agent is designed to always use tools for UI actions (upload, data selection, progress, tables, downloads). Avoid adding text‑only flows that bypass these tools.
• Job lifecycle:
  • The sandbox worker only moves jobs between processing and failed and updates processed_rows / progress.
  • The orchestrator in /api/enrich is responsible for marking jobs completed and attaching output files.
• Extending enrichment types:
  • Add new flags to the enrichment config (EnrichmentConfig + startEnrichment schema).
  • Update ai-coding-agent to include a new type and instructions.
  • Ensure the generated code writes either:
    • enrichedCsv to logs, or
    • outputPath / output_path in the final JSON result, so download discovery continues to work.