epic(discord-memory): native archive + recall integration to control token growth

[drpedapati]

Problem

Discord thread/session context can grow unbounded, driving token cost up and eventually degrading reliability.

Goal

Implement native sciClaw Discord archive + recall so live prompt context remains bounded while preserving continuity via retrieved archived snippets.

Why this is separate from existing issues

Current open issues (#73, #71, #67) do not cover archive/trim/recall memory control for Discord token consumption.

Phased Delivery Plan

Phase 1 (Smallest Fully Functional Chunk) START HERE

• phase-1(mvp): auto-archive+trim with manual lexical recall for Discord #82 auto-archive + safe trim + manual lexical recall (CLI)
• feat(config): add Discord archive+recall settings and validation #75 config defaults/validation
• feat(archive): add sciClaw session adapter and markdown archive pipeline #76 session adapter + archive pipeline
• feat(trim): add safe retention policy for archived Discord sessions #77 trim policy
• feat(cli): add 'sciclaw archive discord' operational command group #80 CLI surface (minimum commands for list/run/recall)

Success criteria:

1. Long Discord sessions stop unbounded growth.
2. Archived context is retrievable by operators/users via recall command.
3. End-to-end integration tests pass for archive->trim->recall path.

Phase 2 (Automatic Recall in Agent Loop)

• feat(agent-loop): trigger auto-archive and inject bounded recall for Discord #79 auto-archive trigger hardening + bounded auto-recall injection per Discord turn

Success criteria:

1. Retrieved snippets are injected automatically with strict size caps.
2. Fail-open behavior verified when recall/index fails.

Phase 3 (Quality and Ops Hardening)

• feat(semantic): add hybrid lexical+vector recall index for Discord archives #78 hybrid lexical+vector retrieval and reranking
• feat(doctor+docs): add archive health checks and Discord memory runbook #81 doctor checks + runbook + tuning docs

Success criteria:

1. Recall relevance improvements over lexical-only baseline.
2. Operators have health checks and troubleshooting docs.

Child Issues

• phase-1(mvp): auto-archive+trim with manual lexical recall for Discord #82 phase-1(mvp): auto-archive+trim with manual lexical recall for Discord
• feat(config): add Discord archive+recall settings and validation #75 feat(config): add Discord archive+recall settings and validation
• feat(archive): add sciClaw session adapter and markdown archive pipeline #76 feat(archive): add sciClaw session adapter and markdown archive pipeline
• feat(trim): add safe retention policy for archived Discord sessions #77 feat(trim): add safe retention policy for archived Discord sessions
• feat(semantic): add hybrid lexical+vector recall index for Discord archives #78 feat(semantic): add hybrid lexical+vector recall index for Discord archives
• feat(agent-loop): trigger auto-archive and inject bounded recall for Discord #79 feat(agent-loop): trigger auto-archive and inject bounded recall for Discord
• feat(cli): add 'sciclaw archive discord' operational command group #80 feat(cli): add 'sciclaw archive discord' operational command group
• feat(doctor+docs): add archive health checks and Discord memory runbook #81 feat(doctor+docs): add archive health checks and Discord memory runbook

TDD/Integration Contract

This epic is blocked unless each phase includes:

1. Deterministic integration fixtures using realistic Discord session histories.
2. End-to-end tests for the exact phase behavior.
3. Failure-mode tests proving fail-open chat behavior.
4. Non-Discord regression coverage.

[drpedapati]

Status update (2026-02-27): Phase 1 + Phase 2 items are now closed.

Closed children:

• feat(config): add Discord archive+recall settings and validation #75 config archive+recall settings
• feat(archive): add sciClaw session adapter and markdown archive pipeline #76 archive pipeline/session adapter
• feat(trim): add safe retention policy for archived Discord sessions #77 trim policy
• feat(cli): add 'sciclaw archive discord' operational command group #80 archive CLI commands
• phase-1(mvp): auto-archive+trim with manual lexical recall for Discord #82 phase-1 MVP
• feat(agent-loop): trigger auto-archive and inject bounded recall for Discord #79 auto-recall injection in agent loop

Still open for this epic:

• feat(doctor+docs): add archive health checks and Discord memory runbook #81 doctor/docs runbook and archive health checks
• feat(semantic): add hybrid lexical+vector recall index for Discord archives #78 hybrid lexical+vector retrieval

Keeping this epic open until #81 and #78 are complete.

[drpedapati]

Phase 3 Design Finalized

The semantic memory architecture for #78 has been designed and documented in the semantic-memory-cli workspace. Key decisions:

vecflow-style Architecture

• Pure Go implementation using chromem-go for embedded vector storage
• Local ONNX embeddings (all-MiniLM-L6-v2 or nomic-embed-text)
• Hybrid BM25 + vector search with Reciprocal Rank Fusion
• Filesystem-native state — memory is a directory of files, not a database

Critical: Per-Workspace Isolation

Each routed Discord channel gets its own memory directory:

nihc3i-dave/memory/          ← Channel A's semantic index
ados-fxs/memory/             ← Channel B's semantic index (separate)
restore-manuscript/memory/   ← Channel C's semantic index (separate)

This matches the existing routing model where sessions are already namespaced per-workspace. No cross-channel memory bleeding.

Preprint Draft

A conceptual paper has been drafted in semantic-memory-cli/manuscript/ covering:

• Context rot problem (Hong et al. 2025)
• Why CLI-native memory tooling matters
• Hybrid retrieval design
• Comparison to Mem0, Letta/MemGPT, Zep, A-MEM

Implementation Path

1. Integrate chromem-go into sciclaw
2. Add embedding wrapper (ONNX or Ollama)
3. Extend sciclaw archive discord with vector indexing
4. Wire hybrid recall into agent loop

Target: <200ms recall latency, fail-open when vectors unavailable.