Add RFC

Author: robacourt

docs/rfcs/algorithm-for-move-ins-out-with-subqueries.md

@@ -0,0 +1,134 @@
+# RFC: Move-in/Move-out Handling for Shapes with Subqueries
+
+_Written with assistance from Claude Opus 4.1_
+
+## Background
+
+Shapes now support subqueries in WHERE clauses (e.g. `SELECT * FROM tasks WHERE project_id IN (SELECT id FROM projects WHERE ...)`). This introduces move-ins (project now satisfies filter → pull in all its tasks) and move-outs (project no longer satisfies filter → remove its tasks). Currently we invalidate shapes on any move, forcing complete resync.
+
+## Core Algorithm: Move-in Query Positioning
+
+When a move-in is detected:
+
+1. Execute query with `REPEATABLE READ READ ONLY` transaction, capture xmin/xmax/xip snapshot info, but don't block processing until results are available
+1. Continue processing replication stream, writing to shape log until either:
+    - A transaction outside the snapshot is encountered → start buffering
+    - Query sends first result → pause processing, start buffering (query takes write lock on shape log, because we need to write results in the correct place without interleaving ongoing transactions or storing entire query result set in memory)
+1. Ongoing transaction processing:
+    - **Before** query results written: write anything that's considered part of the snapshot until first transaction that's not part of the snapshot based on the info, then buffer
+    - **While writing** buffer every transaction for a limited time or memory budget
+    - **After** query results written: apply if transaction is not part of the snapshot
+1. Resume processing buffered transactions after query completes
+
+This ensures causal consistency - the query results are positioned correctly in the replication stream without duplicates and without shadowing things that
+
+## Two Approaches for Move-out Tracking
+
+### Approach 1: Server-side Tracking
+
+Server maintains on-disk index: `{parent_key -> Set<child_keys>}` where we can for now assume sets are disjoint.
+
+**Move-out handling:**
+
+- Lookup all child keys for the parent
+- Send delete/row-gone messages for each child
+
+**Pros:**
+
+- Simpler clients - no tracking logic needed
+- More consistent shape log - server knows exactly what was sent
+- Clean abstraction boundary
+
+**Cons:**
+
+- Disk/memory cost scales with total rows across all shapes
+- Requires persistence and recovery mechanisms
+- Doesn't play well with `changes_only` mode unless we track rows on updates too
+
+### Approach 2: Client-side Tracking
+
+Server annotates each row with "present because of parent_key X" on inserts. Client maintains mapping.
+
+**Move-out handling:**
+
+- Server sends "parent_key X moved out" message
+- Client removes all rows it tracked for that parent_key
+
+**Pros:**
+
+- Minimal server resources - no per-row tracking
+- Works naturally with `offset=now` mode where clients delete only what they've seen, no server assumptions needed
+
+**Cons:**
+
+- More complex client implementation
+- Clients need persistent parent-key mapping storage
+- Doesn't work well with `changes_only` mode unless we annotate updates too
+- All client implementations must support this
+
+## Trade-offs
+
+The key tension is between server resource usage and client complexity. Server-side tracking provides better consistency guarantees and simpler clients at the cost of significant disk usage. Client-side tracking pushes complexity to clients but scales better and handles partial history naturally.
+
+## Edge Cases and Considerations
+
+### Race Conditions
+
+- If project moves in → out → in while first move-in query still executing: cancel in-flight query on symmetrical move-out
+
+### Failure Modes
+
+- Move-in query failure → shape invalidation
+- Buffer overflow (configurable limit) → shape invalidation
+
+### Nested Subqueries
+
+With nested subqueries like `WHERE project_id IN (SELECT ... WHERE owner_id IN (SELECT ...))`, we use internally materialized shapes. Move-ins cascade: innermost → middle → outer. **This introduces lag** as each level must complete before triggering the next.
+
+### Multiple Subqueries (Future)
+
+- AND: Straightforward move-out when any condition fails
+- OR: Complex - query results may overlap with already-sent rows. Would require bidirectional mapping: `{child_key -> Set<parent_keys>}` for server-side tracking. Currently blocked because clients expect inserts, not upserts.
+
+### Performance
+
+Highly volatile filter conditions causing frequent move-ins will saturate the PostgreSQL query connection pool.
+
+## Multi-shape consistency
+
+To make the clients see consistently across shapes like `SELECT * FROM projects WHERE …` and `SELECT * FROM tasks WHERE project_id in (SELECT id FROM projects WHERE …)` we could insert a special "move-in pending" control message on the shape that's outer, so that the client can hold application until move-in resolves. [Sam Willis](https://electric-sql.slab.com/users/xvbarmzz) would this be at all useful/needed?
+
+## Implementation Notes
+
+### Consistency
+
+- All operations through single shape log ensures causal ordering
+- Move-in itself is invisible to clients (client sees resulting row operations, not original triggering operation)
+- Parent row change always visible before its move-in results if 2 shapes are on the same LSN
+
+### Transaction Handling
+
+- Process changes in topological order within same transaction: inner subquery → outer
+- Use PostgreSQL transaction visibility rules to determine if operation is "in snapshot"
+
+# Notes
+
+- Meeting notes 06-05-2025: [https://notes.granola.ai/d/349a0b89-6f4a-41a8-9f98-a447f85eacb4](https://notes.granola.ai/d/349a0b89-6f4a-41a8-9f98-a447f85eacb4?source=copy_link)
+
+# Current iteration/findings from discussions
+
+- Because Electric tends to live behind proxies, clients rarely have access to the `where` clause of a shape they're requesting. It's then unreasonable to expect the client to know how to clean up rows from a move-out without providing it some additional information
+    - Most simple implementation seems to be "tagging" where every insert/update is somehow tagged on the "why" it's present in the shape.
+        - Pro: clients need not understand full where clause evaluation, just be able to keep a set of indices for the cleanup
+        - Con: in order to support `OR`-ed subquery conditions, clients will need to have a reference count of all tags in order to correctly react to just one of the parts of an `OR` clause
+        - Con: move-ins become slightly more complicated, because we need to not only move in new rows when a new "parent" moves in, but, in case of `OR`-ed subqueries, we need to update all already-sent rows for which this parent is true as one part of an `OR` chain to have this a a new tag. 
+            - This is not as complicated as it sounds if we make 2 queries (or a `UNION` query for ease of streaming) - one that's fully disjoint from what the client currently sees (i.e. negate the current conditions) and second that's a strict subset of what the client currently sees (i.e. replace `OR` with `AND` for the new parent part) and send the former as `INSERT` with correct tags, and latter as `UPDATE` with full row value and updated tag set
+            - Or, easier, don't make the query disjoint but make the `INSERT` be treated as `UPSERT` by the client and merge the tags
+        - Con: clients need to be able to maintain this index, which is likely to be separate from normal storage that it's materializing into.
+        - Note: all where clauses can be converted to a [DNF](https://en.wikipedia.org/wiki/Disjunctive_normal_form) form and then each `AND` chain will be a separate (likely composite) tag
+    - Another approach is to make the client somehow aware of the way the tags are constructed from the source values
+        - Pro: less information in each message, saving traffic & disk
+        - Con: shape requests must include in selected columns all those that are referenced in subquery comparison operations
+        - Con: client needs to be made aware of a DNF form (or AST, or both) of a where clause for invalidation, which means the client might need to be able to execute same PostgreSQL function subset that Electric can execute. It's also possible that the constants in this AST might be sensitive if they are currently server-added. 
+- Decision currently reached - go for the disjoint + subset union queries to keep protocol simpler for now
+- Complexity involving move-outs based on tags: when a move-out happens, it happens "inside" one of the subqueries. The tag form however is based upon the where clause structure of the outer shape - if there is a `(id IN (SELECT 1) AND tag IN (SELECT 2))` where clause, then the row will have a tag of `{1, 2}` (in some representation), but we have access only to one part of the tag. This means that in practice, for a where clause involving an `OR` with an `AND` anywhere (like `(id IN (SELECT 1) AND tag IN (SELECT 2)) or (id IN (SELECT 1) AND tag IN (SELECT 3))`) there will be a pair of tags `{1, 2}` and `{1, 3}` and the invalidation would need to be able to specify a partial tag within an `AND` (luckily we don't care if there are any row-content dependent constants `AND`-ed with the subquery contents) (which is not the case with `OR`-ed constants). This means we need to be able to specify a move-out of a `{*, 2}` tag, which might make index-keeping on the client complicated or inefficient. (server-backed move-outs would also need to make this trade-off, but