Full virtual scrolling for entry list (#2152)

* Do morph type filtering server-side

* Add Delayed component to wrap VList items

* Move SortMenu and extract options

* Add entry loader service

* Make agent instructions leaner

* Add virtual scrolling tests

* Add entry-loader-service tests

* Prepare backend API for querying entry index

* v2 test improvements with added event handling

* Add more failing event handling test cases

* Frontend: Add entry index lookup, versions and invalidation

* Use quiet reset for all events

* Debounce quiet resets

* Fix sqlite contains function not always registered

* Handle restoring deleted entries

Author: myieye

.github/copilot-instructions.md

@@ -7,28 +7,14 @@ This is a monorepo containing:
 - **FwLite** - A lightweight FieldWorks application (MAUI desktop app)
 - **FwHeadless** - A headless service for FieldWorks data processing
 
-## Tech Stack
+### Tech Stack
 
 - **Backend**: .NET 9, C#, Entity Framework Core, GraphQL (Hot Chocolate)
 - **Frontend**: SvelteKit, TypeScript
 - **Database**: PostgreSQL
 - **Infrastructure**: Docker, Kubernetes, Skaffold, Tilt
 
-## Development Commands
-
-```bash
-# Backend
-dotnet build backend/LexBoxApi/LexBoxApi.csproj
-dotnet test
-
-# FwLite (Windows)
-dotnet build backend/FwLite/FwLiteMaui/FwLiteMaui.csproj --framework net9.0-windows10.0.19041.0
-
-# Frontend
-cd frontend && pnpm dev
-```
-
-## Project Structure
+### Structure
 
 ```text
 languageforge-lexbox/
@@ -39,68 +25,67 @@ languageforge-lexbox/
 │   ├── FwLite/          # FwLite MAUI app
 │   ├── FwHeadless/      # Headless FW service
 │   └── Testing/         # Test projects
-├── frontend/            # SvelteKit web app
+├── frontend/            # Lexbox SvelteKit web app
+├── frontend/viewer/     # FieldWorks Lite frontend Svelte code
 └── deployment/          # K8s/Docker configs
 ```
 
-**IMPORTANT: Testing Policy**
-- ❌ **Do NOT run integration tests** (`dotnet test`) unless the user explicitly asks
-- Integration tests require full test infrastructure (database, services) and take significant time
-- Only run unit tests locally when verifying critical business logic
-- User must explicitly request test runs before executing them
-
-### Checking GitHub Issues and PRs
-
-When asked to check GitHub issues or PRs, use the `gh` CLI instead of browser tools:
-
-```bash
-# List open issues
-gh issue list --limit 30
-
-# List open PRs
-gh pr list --limit 30
-
-# View specific issue or PR
-gh issue view <number>
-gh pr view <number>
-```
-
-- If the user asks about "the" PR, but does not explicitly name a PR or branch, assume they mean the PR associated with the current branch.
-
-Provide an in-conversation summary highlighting:
-- Urgent/critical issues (regressions, bugs, broken builds)
-- Common themes or patterns
-- Items needing immediate attention
-
-**Why CLI over browser**: Faster, less tokens, easier to scan and discuss.
-
 ### Important Files
 
 Key documentation for this project:
 - `README.md` - Project overview and setup
 - `AGENTS.md` - You are here! Agent instructions
-- `.github/copilot-instructions.md` - GitHub Copilot auto-loaded instructions
 - `.github/AGENTS.md` - **CI/CD and deployment guide** (workflows, K8s, Docker)
 - `docs/DEVELOPER-win.md` - Windows development setup
 - `docs/DEVELOPER-linux.md` - Linux development setup
 - `docs/DEVELOPER-osx.md` - macOS development setup
 - `backend/README.md` - Backend architecture
-- `backend/FwLite/AGENTS.md` - **FwLite/CRDT critical code guide** (data loss risks!)
-- `backend/FwHeadless/AGENTS.md` - **FwHeadless sync guide**
+- `backend/AGENTS.md` - General backend guidelines
+- `backend/LexBoxApi/AGENTS.md` - API & GraphQL specific rules
+- `backend/FwLite/AGENTS.md` - **FwLite/CRDT** (Critical code! Data loss risks!)
+- `backend/FwHeadless/AGENTS.md` - **FwHeadless guide** (Critical code! Data loss risks! Mercurial sync, FwData processing)
+- `frontend/AGENTS.md` - General frontend/SvelteKit rules
+- `frontend/viewer/AGENTS.md` - **FwLite Viewer** (Specific frontend rules)
 - `deployment/README.md` - Deployment and infrastructure
 
+## Guidelines
+
+### Testing
+
+- ❌ **Do NOT run dotnet INTEGRATION tests** unless the user explicitly asks. They require full test infrastructure (database, services) which usually isn't available.
+- ✅ **DO run unit tests locally** and filter to the tests that are relevant to the changes you are making. Use IDE testing tools over the cli.
+
 ### Questions?
 
 - Check existing issues: `gh issue list --limit 30`
 - Look at recent commits: `git log --oneline -20`
 - Read the docs in `docs/` directory
 - Create a GitHub issue if unsure
+- Ask the user to clarify
+
+### Pre-Flight Check
+
+Before implementing any change that will touch many files or is in a 🔴 **Critical** area (FwLite sync, FwHeadless) do a "Pre-Flight Check" and list every component in the chain that will be touched (e.g., MiniLcm -> LcmCrdt -> FwDataBridge -> SyncHelper).
 
 ### Important Rules
 
-- ✅ Use GitHub Issues for task tracking
+- ✅ **ALWAYS read local `AGENTS.md` files** in the directories you are working in (and their parents) before starting.
+- ✅ **ALWAYS review relevant code paths** before asking clarification questions.
+- ✅ New instructions in AGENTS.md files should be SUCCINCT.
 - ✅ Use `gh` CLI for GitHub issues/PRs, not browser tools
+- ✅ When pulling PR comments with `gh` use `api`. It's the only thing that returns review comments.
+- ✅ If the user asks about "the" PR, but does not explicitly name a PR or branch, assume they mean the PR associated with the current branch.
 - ✅ Use **Mermaid diagrams** for flowcharts and architecture (not ASCII art)
+- ✅ Prefer IDE diagnostics (compiler/lint errors) over CLI tools for identifying issues. Fixing these diagnostics is part of completing any instruction.
 - ✅ Do NOT run integration tests unless user explicitly requests
-- ❌ Do NOT use ASCII art for diagrams (use Mermaid instead)
+- ✅ When handling a user prompt ALWAYS ask for clarification if there are details to clarify, important decisions that must be made first or the plan sounds unwise
 - ❌ Do NOT git commit or git push without explicit user approval
+
+### 🛡️ VIGILANCE
+
+- ❌ **NEVER "fix" a failure** by removing assertions, commenting out code, or changing data to match a broken implementation.
+- ✅ **ALWAYS fix the root cause** when a test or check fails.
+- ✅ **ALWAYS double-check** that your "fix" hasn't made a check or test meaningless (e.g., asserting `expect(true).toBe(true)`).
+- ✅ **Assert that E2E test user actions** e.g. (scroll, click, etc.) actually have the expected effect before proceeding further.
+
+If you are struggling, explain the difficulty to the user instead of cheating. **Integrity is non-negotiable.**

AGENTS.md

@@ -30,6 +30,17 @@ dotnet test FwLiteOnly.slnf
 dotnet build FwLiteMaui/FwLiteMaui.csproj --framework net9.0-windows10.0.19041.0
 ```
 
+## Generated Types (TypeScript)
+
+The frontend viewer uses TypeScript types and API interfaces generated from .NET using **Reinforced.Typings**. These are automatically updated when you build the **FwLiteShared** project (or any project that depends on it like `FwLiteMaui` or `FwLiteWeb`).
+
+```bash
+# To manually update generated types:
+dotnet build backend/FwLite/FwLiteShared/FwLiteShared.csproj
+```
+
+The configuration for this lives in `FwLiteShared/TypeGen/ReinforcedFwLiteTypingConfig.cs` and `FwLiteShared/Reinforced.Typings.settings.xml`.
+
 ## Project Structure
 
 | Directory | Priority | Purpose |

backend/FwLite/AGENTS.md

@@ -0,0 +1,12 @@
+using FwDataMiniLcmBridge.Tests.Fixtures;
+
+namespace FwDataMiniLcmBridge.Tests.MiniLcmTests;
+
+[Collection(ProjectLoaderFixture.Name)]
+public class EntryIndexTests(ProjectLoaderFixture fixture) : EntryIndexTestsBase
+{
+    protected override Task<IMiniLcmApi> NewApi()
+    {
+        return Task.FromResult<IMiniLcmApi>(fixture.NewProjectApi("entry-index-test", "en", "en"));
+    }
+}

backend/FwLite/FwDataMiniLcmBridge.Tests/MiniLcmTests/EntryIndexTests.cs

@@ -916,23 +916,27 @@ public IAsyncEnumerable<Entry> GetEntries(
         Func<ILexEntry, bool>? predicate, QueryOptions? options = null, string? query = null)
     {
         options ??= QueryOptions.Default;
-        var entries = GetLexEntries(predicate, options);
-
-        entries = ApplySorting(options, entries, query);
+        var entries = GetFilteredAndSortedEntries(predicate, options, options.Order, query);
         entries = options.ApplyPaging(entries);
 
         return entries.ToAsyncEnumerable().Select(FromLexEntry);
     }
 
-    private IEnumerable<ILexEntry> ApplySorting(QueryOptions options, IEnumerable<ILexEntry> entries, string? query)
+    private IEnumerable<ILexEntry> GetFilteredAndSortedEntries(Func<ILexEntry, bool>? predicate, FilterQueryOptions? filterOptions, SortOptions order, string? query)
+    {
+        var entries = GetLexEntries(predicate, filterOptions);
+        return ApplySorting(order, entries, query);
+    }
+
+    private IEnumerable<ILexEntry> ApplySorting(SortOptions order, IEnumerable<ILexEntry> entries, string? query)
     {
-        var sortWs = GetWritingSystemHandle(options.Order.WritingSystem, WritingSystemType.Vernacular);
-        if (options.Order.Field == SortField.SearchRelevance)
+        var sortWs = GetWritingSystemHandle(order.WritingSystem, WritingSystemType.Vernacular);
+        if (order.Field == SortField.SearchRelevance)
         {
-            return entries.ApplyRoughBestMatchOrder(options.Order, sortWs, query);
+            return entries.ApplyRoughBestMatchOrder(order, sortWs, query);
         }
 
-        return options.ApplyOrder(entries, e => e.LexEntryHeadword(sortWs));
+        return order.ApplyOrder(entries, e => e.LexEntryHeadword(sortWs));
     }
 
     public IAsyncEnumerable<Entry> SearchEntries(string query, QueryOptions? options = null)
@@ -955,6 +959,24 @@ public IAsyncEnumerable<Entry> SearchEntries(string query, QueryOptions? options
         return Task.FromResult(lexEntry is null ? null : FromLexEntry(lexEntry));
     }
 
+    public Task<int> GetEntryIndex(Guid entryId, string? query = null, IndexQueryOptions? options = null)
+    {
+        var predicate = EntrySearchPredicate(query);
+        var entries = GetFilteredAndSortedEntries(predicate, options, options?.Order ?? SortOptions.Default, query);
+
+        var rowIndex = 0;
+        foreach (var entry in entries)
+        {
+            if (entry.Guid == entryId)
+            {
+                return Task.FromResult(rowIndex);
+            }
+            rowIndex++;
+        }
+
+        return Task.FromResult(-1);
+    }
+
     public async Task<Entry> CreateEntry(Entry entry, CreateEntryOptions? options = null)
     {
         options ??= CreateEntryOptions.Everything;

backend/FwLite/FwDataMiniLcmBridge/Api/FwDataMiniLcmApi.cs

@@ -32,6 +32,7 @@ public class LexEntryFilterMapProvider : EntryFilterMapProvider<ILexEntry>
 
     public override Expression<Func<ILexEntry, string, object>> EntryCitationForm => (entry, ws) => entry.PickText(entry.CitationForm, ws);
     public override Expression<Func<ILexEntry, string, object>> EntryLiteralMeaning => (entry, ws) => entry.PickText(entry.LiteralMeaning, ws);
+    public override Expression<Func<ILexEntry, object?>> EntryMorphType => e => LcmHelpers.FromLcmMorphType(e.PrimaryMorphType);
     public override Expression<Func<ILexEntry, object?>> EntryComplexFormTypes => e => EmptyToNull(e.ComplexFormEntryRefs.SelectMany(r => r.ComplexEntryTypesRS));
     public override Func<string, object>? EntryComplexFormTypesConverter => EntryFilter.NormalizeEmptyToNull<ILexEntryType>;
 }

backend/FwLite/FwDataMiniLcmBridge/LexEntryFilterMapProvider.cs

@@ -95,6 +95,12 @@ public Task<int> CountEntries(string? query, FilterQueryOptions? options)
         return Task.Run(() => _wrappedApi.CountEntries(query, options));
     }
 
+    [JSInvokable]
+    public Task<int> GetEntryIndex(Guid id, string? query, IndexQueryOptions? options)
+    {
+        return Task.Run(() => _wrappedApi.GetEntryIndex(id, query, options));
+    }
+
     [JSInvokable]
     public Task<Entry[]> GetEntries(QueryOptions? options = null)
     {

backend/FwLite/FwLiteShared/Services/MiniLcmJsInvokable.cs

@@ -119,6 +119,7 @@ private static void ConfigureMiniLcmTypes(ConfigurationBuilder builder)
         builder.ExportAsInterfaces([
                 typeof(QueryOptions),
                 typeof(FilterQueryOptions),
+                typeof(IndexQueryOptions),
                 typeof(SortOptions),
                 typeof(ExemplarOptions),
                 typeof(EntryFilter),

backend/FwLite/FwLiteShared/TypeGen/ReinforcedFwLiteTypingConfig.cs

@@ -99,6 +99,7 @@ await projectProvider.OpenProject(project, context.HttpContext.RequestServices)
         api.MapGet("/entries", MiniLcm.GetEntries);
         api.MapGet("/entries/{search}", MiniLcm.SearchEntries);
         api.MapGet("/entry/{id:Guid}", MiniLcm.GetEntry);
+        api.MapGet("/entry/{id:Guid}/index", MiniLcm.GetEntryIndex);
         api.MapGet("/parts-of-speech", MiniLcm.GetPartsOfSpeech);
         api.MapGet("/semantic-domains", MiniLcm.GetSemanticDomains);
         api.MapGet("/publications", MiniLcm.GetPublications);
@@ -137,6 +138,16 @@ public static IAsyncEnumerable<Entry> SearchEntries([FromServices] MiniLcmHolder
             return api.GetEntry(id);
         }
 
+        public static Task<int> GetEntryIndex(
+            Guid id,
+            [FromQuery] string? search,
+            [AsParameters] MiniLcmQueryOptions options,
+            [FromServices] MiniLcmHolder holder)
+        {
+            var api = holder.MiniLcmApi;
+            return api.GetEntryIndex(id, search, options.ToIndexQueryOptions());
+        }
+
         public static IAsyncEnumerable<PartOfSpeech> GetPartsOfSpeech([FromServices] MiniLcmHolder holder)
         {
             var api = holder.MiniLcmApi;
@@ -185,6 +196,19 @@ public QueryOptions ToQueryOptions()
                 string.IsNullOrEmpty(GridifyFilter) ? null : new EntryFilter {GridifyFilter = GridifyFilter});
         }
 
+        public IndexQueryOptions ToIndexQueryOptions()
+        {
+            ExemplarOptions? exemplarOptions = string.IsNullOrEmpty(ExemplarValue) || ExemplarWritingSystem is null
+                ? null
+                : new(ExemplarValue, ExemplarWritingSystem);
+            var sortField = SortField ?? SortOptions.Default.Field;
+            return new IndexQueryOptions(new SortOptions(sortField,
+                    SortWritingSystem ?? SortOptions.Default.WritingSystem,
+                    Ascending ?? SortOptions.Default.Ascending),
+                exemplarOptions,
+                string.IsNullOrEmpty(GridifyFilter) ? null : new EntryFilter {GridifyFilter = GridifyFilter});
+        }
+
         public SortField? SortField { get; set; } = SortOptions.Default.Field;
 
         [DefaultValue(SortOptions.DefaultWritingSystem)]

backend/FwLite/FwLiteWeb/Routes/MiniLcmRoutes.cs

@@ -0,0 +1,26 @@
+using Xunit.Abstractions;
+
+namespace LcmCrdt.Tests.MiniLcmTests;
+
+public class EntryIndexTests(ITestOutputHelper output) : EntryIndexTestsBase
+{
+    private readonly MiniLcmApiFixture _fixture = new();
+
+    public override async Task InitializeAsync()
+    {
+        _fixture.LogTo(output);
+        await _fixture.InitializeAsync();
+        await base.InitializeAsync();
+    }
+
+    protected override Task<IMiniLcmApi> NewApi()
+    {
+        return Task.FromResult<IMiniLcmApi>(_fixture.Api);
+    }
+
+    public override async Task DisposeAsync()
+    {
+        await base.DisposeAsync();
+        await _fixture.DisposeAsync();
+    }
+}