Skip to content

[release] v3.9.0 - Conversation timestamps, RAG improvements, Qdrant 1.16#32

Merged
byerlikaya merged 24 commits intomainfrom
feat/dashboard-web-ui
Feb 5, 2026
Merged

[release] v3.9.0 - Conversation timestamps, RAG improvements, Qdrant 1.16#32
byerlikaya merged 24 commits intomainfrom
feat/dashboard-web-ui

Conversation

@byerlikaya
Copy link
Owner

🚀 SmartRAG Pull Request

📝 Description

Release preparation for v3.9.0: conversation timestamps and sources, major RAG search improvements, duplicate upload prevention, Whisper native bootstrap, MCP on-demand connection, and Qdrant.Client 1.16.1 compatibility.

🔗 Related Issue

N/A - Release preparation

🔄 Type of Change

Please mark the relevant options:

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🧪 Test improvements
  • 🔧 Code refactoring
  • 🚀 Release preparation

🧪 Testing

Please describe the tests that you ran to verify your changes:

  • Unit tests pass
  • Integration tests pass
  • Manual testing completed
  • SmartRAG.Demo runs without errors
  • Build succeeds with 0 errors, 0 warnings

📋 Checklist

  • My code follows the SmartRAG style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (EN + TR)
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • LoggerMessage definitions are correct (no parameter mismatches)
  • EventId assignments are unique (no conflicts)
  • Code is generic and provider-agnostic (no hardcoded domain-specific names)
  • All public APIs have XML documentation

🚨 Critical Rules Compliance

  • Generic Code: No hardcoded table/database/column names
  • Error Fixing: Only fixed the reported error, no refactoring
  • Build Quality: 0 errors, 0 warnings, 0 messages
  • Language: All code elements in English only
  • SOLID/DRY: Principles followed

📸 Screenshots (if applicable)

N/A

📋 Additional Context

  • Version: 3.9.0
  • Breaking Changes: IStorageFactory.GetCurrentRepository(IServiceProvider), IConversationRepository new methods, IQdrantCacheManager removed
  • Migration Guide: Added in docs/en/changelog/migration-guides.md and docs/tr/changelog/migration-guides.md

🔄 Migration Guide (if breaking changes)

See docs/changelog/migration-guides.md for v3.8.x → v3.9.0 migration:

IStorageFactory:

// Before (v3.8.x)
var repository = _storageFactory.GetCurrentRepository();

// After (v3.9.0)
var repository = _storageFactory.GetCurrentRepository(_serviceProvider);

IConversationRepository (custom implementations): Add AppendSourcesForTurnAsync, GetSourcesForSessionAsync, GetAllSessionIdsAsync.

IQdrantCacheManager: Removed - remove any references.

📊 Performance Impact

  • No performance impact
  • Performance improvement
  • Performance regression (explain below)

🔒 Security Considerations

  • No security implications
  • Security improvement
  • Security concern (explain below)

byerlikaya and others added 24 commits January 28, 2026 21:02
Implement conversation source persistence across repositories and expose it via dashboard chat, and add a structured settings view with flags and layout improvements.
… chunks

- Exclude schema chunks and documents when EnableDatabaseSearch is false
- Do not prepend chunk 0 to top when it has no significant query overlap
- IndicatesMissingData returns false when sources contain majority of query terms
- Use fallback relevance threshold when few results but many candidates
- Use effective search options in aggressive search path
Enable MCP search for requests tagged with -mcp even when EnableMcpSearch
is false in config. McpIntegrationService now calls ConnectAllAsync when
no servers are connected, so first -mcp query triggers connection.
- Increase max conversation turns passed to RAG pipeline
- Boost previous-query chunks and retain top document chunks for follow-ups
- Add previous assistant answer to document context for LLM
- Refine last user query selection and prompt instructions for follow-ups
…er and docs

- DocumentService: return existing document when FileHash matches (no re-parse/OCR)
- SmartRagStartupService: initialize DatabaseConnectionManager for schema analysis
- Whisper: ForceTranscribeOnly, language handling, WhisperNativeBootstrap
- DocumentSkippedException, chunk deduplication, audio-ocr docs and examples
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…sponse building

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…gical matching

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ency patterns

Co-authored-by: Cursor <cursoragent@cursor.com>
- Add MatchesFilenameForQuery: exact word + token-prefix match (kasko/kaskosu)
- Fix sürecektir/süreli false positive in filename matching
- ChunkPrioritizer: RelevanceScore as primary sort
- Skip filename early return when PreCalculatedResults provided
- Preserve PreCalculatedResults when Options filter yields empty
- Remove Qdrant cache, remove debug logs, keyword fallback normalization
Track conversation created/updated timestamps across storage providers, expose them via dashboard chat API, and update the dashboard UI for better chat and documents usability (copyable IDs, improved layout and alignment).
- QdrantDocumentRepository: replace obsolete Vector.Data with Vector.Dense.Data
- QdrantDocumentRepository: use Vectors implicit conversion for PointStruct
- QdrantDocumentRepository: use PointsCount instead of removed VectorsCount
- SmartRAG.API: fix Swashbuckle/OpenAPI package references for successful build
- SmartRAG: Qdrant.Client 1.16.1, StackExchange.Redis, MySql.Data, itext, EPPlus, PDFtoImage
- SmartRAG.Dashboard: Microsoft.Extensions.FileProviders.Embedded 9.0.12
- SmartRAG.Demo: Microsoft.Extensions.Logging.Console, StackExchange.Redis
@byerlikaya byerlikaya self-assigned this Feb 5, 2026
@byerlikaya byerlikaya force-pushed the feat/dashboard-web-ui branch 2 times, most recently from 0ddc193 to e3265e3 Compare February 5, 2026 20:54
@byerlikaya byerlikaya merged commit d91a25e into main Feb 5, 2026
18 checks passed
@byerlikaya byerlikaya deleted the feat/dashboard-web-ui branch February 5, 2026 20:56
byerlikaya added a commit that referenced this pull request Feb 5, 2026
…1.16 (#32)

* refactor: fix storage factory scoping for document repository

* feat(dashboard): add web UI and API integration

* feat(dashboard): add chat sources and settings UI

Implement conversation source persistence across repositories and expose it via dashboard chat, and add a structured settings view with flags and layout improvements.

* fix: improve RAG chunk selection and respect DB search off for schema chunks

- Exclude schema chunks and documents when EnableDatabaseSearch is false
- Do not prepend chunk 0 to top when it has no significant query overlap
- IndicatesMissingData returns false when sources contain majority of query terms
- Use fallback relevance threshold when few results but many candidates
- Use effective search options in aggressive search path

* feat(dashboard): UI and endpoint updates

* feat(mcp): connect MCP servers on-demand when -mcp tag is used

Enable MCP search for requests tagged with -mcp even when EnableMcpSearch
is false in config. McpIntegrationService now calls ConnectAllAsync when
no servers are connected, so first -mcp query triggers connection.

* feat(rag): improve follow-up question handling with conversation context

- Increase max conversation turns passed to RAG pipeline
- Boost previous-query chunks and retain top document chunks for follow-ups
- Add previous assistant answer to document context for LLM
- Refine last user query selection and prompt instructions for follow-ups

* feat(dashboard): pipeline and UI updates for chat

* refactor: fix storage factory scoping for document repository

* fix: skip duplicate uploads by hash, init DB schema on startup, Whisper and docs

- DocumentService: return existing document when FileHash matches (no re-parse/OCR)
- SmartRagStartupService: initialize DatabaseConnectionManager for schema analysis
- Whisper: ForceTranscribeOnly, language handling, WhisperNativeBootstrap
- DocumentSkippedException, chunk deduplication, audio-ocr docs and examples

* feat(dashboard): improve web UI styling and endpoints
* feat(api): integrate dashboard with SmartRAG API
* refactor(document): improve search strategy, relevance scoring and response building
* fix: always persist sources for chat turn in dashboard
* refactor: improve chunk prioritization with phrase words and morphological matching
* feat: add filename-based early return and improve document search
* fix: improve PDF text extraction logic, Turkish encoding and OCR currency patterns
* fix: improve RAG document search and filename matching

- Add MatchesFilenameForQuery: exact word + token-prefix match (kasko/kaskosu)
- Fix sürecektir/süreli false positive in filename matching
- ChunkPrioritizer: RelevanceScore as primary sort
- Skip filename early return when PreCalculatedResults provided
- Preserve PreCalculatedResults when Options filter yields empty
- Remove Qdrant cache, remove debug logs, keyword fallback normalization

* feat: add conversation timestamps and dashboard UX improvements

Track conversation created/updated timestamps across storage providers, expose them via dashboard chat API, and update the dashboard UI for better chat and documents usability (copyable IDs, improved layout and alignment).

* style(dashboard): add WhatsApp-like dark theme and align settings badges with chat

* fix: allow Dashboard Models folder to be tracked (was ignored by **/models/)

* fix: resolve Qdrant.Client 1.16.x and SmartRAG.API build errors

- QdrantDocumentRepository: replace obsolete Vector.Data with Vector.Dense.Data
- QdrantDocumentRepository: use Vectors implicit conversion for PointStruct
- QdrantDocumentRepository: use PointsCount instead of removed VectorsCount
- SmartRAG.API: fix Swashbuckle/OpenAPI package references for successful build

* build: update NuGet package versions

- SmartRAG: Qdrant.Client 1.16.1, StackExchange.Redis, MySql.Data, itext, EPPlus, PDFtoImage
- SmartRAG.Dashboard: Microsoft.Extensions.FileProviders.Embedded 9.0.12
- SmartRAG.Demo: Microsoft.Extensions.Logging.Console, StackExchange.Redis

* [release] release: prepare v3.9.0 - conversation timestamps, RAG improvements, Qdrant 1.16
byerlikaya added a commit that referenced this pull request Feb 7, 2026
…1.16 (#32)

* refactor: fix storage factory scoping for document repository

* feat(dashboard): add web UI and API integration

* feat(dashboard): add chat sources and settings UI

Implement conversation source persistence across repositories and expose it via dashboard chat, and add a structured settings view with flags and layout improvements.

* fix: improve RAG chunk selection and respect DB search off for schema chunks

- Exclude schema chunks and documents when EnableDatabaseSearch is false
- Do not prepend chunk 0 to top when it has no significant query overlap
- IndicatesMissingData returns false when sources contain majority of query terms
- Use fallback relevance threshold when few results but many candidates
- Use effective search options in aggressive search path

* feat(dashboard): UI and endpoint updates

* feat(mcp): connect MCP servers on-demand when -mcp tag is used

Enable MCP search for requests tagged with -mcp even when EnableMcpSearch
is false in config. McpIntegrationService now calls ConnectAllAsync when
no servers are connected, so first -mcp query triggers connection.

* feat(rag): improve follow-up question handling with conversation context

- Increase max conversation turns passed to RAG pipeline
- Boost previous-query chunks and retain top document chunks for follow-ups
- Add previous assistant answer to document context for LLM
- Refine last user query selection and prompt instructions for follow-ups

* feat(dashboard): pipeline and UI updates for chat

* refactor: fix storage factory scoping for document repository

* fix: skip duplicate uploads by hash, init DB schema on startup, Whisper and docs

- DocumentService: return existing document when FileHash matches (no re-parse/OCR)
- SmartRagStartupService: initialize DatabaseConnectionManager for schema analysis
- Whisper: ForceTranscribeOnly, language handling, WhisperNativeBootstrap
- DocumentSkippedException, chunk deduplication, audio-ocr docs and examples

* feat(dashboard): improve web UI styling and endpoints
* feat(api): integrate dashboard with SmartRAG API
* refactor(document): improve search strategy, relevance scoring and response building
* fix: always persist sources for chat turn in dashboard
* refactor: improve chunk prioritization with phrase words and morphological matching
* feat: add filename-based early return and improve document search
* fix: improve PDF text extraction logic, Turkish encoding and OCR currency patterns
* fix: improve RAG document search and filename matching

- Add MatchesFilenameForQuery: exact word + token-prefix match (kasko/kaskosu)
- Fix sürecektir/süreli false positive in filename matching
- ChunkPrioritizer: RelevanceScore as primary sort
- Skip filename early return when PreCalculatedResults provided
- Preserve PreCalculatedResults when Options filter yields empty
- Remove Qdrant cache, remove debug logs, keyword fallback normalization

* feat: add conversation timestamps and dashboard UX improvements

Track conversation created/updated timestamps across storage providers, expose them via dashboard chat API, and update the dashboard UI for better chat and documents usability (copyable IDs, improved layout and alignment).

* style(dashboard): add WhatsApp-like dark theme and align settings badges with chat

* fix: allow Dashboard Models folder to be tracked (was ignored by **/models/)

* fix: resolve Qdrant.Client 1.16.x and SmartRAG.API build errors

- QdrantDocumentRepository: replace obsolete Vector.Data with Vector.Dense.Data
- QdrantDocumentRepository: use Vectors implicit conversion for PointStruct
- QdrantDocumentRepository: use PointsCount instead of removed VectorsCount
- SmartRAG.API: fix Swashbuckle/OpenAPI package references for successful build

* build: update NuGet package versions

- SmartRAG: Qdrant.Client 1.16.1, StackExchange.Redis, MySql.Data, itext, EPPlus, PDFtoImage
- SmartRAG.Dashboard: Microsoft.Extensions.FileProviders.Embedded 9.0.12
- SmartRAG.Demo: Microsoft.Extensions.Logging.Console, StackExchange.Redis

* [release] release: prepare v3.9.0 - conversation timestamps, RAG improvements, Qdrant 1.16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant