[Bug]: search()/find() returns non-deterministic results for identical queries

[ponsde]

Bug: search() returns non-deterministic results for identical queries

Version: openviking 0.1.17, Python 3.12, Ubuntu 24.04

Description:
Calling client.search(query) multiple times with the same query, on the same client instance, against the same unmodified dataset, returns completely different result sets each time.

5 consecutive calls yield an average Jaccard similarity of only 0.11, with zero URIs appearing in all 5 results.

Reproduction:

import openviking as ov
import os, time

os.environ['OPENVIKING_CONFIG_FILE'] = 'ov.conf'

c = ov.SyncOpenViking(path='./data')
c.initialize()
time.sleep(3)

query = "Docker 容器日志怎么看"
for run in range(5):
    r = c.search(query)
    uris = [x.uri for x in r.resources]
    print(f"Run {run+1}: {uris[:4]}")
    time.sleep(0.5)

c.close()

Actual result (same client, same query, no restart):

Run 1: ['viking://resources/A/A.md']
Run 2: ['viking://resources/B/B.md', 'viking://resources/C/C.md', ...]
Run 3: ['viking://resources/D/D.md', 'viking://resources/E/E.md', ...]
Run 4: ['viking://resources/F/F.md', 'viking://resources/G/G.md', ...]
Run 5: ['viking://resources/H/H.md', 'viking://resources/I/I.md', ...]

• Average Jaccard similarity across pairs: 0.11
• URIs common to all 5 runs: 0

Expected result:
Same client + same query + same data → identical (or near-identical) result set.

Environment details:

• Embedding model: BAAI/bge-m3 (1024 dim, via OpenAI-compatible API)
• Dataset: 38 resources (markdown files)
• No data modifications between calls
• find() exhibits the same behavior

Impact:
Downstream logic that depends on search consistency (coverage scoring, dedup, caching) becomes unreliable. Short queries (e.g. 3-character abbreviations) are affected more severely.

[bljfy]

这个问题应该是模型的温度设置设定的吧
如果想要一样的我觉得要么加一个问题的缓存机制或者降低模型的温度参数试试.

[ponsde]

感谢回复！我看了一下源码想补充几点：

我的调用方式是 client.search(query) 和 client.find(query)，都不传 session。这种情况下：

• find() → 直接构造 TypedQuery → HierarchicalRetriever，不经过 LLM
• search() 无 session → 也是直接构造 TypedQuery，跳过 IntentAnalyzer

（参考路径：storage/viking_fs.py 的 find() 和 search() 中无 session 分支）

所以温度参数应该不影响这条路径。query 就是传入的原始字符串，embedding 由 bge-m3 生成（确定性），没有 LLM 参与改写。

非确定性更可能来自底层 engine.IndexEngine.search() 的 ANN（HNSW）算法——图遍历路径在某些实现中不固定，加上 HierarchicalRetriever 的递归搜索机制会放大首轮结果的微小差异。

如果方便的话，能确认一下 C++ 引擎层是否有 seed 或确定性模式的选项？

我们目前用本地关键词索引 + 多路检索取并集来缓解，效果还行，但原生层面能稳定的话更理想。

[MaojiaSheng]

候选数据规模有多大，能大致复现吗

[r266-tech]

数据规模：

• Resources: ~190 markdown 文件（日记、文档、人物档案等）
• Memories: ~600 条（entities/events/preferences/patterns）
• Vectors: ~1500 条（text-embedding-3-large, dim=3072）
• Sessions: 5 个活跃 session

复现方法：
连续执行相同的 search() 调用（same query, same parameters），对比返回的 memory/resource URIs 集合。我观察到约 10-20% 的结果差异（Jaccard ~0.11）。

import requests, time

results = []
for i in range(5):
    r = requests.post("http://localhost:1933/api/v1/search/search", 
        json={"query": "V 的编码偏好", "limit": 5})
    uris = [m["uri"] for m in r.json()["result"]["memories"]]
    results.append(set(uris))
    time.sleep(1)

# Compare consecutive results
for i in range(len(results)-1):
    a, b = results[i], results[i+1]
    jaccard = len(a & b) / len(a | b) if a | b else 1.0
    print(f"Run {i} vs {i+1}: Jaccard={jaccard:.2f}, diff={a.symmetric_difference(b)}")

怀疑是 VLM intent analysis 的 temperature > 0 导致 TypedQuery 每次略有不同，进而 embedding 不同。find() 没有这个问题（跳过 VLM 阶段）。

[ponsde]

补充一下最新测试数据和代码路径分析。

环境

• openviking 0.1.17, Python 3.12, Ubuntu 24.04
• Embedding: bge-m3 (1024 dim, OpenAI-compatible API)
• 数据集：52 resources, 176 .md files (含 overview/abstract), VectorDB ~1.7MB
• 调用方式：Python SDK，不传 session

测试结果

3 个 query × 5 次连续调用，find() 和 search() 均不传 session：

Method	Query	Avg Jaccard	5次公共 URI
find()	"Docker 容器日志怎么看"	0.091	0
find()	"grok2api 部署"	0.168	0
find()	"Nginx 反向代理配置"	0.070	0
search()	"Docker 容器日志怎么看"	0.173	0
search()	"grok2api 部署"	0.054	0
search()	"Nginx 反向代理配置"	0.096	0

6 组全部非确定性，5 次调用零公共结果。find() 和 search() 表现一致。

关于 VLM 温度的说明

看了一下源码（storage/viking_fs.py）：

• find() → 直接构造 TypedQuery(intent="") → HierarchicalRetriever，不经过 IntentAnalyzer
• search() 不传 session → session_info 为 None → 走 L510 的 else 分支，同样直接构造 TypedQuery，不经过 IntentAnalyzer

# viking_fs.py L494-525
if session_summary or recent_messages:
    analyzer = IntentAnalyzer(max_recent_messages=5)
    # ... VLM path
else:
    # No session context: create query directly
    TypedQuery(query=query, context_type=ctx_type, intent="", priority=1)

所以在不传 session 的调用路径下，两个方法都不经过 VLM，temperature 不影响。非确定性应该来自更底层（HierarchicalRetriever → IndexEngine.search()）。

复现脚本

import asyncio, os, time
from itertools import combinations

os.environ['OPENVIKING_CONFIG_FILE'] = 'ov.conf'
import openviking as ov

async def test():
    c = ov.AsyncOpenViking(path='./data')
    await c.initialize()
    await asyncio.sleep(2)

    queries = ["Docker 容器日志怎么看", "grok2api 部署", "Nginx 反向代理配置"]

    for method_name in ["find", "search"]:
        method = getattr(c, method_name)
        for query in queries:
            all_uris = []
            for run in range(5):
                result = await method(query)
                uris = set()
                if hasattr(result, 'resources'):
                    uris.update(r.uri for r in result.resources)
                if hasattr(result, 'memories'):
                    uris.update(m.uri for m in result.memories)
                all_uris.append(uris)
                print(f"[{method_name}] '{query}' run {run+1}: {len(uris)} results")
                await asyncio.sleep(0.5)

            jaccards = [len(a&b)/len(a|b) if a|b else 1.0
                        for a, b in combinations(all_uris, 2)]
            common = all_uris[0].intersection(*all_uris[1:])
            print(f"  Avg Jaccard: {sum(jaccards)/len(jaccards):.3f}, Common: {len(common)}")

    await c.close()

asyncio.run(test())

[ponsde]

补充：纯净环境 vs 大数据量对比（定位到数据规模是触发条件）

基于上次的测试，又做了一轮对比实验，想搞清楚非确定性的触发条件。

实验设计

在同一台机器、同一份 ov.conf、同一个 openviking 0.1.17 下，建两个完全独立的数据目录：

环境	数据目录	资源数	说明
纯净	/tmp/ov_clean_test/data	5	全新目录，5 个不同主题的短文档
正式	~/OpenViking_test/data	56+	长期使用，多种来源的真实知识库

每组测试：3 个 query × 6 次连续调用 × find() + search()（均不传 session）。

纯净环境结果（5 个文档）

Query	find() Jaccard	search() Jaccard
"Docker 怎么部署"	1.00, 1.00, 1.00, 1.00, 1.00 ✅	1.00, 1.00, 1.00, 1.00, 1.00 ✅
"Python 虚拟环境"	1.00, 1.00, 1.00, 1.00, 1.00 ✅	1.00, 1.00, 1.00, 1.00, 1.00 ✅
"Nginx 反向代理"	1.00, 1.00, 1.00, 1.00, 1.00 ✅	1.00, 1.00, 1.00, 1.00, 1.00 ✅

36 次调用，全部完全一致。

正式环境结果（56+ 资源）

Query	find() Jaccard (vs #1)	search() Jaccard (vs #1)
"Docker 部署"	0.00, 0.11, 0.00, 0.00, 0.00 ❌	1.00, 0.80, 0.00, 0.43, 0.00 ❌
"grok2api 注册"	0.00, 0.67, 0.40, 0.40, 0.40 ❌	0.40, 0.40, 0.17, 0.40, 0.40 ❌
"API 代理配置"	0.67, 0.67, 0.67, 0.00, 0.25 ❌	0.11, 0.11, 0.11, 0.11, 0.11 ❌

36 次调用，全部不一致。find() 最差 Jaccard 0.00（完全不同的结果集）。

分析

1. 同机器、同配置、同版本，唯一变量是数据量 → 数据规模是触发条件
2. 纯净环境 5 个文档时 flat index 的候选集很小，排序几乎没有歧义
3. 56+ 资源时，向量相似度接近的候选多了，底层检索的非确定性就暴露了
4. find() 不经过 VLM（上次已确认），排除 temperature 因素
5. 问题大概率在 HierarchicalRetriever → 向量检索层（AGFS flat index）

环境信息

• openviking 0.1.17
• embedding: BAAI/bge-m3 (dim 1024), provider openai
• OS: Ubuntu 24.04, Python 3.12
• 向量索引: local flat, cosine, int8 量化

纯净环境搭建 + 测试脚本

"""
纯净环境: 5 个文档，全新数据目录
ov.conf 配置同正式环境，仅 path 不同
"""
import os, asyncio
os.environ['OPENVIKING_CONFIG_FILE'] = '/tmp/ov_clean_test/ov.conf'

import openviking as ov

def jaccard(a, b):
    sa, sb = set(a), set(b)
    return len(sa & sb) / len(sa | sb) if (sa | sb) else 1.0

async def test(data_path, label):
    client = ov.AsyncOpenViking(path=data_path)
    await client.initialize()
    
    queries = ["Docker 怎么部署", "Python 虚拟环境", "Nginx 反向代理"]
    
    for q in queries:
        # find
        find_results = []
        for _ in range(6):
            r = await client.find(q, limit=5)
            uris = [x.uri for x in (r.resources or [])]
            find_results.append(uris)
        
        jaccards = [jaccard(find_results[0], find_results[i]) for i in range(1, 6)]
        print(f"[{label}] find '{q}': {[f'{j:.2f}' for j in jaccards]}")
        
        # search (no session)
        search_results = []
        for _ in range(6):
            r = await client.search(q, limit=5)
            uris = [x.uri for x in (r.resources or [])]
            search_results.append(uris)
        
        jaccards = [jaccard(search_results[0], search_results[i]) for i in range(1, 6)]
        print(f"[{label}] search '{q}': {[f'{j:.2f}' for j in jaccards]}")
    
    await client.close()

asyncio.run(test('/tmp/ov_clean_test/data', 'clean'))

[ponsde]

Update: Root cause found — our configuration issue

After further investigation, we identified the root cause on our side:

Our production environment vector index was created with Dimension: 3072, but our current embedding model (BAAI/bge-m3) outputs 1024-dim vectors, and our ov.conf also specifies dimension: 1024.

The dimension was likely different when the collection was first created. After changing the config, OpenViking (correctly) skips re-creating an existing collection, so the old 3072-dim index remained while new 1024-dim vectors were being inserted — causing the non-deterministic retrieval behavior.

Evidence

Environment	Collection Dim	Embedding Dim	Deterministic?
Clean (60 docs, correct dim)	1024	1024	✅ Jaccard 1.00
Production (56+ resources, mismatched)	3072	1024	❌ Jaccard 0.00–0.67
Production via HTTP serve mode	3072	1024	❌ Same as embedded

Solution

Delete data/vectordb/ and let OpenViking rebuild the index with the correct dimension.

Suggestion

It would be helpful if OpenViking logged a warning (or raised an error) at startup when the existing collection dimension does not match config.embedding.dimension. This would catch configuration drift early. Something like:

# In init_context_collection or service startup
existing_dim = collection_meta["Dimension"]
config_dim = config.embedding.dimension
if existing_dim != config_dim:
    logger.warning(f"Collection dimension ({existing_dim}) != config dimension ({config_dim}). "
                   "Consider rebuilding the index.")

Sorry for the noise — closing this as a configuration issue on our end. Thanks for your time.