[BUG] elastic_transport.ConnectionTimeout: Connection timed out

[iziang]

Describe the bug

2025-07-10 12:40:23,629 - INFO - 10.183.236.197:12979 - "POST /api/v1/available_models HTTP/1.1" 200
2025-07-10 12:40:23,666 - INFO - 10.183.236.197:12652 - "GET /api/v1/bots/botd50d075defa04e63/flow HTTP/1.1" 200
2025-07-10 12:40:23,667 - INFO - 10.183.236.197:12078 - "GET /api/v1/collections HTTP/1.1" 200
2025-07-10 12:40:24,832 - INFO - 10.183.236.197:12652 - "GET /api/v1/bots/botd50d075defa04e63/chats/chate2775dd27e1a9efd HTTP/1.1" 200
2025-07-10 12:40:24,863 - INFO - ('10.183.236.197', 11967) - "WebSocket /api/v1/bots/botd50d075defa04e63/chats/chate2775dd27e1a9efd/connect" [accepted]
2025-07-10 12:40:24,863 - INFO - connection open
2025-07-10 12:40:37,334 - INFO - Starting flow execution 84763f37 for flow rag_flow
2025-07-10 12:40:37,334 - INFO - Flow event: flow_start for None node None
2025-07-10 12:40:37,334 - INFO - Executing node group: {'mcxco9pp'}
2025-07-10 12:40:37,335 - INFO - Flow event: node_start for start node mcxco9pp
2025-07-10 12:40:37,335 - INFO - Flow event: node_end for start node mcxco9pp
2025-07-10 12:40:37,335 - INFO - Executing node group: {'mcxco9ps', 'mcxco9pr', 'mcxco9pq'}
2025-07-10 12:40:37,336 - INFO - Flow event: node_start for graph_search node mcxco9ps
2025-07-10 12:40:37,337 - INFO - Flow event: node_start for fulltext_search node mcxco9pr
2025-07-10 12:40:37,338 - INFO - Flow event: node_start for vector_search node mcxco9pq
2025-07-10 12:40:37,340 - INFO - get_collection_embedding_model_sync alibabacloud text-embedding-v4
2025-07-10 12:40:37,343 - INFO - get_collection_embedding_model https://dashscope.aliyuncs.com/compatible-mode/v1
2025-07-10 12:40:37,347 - INFO - get_collection_embedding_model_sync alibabacloud text-embedding-v4
2025-07-10 12:40:37,350 - INFO - get_collection_embedding_model https://dashscope.aliyuncs.com/compatible-mode/v1
2025-07-10 12:50:37,534 - INFO - Retrying request to /embeddings in 0.387707 seconds
2025-07-10 12:50:38,167 - INFO - Flow event: node_end for vector_search node mcxco9pq
2025-07-10 12:50:38,168 - INFO - HEAD http://es-cluster-mdit-http:9200/col460666f28862d7c1 [status:N/A duration:600.821s]
2025-07-10 12:50:38,168 - WARNING - Node <AiohttpHttpNode(http://es-cluster-mdit-http:9200)> has failed for 1 times in a row, putting on 1 second timeout
2025-07-10 12:50:38,170 - INFO - Flow event: node_error for fulltext_search node mcxco9pr
2025-07-10 12:50:38,171 - INFO - connection closed
2025-07-10 12:50:38,174 - INFO - Flow event: flow_error for None node None
2025-07-10 12:50:38,174 - ERROR - Error processing WebSocket message: Connection timed out
Traceback (most recent call last):
  File "/app/aperag/service/chat_service.py", line 401, in handle_websocket_chat
    _, system_outputs = await engine.execute_flow(flow, initial_data)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/aperag/flow/engine.py", line 167, in execute_flow
    raise e
  File "/app/aperag/flow/engine.py", line 140, in execute_flow
    await self._execute_node_group(flow, node_group)
  File "/app/aperag/flow/engine.py", line 256, in _execute_node_group
    await asyncio.gather(*tasks)
  File "/app/aperag/flow/engine.py", line 433, in _execute_node
    raise e
  File "/app/aperag/flow/engine.py", line 406, in _execute_node
    outputs = await runner.run(user_input, sys_input)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/aperag/flow/runners/fulltext_search.py", line 99, in run
    docs = await self.service.execute_fulltext_search(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/aperag/flow/runners/fulltext_search.py", line 72, in execute_fulltext_search
    keywords = await extractor.extract(query)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/aperag/index/fulltext_index.py", line 265, in extract
    resp = await self.client.indices.exists(index=self.index_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/elasticsearch/_async/client/indices.py", line 1523, in exists
    return await self.perform_request(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/elasticsearch/_async/client/_base.py", line 422, in perform_request
    return await self._client.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/elasticsearch/_async/client/_base.py", line 271, in perform_request
    response = await self._perform_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/elasticsearch/_async/client/_base.py", line 315, in _perform_request
    meta, resp_body = await self.transport.perform_request(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/elastic_transport/_async_transport.py", line 264, in perform_request
    resp = await node.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/elastic_transport/_node/_http_aiohttp.py", line 228, in perform_request
    raise err from None
elastic_transport.ConnectionTimeout: Connection timed out
2025-07-10 12:50:38,175 - INFO - WebSocket disconnected for bot botd50d075defa04e63, chat chate2775dd27e1a9efd

api-5b947cb4c4-fr4n2.log

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[iziang]

api-5b947cb4c4-fr4n2.log