why use websocket #93
Description
Hello, thanks for this amazing repo.
I am using the NALLM backend with a custom Streamlit app.
I have been facing issues with the websocket (mainly timeout issues and disconnections) implemented in api/src/main.py under @app.websocket("/text2text") endpoint.
Apologies if my question is basic but I am just trying to understand.
Is there a good reason for using websockets instead of HTTP endpoint?
My understanding is that websockets are mainly used for chat applications between users (where users don't know when they will be getting a new message). If the chat application is between a human and AI where a human always initiate the conversation by asking the question and the AI/ LLM responds with an answer, why do you use a websocket? I assume using HTTP for the "/text2text" endpoint could work as well?