UN-3310 Faster server startup. by andreidenissov-cog · Pull Request #290 · cognizant-ai-lab/neuro-san

andreidenissov-cog · 2025-07-09T00:58:51Z

Main idea of this PR is switch to "lazy" service instances instantiation.
Right now, we always create service instances - 2 of them in fact for each agent defined in server manifest;
one for gRPC, one for http, even if say we don't use gRPC for this particular server invocation,
and only use a single agent for chat.
Each service creation involves reading at least 2 files, which adds up.
Changing to service instantiation on demand makes server start up much faster.
There is an initial lag of course for loading all Python classes, but nothing to be done for that I guess.

andreidenissov-cog · 2025-07-09T01:07:50Z

neuro_san/service/http/server/http_sidecar.py

+                agent_network_provider,
+                agent_server_logging)
+        self.allowed_agents[agent_name] = agent_service_provider



Instead of AsyncAgentService instances, "allowed_agents" map now contains AsyncAgentServiceProvider instances for lazy instantiation of AsyncAgentService.

andreidenissov-cog · 2025-07-09T01:16:52Z

neuro_san/service/generic/agent_service_provider.py

+        self.server_logging: AgentServerLogging = server_logging
+        self.agent_network_provider: AgentNetworkProvider = agent_network_provider
+        self.agent_name: str = agent_name
+        self.lock: Lock = Lock()


Save constructor parameters for later - when we will need to instantiate AgentService object on the first use.

andreidenissov-cog · 2025-07-09T01:17:58Z

neuro_san/service/generic/agent_service_provider.py

+                        self.security_cfg,
+                        self.agent_name,
+                        self.agent_network_provider,
+                        self.server_logging)


Standard instance creation pattern for multi-threaded environment.

andreidenissov-cog · 2025-07-09T01:20:39Z

neuro_san/service/generic/async_agent_service_provider.py

+from typing import Dict
+from threading import Lock
+import copy
+


This class more or less duplicates AgentServiceProvider logic, but for AsyncAgentService instances.
Maybe we could cook up something more generic and elegant Pythonic style for both classes,
but not in this PR.

andreidenissov-cog · 2025-07-09T01:23:03Z

neuro_san/service/grpc/grpc_agent_service.py

+            # Service is not yet instantiated - it has no requests
+            return 0
+        service: AgentService = self.service_provider.get_service()
+        return service.get_request_count()


That's perf optimization: if service instance has not been instantiated, then its request count is obviously 0.
Otherwise with current gRPC server logic, we'll always instantiate all agent services,
even when not using gRPC service at all.

andreidenissov-cog · 2025-07-09T01:24:10Z

neuro_san/service/grpc/grpc_agent_service.py

+                agent_name,
+                agent_network_provider,
+                server_logging)



Create "lazy" service provider - instantiate actual AgentService only when needed.

andreidenissov-cog · 2025-07-09T01:25:26Z

neuro_san/service/grpc/grpc_agent_service.py

+        service: AgentService = self.service_provider.get_service()
+        response_dict: Dict[str, Any] =\
+            service.function(request_dict, request_metadata, context)



Here and below - get AgentService from AgentServiceProvider for actual request processing.

andreidenissov-cog · 2025-07-09T01:27:18Z

neuro_san/service/http/handlers/base_request_handler.py

+            self.do_finish()
+            return None
+        return service_provider.get_service()
+


Factor out common logic for getting AsyncAgentService for request processing.
Will be used in all (3) neuro-san API requests.

andreidenissov-cog · 2025-07-09T01:27:35Z

neuro_san/service/http/handlers/base_request_handler.py


-        self.logger.info(self.get_metadata(), f"[REQUEST RECEIVED] {self.request.method} {self.request.uri}")
+        self.logger.debug(self.get_metadata(), f"[REQUEST RECEIVED] {self.request.method} {self.request.uri}")



Less chatty.

andreidenissov-cog · 2025-07-09T01:28:32Z

neuro_san/service/http/handlers/connectivity_handler.py

-
-        service: AsyncAgentService = self.agent_policy.allow(agent_name)
+        service: AsyncAgentService = await self.get_service(agent_name, metadata)
        if service is None:


Here and below - use common function from base_request_handler module.

Nice that this consolidates all that other status/error/do_finish business.

andreidenissov-cog · 2025-07-09T01:29:09Z

neuro_san/service/http/handlers/function_handler.py

-            self.set_status(404)
-            self.logger.error({}, "error: Invalid request path %s", self.request.path)
-            self.do_finish()
            return


Use common function from base_request_handler module.

andreidenissov-cog · 2025-07-09T01:29:18Z

neuro_san/service/http/handlers/streaming_chat_handler.py

-            self.logger.error({}, "error: Invalid request path %s", self.request.path)
-            self.do_finish()
            return



Use common function from base_request_handler module.

andreidenissov-cog · 2025-07-09T01:30:32Z

neuro_san/service/http/server/http_sidecar.py

+        app.listen(self.http_port)
+        self.logger.info({}, "HTTP server is running on port %d", self.http_port)
+        self.logger.info({}, "HTTP server is shutting down after %d requests", self.requests_limit)



Start actually listening after all preparations are done.
Log server start at this point.

…artup01

d1donlydfink

Please consider the nits I mention below for a future PR.

It's conceivable we could get some complaints about initial start up time of individual service. (Thinking: 1C) Cross that bridge when we get there.

d1donlydfink · 2025-07-09T15:48:34Z

neuro_san/service/generic/agent_service_provider.py

+                        self.server_logging)
+        return self.service_instance
+
+    def service_created(self) -> bool:


Nit: Since this is a boolean, perhaps a better name would be is_service_created()
Here and below in the async version.

d1donlydfink · 2025-07-09T15:54:10Z

neuro_san/service/generic/async_agent_service_provider.py

+                        self.security_cfg,
+                        self.agent_name,
+                        self.agent_network_provider,
+                        self.server_logging)


For next PR consideration: Since all of this is the same except for the type of service_instance, maybe there is a path to abstraction here.
However well functioning all the async stuff is, someday Python will clean up this copy-paste cancer on the language.

d1donlydfink · 2025-07-09T15:57:13Z

neuro_san/service/http/handlers/connectivity_handler.py

-
-        service: AsyncAgentService = self.agent_policy.allow(agent_name)
+        service: AsyncAgentService = await self.get_service(agent_name, metadata)
        if service is None:


Nice that this consolidates all that other status/error/do_finish business.

…artup01

andreidenissov-cog added 3 commits July 8, 2025 10:26

WIP.

41aa184

Added lazy agent services instantiations.

33cd8eb

Merge with main.

9f59b09

andreidenissov-cog self-assigned this Jul 9, 2025

andreidenissov-cog commented Jul 9, 2025

View reviewed changes

Move http server start point.

c948b80

andreidenissov-cog commented Jul 9, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into ASD-UN-3310-faster-st…

58ea203

…artup01

andreidenissov-cog requested a review from d1donlydfink July 9, 2025 01:52

d1donlydfink approved these changes Jul 9, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into ASD-UN-3310-faster-st…

7968711

…artup01

andreidenissov-cog merged commit 172ca02 into main Jul 9, 2025
4 checks passed

andreidenissov-cog deleted the ASD-UN-3310-faster-startup01 branch July 9, 2025 17:26


		self.logger.info(self.get_metadata(), f"[REQUEST RECEIVED] {self.request.method} {self.request.uri}")
		self.logger.debug(self.get_metadata(), f"[REQUEST RECEIVED] {self.request.method} {self.request.uri}")

Comments

Conversation

andreidenissov-cog commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d1donlydfink left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d1donlydfink Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andreidenissov-cog commented Jul 9, 2025 •

edited

Loading

d1donlydfink left a comment •

edited

Loading

d1donlydfink Jul 9, 2025 •

edited

Loading