UN-3276 support for user defined llm in hocon file with class key by Noravee · Pull Request #270 · cognizant-ai-lab/neuro-san

Noravee · 2025-06-26T17:33:58Z

Changes

Add UserSpecifiedLangchainLlmFactory

Tests

Ran agent network with gpt-4.1-mini, which is not in the default llm info.

…n_file_with_class_key

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

neuro_san/internals/run_context/langchain/core/langchain_run_context.py

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

neuro_san/internals/run_context/langchain/llms/user_specified_langchain_llm_factory.py

…n_file_with_class_key

d1donlydfink

A lot to address in here.

Same advice as always:

Slow down. Fast is not the answer in public API development.
Think through how your change effects existing installations
Be your own first reviewer. The best thing you could do is ask yourself "What is Dan going to say about this?" before handing it off to me.

neuro_san/internals/run_context/langchain/llms/user_specified_langchain_llm_factory.py

…n_file_with_class_key

- Resolve llm class in default_llm_factory when all factories fail - Some test hocons

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

Noravee · 2025-06-26T18:30:10Z

neuro_san/internals/run_context/langchain/llms/user_specified_langchain_llm_factory.py

+
+class UserSpecifiedLangChainLlmFactory(LangChainLlmFactory):
+    """
+    A factory for constructing LLMs based on user-specified configurations provided under the "llm_config"


Add UserSpecifiedLangChainLlmFactory that uses arguments from given by the user.

Noravee · 2025-06-26T18:31:52Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

+        if self.llm_class:
+            # If config has "class", it is a user-specified llm so return config as is,
+            # and replace "StandardLangChainLlmFactory" with "UserSpecifiedLangChainLlmFactory".
+            self.llm_factories[0] = UserSpecifiedLangChainLlmFactory()


If llm_config has class, use UserSpecifiedLangChainLlmFactory.

Noravee · 2025-06-26T18:33:44Z

neuro_san/internals/run_context/langchain/core/langchain_run_context.py

            try:
                return_dict: Dict[str, Any] = await agent_executor.ainvoke(inputs, invoke_config)
-            except (APIError, BadRequestError, AuthenticationError, ChatGoogleGenerativeAIError) as api_error:
+            except (openai.APIError, anthropic.APIError, ChatGoogleGenerativeAIError) as api_error:


Use APIError for anthropic as well.

Noravee · 2025-06-26T18:34:24Z

neuro_san/internals/run_context/langchain/core/langchain_run_context.py

                retries = retries - 1
                exception = key_error
                backtrace = traceback.format_exc()
+            except TypeError as type_error:


Add TypeError for non-existing arguments user may pass.

neuro_san/internals/run_context/langchain/util/api_key_error_check.py

Noravee · 2025-07-02T09:04:05Z

neuro_san/internals/run_context/langchain/toolbox/toolbox_factory.py

-        class_args_set: Set[str] = set(signature(method_class).parameters.keys())
+        pydantic_args: Set[str] = set()
+        # Check for if it is a class that extends pydantic BaseModel
+        if isclass(method_class) and issubclass(method_class, BaseModel):


Check if the class extends BaseModel. This is necessary since pydantic allows arguments to have alias.

Noravee · 2025-07-02T09:04:49Z

neuro_san/registries/manifest.hocon

-
+    "test_new_model_default_class.hocon": true,
+    "test_new_class.hocon": true,
+    "test_new_model_extended_class.hocon": true


This is for testing only. It will not be in the final PR.

Noravee · 2025-07-02T09:05:06Z

neuro_san/registries/test_new_class.hocon

+    "llm_config": {
+        "class": "langchain_groq.chat_models.ChatGroq",
+        "model_name": "deepseek-r1-distill-llama-70b",
+        "temperature": 0.5


This is for testing only. It will not be in the final PR.

Noravee · 2025-07-02T09:05:14Z

neuro_san/registries/test_new_model_default_class.hocon

+    "llm_config": {
+        "class": "openai",
+        "model_name": "gpt-4.1-mini",
+        "temperature": 0.5


This is for testing only. It will not be in the final PR.

Noravee · 2025-07-02T09:05:22Z

neuro_san/registries/test_new_model_extended_class.hocon

+    "llm_config": {
+        "class": "groq",
+        "model_name": "deepseek-r1-distill-llama-70b",
+        "temperature": 0.5,


This is for testing only. It will not be in the final PR.

Noravee · 2025-07-02T09:20:18Z

There are three types of LLM extensions using the class key:

Using a new model with an existing class
The class is already defined in the standard factory.
Example: test_new_model_default_class.hocon
→ Uses StandardLangChainLlmFactory to instantiate the LLM.
Using a new model with a new class
The class is not pre-defined and is resolved dynamically.
Example: test_new_class.hocon
→ Instantiated via DefaultLlmFactory.
Using a new model with a class added to classes.factories
The factory class explicitly extends LangChainLlmFactory.
Example: test_new_model_extended_class.hocon
→ Instantiation is handled by the custom factory.

Note: Some checks fail due to missing dependencies (e.g., langchain-groq is not included in requirements.txt).
All temporary test files will be removed in the final PR.

neuro_san/internals/run_context/langchain/core/langchain_run_context.py

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

neuro_san/internals/run_context/langchain/llms/standard_langchain_llm_factory.py

neuro_san/internals/run_context/langchain/toolbox/toolbox_factory.py

neuro_san/internals/run_context/langchain/util/api_key_error_check.py

…n_file_with_class_key

- Refactor "check_invalid_arguments" into a util class - Revert the error message detection in api key error checker - Modified test_toolbox_factory - Add test for ArgumentValidator

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

Noravee · 2025-07-03T19:00:01Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py


+        # Try resolving via 'class' in config if factories failed
+        class_path: str = config.get("class")
+        if llm is None and found_exception is not None and class_path:


The conditions to instantiate using class in llm_config in agent network hocon are

Failed instantiation from the llm factories (thus llm is None)

Consequently, found_exception is not None

There is class in the config.

Noravee · 2025-07-03T19:01:54Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py


+    def create_base_chat_model_from_user_class(self, class_path: str, config: Dict[str, Any]) -> BaseLanguageModel:
+        """
+        Create a BaseLanguageModel from the user-specified langchain model class.


Refactor the logic of instantiating using "class" and user-defined args to create_base_chat_model_from_user_class() method.

Noravee · 2025-07-03T19:03:53Z

neuro_san/internals/run_context/langchain/util/argument_validator.py

+
+class ArgumentValidator:
+    """
+    A utility class for inspecting method and class arguments, particularly useful for


Refactor checking for invalid args to a utility class ArgumentValidator.

This is very nicely abstracted.

Noravee · 2025-07-03T19:08:12Z

tests/neuro_san/internals/run_context/langchain/util/test_argument_validator.py

+# ------------------------- Tests -------------------------------
+
+class TestArgumentValidator:
+    """Tests for the ArgumentValidator class and its static validation utilities."""


Add test for ArgumentValidator.

Noravee · 2025-07-03T19:08:58Z

tests/neuro_san/internals/run_context/langchain/toolbox/test_toolbox_factory.py

 from neuro_san.internals.run_context.langchain.toolbox.toolbox_factory import ToolboxFactory

+RESOLVER_PATH = "leaf_common.config.resolver.Resolver.resolve_class_in_module"
+VALIDATIOR_PATH = (


Modify test for toolbox factory to reflect the refactoring.

Noravee · 2025-07-03T19:11:28Z

Refactor the instantiation of LLM from class in DefaultLlmFactory
Refactor the check for invalid args into a util class ArgumentValidator
Create test for ArgumentValidator

Noravee · 2025-07-04T00:46:06Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

+            # To prevent this, we first fetch the default arguments for the given class from llm_info,
+            # then merge them with the user-provided config. This ensures all expected arguments are present,
+            # and the user’s config values take precedence over the defaults.
+            config_from_class_in_llm_info: Dict[str, Any] = self.get_chat_class_args(class_from_llm_config)


Combine user-specified config with the one in class in llm_info.

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

d1donlydfink · 2025-07-07T20:52:09Z

neuro_san/internals/run_context/langchain/util/argument_validator.py

+    @staticmethod
+    def get_base_model_args(base_model_class: Type[BaseModel]) -> Set[str]:
+        """
+        Extract all field names and aliases from a Pydantic BaseModel class.


It's good that you call out the BaseModel class as coming from Pydantic.
I always get confused thinking it's an LLM, but it's not, it's lower level than that.

d1donlydfink

This is looking really good... good enough at least to take out of Draft mode.
A couple more little things and we are there I think.

…n_file_with_class_key

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

…n_file_with_class_key

Noravee

Changes

Refactor ResolverUtil.create_instance() to ResolverUtil.create_class()
Use ResolverUtil in DefaultLlmFactory.

Noravee · 2025-07-08T18:26:40Z

neuro_san/internals/utils/resolver_util.py

+
+    @staticmethod
+    def create_class(class_name: str, class_name_source: str, type_of_class: Type) -> Type:
+        """


I refactor create_instance() to create_class() method because I need we need constructor with args for user-defined llm.

Makes sense

Noravee · 2025-07-08T18:28:23Z

neuro_san/internals/utils/resolver_util.py

-        if not isinstance(instance, type_of_class):
-            raise ValueError(f"Class '{class_name}' from {class_name_source} "
-                             "must be of type {type_of_class.__name__}")
+        if not issubclass(class_reference, type_of_class):


Move check correct type to create_class().

Noravee · 2025-07-08T18:33:00Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

+            raise ValueError("'class' in llm_config must be a string")
+
+        # Resolve the 'class'
+        llm_class: Type[BaseLanguageModel] = ResolverUtil.create_class(


Use ResolveUtil.create_class for the llm since we need to instantiate it with args.

Noravee · 2025-07-08T18:33:37Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

+        # Check for invalid args and throw error if found
+        ArgumentValidator.check_invalid_args(llm_class, user_config)
+
+        return llm_class(**user_config)


LLM instantiation with args.

Noravee · 2025-07-08T18:39:03Z

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py

-            raise ValueError(f"Class {llm_factory_class_name} in {llm_info_file} "
-                             "must be of type LangChainLlmFactory")
+        # Resolve and instantiate the factory class
+        llm_factory = ResolverUtil.create_instance(


Use ResolverUtil.create_instance for resolve and instantiation of the factory class.

d1donlydfink · 2025-07-08T21:38:54Z

tests/neuro_san/internals/run_context/langchain/toolbox/test_toolbox_factory.py


-        with patch("leaf_common.config.resolver.Resolver.resolve_class_in_module") as mock_resolver, \
-                patch.object(factory, "_check_invalid_args") as mock_check_invalid:
+        with patch(RESOLVER_PATH) as mock_resolver, patch(VALIDATIOR_PATH) as mock_check_invalid:


Ooo. fancy!

d1donlydfink

This looks great Nor. Overall things actually look a lot simpler as this has evolved over time. Nice work!

…n_file_with_class_key

Noravee added 3 commits June 24, 2025 14:38

add user_specified_langchain_llm_factory

de0a29a

fix indent and pylint

ecdee78

Add more comments

d23186e

Noravee marked this pull request as draft June 26, 2025 17:34

Noravee and others added 3 commits June 26, 2025 13:35

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

928ea36

…n_file_with_class_key

remove ollama response error

58b3b8b

Merge branch 'UN-3276_Support_for_user-defined_llm_in_hocon_file_with…

5906c6f

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

Noravee requested review from d1donlydfink and ofrancon June 26, 2025 18:36

Noravee marked this pull request as ready for review June 26, 2025 20:26

d1donlydfink reviewed Jun 26, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/core/langchain_run_context.py Outdated Show resolved Hide resolved

d1donlydfink reviewed Jun 26, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/core/langchain_run_context.py Outdated Show resolved Hide resolved

d1donlydfink reviewed Jun 26, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py Outdated Show resolved Hide resolved

d1donlydfink reviewed Jun 26, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/llms/user_specified_langchain_llm_factory.py Outdated Show resolved Hide resolved

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

8a8128f

…n_file_with_class_key

d1donlydfink requested changes Jun 26, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/llms/user_specified_langchain_llm_factory.py Outdated Show resolved Hide resolved

d1donlydfink marked this pull request as draft June 27, 2025 18:44

Noravee and others added 4 commits June 30, 2025 14:45

minor changes

23891db

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

2a8131f

…n_file_with_class_key

- Remove user_specified_langchain_llm_factory

8e4d881

- Resolve llm class in default_llm_factory when all factories fail - Some test hocons

Merge branch 'UN-3276_Support_for_user-defined_llm_in_hocon_file_with…

962abf7

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

Noravee requested a review from d1donlydfink July 2, 2025 08:31

Noravee added 4 commits July 2, 2025 01:35

use alias for api error in langchain run context

17974b5

change comments

9d90aaa

remove space

7647dc9

Add comments

741e50e

Noravee commented Jul 2, 2025

View reviewed changes

d1donlydfink reviewed Jul 2, 2025

View reviewed changes

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

f8ce89d

…n_file_with_class_key

Noravee added 3 commits July 3, 2025 11:50

- Refactor logic on creating llm based on "class" into another method

69889ad

- Refactor "check_invalid_arguments" into a util class - Revert the error message detection in api key error checker - Modified test_toolbox_factory - Add test for ArgumentValidator

Merge branch 'UN-3276_Support_for_user-defined_llm_in_hocon_file_with…

9363936

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

Add type hints

a796cf8

Noravee commented Jul 3, 2025

View reviewed changes

Noravee added 2 commits July 3, 2025 15:20

Add callbacks

bbc3fbc

combine user config with the one in class in llm_info

50bd7f0

Noravee commented Jul 4, 2025

View reviewed changes

d1donlydfink reviewed Jul 7, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py Outdated Show resolved Hide resolved

d1donlydfink reviewed Jul 7, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py Outdated Show resolved Hide resolved

d1donlydfink reviewed Jul 7, 2025

View reviewed changes

neuro_san/internals/run_context/langchain/llms/default_llm_factory.py Outdated Show resolved Hide resolved

d1donlydfink reviewed Jul 7, 2025

View reviewed changes

Noravee and others added 6 commits July 7, 2025 18:36

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

1aac685

…n_file_with_class_key

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

69f38f2

…n_file_with_class_key

remove optional

e6dc921

Merge branch 'UN-3276_Support_for_user-defined_llm_in_hocon_file_with…

0b98a39

…_class_key' of https://github.com/leaf-ai/neuro-san into UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key

refactor default llm factory with resolver util

cca1d03

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

b021671

…n_file_with_class_key

Noravee marked this pull request as ready for review July 8, 2025 18:22

Noravee commented Jul 8, 2025

View reviewed changes

Noravee requested a review from d1donlydfink July 8, 2025 18:34

Noravee commented Jul 8, 2025

View reviewed changes

d1donlydfink reviewed Jul 8, 2025

View reviewed changes

d1donlydfink approved these changes Jul 8, 2025

View reviewed changes

Merge branch 'main' into UN-3276_Support_for_user-defined_llm_in_hoco…

35c0362

…n_file_with_class_key

Noravee merged commit c26d49b into main Jul 9, 2025
4 checks passed

Noravee deleted the UN-3276_Support_for_user-defined_llm_in_hocon_file_with_class_key branch July 9, 2025 01:39

Comments

Conversation

Noravee commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

d1donlydfink left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Noravee commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Noravee commented Jul 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d1donlydfink left a comment

Choose a reason for hiding this comment

Uh oh!

Noravee left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Noravee commented Jun 26, 2025 •

edited

Loading

d1donlydfink left a comment •

edited

Loading

Noravee commented Jul 2, 2025 •

edited

Loading

Noravee left a comment •

edited

Loading