Conversation
…neuro-san into langchain_tool_callback
| # Replace langchain tool's name with agent name | ||
| instance.name = agent_name | ||
| # Add "langchain_tool" tags so journal callback can idenitify it | ||
| instance.tags = ["langchain_tool"] |
There was a problem hiding this comment.
Override langchain tool's name with agent_anme and add langchain_tool to tags as identifier.
| toolkit: List[BaseTool] = instance.get_tools() | ||
| for tool in toolkit: | ||
| # Prefix the name of the agent to each tool | ||
| tool.name = f"{agent_name}_{tool.name}" |
There was a problem hiding this comment.
For agent with multiple tools (toolkit), prefix the tool names with the agent name.
There was a problem hiding this comment.
Worth noting for above argument that you are not really checking for a None agent_name value,
so that kinda bolsters my argument as to its necessity of not being None.
| "session_id": run.get_id() | ||
| } | ||
| }, | ||
| "callbacks": callbacks |
There was a problem hiding this comment.
Use callbacks as invoke_config in wait_on_run() to have access to on_tool_start() and on_tool_run()
| passed to the tool. | ||
| """ | ||
|
|
||
| if "langchain_tool" in tags: |
There was a problem hiding this comment.
Check if the tool is langchain's tool with tags.
| # Create a journal entry for this invocation and log the combined inputs | ||
| self.langchain_tool_journal = OriginatingJournal(self.base_journal, origin) | ||
| message: BaseMessage = AgentMessage(content=f"Received arguments {combined_args}") | ||
| await self.langchain_tool_journal.write_message(message) |
There was a problem hiding this comment.
Creating journal for langchain tool and write message.
|
|
||
| # Also log the tool output to the LangChain tool-specific journal | ||
| message: BaseMessage = AgentMessage(content=f"Got result: {output}") | ||
| await self.langchain_tool_journal.write_message(message) |
There was a problem hiding this comment.
Write message in the langchain journal with tool output. This can be used as an indicator that the tool is finished.
There was a problem hiding this comment.
@Noravee Since the UI already checks for "Got result:" for coded tool end events on the text field, this should work fine.
We also may want to consider adding a way to signify an end event for coded tools or langchain tools on the structure key instead, so that we don't have to do string parsing on the text field.
cc: @d1donlydfink
There was a problem hiding this comment.
Can do that in a separate PR @swensel for full consistency
There was a problem hiding this comment.
Thanks @d1donlydfink , that would be great.
cc: @Noravee
neuro_san/internals/run_context/langchain/core/langchain_run_context.py
Outdated
Show resolved
Hide resolved
neuro_san/internals/run_context/langchain/journaling/journaling_callback_handler.py
Show resolved
Hide resolved
d1donlydfink
left a comment
d1donlydfink
left a comment
There was a problem hiding this comment.
Please answer the questions sprinkled throughout the PR.
dsargent
left a comment
dsargent
left a comment
There was a problem hiding this comment.
Don't really know enough to be useful here. I will defer to @d1donlydfink's questions.
|
|
||
| def create_agent_with_fallbacks(self, prompt_template: ChatPromptTemplate, | ||
| callbacks: List[BaseCallbackHandler]) -> Agent: | ||
| def create_agent_with_fallbacks(self, prompt_template: ChatPromptTemplate) -> Agent: |
There was a problem hiding this comment.
Remove callbacks from the argument list of create_agent_with_fallbacks. This eliminates the need for
a callbacks parameter in both default_llm_factory and standard_langchain_llm_factory.
However, that cleanup is outside the scope of this PR and will be addressed separately.
d1donlydfink
left a comment
d1donlydfink
left a comment
There was a problem hiding this comment.
- Please add nice explanation of need for 2 Journals you added in the PR to the code itself
- Make agent_name a required argument of create_tool_from_toolbox(()
- Minor modification in toolbox_factory.create_tool_from_toolbox()
…neuro-san into langchain_tool_callback
4 participants
Summary
This PR introduces a proof of concept for logging (journaling) output from a LangChain tool using a callback-based approach.
Problem
For coded tools, journaling is handled within the activation class. However, LangChain tools are subclasses of LangChain’s
BaseTool, and do not invoke our custom implementation (LangchainOpenaiFunctionTool) that creates an activation class and handles logging. As a result:No journal is created for LangChain tools.
There is no clear indication when such a tool finishes execution.
Solution
Leverage
JournalingCallbackHandler, which defineson_tool_start()andon_tool_end()—methods that are triggered at the start and end of tool execution.Previously, callbacks were passed as arguments when instantiating the LLM, which limited their use to LLM-related events (e.g.,
on_llm_end()). By instead using the invoke config, callbacks can now respond to a broader set of events, including tool-level events likeon_tool_start()andon_tool_end().Changes
ToolboxFactory.create_tool_from_toolbox()Added
agent_nameargument to override or prefix the LangChain tool name. This supports naming for single-tool agents and for agents with toolkits.Added
"langchain_tool"tag to identify tools that should trigger journaling.JournalingCallbackHandler__init__:Renamed
journaltocalling_agent_journal.Added new parameters:
base_journal,parent_origin, andorigination.on_tool_start:Checks for the
"langchain_tool"tag.Creates a
langchain_tool_journaland logs the tool’s input.Prevents double journaling for coded or branch tools.
on_tool_end:calling_agent_journaland thelangchain_tool_journal.LangChainRunContextcreate_resources()towait_from_run()to ensure proper callback usage for tools as well as LLMs.'TestToolboxFactory
: Added attributenameandtags` to mock tools.Tests
Tested by running the following agents and inspecting their corresponding thinking files to verify tool-level logging:
bing_searchmusic_nerd_promusic_nerd_pro_multi_agentConfirmed that:
Tool input is logged at the start (
on_tool_start)Tool output is logged at the end (
on_tool_end)Journals are correctly scoped and do not duplicate entries for coded or branch tools
Future Development
This callback-based approach could be extended to handle journaling for coded tools, branch tools, and other tool types—providing a unified and consistent logging mechanism across tool types.