Agent that evaluates output from the pre-defined set of agents, recommeding addition or replacements of agents

[swanidhi]

Thank you for this project. I am really excited about this!

I'm currently envisioning a solution that has the potential to bring about a revolutionary shift in how teams are structured and optimized across various functions and industries. (revolutionary, at least in my mind)

Imagine a scenario where we embark on a comprehensive research initiative to determine the exact skill set required for a new project. This would involve a meticulous analysis of outcomes derived from proxy tasks, followed by a thorough comparison and contrast of the results. Subsequently, we would fine-tune these insights to maximize project outcomes. I believe that this approach has the potential to significantly enhance resource management within Project Management and streamline the process of identifying the most suitable resources for a given project. The implications for projects developed by Autogen are nothing short of remarkable.

Can we consider the possibility of implementing an agent that measures key performance indicators (KPIs) or conducts specific tests to evaluate individual agent performance and team dynamics? This agent could then provide invaluable recommendations to fine-tune both individual agents and the overall team, thereby elevating the project's success potential to new heights. The possibilities are truly exhilarating.

[KeepALifeUS]

Great idea! This is exactly the kind of meta-level orchestration that production multi-agent systems need.

I've been working on something similar and found that indirect coordination (stigmergy pattern) helps a lot with agent evaluation:

What worked for me:

• Each agent writes its outputs to a shared state with metadata (cost, latency, token count)
• A "Coordinator" agent periodically reviews this state and identifies underperforming patterns
• Instead of replacing agents, we adjust their context or scope based on performance

Key metrics I track:

• Token efficiency (output value / input tokens)
• Task completion rate per agent
• Downstream impact (did this agent's output trigger expensive follow-up work?)

Challenges:

• Attribution is hard when one agent's poor output causes another to retry multiple times
• "Performance" depends heavily on task type - a verbose agent might be bad for summaries but great for detailed analysis

For the recommendation system, I'd suggest starting with rule-based thresholds before adding an LLM-based evaluator - it's more predictable and easier to debug.

I documented some of these patterns here: https://github.com/KeepALifeUS/autonomous-agents

Would be interested to collaborate if you're actively building this!