Mit Patel's Blog

Building a Multi-Agent Financial AI System: My Journey from Confusion to Clarity

Or how I learned to stop worrying and love coordinating AI agents

Why I Built This (And Why You Might Want To Too)

Let me be honest – I didn't start this project thinking I'd build a multi-agent system. I just wanted to automate my stock research because manually checking Yahoo Finance, reading analyst reports, and googling market sentiment was eating up way too much of my time.

What started as "let me just pull some stock data with Python" turned into a fascinating deep dive into AI agents, and honestly, some pretty frustrating debugging sessions. But the end result? A system that can analyze any stock in seconds, pulling real-time data and current market sentiment simultaneously.

If you're curious about AI agents, frustrated with manual financial research, or just want to see how different AI models can work together, this might be interesting for you.

The "Aha!" Moment: Why Multiple Agents?

My first attempt was simple – one AI agent with access to Yahoo Finance. It worked, but the analysis felt incomplete. I'd get great financial data but miss the context of what was happening in the market.

Then I tried giving one agent access to both financial data AND web search. Big mistake. The agent got confused about which tool to use when, and the responses were inconsistent. Sometimes it would search the web for stock prices (which it already had access to via Yahoo Finance), other times it would try to get news from financial APIs.

That's when I realized: different tasks need different specialists.

  • Financial data extraction: Needs precision, structured queries, and understanding of financial metrics
  • Market research: Needs broad search capabilities, source evaluation, and context synthesis

So I built two specialized agents and let a coordinator manage them. Game changer.

Setting Up: Where I Went Wrong (So You Don't Have To)

The API Key Dance

Let's start with the painful part – getting your API keys sorted. I made this harder than it needed to be.

What I did wrong first:

# I thought I could just export these in terminal
export GROQ_API_KEY="your_key_here"
export AGNO_API_KEY="your_other_key_here"

This works... until you close your terminal. Then you're back to square one.

What actually works:

# Copy the sample file
cp .env.sample .env

# Edit .env with your actual keys
GROQ_API_KEY=your_actual_groq_key_here
AGNO_API_KEY=your_actual_agno_key_here

Pro tip: Get your Groq API key from console.groq.com first. It's free and fast. The Agno key is only needed if you want the playground interface.

Dependencies Hell (And How I Escaped)

My requirements.txt went through several iterations:

# Version 1 (didn't work)
agno
groq
yfinance
duckduckgo-search
python-dotenv
fastapi

# Version 2 (worked but was slow)
agno==0.1.0
groq==0.8.0
# ... specific versions

# Final version (current)
agno 
groq
yfinance
duckduckgo-search
python-dotenv
fastapi[standard]  # This [standard] part was key!

The fastapi[standard] inclusion was crucial for the playground to work properly. Spent two hours debugging why the web interface wasn't starting before I figured this out.

The Code: How It Actually Works

Agent Architecture (The Fun Part)

Here's the core of my system:

# Web Search Agent - My "market researcher"
web_search_agent = Agent(
    name="Web Search Agent",
    role="Search the web for the information",
    model=Groq(id="llama-3.1-8b-instant"),
    tools=[DuckDuckGoTools()],
    instructions=["Always include sources"],
    show_tool_calls=True,
    markdown=True,
)

# Finance Agent - My "data analyst"
finance_agent = Agent(
    name="Finance AI Agent",
    model=Groq(id="gemma2-9b-it"),
    tools=[
        YFinanceTools(
            stock_price=True,
            analyst_recommendations=True,
            stock_fundamentals=True,
            company_news=True,
        ),
    ],
    instructions="Use tables to display the data",
    show_tool_calls=True,
    markdown=True,
)

# The coordinator - brings it all together
multi_ai_agent = Agent(
    team=[web_search_agent, finance_agent],
    model=Groq(id="gemma2-9b-it"),
    instructions=["Always include sources", "Use table to display the data"],
    show_tool_calls=True,
    markdown=True,
)

Why These Specific Models?

I experimented with different model combinations:

  • Llama 3.1 8B Instant for web search: Fast, good at parsing search results, doesn't overthink
  • Gemma2 9B IT for financial analysis: Better at structured data interpretation and mathematical reasoning

Initially, I used the same model for both agents, but the web search agent was too slow, and the finance agent made too many unnecessary web searches.

The Tools: My Digital Swiss Army Knife

YFinanceTools - This was surprisingly comprehensive:

YFinanceTools(
    stock_price=True,              # Real-time prices
    analyst_recommendations=True,   # Buy/sell/hold ratings
    stock_fundamentals=True,       # P/E, market cap, etc.
    company_news=True,            # Latest company-specific news
)

DuckDuckGoTools - My web search workhorse:

  • No API limits (unlike Google)
  • Good at finding recent news
  • Automatically includes sources
  • Doesn't get blocked by rate limits

The Workflow in Action

When I ask: "Analyze Apple stock and give me the current sentiment"

  1. Coordinator receives the query and thinks: "This needs both financial data and market research"
  2. Finance Agent kicks in:
    • Calls YFinanceTools.stock_price("AAPL")
    • Calls YFinanceTools.analyst_recommendations("AAPL")
    • Formats data into neat tables
  3. Web Search Agent simultaneously:
    • Searches for "Apple stock market sentiment news"
    • Finds recent articles and analyst reports
    • Provides source links
  4. Coordinator combines everything into a comprehensive analysis

The whole process takes about 30-60 seconds and gives me information that would take me 15-20 minutes to gather manually.

My Biggest Challenges (And How I Solved Them)

Challenge 1: Agent Confusion

Problem: My agents were stepping on each other's toes. The finance agent would try to search the web for stock prices it already had access to.

Solution: Clear role definitions and specific instructions. I learned that AI agents need very explicit boundaries.

What worked:

# Instead of vague instructions
instructions="Get financial information"

# I became specific
instructions="Use tables to display the data. Only use YFinance tools for financial data."

Challenge 2: Inconsistent Output Formats

Problem: Sometimes I'd get markdown tables, sometimes plain text, sometimes a mix.

Solution: Standardized instructions across all agents and enabled markdown=True everywhere.

Challenge 3: API Rate Limits

Problem: During testing, I kept hitting rate limits with various APIs.

Solution: This is why I switched to DuckDuckGo for web search – no API limits, and it's actually quite good for financial news.

Challenge 4: Environment Variables Not Loading

Problem: The classic "it works on my machine" scenario. Environment variables weren't loading consistently.

Solution: 

from dotenv import load_dotenv
load_dotenv()  # This MUST be called before accessing env vars

Such a simple fix, but it took me embarrassingly long to figure out.

What I Learned Along the Way

  1. Agent Specialization is Key Don't try to make one agent do everything. Specialized agents with clear roles perform much better than generalists.
  1. Instructions Matter More Than You Think The difference between "get financial data" and "use tables to display financial data from YFinance tools only" is huge in terms of output quality.
  1. Tool Selection is Critical I initially wanted to use multiple financial APIs, but YFinance alone covers 90% of what I need. Sometimes less is more.
  1. Debugging Multi-Agent Systems is... Interesting Setting show_tool_calls=True was a lifesaver. Being able to see which agent called which tool when helped me understand the flow and debug issues.
  1. The Playground is Worth the Setup Initially, I thought the playground was just a nice-to-have. Wrong. It's incredibly useful for testing different queries and understanding how agents respond to various inputs.

My Takeaways and What's Next

What Worked Really Well:

  • Multi-agent architecture: The specialization approach paid off
  • Tool integration: YFinance + DuckDuckGo covers most financial research needs
  • Groq models: Fast, reliable, and cost-effective
  • Interactive playground: Great for experimentation

What I'd Do Differently:

  • Start with clearer agent roles: Would have saved hours of debugging
  • Set up better error handling: Currently, if one agent fails, everything stops
  • Add more financial metrics: Could integrate more sophisticated analysis tools

What's Next:

I'm thinking about adding:

  • Portfolio analysis agent: For analyzing multiple stocks at once
  • Technical analysis tools: Chart patterns, indicators, etc.
  • Sentiment scoring: Quantifying market sentiment from news
  • Alert system: Notifications when stocks hit certain conditions

Want to Try It Yourself?

If you're interested in building something similar:

  1. Start simple: One agent, one tool, one specific task
  2. Add complexity gradually: Don't try to build everything at once
  3. Test extensively: Use the playground to understand agent behavior
  4. Focus on instructions: They're more important than you think

The full code is available [in my repository], and honestly, half the learning was in the debugging process. Don't be discouraged if it doesn't work perfectly the first time – that's part of the fun.

Final Thoughts

Building this system taught me that AI agents are powerful, but they're not magic. They need clear instructions, appropriate tools, and careful coordination. The technology is there, but the art is in the orchestration.

Also, there's something deeply satisfying about asking your computer "How's Tesla doing?" and getting a comprehensive analysis with real-time data, analyst opinions, and market context in under a minute.

If you build something similar, I'd love to hear about it. What tools did you use? What challenges did you face? How did you solve them?

Have questions about the implementation or want to share your own multi-agent adventures? Feel free to reach out or check out the code repository.