Building a Meeting Notes AI Agent: From Manual to Automated in One Week

Every week, I used to spend roughly two hours writing up meeting notes.

Not because meetings were complex — but because turning raw conversation into something structured and useful takes real cognitive effort. You have to recall context, identify what mattered, assign owners to action items, and format everything legibly before the detail fades from memory.

I finally got tired of it. So I built an agent to do it instead.

The Problem with Manual Notes

The standard approach to meeting notes is broken in a few ways:

1. They’re written after the fact. Memory degrades fast. Notes written 30 minutes after a call are noticeably worse than notes written during — and notes written the next day are often little more than a skeleton.
2. They’re inconsistent. Everyone formats their notes differently. One person captures decisions but misses action items. Another lists todos without context. Neither version is fully useful to someone who wasn’t in the room.
3. They don’t distribute themselves. Even good notes sit in a personal doc that nobody reads. Getting the right information to the right people requires another manual step.

The real cost isn’t just the time spent writing — it’s the organizational memory that evaporates when notes aren’t taken well.

How the Agent Works

The architecture is straightforward. Here’s the full pipeline:

Recording → Transcription → Extraction → Formatting → Distribution

Step 1: Recording & Transcription

I use a combination of Fireflies.ai for automatic transcription and a custom webhook to capture the raw text when the meeting ends. For local recordings, I use Whisper (OpenAI’s transcription model) via a small Python script.

The transcript arrives as a blob of speaker-attributed text like:

[Long Bia]: We need to finalize the pricing before Thursday.
[Team]: Agreed. Can you send the updated deck?
[Long Bia]: Yes, I'll send it by EOD tomorrow.

Step 2: Structured Extraction with Claude

This is where the agent earns its keep. I send the raw transcript to Claude with a carefully designed prompt that extracts:

• Summary — 2–3 sentences covering the core discussion
• Decisions made — explicit commitments or conclusions reached
• Action items — tasks with owners and deadlines
• Open questions — unresolved items that need follow-up
• Key context — background information that future readers need

The prompt is structured to return clean JSON, which makes downstream formatting reliable:

import anthropic

client = anthropic.Anthropic()

def extract_meeting_structure(transcript: str) -> dict:
    message = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=2048,
        messages=[
            {
                "role": "user",
                "content": f"""You are an expert meeting facilitator. Extract structured information from this meeting transcript.

Return a JSON object with these exact keys:
- summary: string (2-3 sentences)
- decisions: array of strings
- action_items: array of objects with {{"owner": string, "task": string, "due": string or null}}
- open_questions: array of strings
- context: string (key background info, optional)

Transcript:
{transcript}

Return only valid JSON, no markdown."""
            }
        ]
    )

    return json.loads(message.content[0].text)

Step 3: Formatting

The JSON gets rendered into a Markdown document using a Jinja2 template. The output looks like this:

# Meeting Notes — Product Sync
**Date:** March 15, 2025 | **Duration:** 47 min

## Summary
Discussed Q2 pricing strategy and finalized the go-to-market timeline.
The team aligned on a $29 entry price point for the initial launch.

## Decisions
- Entry price set at $29 (one-time purchase)
- Launch date moved to April 1st

## Action Items
| Owner | Task | Due |
|-------|------|-----|
| Long Bia | Send updated pricing deck | March 16 |
| Team | Review competitive analysis | March 18 |

## Open Questions
- How do we handle refund requests after 30 days?
- Do we need a separate enterprise pricing tier at launch?

Step 4: Distribution

The formatted notes get sent to:

• Slack — posted to the relevant channel via Incoming Webhooks
• Notion — created as a new page in the team database via the Notion API
• Email — sent to all meeting participants using SendGrid

The whole pipeline runs automatically within 5 minutes of the meeting ending.

What Surprised Me

A few things I didn’t expect:

The extraction quality is remarkably high. Claude correctly identifies the difference between a decision (“we agreed to X”) and a discussion point (“we talked about X”). It catches implied owners even when someone says “can someone handle this?” rather than explicitly assigning the task.

Edge cases are easy to handle in the prompt. When participants use shorthand or refer to past context, Claude usually infers correctly. When it doesn’t, a few examples in the prompt (few-shot prompting) solve the issue within one iteration.

The distribution step matters more than the extraction. Even perfect notes don’t help if nobody reads them. Pushing to Slack dramatically increased engagement — people actually reference the notes in follow-up conversations now.

What’s Next

A few improvements I’m working on:

• Recurring meeting awareness: Detect when a meeting is a recurring sync and carry forward unresolved open questions automatically
• Meeting analytics: Track action item completion rates over time
• Calendar integration: Auto-tag notes with the meeting title and attendees from the calendar event

If you want to try this yourself, the core extraction logic is less than 50 lines of Python. The hard part is wiring up the integrations — but each one is well-documented and straightforward.

The best automation is the kind you forget about. Two weeks in, I genuinely stopped thinking about meeting notes as a task. That’s the goal.

If you’re building something similar or want to talk through the architecture, get in touch.