Multi-Agent Workflows for AI-103: Agent Framework, Approvals and Monitoring

Introduction

Multi-agent orchestration in Microsoft Foundry combines specialist roles with explicit routing, durable state and enforceable action limits. This lesson completes the agent part of AI-103's largest domain, Implement generative AI and agentic solutions (30–35%). One agent with a handful of tools is often enough, but real systems frequently need several specialised agents, long-running processes, and controls that decide how much freedom each agent has. The official skills list three areas covered here: orchestrated multi-agent solutions, autonomous and semiautonomous workflows with safeguards and approvals, and monitoring, evaluation and error analysis of agents.

After this lesson you will be able to choose an orchestration pattern, design hand-offs between agents, place approvals and limits in an autonomous workflow, and diagnose agent failures from traces and evaluations.

Learning objectives

• Decide when a multi-agent design is justified, and when one agent or a fixed workflow is better.
• Compare orchestration patterns: sequential, concurrent, group chat, hand-off and manager-planned.
• Build workflows in Foundry and with the Microsoft Agent Framework, including agent-to-agent calls.
• Design autonomy levels with safeguards: limits, approvals, timeouts, idempotency and kill switches.
• Monitor agents with traces and metrics, evaluate behaviour and perform error analysis.

Start with Foundry agent tools and conversation isolation before adding orchestration.

1. Why and when to use several agents

A multi-agent solution splits a job among agents that each have a narrow role, their own instructions and their own tools. Potential benefits include smaller tool sets, separate roles, independent testing and specialized models (a small model for classification, a larger one for reasoning). Costs: more model calls, latency, tokens, coordination bugs and a bigger surface to monitor.

Rule of thumb. Add an agent only when it removes a concrete problem such as tool confusion, a security boundary or a quality check; never for architectural elegance.

2. Orchestration patterns

2.1 Foundry workflows

The Foundry visual workflows preview is scheduled to retire on December 1, 2026. Microsoft recommends Agent Framework for new workflows; plan migration of existing visual workflows. Historical templates include human input, sequential execution and group chat, with variables and branching. Recognize those concepts in older material while distinguishing the retiring product from the general workflow pattern. For business connectors or long waits, evaluate a supported durable orchestrator such as Azure Logic Apps and check the integration lifecycle separately.

2.2 Microsoft Agent Framework

The Microsoft Agent Framework is the open-source SDK (Python and .NET) for building agents and multi-agent workflows that run against Foundry. Concepts you should recognise:

• An agent wraps a chat client (a Foundry deployment), instructions and tools.
• A workflow is a graph of executors (agents or functions) connected by edges, with typed messages flowing between them. It supports sequential, concurrent, hand-off and group-chat builders, checkpoints and request/response steps for human input.
• Agents can be hosted in Foundry Agent Service so that identity, tracing and scale are managed.

import asyncio
import os
from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from agent_framework.orchestrations import SequentialBuilder
from azure.identity import AzureCliCredential

async def main():
    # Development credential; sign in with az login first.
    with AzureCliCredential() as credential:
        client = FoundryChatClient(
            project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
            model=os.environ["FOUNDRY_MODEL"],
            credential=credential,
        )
        writer = Agent(
            client=client, name="writer",
            instructions="Draft one sentence about the return window supplied in the request.",
        )
        reviewer = Agent(
            client=client, name="reviewer",
            instructions=(
                "Check the draft against the supplied 45-day return policy. "
                "Report discrepancies. Your feedback does not authorize publication."
            ),
        )
        workflow = SequentialBuilder(participants=[writer, reviewer]).build()
        events = await workflow.run("Draft a notice: the return window is 45 days.")
        for result in events.get_outputs():
            for message in result.messages:
                print(message.text)

if __name__ == "__main__":
    asyncio.run(main())

Use a compatible current Agent Framework installation (the agent-framework metapackage plus azure-identity) and lock tested versions. Set the project endpoint and deployed model name in the two environment variables. The former AzureAIAgentClient API has been removed from current Python packages; the example uses FoundryChatClient and the current orchestration namespace. It runs local orchestration over project inference, not a published service-managed workflow. Both agents share a development credential and have no write tools; this is not a separation-of-permissions example.

SequentialBuilder passes earlier conversation messages by default. Configure the context passed to each participant deliberately; chain_only_agent_responses can reduce propagation but does not enforce access control. A reviewer response is probabilistic feedback, not an authenticated human approval or a deterministic policy check.

2.3 Agent-to-agent (A2A) and tools as agents

Distinguish three related integration styles:

With connected agents the parent delegates and keeps ownership; with hand-off the user's next message goes to the new agent. Choose by who should own the answer.

Worked example 1: choosing the pattern

Scenario. A bank wants to analyse a loan application. Credit risk, fraud signals and regulatory checks are independent. A final summary must combine them and a human officer must approve any loan above 100,000 EUR.

Reasoning, step by step.

1. The three checks are independent and read different systems, so run them concurrently as three specialised agents, each with only its own tools and data access.
2. A summary agent merges the three results (fan-in). Keep it free of data-source tools: it only sees structured outputs.
3. Add a human-in-the-loop step after the summary when amount > 100,000 EUR; below that, any automated step must still pass the approved deterministic policy, data validation and authorization. Agent agreement alone is not a credit decision control.
4. Validate structured fields and apply explicit merge rules. A JSON schema constrains shape on supported model/API paths; it does not make model judgments deterministic or prove their facts. Route missing, conflicting or invalid evidence to review.
5. Set per-agent timeouts, so one slow check does not block forever, and a rule: a missing check means "refer to human", never "assume pass".

Why not group chat? The agents have no need to debate; a free-form conversation costs more, is harder to audit and may drift. Why not a single agent with all tools? It would hold access to every data source, weaken separation of duties and choose among many tools less reliably.

3. Hand-offs and shared context

Hand-offs fail most often because context is lost. Techniques:

• Pass only the authorized context the next agent needs, often a structured summary (issue, prior steps, source identifiers). Model-generated identity fields are hints, not authority; keep tenant/user identity bound to a verified backend session.
• Keep shared state (a case record) in your own store; agents read and write it through tools.
• Give each agent a clear description of when it should receive control, because routing relies on those descriptions.
• Limit hand-offs per request (for example 4) and define a fallback owner, usually a human queue.

4. Autonomous and semiautonomous workflows

4.1 The autonomy spectrum

Autonomy should follow risk and reversibility. Reading sensitive data or sending it to a remote agent can itself be high risk; creating a local draft often has fewer external effects. Classify the actual data flow and action: sending money, deleting data or contacting customers are high risk and need approval or tight limits.

4.2 Safeguards checklist

1. Least privilege. Use separate identities, scopes and backend checks where duties need isolation. Separate Agent objects or instructions do not automatically create separate Azure identities; avoid a shared privileged executor.
2. Hard limits in code. Maximum amount, maximum records per run, maximum tool calls, maximum tokens and a wall-clock timeout.
3. Approvals. Gate sensitive actions in the backend. MCP has an approval-request protocol; custom function tools need application-enforced approval or a framework wrapper. Bind authorization to the authenticated approver, exact arguments, resource, amount and request ID; expire it and prevent replay.
4. Idempotency. Retries must not duplicate effects; use idempotency keys.
5. Allow lists. Restrict which tools, domains and MCP servers an agent may use.
6. Content and attack protection. Prompt-attack detection on untrusted inputs such as emails and web pages; treat retrieved or tool-returned text as data.
7. Circuit breakers and kill switch. Pause the workflow automatically on anomalies (error rate, spend, unusual volume) and let operators stop new dispatches. Account for in-flight calls: disabling the agent does not roll back a payment or deletion already accepted.
8. Audit. Record observable routing outcomes, tool/action IDs, necessary sanitized arguments/results and approvals. Instrument and sample traces deliberately; do not assume raw internal reasoning is available. Store required audit records durably with retention and access controls.
9. Fallback to people. Define what happens when the agent is unsure or exceeds limits: create a ticket or hand over.

The Responsible AI lesson on tool approvals and backend enforcement explains how to bind approval to an exact action.

4.3 Long-running work

Autonomous tasks can run for minutes or days. Design for it: persist state in a store or workflow checkpoints, make steps resumable after failure, set deadlines, and notify people when approvals are waiting. A workflow that waits for approval must survive restarts, which is why checkpointing or a durable workflow service is used rather than keeping state in a process.

Worked example 2: placing approvals

Scenario. An IT-operations agent can restart services, resize virtual machines and delete unused disks. It monitors alerts overnight.

Design, step by step.

1. Classify actions by risk: read metrics (low), restart a stateless service (medium, reversible), resize (medium, costs money), delete disk (high, irreversible).
2. Autonomous: reading metrics and restarting stateless services, with a limit of 3 restarts per service per hour (a loop would otherwise hammer production).
3. Semiautonomous: resizing within a pre-approved size list; anything larger needs approval.
4. Approval always: deleting disks; an authorized preparation step creates a supported snapshot, verifies it completed and requests approval with the exact disk, snapshot, cost and retention details. The executor rechecks that the disk is still unused before deletion; a snapshot is not proof that deletion is safe.
5. Give the agent an identity with roles scoped to one resource group, not the subscription.
6. Add a kill switch (feature flag) that the on-call engineer can flip, and alerts if the agent exceeds its action budget.

Why not rely on "be careful" instructions? An overnight loop without enforced limits could repeat a destructive action thousands of times before anyone notices; limits and approvals are controls, instructions are not.

5. Monitoring, evaluation and error analysis

5.1 What to observe

Use the GenAI optimization lesson on latency, tokens and model routing to interpret the resulting costs and timing.

Multi-agent systems need correlated traces: a trace id passed through every hand-off so one user request can be followed across agents. Without it, error analysis becomes guesswork.

5.2 Agent evaluators

Beyond text-quality evaluators, agent-focused evaluators examine behaviour:

• Intent resolution (preview): did the agent correctly understand what the user wanted?
• Tool call accuracy: did it choose the right tool with correct arguments?
• Task adherence (preview): did it follow the instructions and the task scope?
• Groundedness and relevance: for retrieval-based answers, as in the previous lesson.
• Safety evaluators: harmful content, prompt-attack vulnerability and unsafe actions, often tested with simulated adversarial conversations.

Evaluate with a golden dataset of scenarios with expected tool use, include multi-turn conversations, and run evaluations on every agent version in CI. For live checks, configure instrumented sampling, the required evaluator permissions/resources and supported alert export; Foundry continuous evaluation is currently preview and can add judge-model cost. Offline evaluation and runtime guardrail signals are different mechanisms.

5.3 Error analysis method

1. Collect failing cases from evaluations, user feedback and alerts.
2. Locate the failing step in the trace: wrong routing, wrong tool, bad arguments, tool error, retrieval miss, hallucinated answer or policy violation.
3. Classify by root cause and count each class; prioritize by severity and risk as well as frequency; a rare unauthorized payment can outrank many harmless wording errors.
4. Fix at the right level: instructions for ambiguity, tool schema for wrong arguments, retrieval for missing evidence, code for limits, model choice for reasoning shortfalls.
5. Retest on the same dataset plus new regression cases, and compare versions before promoting.

Worked example 3: diagnosing a flaky triage agent

Scenario. A hand-off system routes tickets to billing or technical agents. Evaluations show intent resolution 3.1 out of 5 for ambiguous tickets, and traces show 22% of tickets bounce between agents at least twice.

Reasoning, step by step.

1. Traces show that bouncing tickets mention both invoices and login problems; the triage descriptions overlap ("handles account issues" in both).
2. Root cause: overlapping role descriptions and no tie-break rule. It is not a model-capacity problem, because the inspected failures show overlapping routing criteria. Correct simple tickets alone would not rule out a harder-model limitation.
3. Fix: split the descriptions by trigger ("billing: invoices, payments, refunds"; "technical: login, errors, performance"), add a rule that mixed tickets go to the technical agent first with a billing note, and add a limit of two hand-offs before a human queue.
4. Re-run the dataset including 30 new mixed tickets; target bounce rate under 3%.

Why not switch to a bigger model first? It costs more and may not fix an ambiguity that humans would also face; test the cheap, targeted fix first.

Comparison: choosing a structure

Common pitfalls

• Using many agents where one agent or a workflow would do.
• Letting group chats run without a stop condition or turn limit.
• Passing the whole transcript on every hand-off, or passing nothing.
• Giving all agents the same powerful identity.
• Relying on instructions instead of enforced limits and approvals.
• Treating the output of one agent as trusted when it processed untrusted content.
• Forgetting trace correlation across agents.
• Evaluating only final answers and ignoring tool-call correctness.
• Averaging scores without reading the failing traces.

Exam tips

• "Independent analyses" suggests concurrent; "draft then check" suggests sequential; "route to a specialist" suggests hand-off.
• "Irreversible or financial action" suggests approval and least privilege.
• "Loops or repeated calls overnight" suggests iteration limits, circuit breakers and alerts.
• "Cannot tell which agent caused the error" suggests correlated tracing.
• Fix the layer where the defect appears: schema, instructions, retrieval, code or model.
• A reviewer agent should not share the same write permissions as the producer.

Mini case study: Adventure Works Claims

Adventure Works Claims processes insurance claims. A claim arrives with photos, a form and a policy number. Needed: extract data, verify coverage, assess damage, detect fraud signals, propose a payout, and obtain approval for payouts above 5,000 EUR. Leadership wants overnight processing with no duplicate payments, full audit and an evaluation gate before each release.

Solution.

1. Structure. A workflow with concurrent specialist agents: coverage (reads policy store), damage (multimodal analysis), fraud (reads historical claims). An intake step extracts structured data; a decision agent merges the three outputs.
2. Autonomy. Payouts up to 5,000 EUR may execute only after validated evidence passes approved deterministic eligibility, authorization and amount checks; agent agreement and low predicted fraud risk are supporting signals, not permission; otherwise the claim goes to a human review step. Above 5,000 EUR a human approval step is always required.
3. Safeguards. One managed identity per agent with scoped roles; the payment executor records a stable operation ID bound to the claim, payee, amount and currency; retries reuse it, while later legitimate installments require separate authorized operations; a daily payout budget and a kill switch; prompt-attack detection on uploaded documents because they are untrusted.
4. Context. Agents exchange JSON with a schema; the case record lives in a database, and one trace id follows the claim.
5. Monitoring. Traces to Application Insights, alerts on error rate, spend and approvals waiting more than 24 hours.
6. Evaluation. Golden claims including fraud attempts and injection text inside photos' captions; agent evaluators for intent resolution, tool-call accuracy and task adherence run in CI, and sampled production traces are evaluated daily.

Commentary. Concurrency fits the independent checks, properly enforced identity and data scopes create security boundaries, limits and approvals bound the risk of full autonomy, and correlated traces plus evaluations make failures diagnosable.

Key terms

• Orchestration: coordination of agents and steps to complete a task.
• Hand-off: transfer of control from one agent to another.
• Group chat: several agents sharing a conversation with a coordinator.
• Workflow: a defined graph of steps, agents and conditions.
• A2A: Agent2Agent protocol for calling remote agents.
• Human in the loop: a workflow step that waits for a person.
• Kill switch: control stopping new work; in-flight side effects require separate handling.
• Idempotency key: identifier that makes a repeated operation harmless.
• Trace: recorded tree of an agent run's calls and results.
• Error analysis: classifying failures by root cause to prioritise fixes.

Summary checklist

• I can justify when to use several agents and when not to.
• I can match sequential, concurrent, group chat, hand-off and manager-planned patterns to scenarios.
• I can design hand-offs with structured context and limits.
• I can place approvals, limits, identities and kill switches according to risk.
• I can read traces and pick the layer to fix.
• I can choose agent evaluators and run them in CI and on production samples.

Sources

• Agent Framework sequential orchestration and context propagation (opens in a new tab)

• Current Foundry model provider (opens in a new tab)

• Agent Framework Python migration guide (opens in a new tab)

• Agent Framework overview and installation (opens in a new tab)

• Agent evaluators (opens in a new tab)

• Monitoring and continuous evaluation (opens in a new tab)

• Microsoft Learn, AI-103 study guide: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-103 (opens in a new tab)

• Build a workflow in Microsoft Foundry: https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/workflow (opens in a new tab)

• Transparency note for Foundry Agent Service: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/agents/transparency-note (opens in a new tab)

• Connect agents to MCP server endpoints: https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/tools/model-context-protocol (opens in a new tab)

• Develop AI agents on Azure (learning path): https://learn.microsoft.com/en-us/training/paths/develop-ai-agents-azure/ (opens in a new tab)

• Automate Foundry agents with workflows in Azure Logic Apps: https://learn.microsoft.com/en-us/azure/logic-apps/automate-foundry-agents-with-workflows (opens in a new tab)