Skip to content
FR

Monitor, Analyze, and Tune AI Agents

Step 15 of 22

AI Agent Monitoring: Telemetry, Feedback, and Performance

AI agent monitoring connects telemetry, user feedback, and business outcomes to decisions about what to improve. For AB-100, this lesson covers Analyze, monitor, and tune AI-powered business solutions within Deploy AI-powered business solutions (40–45% in the course outline before the announced November 16, 2026 update). You will choose observability tools, distinguish operational signals from quality evidence, and measure changes against a baseline.

Learning objectives

After this lesson you can:

  • Recommend a monitoring process and tools for agents built with Copilot Studio, Microsoft Foundry and Microsoft 365 Copilot.
  • Distinguish built-in analytics from telemetry exported to Azure Monitor and Application Insights.
  • Select performance, quality, cost and adoption metrics for an agent.
  • Analyze user feedback and the improvement backlog to prioritize changes.
  • Interpret telemetry (traces, latency, token usage, errors) to decide what to tune: instructions, knowledge, tools, model or orchestration.
  • Close the loop with a controlled, measured tuning cycle.

1. Why monitoring is an architectural deliverable

An agentic solution is non-deterministic. The same request can produce different answers, call different tools and cost a different amount. Monitoring therefore answers four questions continuously:

Question Typical evidence
Is it used? Sessions, active users, retention, channels
Is it useful? Resolution or task-completion rate, escalation rate, satisfaction ratings
Is it correct and safe? Groundedness, wrong or unsupported answers, policy or content-safety flags
Is it sustainable? Latency, error rate, token and capacity consumption, cost per resolved task

A good design states these questions, the metric behind each one, the target value and the owner who acts when a target is missed. Without targets, dashboards only describe; they do not govern.

2. The monitoring toolbox

Copilot Studio analytics

Copilot Studio includes built-in analytics for each agent. Typical views cover usage and engagement, resolution and escalation outcomes, satisfaction scores, topic-level and knowledge-source performance, and tool or action execution. Start here before building a custom dashboard, but check permissions, agent type, and metric prerequisites. The standard-harness Monitor page excludes authoring test-panel activity; Application Insights agent telemetry can include it. Active-user metrics require agent authentication. Transcript availability and Dataverse retention are separate considerations.

Agent Insights Hub in Copilot Agent Kit aggregates configured telemetry, transcripts, and credit usage. It requires setup, including inventory synchronization, Application Insights connectivity, appropriate application access, and kit permissions. Its synchronized dashboards should not be assumed to be instantaneous views of every agent.

Application Insights and Azure Monitor

Built-in analytics are agent-centric. For operations teams and for cross-system correlation, export telemetry to Azure Application Insights (part of Azure Monitor):

  • Agent-level telemetry: connect a Copilot Studio agent to an Application Insights resource; interactions, including tests in the authoring canvas, are logged and can be queried with Kusto Query Language (KQL).
  • Environment-level telemetry (preview): managed environments can export OpenTelemetry-aligned agent spans. The current schema writes invocation and tool spans to dependencies; older invocation data may be in requests. Validate the current harness support and schema before reusing queries.
  • Agent details view in Application Insights, reached through Agents (Preview): a cross-source view after compatible data collection is configured. Token, cost, error, and latency visibility depends on emitted telemetry.

Use Azure Monitor alerts for operational thresholds (error spikes, latency, failed tool calls) and workbooks or dashboards for trends.

Microsoft Foundry observability

For supported Foundry agents, tracing captures instrumented operations such as model calls, tool calls, retries, timing, and usage in Application Insights. Enable tracing, connect the resource, and check reader permissions. A missing span can reflect instrumentation or sampling rather than proof that an operation never happened. Foundry also provides evaluations that score outputs on quality and safety dimensions (for example relevance, groundedness, coherence and harmful content), both before release on test data and after release on sampled production traffic.

Microsoft 365 and administration surfaces

For agents running in Microsoft 365 Copilot, administrators rely on the Microsoft 365 admin center and Copilot usage reporting to see adoption and agent inventory, and on Microsoft Purview for audit and compliance records. Power Platform admin center reports capacity and consumption of Copilot Studio agents.

Choosing among them

Need Best fit
Maker checks resolution and satisfaction of one Copilot Studio agent Built-in Copilot Studio analytics
Central view over many Copilot Studio agents Agent Insights Hub
Operations team needs alerts, KQL and correlation with other services Application Insights / Azure Monitor
Debug a multi-step Foundry agent run Foundry tracing
Measure answer quality against an explicit, reviewed rubric Foundry evaluations or a reviewed test set with suitable graders
Adoption, licensing and audit across the tenant Admin centers and Purview

3. Metrics that matter

Group metrics so the dashboard supports decisions:

  • Adoption: active users, sessions per user, return rate, share of eligible population.
  • Effectiveness: task completion or resolution rate, escalation or handoff rate, abandonment, time saved per task.
  • Quality: user satisfaction (thumbs, CSAT), groundedness, answer accuracy against a reference set, fallback and "I don't know" rate.
  • Performance: end-to-end latency, tool-call latency and failure rate, retries, time to first response.
  • Cost: tokens per conversation, Copilot credits or capacity consumed, cost per successful outcome (compare with the AI agent ROI and TCO business case).
  • Risk: content-safety blocks, prompt-manipulation attempts, access denials, data-loss-prevention hits.

Two cautions are common exam material. First, a single metric misleads: a falling escalation rate is good only if satisfaction and accuracy hold. Second, averages hide problems; look at percentiles for latency and at segments (channel, language, department) for quality.

4. User feedback and backlog analysis

Quantitative telemetry shows observed behavior; feedback offers clues about causes. Neither alone proves causation.

Sources include in-conversation ratings and comments, human-agent handoff notes, support tickets, surveys, and the transcripts of failed or escalated conversations. The architect sets up a repeatable analysis:

  1. Collect feedback in one backlog (for example Azure DevOps, Planner or Dataverse) with a restricted reference to the conversation. Keep sensitive transcript content out of broadly shared backlog items.
  2. Classify items: missing knowledge, wrong or outdated knowledge, wrong routing or topic, tool failure, misunderstood intent, tone or format, security or policy issue.
  3. Quantify by frequency and business impact, using telemetry to confirm that a complaint is systemic and not an outlier.
  4. Prioritize with impact versus effort; safety, compliance and correctness defects outrank convenience requests.
  5. Feed back the result to users so they see that feedback is acted on, which keeps ratings flowing.

AI can help here: use a model or Copilot to cluster free-text comments and unanswered questions into themes, then validate the clusters manually before acting.

5. Interpreting telemetry to decide what to tune

Telemetry becomes useful when each symptom maps to a likely cause and a lever.

Symptom in telemetry or feedback Likely cause Tuning lever
High fallback rate for a topic area Missing or poorly described knowledge or topic Add or restructure knowledge sources, improve topic descriptions
Confident but wrong answers; low groundedness Weak or conflicting sources, loose instructions Curate sources, guide grounded answers, and verify cited evidence; instructions and citations do not guarantee correctness
Wrong tool chosen or tool not called Ambiguous tool names or descriptions Rewrite tool descriptions and input definitions, reduce overlapping tools
Frequent tool timeouts or errors Backend or connector problem, throttling Investigate throttling and failures; use bounded retries with backoff, timeouts, and idempotency for writes
High latency Many sequential steps, large prompts, slow model Reduce unnecessary steps and context; parallelize independent supported operations only; evaluate model changes
High token cost per conversation Long context, verbose output, unnecessary retrieval Shorten instructions and history, limit retrieved passages, route simple tasks to cheaper models
Escalations concentrated in one intent Agent lacks authority or process knowledge Add the action or knowledge, or document that the intent is for humans
Quality drops after a platform or model update Model or service change Re-run the regression set, pin or switch model versions where supported
Unusual spikes of blocked prompts Misuse or attack Review content-safety and access controls, alert security

In multi-agent designs, traces show which sub-agent or tool consumed time or produced the error; tuning one component in isolation without a trace often moves the problem elsewhere.

6. The tuning cycle

Tune deliberately, one lever at a time:

  1. Baseline: record current metrics on a fixed test set and on production segments.
  2. Hypothesis: state the cause suspected from telemetry.
  3. Change in a development environment (instructions, knowledge, tool descriptions, model, thresholds).
  4. Evaluate against the same test set plus new cases drawn from the failures; check safety evaluations too.
  5. Release through the normal ALM pipeline (covered later in this chapter's domain) and consider a limited rollout.
  6. Compare to the baseline after release and keep or revert the change.

Make routine tuning changes through development, evaluation, and controlled release. An authorized emergency mitigation may need a production control, but record the change and reproduce it in the release artifacts. Neither a dashboard nor an old artifact by itself guarantees a safe rollback.

Telemetry privacy and coverage

Message text, tool arguments, and results can contain customer data. Agent-level Application Insights logging has separate conversation-detail and sensitive-property controls; choose them deliberately. Minimize or redact sensitive fields before export, restrict log access, and set retention. Telemetry storage and ingestion also contribute to operating cost.

Monitor the trigger path as well as the agent. If an event never reaches an autonomous agent, its agent analytics cannot contain that run. Add upstream delivery/failure evidence so a silent trigger failure is distinguishable from a quiet day. Record the reporting timezone, window, and sample coverage when comparing dashboards.

7. Worked example

A retailer runs a Copilot Studio customer-service agent connected to Dynamics 365 Customer Service. After two months the telemetry shows: satisfaction fell from 4.3 to 3.9, escalations for "returns" doubled, p95 latency is 14 seconds, and cost per conversation rose 30%.

Analysis: transcripts of escalated returns show the agent quoting an old policy, because a superseded PDF and the new policy page are both knowledge sources. Traces show the returns flow calls the order-lookup tool three times in sequence, and the long conversation history is sent on every turn.

Recommended actions: remove the outdated source and set a review owner for policy content; rewrite the order-lookup tool description and instructions so it is called once and reused; summarize history to limit tokens; add returns scenarios to the regression test set; use a supported pilot group or a controlled traffic split where the delivery architecture supports one, and compare satisfaction, escalation, latency, and cost before wider release. Alerts on tool-failure rate and a weekly review of the feedback backlog keep the loop running.

Step-by-step investigation

  1. Confirm that the comparison uses the same reporting window, channel mix, and deployed version. A new channel can change both average latency and satisfaction without a regression in the original channel.
  2. Sample failed returns and verify the actual policy citations. Remove obsolete material from the retrieval path and retain the required document history under its separate retention policy.
  3. Correlate the repeated order lookups by operation ID and inputs. Calls with identical names can still be necessary if their inputs or underlying business state differ. Reuse a result only while it remains valid.
  4. Measure the contribution of the model and each backend. If three unnecessary serial lookups each take two seconds, removing two could save about four seconds for that path; this is an illustrative estimate to verify, not a guaranteed p95 improvement.
  5. Add reviewed returns cases to the agent evaluation and end-to-end test set. Require correctness and permission checks alongside the latency target.
  6. Promote the change through ALM for agents, data, and models, define who can stop the pilot, and compare results after enough representative usage.

For an illustrative cost calculation, €180 for 900 conversations is €0.20 per conversation. If only 600 meet the agreed successful-outcome definition, the cost is €0.30 per successful outcome. A release that reduces total spending while reducing successful outcomes even faster may be less economical. Use the same denominator across the baseline and candidate.

When comparing cost or resolution, record the denominator explicitly: all sessions, engaged sessions, or verified completed tasks. Keep failed requests in the operational error view even if a business-success report excludes them. Otherwise, a change in report filters can look like improved performance while the underlying service has not improved.

Mini case: fewer escalations, worse service

A new instruction discourages human transfers. Escalation falls from 20% to 8%, but complaints increase and a reviewed sample contains unsupported refund promises. Should the team celebrate the reduction?

Solution: investigate the incorrect promises, stop or mitigate the harmful behavior through the release process, and restore the required transfer path. Escalation is sometimes the correct outcome. Grade each scenario against its intended result rather than rewarding an agent for retaining every conversation. Track complaint rate and verified policy compliance alongside resolution and handoff counts.

Key terms

  • Trace: linked observations for an instrumented request or workflow.
  • Span: one timed operation within a trace, such as a model or tool call.
  • p95 latency: the duration at or below which 95% of measured requests finish, for a stated population and window.
  • Groundedness: support for an answer in the supplied evidence; it does not establish that the evidence is current or true.
  • Baseline: the version, population, and measurements used for comparison.

Exam tips

  • Start with built-in analytics for a single agent; use Application Insights when you need alerts, KQL queries or cross-service correlation.
  • Foundry tracing explains how an agent run executed; evaluations measure how good the outputs are.
  • Form a hypothesis from the evidence: conflicting sources call for source review, while latency can involve topics, models, tools, and orchestration. Confirm the cause before selecting a change.
  • Combine metrics (resolution with satisfaction and accuracy) and use percentiles and segments, not only averages.
  • Tune in a non-production environment against a stable test set, then release through ALM and compare with the baseline.
  • Feedback needs a backlog, classification, and prioritization by impact; telemetry confirms whether an issue is systemic.

Sources