Microsoft Foundry Operations for AI-103: Quotas, Monitoring and Security

Introduction

This lesson covers production management objectives in the domain Plan and manage an Azure AI solution (25–30% of AI-103). The preceding lesson covers Foundry deployment setup and CI/CD. Here you learn to run an AI system in production: control quota and cost, watch quality and safety signals, keep search indexes healthy, and lock down identity and networking. Exam questions in this area are almost always short operations scenarios ("users report X, what do you check or change first?"), so the goal is to build a reliable diagnostic reflex.

After this lesson you will be able to size and throttle a deployment, choose the right monitoring signal for a symptom, detect drift and grounding problems, maintain an Azure AI Search index, and design keyless, private access with least-privilege roles.

Learning objectives

• Explain quotas, tokens per minute, requests per minute, scaling choices and cost controls.
• Choose telemetry, tracing and evaluation signals for performance, drift, safety events and grounding quality.
• Diagnose poor retrieval by inspecting index health, freshness and relevance.
• Apply managed identity, private endpoints, keyless authentication and role-based access control.
• Pick the least-privilege fix for an access or network problem.

1. Quotas, scaling, rate limits and cost

1.1 How capacity is measured

For Standard model deployments, capacity is allocated as tokens per minute (TPM) with a model-dependent requests per minute (RPM) ratio. Standard quota is scoped by subscription, region, model and deployment type, then divided among deployments. Spare subscription quota must be assigned to the deployment before it can use it; an additional deployment drawing from the same exhausted pool creates no extra quota. Provisioned deployments instead use PTUs, with model-independent quota pools for the applicable region and deployment type. Quota approval does not guarantee that deployment capacity is available.

A token is roughly three to four characters of English text. Both the prompt (instructions, conversation history, retrieved passages) and the completion (generated output) contribute to billed usage. Rate-limit admission uses an estimated token budget rather than the final billed count. A high maximum-output setting can increase that estimate and trigger throttling sooner; use the output-token parameter supported by the chosen API and model.

1.2 What a 429 means and how to react

HTTP 429 can indicate a deployment TPM/RPM limit, backend capacity pressure or a temporarily reduced effective limit. Inspect the error body and rate-limit headers before choosing a remedy. Good clients:

1. Honor the retry delay returned by the service, including retry-after-ms where supplied; do not interpret milliseconds as seconds.
2. Use exponential backoff with jitter so many clients do not retry in lockstep.
3. Cap retries and fail gracefully. Ensure retries cannot duplicate external tool side effects, and use a fallback only when quality, region and data-residency requirements still hold.

For persistent 429s, correlate usage with allocated TPM/RPM and service health. Reduce bursts and oversized token budgets first; request and allocate more quota when the pool is the bottleneck. A second deployment helps only when it has independently available quota and capacity. Provisioned throughput can suit steady workloads, subject to sizing and availability.

1.3 Scaling options compared

1.4 Controlling cost

For Standard deployments, token usage is a major model-cost driver. Provisioned capacity is billed even when idle; retrieval, tools, telemetry and evaluations add separate costs. The main levers:

• Right-size the model. Route simple tasks (classification, extraction) to a small model and reserve the large model for hard reasoning.
• Shorten prompts. Trim history, summarize old turns, retrieve fewer but better passages (top 3–5 instead of 20).
• Cap outputs with a sensible maximum token setting and concise-answer instructions.
• Cache repeated answers where correctness allows it, and use prompt caching features when the platform supports them.
• Budgets and alerts. Create Azure Cost Management budgets with alerts at thresholds such as 50%, 80% and 100%, scoped to the resource group. Tag resources by team, application and environment for cost allocation. Budget alerts notify you; they do not automatically stop resources or token consumption.
• Gateway quotas. Limit tokens per consumer so one application cannot consume everything.

Worked example 1: diagnosing a bill spike

Scenario. Contoso Retail runs a support assistant. Monthly cost doubled from about 2,000 EUR to 4,100 EUR, but ticket volume rose only 5%. Token analytics in Application Insights show the average prompt grew from 1,800 to 5,200 tokens.

Reasoning, step by step.

1. Prompt tokens per request rose by about 189%: (5,200 − 1,800) / 1,800. With 5% more requests, total prompt-token volume is about 3.03 times its previous level. This is a strong lead, not a complete explanation of a 2.05-fold bill: compare output tokens, model prices, caching and other costs too.
2. Look at what makes up the prompt: system instructions, conversation history, retrieved passages.
3. Traces show the team changed retrieval from top 4 passages to top 15 and stopped truncating history.
4. Fix: return to top 4–5 passages with reranking, summarize history after a few turns, set a prompt-size budget, and add an alert on average tokens per request.

Why other fixes are weaker. Buying provisioned throughput does not reduce tokens; it only changes billing. Switching off logging hides the problem and removes your evidence. Lowering the temperature does not affect prompt size.

2. Monitoring performance, drift, safety events and grounding quality

2.1 The four families of signals

Monitoring should answer a question of the form "what changed?" Always correlate a symptom with a deployment event, a model version change, an index rebuild or a traffic change.

2.2 Tracing

Foundry supports tracing built on OpenTelemetry, sent to Application Insights. With suitable instrumentation and sampling, traces can show retrieval calls, model calls with token counts, tool calls and output. Prompt and response content is available only when that capture is configured; it is not guaranteed for every request. Use traces to find which step is slow, which tool failed and which passages were retrieved. Remember that traces can contain sensitive prompt content; control who can read them, apply retention limits and consider redaction.

2.3 Evaluation as monitoring

Evaluators score outputs automatically. Built-in evaluators cover areas such as groundedness, relevance, coherence, fluency, similarity and safety categories. Foundry continuous evaluation is currently a preview capability. With tracing, sampling, evaluator configuration and the necessary managed-identity permissions in place, it scores sampled live traffic; judge-model evaluations also consume resources and can incur cost. Export the resulting signals into supported dashboard and alert paths. These signals help detect drift: a gradual decline in quality caused by a changed model version, changed documents, changed user behaviour or a stale index.

Types of drift to distinguish:

• Data drift: user questions change (new products, new vocabulary) and the retrieval content no longer covers them.
• Model drift: a model version update changes behaviour; mitigate through controlled versions and testing before upgrading, while planning migration before model retirement.
• Content drift: the source documents change or become outdated, so grounded answers become wrong.

2.4 Grounding quality

For RAG systems, groundedness measures whether claims in the answer are supported by retrieved context. Groundedness alone does not identify the root cause or prove that the sources are current or true. An answer can be fully supported by an obsolete policy and still be wrong for today. Evaluate retrieval (did the right, current passage come back?) separately from generation (did the answer stay inside that evidence?).

Worked example 2: groundedness fell after a release

Scenario. An HR policy assistant had groundedness around 4.5/5 for weeks. After Tuesday's release it dropped to 3.1. Latency and error rates are unchanged. The release included a new model version (auto-upgrade) and a re-indexed document set.

Reasoning.

1. Operational metrics are normal, so this is a quality problem, not an availability one.
2. Two changes coincided; isolate them. Replay the golden test set against the old and new model versions using the same index, then against the old and new index using the same model.
3. Results: the same model with the new index scores 4.4; the new model with the old index scores 3.2. The model version is the cause.
4. Fix: restore the previous version only if it is still supported and available. Use an appropriate version-upgrade policy while testing the replacement in staging with an evaluation gate, and complete migration before the retirement deadline. Pinning cannot preserve a retired model indefinitely.

Why other choices are worse. Increasing capacity will not change answer quality. Re-embedding all documents wastes effort when the index tested fine. Lowering the content filter threshold does not affect groundedness.

2.5 Alerts and dashboards

Create Azure Monitor metric or log alert rules for p95 latency, 429/5xx rates, blocked requests and abnormal token usage. For groundedness, first make evaluator results available through a supported log or custom-metric path; a quality score is not automatically a native Azure Monitor metric. Route alerts to action groups, using email or an appropriate integration such as a Logic App for Teams or an incident tool. Put operational and quality charts in one dashboard so on-call engineers see both. The Azure Monitor lesson on logs, alerts and action groups explains these routing mechanisms.

3. Search index health and relevance

Retrieval is a major source of RAG failures. Revisit RAG application design and grounding with Foundry when separating retrieval and generation. Azure AI Search health has several dimensions.

3.1 Index health checklist

3.2 Improving relevance

1. Use hybrid search: keyword search catches exact terms (part numbers, acronyms), vector search catches meaning.
2. Add semantic ranking to rerank the top results.
3. Use metadata filters (language, product, date) to reduce the candidate set.
4. Tune chunking: small chunks are precise but lose context; large chunks carry context but dilute the match. Overlap of 10–20% is a common start.
5. Evaluate with a golden set: questions with the expected source passage. Measure whether the expected passage is in the top k.
6. Use scoring profiles to boost freshness or specific fields when business rules require it.

Document and query vectors must share a compatible embedding vector space and dimensions. When switching to an incompatible embedding model, regenerate the affected document vectors and rebuild or repopulate the index before serving queries with the replacement. A deployment-name change alone does not require re-embedding.

Worked example 3: the right document ranks too low

Scenario. Users ask "What is the warranty for model XR-440?" The correct manual is in the index but ranks 12th, so with top 5 it never reaches the model. Pure vector search is configured.

Reasoning.

1. The identifier "XR-440" is an exact token; embeddings blur such strings.
2. Enable hybrid search so BM25 keyword matching contributes, and enable semantic ranking.
3. Add a filter by product family when the app can infer it.
4. Re-run the golden set and confirm the expected passage appears in the top 3.

Wrong fixes. Sending all 50 retrieved candidates to the chat model inflates tokens and noise. A larger retrieval candidate set can still help reranking: distinguish vector-query k, the results returned by search, and the small final context passed to the model. For hybrid semantic ranking, 50 candidates may be useful even when only 3–5 passages reach the prompt. Switching to a larger chat model does not help when the passage never arrives.

4. Managed identity, private networking, keyless credentials and role policies

4.1 Identity first

Prefer Microsoft Entra ID authentication to API keys. Keys are shared secrets that cannot be tied to a person or workload, are hard to rotate across teams, and API keys do not support managed identity. Entra ID gives per-identity auditing and short-lived tokens.

In code, DefaultAzureCredential tries a configured credential chain; environment credentials can take precedence over managed identity. Verify which principal is selected. For production, an explicitly configured managed-identity credential can avoid ambiguity.

4.2 Role-based access control

Assign the least privilege role at the narrowest supported scope. Foundry Agent Consumer is designed for invoking published agent endpoints without authoring privileges. Foundry User supports building and running project solutions; Foundry Project Manager adds project management and agent publishing; publishing requires the role on the Foundry resource scope. Model-deployment management requires account-level control permissions, available through Foundry Account Owner or a suitably scoped custom role; project management alone does not grant it. Foundry Account Owner and Foundry Owner support broader resource administration. For project- or agent-scoped Agent Consumer assignments, use Azure CLI; the Azure portal currently exposes that role at account scope only. Some portal screens retain the earlier Azure AI role names; check the role definition rather than assuming that all higher-sounding titles grant every data action.

Other roles frequently needed together with Foundry:

• Search Index Data Reader to query an Azure AI Search index; Search Index Data Contributor to write documents; Search Service Contributor to manage indexes.
• Storage Blob Data Reader/Contributor for documents.
• Key Vault Secrets User to read secrets when unavoidable.
• Log Analytics Reader / Monitoring Reader to read traces and run trace-based evaluations.

Roles on the resource are data-plane vs control-plane aware: "Owner" or "Contributor" does not necessarily allow data actions such as calling a model or reading index documents, which is why dedicated data roles exist.

4.3 Private networking

To prevent public exposure:

1. Create private endpoints for the Foundry resource, Azure AI Search and Storage, and integrate them with private DNS zones so names resolve to private IPs.
2. Disable public network access for a private-only requirement. Selected public networks or IP ranges still use a public endpoint and are a different policy.
3. Run the application in a virtual network (Container Apps environment, App Service with VNet integration, Functions premium).
4. For private outbound indexer access from Search to supported storage or embedding services, configure and approve shared private links and use private indexer execution where required. A client-facing Search private endpoint does not create that outbound path. Trusted-service firewall exceptions with managed identity are a separate public-endpoint mechanism; they do not satisfy a strict Private Link requirement.
5. For developer access, use a jump host, VPN or Bastion rather than reopening public access.

4.4 Secrets that remain

Some third-party keys cannot be avoided. Store them in Azure Key Vault with RBAC, read them with the managed identity, enable soft delete and purge protection, and use Key Vault references or runtime retrieval rather than copying values into app settings. Customer-managed keys (CMK) can encrypt data at rest when policy requires it.

4.5 Governance with Azure Policy

Azure Policy can enforce guardrails across subscriptions, such as denying AI resources with public network access enabled, requiring a diagnostic setting, or requiring approved regions. Use policy for preventive control and Defender for Cloud for posture and threat detection.

Worked example 4: private by design, and it still fails

Scenario. After enabling a private endpoint on Azure AI Search and disabling public access, a container app cannot connect. It uses Azure-provided DNS, the private endpoint is approved and routing is ready, but nslookup returns a public IP. The developer can still open the Azure portal.

Reasoning.

1. Opening the portal proves access to the management UI, not a successful query to the Search data endpoint. Inspect DNS from the application network.
2. The public-IP answer points to a missing or incorrect private DNS zone (privatelink.search.windows.net), record or virtual-network link. An actual DNS timeout instead requires checking resolver reachability and forwarding; it is not the same symptom.
3. Fix the private DNS record and virtual-network link, then verify resolution to the private endpoint address. For custom DNS, configure the appropriate conditional forwarding as well.
4. Then confirm the app's managed identity holds Search Index Data Reader.

Wrong fixes. Re-enabling public access defeats the requirement. Adding a key does not solve DNS. Granting Owner does not resolve a network path problem.

5. Comparison tables

Common pitfalls

• Assuming every 429 proves subscription quota exhaustion; inspect service pressure, allocation, RPM and client behavior.
• Assuming Contributor or Owner grants data-plane access to models or index documents.
• Disabling public access before verifying private DNS and routing. Missing private DNS can return a public IP rather than cause a DNS timeout.
• Changing the embedding model without re-indexing.
• Fixing a quality problem with more capacity. Conversely, shorter prompts and bounded output budgets can help genuine token-capacity problems.
• Leaving automatic model upgrades enabled on production deployments with no evaluation gate.
• Logging full prompts and responses without access control or retention limits.

Exam tips

• "No secrets", "keyless" or "least privilege" points to managed identity plus a narrow role.
• "Must not traverse the public internet" points to private endpoints plus private DNS plus disabled public access.
• When two causes coincide, the best answer isolates the variable (test old model with new index, and so on) rather than changing both.
• Exact identifiers failing in search suggest hybrid search; meaning-based misses suggest better chunking, vectors or semantic ranking.
• Distinguish control-plane roles (manage resources) from data-plane roles (call, read, write).

Mini case study: Fabrikam Logistics

Fabrikam runs a shipment-tracking agent in West Europe. Observations after a month in production:

1. p95 latency doubled during weekday mornings, with 429 errors from 8:00 to 10:00 only.
2. Finance reports spend 30% above forecast.
3. A security review found the app uses an API key stored in an app setting, and the Search service is reachable from the internet.
4. Users report that answers about new customs rules are outdated.

Solution.

• Items 1 and 2: the morning peak exceeds the deployment TPM. Add retry with backoff, request and allocate more quota or use an independently provisioned deployment behind API Management when residency rules permit, and consider provisioned throughput if the peak is predictable. Set budgets and alerts, and lower top-k and trim history to reduce tokens.
• Item 3: switch to a managed identity with the least-privilege Foundry role for the exact API (Agent Consumer for published-agent invocation, User for project authoring) and Search Index Data Reader for retrieval, remove the key, add private endpoints with private DNS, and disable public access. Apply Azure Policy to deny public access in future.
• Item 4: content drift. Check the indexer schedule and last run, enable change detection, and evaluate freshness and answer correctness on current customs examples. A freshness-aware scoring profile may help; groundedness alone cannot detect an answer supported by obsolete rules.

Commentary. Notice that four symptoms map to four different areas: capacity, cost, identity/network and index freshness. A good answer treats each with the smallest change that fixes its root cause.

Key terms

• TPM / RPM: tokens and requests per minute limits for a deployment.
• 429: rate limit response; honour Retry-After.
• Provisioned throughput: reserved model capacity sized for sustained throughput and latency targets.
• Tracing: step-by-step records of requests, usually via OpenTelemetry.
• Continuous evaluation: automatic scoring of sampled live traffic.
• Groundedness: degree to which an answer is supported by retrieved context.
• Drift: quality change over time from model, data or content changes.
• Hybrid search: keyword and vector retrieval combined.
• Private endpoint: private IP address for a PaaS service in your virtual network.
• Managed identity: Azure-managed workload identity with automatic credential rotation.
• Least privilege: the minimum role at the narrowest scope.

Summary checklist

• I can explain TPM/RPM, handle 429 with backoff and decide when to add capacity.
• I can identify the main cost levers: model size, prompt size, top-k, output caps and budgets.
• I can pick signals for latency, usage, quality and safety and set alerts.
• I can separate retrieval problems from generation problems.
• I can maintain index freshness, chunking, hybrid search and semantic ranking.
• I can design keyless access with managed identities and narrow roles.
• I can design private connectivity with private endpoints and private DNS.

Sources

• Microsoft Learn, AI-103 study guide: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-103 (opens in a new tab)

• Authentication and authorization in Microsoft Foundry: https://learn.microsoft.com/en-us/azure/foundry/concepts/authentication-authorization-foundry (opens in a new tab)

• Monitor agents with the Agent Monitoring Dashboard: https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/how-to-monitor-agents-dashboard (opens in a new tab)

• Troubleshoot evaluation and observability issues: https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/troubleshooting (opens in a new tab)

• Microsoft Foundry architecture: https://learn.microsoft.com/en-us/azure/foundry/concepts/architecture (opens in a new tab)

• Foundry role definitions and agent scopes (opens in a new tab)

• Model quotas and rate-limit diagnostics (opens in a new tab)

• Model retirement schedule (opens in a new tab)

• Cost Management budgets (opens in a new tab)

• Azure AI Search vector ranking (opens in a new tab)

• Search indexer troubleshooting and private execution (opens in a new tab)