Optimize Microsoft Foundry GenAI for AI-103: Prompts, Cost and Latency

Introduction

Optimizing Microsoft Foundry GenAI means measuring quality, cost and latency together before choosing a prompt, model or orchestration change. This lesson closes AI-103's largest domain, Implement generative AI and agentic solutions (30–35%). Building a working prompt, a RAG pipeline or an agent is only the beginning. The official skills for this part of the exam ask whether you can make a solution better, cheaper, faster and observable once real users arrive: prompt engineering and model parameters; reflection, chain-of-thought evaluation and self-critique loops; tracing, token analytics, safety signals and latency; and orchestrating multiple models, including hybrid designs that combine an LLM with a rules engine.

After this lesson you will be able to tune prompts and sampling parameters for a stated goal, design a self-critique loop without paying for it needlessly, read traces and token metrics to find the real bottleneck, and decide when a rule, a small model or a large model is the right tool for each step.

Learning objectives

• Write system prompts and few-shot examples that produce stable, structured behaviour.
• Choose generation and reasoning controls supported by the specific model and API.
• Design reflection and self-critique loops, and evaluate reasoning quality without trusting raw chain-of-thought.
• Use tracing, token analytics, safety signals and latency percentiles to diagnose and tune a production app.
• Reduce cost and latency with caching, model routing, batching and streaming.
• Orchestrate several models and combine an LLM with a deterministic rules engine.

1. Prompt engineering as an engineering discipline

A prompt is configuration for a probabilistic component. Treat it like code: version it, test it against a dataset, review changes and roll back if quality drops.

For evidence-dependent answers, revisit Foundry RAG design and retrieval evaluation before attributing every defect to the prompt.

1.1 Anatomy of a good system prompt

Principles that matter for the exam

1. Be specific and positive. "Return a JSON object with keys intent and confidence" works better than "don't be verbose".
2. Separate instructions from data. Put retrieved documents and user text in clearly delimited sections, so injected text inside a document is less likely to be treated as an instruction. Delimiters help but never replace safety controls such as Prompt Shields.
3. Prefer structured outputs. When the model supports it, request a JSON schema (structured outputs) so the response is validated against a schema rather than parsed from free text. This reduces format failures on supported model/API paths. Check refusal, truncation and completion status, then validate schema and business rules; a valid JSON shape can still contain wrong facts.
4. Use few-shot examples for format and edge cases, not for knowledge. Pick examples that include a hard case such as an ambiguous input and the correct "unknown" answer.
5. Keep the prompt short enough to be cheap. Input tokens contribute to usage, but eligible cached reads can be discounted. Avoid removing necessary context merely to minimize token count.

1.2 Zero-shot, few-shot and chain-of-thought prompting

Reasoning models (such as the o-series and GPT-5 family models deployed in Foundry) do internal reasoning before answering. You generally should not add "think step by step" to them; instead you control effort with a reasoning-effort setting when the model offers one. Their reasoning tokens count as output tokens for cost and for the context limit, even though the raw reasoning is not returned.

Worked example 1: stabilising a classifier

Scenario. A telecom app classifies customer messages into billing, outage, sales or other. The prompt says "Classify the message." In testing, 8% of responses contain extra words such as "Sure! This looks like billing", which breaks a downstream switch statement. The same message sometimes gets different labels.

Step 1 – diagnose. The missing output contract is visible in the failures. Sampling is a possible source of label variation, alongside ambiguous examples, model/version changes and input differences.

Step 2 – fix the contract. Use structured outputs with a schema with a required string field named label, enum values billing/outage/sales/other, and additionalProperties set to false and an instruction: "Choose exactly one label. Use other if none apply."

Step 3 – fix variability. For a model/API supporting sampling controls, try a low temperature. Use a seed only where supported (not as a general Responses parameter), and do not treat either control as a determinism guarantee. Add three few-shot examples, including one ambiguous message mapped to other.

Step 4 – verify. Run 500 labelled messages and measure label accuracy, schema validity among completed non-refusal responses, refusal rate and incomplete-response rate separately. Do not count an unparsed refusal as an incorrect label.

Why not the alternatives? Switching to a larger model costs more and does not remove the format problem. Post-processing with regex hides failures. Raising temperature makes things worse. The best answer fixes the contract at the source and constrains sampling.

2. Model parameters

Some reasoning models do not accept temperature or top_p; the platform ignores or rejects them. When a question mentions an error after switching to such a model with temperature set, that is the likely cause.

Determinism is best-effort. Even with temperature 0 and a seed, outputs can differ slightly across model versions or infrastructure. Control and record supported model versions and test upgrades; pinning cannot prevent retirement, forced migrations or every service-side change.

For multi-agent variants, compare the Agent Framework orchestration and bounded-autonomy patterns.

3. Reflection, chain-of-thought evaluation and self-critique

3.1 Reflection and self-critique loops

A self-critique loop asks a model (or a second model) to review its own output against criteria and then revise. Common structures:

• Generate → critique → revise: one draft, one critique, one revision.
• Generator and verifier: a second agent or a different model checks the result against rules (facts in the source, schema, policy).
• Best-of-N with a judge: generate several candidates and have a judge pick one.
• Tool-grounded reflection: the critic runs code, a unit test or a search query instead of trusting its own opinion.

Design rules

1. Give the critic explicit criteria (a rubric), not "is this good?".
2. Cap iterations (usually one or two) and define a stop condition such as "no issues found".
3. Prefer external verification (schema validation, a calculator, retrieval check) over asking the model to re-read its answer.
4. Apply loops only where the quality gain justifies the cost, for example contract drafting, not greeting messages.

3.2 Evaluating chain-of-thought and reasoning quality

Visible reasoning can be useful for debugging, but it is not proof that the final answer is right, and a plausible explanation can still hide a wrong step. Good practice:

• Evaluate the final answer against ground truth first.
• Use evaluators that judge reasoning quality only as a secondary signal, such as coherence, relevance, groundedness and, for agents, task adherence and tool-call accuracy.
• Do not show raw chain-of-thought to end users by default; it may leak instructions or confusing partial thoughts. Reasoning models in Foundry may return only a summary of their reasoning, depending on the model.
• Use step-level checks for multistep tasks: did the plan call the correct tool, with correct arguments, in the right order?

In Microsoft Foundry you can run built-in evaluators (for example groundedness, relevance, coherence, fluency, similarity, and agent evaluators such as intent resolution, tool call accuracy and task adherence) on a dataset or on sampled production traces, and add custom evaluators (code-based or prompt-based "LLM-as-judge"). Calibrate an LLM judge against a small human-labelled set before relying on it, and select a judge validated for the rubric, not merely a larger model. Judge outputs can vary and share blind spots with the generator. Distinguish observable answer/tool evidence from a private reasoning trace; never require hidden chain-of-thought as proof.

Worked example 2: a contract summary loop

Scenario. A legal-tech app summarises contracts. Reviewers report that 6% of summaries state an incorrect notice period. Latency budget: 12 seconds. Current latency: 7 seconds.

Option A: add "think carefully" to the prompt. Cheap, but no verification. Option B: a second call where the same model "re-reads" its summary and confirms it. Adds about 5 seconds, but the same blind spot may remain. Option C: a verifier step that extracts the notice period with structured output, then checks that the quoted clause text actually appears in the source (string match plus a short LLM check), and retries once if not.

Evaluate C first. The quoted source check provides external evidence, but a matching string alone does not establish that exceptions and effective dates were interpreted correctly. Measure the verifier and bounded retry: an assumed 3–4 second pass can fit 12 seconds, while a retry can exceed the remaining budget. Use a deadline and a review fallback. A is unverified and B repeats the bias. Run the evaluators on a 200-contract test set and compare groundedness before and after.

4. Tracing, token analytics, safety signals and latency

Connect these signals to Foundry production monitoring, quotas and security.

4.1 What to collect

Use percentiles alongside averages, throughput and error rates. An average latency of 2 seconds can hide a p99 of 20 seconds that users remember.

Operational tips

• Tag each call with metadata (feature, tenant, prompt version, model deployment) so you can slice cost and quality.
• Record prompts and responses in traces only when policy allows; they can contain personal data. Redact before export, then configure sampling and access controls. Sampling reduces volume but does not sanitize sensitive data. Use pseudonymous tenant identifiers rather than personal data in telemetry tags.
• Alert on rate of 429 responses, p95 latency, token spend per day and content-filter block rate.

4.2 Reducing latency and cost

Prompt-cache details. Eligible requests need an identical reusable prefix of at least 1,024 tokens. Stable instructions first and variable input last improve the opportunity for hits; inspect cached_tokens rather than assuming a hit. Current GPT-5.6-and-later Standard paths can charge for cache writes and support model-specific explicit breakpoints, while earlier models use different cache controls. Compare billed reads/writes and retention settings for the actual deployment. A prompt cache reuses computation; a response cache reuses an answer and needs separate authorization and freshness checks.

Standard vs provisioned vs batch. Standard (pay-as-you-go) is flexible but shares capacity, so latency can vary and 429s occur at limits. Provisioned throughput reserves sized capacity for steady workloads; test tail latency and availability. Batch is for offline jobs with separate enqueued-token quota and deployment/residency choices. The documented 24-hour window is a target, not guaranteed completion or automatic cancellation at 24 hours.

Worked example 3: finding the real bottleneck

Scenario. A support copilot has p95 latency of 14 seconds. A trace shows: retrieval 0.6 s, model call 1 (planner) 3.1 s, three sequential tool calls of 1.2 s each, model call 2 (answer) 6.4 s producing about 900 tokens. Average user question needs about 150 words.

Reasoning.

1. The biggest single span is the answer generation (6.4 s) because the output is long. Add a length instruction and the output-token parameter supported by the chosen API/model, and enable streaming so users read while text is produced.
2. The three tool calls are independent. Run them in parallel: 3.6 s becomes about 1.2 s.
3. The planner call is simple routing. Move it to a small model or to rules.

On this request path, ideal parallel execution reduces tool time from 3.6 s to about 1.2 s, saving roughly 2.4 s before overhead. A shorter answer and cheaper planner need measurement. One trace cannot predict the new p95; benchmark the full workload and track time to first token separately from total completion latency. Why not just buy provisioned throughput? Throughput reservation helps queueing and throttling, not sequential tool calls or long outputs; the trace shows neither is the cause.

5. Orchestrating multiple models and hybrid LLM/rules designs

5.1 Why use more than one model

• Cost tiers. A small language model (SLM) or mini model handles classification, extraction and routing; a larger model handles hard reasoning.
• Capabilities. A multimodal model reads images; an embedding model builds the vector index; a speech model transcribes audio.
• Resilience. Use an approved fallback only if error policy, data residency, model/tool capabilities and capacity permit it. Bound retries and prevent duplicate side effects; another deployment in the same exhausted quota pool does not create quota.
• Governance. A separate judge or safety model reviews outputs.

Foundry offers model routing options such as a model router deployment that selects a suitable underlying model per prompt for cost-quality balance. You still must evaluate routing quality on your own dataset, because generic routing does not establish your business policy. Current model-router configurations expose Balanced, Cost and Quality modes and supported model subsets. Restrict the eligible pool to approved capabilities/residency/context sizes, record the actual model returned and reevaluate after changing mode or subset.

5.2 Hybrid LLM and rules engines

Not every decision should be generative. Rules engines (code, decision tables, policy engines) are deterministic, auditable and cheap; LLMs handle unstructured language and fuzzy judgement.

Typical hybrid flow: LLM extracts candidate fields → backend verifies them against trusted records and a versioned policy → rules engine decides (approve, reject, escalate) → LLM drafts a polite explanation using only the rule's decision and reason codes → validator checks the draft does not contradict the decision.

The key principle: the LLM proposes or explains; the rules decide where an approved deterministic policy applies. Deterministic code can still be wrong if inputs, policy data or implementation are wrong; validate all three and preserve human review where required. Never let the model compute a loan limit or apply a discount by itself when a formula exists.

Comparison: choosing an optimisation lever

Common pitfalls

• Changing temperature and top_p together, then being unable to tell which one mattered.
• Assuming a step-by-step instruction improves a reasoning model; it may add unnecessary text or cost without improving correctness.
• Treating the model's own explanation as evidence of correctness.
• Self-critique loops with no iteration cap, so cost and latency explode.
• Looking at average latency and ignoring p95 and p99.
• Logging full prompts with personal data in traces without access control.
• Using a router or an LLM judge without evaluating it on your own data.
• Using an LLM for exact calculations where a rule or a function call is available.
• Forgetting that streaming makes output-side moderation more complex (partial text may already be displayed).

Exam tips

• "Consistent output" → supported sampling settings, schema-constrained output and versioned evaluations; no seed or temperature guarantees identical answers. "Creative variety" → higher temperature.
• "Reduce cost for simple requests while keeping quality for hard ones" → router or cascade.
• "Reduce perceived latency in a chat UI" → streaming. "Reduce cost of a nightly job" → batch.
• "Decision must be auditable and exact" → rules engine; LLM only for language tasks.
• "Where is time spent?" → traces with spans; "what is spent?" → token analytics by tag.
• When a question gives two plausible fixes, prefer the one that attacks the cause visible in the data (trace, metric, evaluator result).

Mini case study: Northwind Travel Desk

Northwind Travel Desk lets employees request trips in free text. Today one large model reads the request, checks the travel policy from a long prompt, calculates the allowed hotel budget and writes the confirmation. Findings after a month: average cost per request is high; 4% of confirmations show a wrong hotel budget; p95 latency is 11 s; the target is 6 s; trace analytics show 70% of requests are simple one-city trips; the policy is a table by city and grade.

Task. Propose a redesigned flow and the metrics to track.

Solution.

1. Extraction: a small model with structured output turns the free text into fields (cities, dates, grade). Use supported sampling controls; employee grade must come from the trusted HR profile, not user free text.
2. Rules engine: hotel budget and approval requirement are computed from the policy table by code, which removes model arithmetic from that step and shortens the prompt; validate city, dates, grade and the policy version before computing.
3. Routing: simple trips take the fast path; multi-city or ambiguous requests go to a larger model with a self-critique step limited to one pass and validated by the rules engine.
4. Confirmation: a small model writes the message from the rule output only; a validator checks that the amounts match the rules output.
5. Latency: stream the confirmation; run calendar and flight-price lookups in parallel.
6. Operations: trace every request with tags (path, prompt version), track p50/p95 latency, tokens per path, wrong-budget rate (with tests for extraction, policy data and implementation errors), groundedness of confirmation text and content-filter blocks; alert on 429s.

Why this works. Deterministic logic owns the numbers, small models handle the 70% easy traffic, the expensive model is used only where it adds value, and measured quality, cost and latency comparisons establish whether the changes improve this workload.

Key terms

• System prompt: instructions that define role, rules and output contract.
• Few-shot prompting: including examples in the prompt.
• Structured outputs: responses constrained to a JSON schema.
• Temperature / top_p: sampling controls for randomness.
• Reasoning effort: setting that controls how much a reasoning model thinks.
• Self-critique loop: a review-and-revise cycle on a model's output.
• LLM-as-judge: a model used to score outputs against criteria.
• Span / trace: timed operation and the tree of operations for a request.
• p95 / p99 latency: the time under which 95% / 99% of requests complete.
• Model router / cascade: logic that selects or escalates between models.
• Provisioned throughput: sized reserved model capacity; validate sustained throughput and tail latency.
• Rules engine: deterministic decision logic, often tables or code.

Summary checklist

• I can write a system prompt with scope, grounding rule, output contract and examples.
• I can pick temperature, top_p, max tokens, seed and stop for a scenario, and know reasoning models differ.
• I can design a bounded self-critique loop with external verification.
• I can evaluate reasoning quality without trusting raw chain-of-thought.
• I can use traces, token analytics, percentiles and safety signals to find a bottleneck.
• I can choose among streaming, caching, routing, batch and provisioned throughput.
• I can split work between LLMs and a rules engine and justify it.

Sources

• Structured outputs with Responses and Chat Completions (opens in a new tab)

• Prompt caching, matching and cache-write pricing (opens in a new tab)

• Batch processing and target completion window (opens in a new tab)

• Model router modes and supported subsets (opens in a new tab)

• How model router chooses models (opens in a new tab)

• Configure model-router subsets (opens in a new tab)

• Model lifecycle and retirement (opens in a new tab)

• Monitoring and sampled evaluation (opens in a new tab)

• Microsoft Learn, AI-103 study guide: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-103 (opens in a new tab)

• Prompt engineering techniques (Foundry Models): https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering (opens in a new tab)

• Structured outputs: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/structured-outputs (opens in a new tab)

• Reasoning models: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/reasoning (opens in a new tab)

• Evaluate generative AI apps in Foundry: https://learn.microsoft.com/en-us/azure/foundry/concepts/observability (opens in a new tab)

• Foundry Models, deployment types and provisioned throughput: https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/deployment-types (opens in a new tab)

• Develop generative AI apps (learning path): https://learn.microsoft.com/en-us/training/paths/create-custom-copilots-ai-studio/ (opens in a new tab)