Choosing Models and Foundry Services

Introduction

This lesson opens the domain Plan and manage an Azure AI solution, which carries 25–30% of the AI-103 skills measured. Before you write a single line of Python, you have to decide which model and which Microsoft Foundry capabilities a solution needs. The exam rewards candidates who choose the simplest combination that meets the requirements for quality, latency, cost, data residency and safety, rather than the most powerful one.

After this lesson you will be able to choose between large language models, small language models and multimodal models, map a requirement to a Foundry service (generation, grounding, vector search, agent workflows, multimodal processing), pick a retrieval and indexing method, and decide which memory, tool and knowledge services an agent needs.

Learning objectives

• Compare LLMs, small language models (SLMs), reasoning models and multimodal models using selection criteria.
• Match requirements to Foundry Tools and to Foundry Agent Service building blocks.
• Choose a retrieval method (keyword, vector, hybrid, semantic ranking) and an indexing strategy for a given content type.
• Choose between short-term thread state, long-term memory, file search, Azure AI Search and web grounding for an agent.
• Justify a design in terms of cost, latency, quality and governance.

1. The Foundry mental model

Microsoft Foundry is the platform for building AI apps and agents on Azure. Keep four layers in mind:

1. The Foundry resource and project. A resource holds shared settings, networking and model deployments; a project is the working container for your agents, evaluations, connections and files. Apps connect to a project endpoint.
2. The model catalog. Models from Microsoft, OpenAI and other providers are listed with model cards, benchmarks and deployment options. You deploy a model to get an endpoint and a deployment name.
3. Foundry Tools. Task-specific services (formerly branded Azure AI services) such as Azure AI Vision, Azure AI Speech, Azure AI Language, Azure Translator, Azure AI Document Intelligence and Azure Content Understanding. They are prebuilt, billed per transaction and usually need no prompt.
4. Foundry Agent Service. A managed runtime for agents: instructions, a model, tools, threads/conversations and runs, with built-in tools such as file search, code interpreter, Azure AI Search, web grounding and your own functions or OpenAPI/MCP tools.

Exam tip: when a question says "prebuilt," "no model training," "extract fields," or "detect language," think Foundry Tools first. When it says "reason over," "generate," "plan," or "call tools," think a deployed model, often wrapped in an agent.

2. Selecting a model

2.1 Large, small and reasoning models

2.2 Selection criteria

Evaluate every candidate against the same checklist: quality on your own task (use your evaluation set, not only public benchmarks), latency (time to first token and total), cost (input, output and reasoning tokens), context window, modalities, tool-calling and structured-output support, regional availability and data residency, quota and responsible-AI requirements.

2.3 Deployment types in brief

The deployment type changes data-processing location, throughput guarantees and price, not the model's behaviour:

Worked example 1: choosing a model for a support triage app

Scenario. Contoso Retail receives 40,000 support emails a day. The app must (a) classify each email into one of 12 categories, (b) extract the order number, and (c) for the 3% flagged as "complaint with legal risk" produce a careful, policy-aware summary for a human. The budget is tight, and classification must finish in under one second.

Reasoning, step by step.

1. Steps (a) and (b) are narrow, high-volume and latency-sensitive. An SLM or a small, cheap general model with structured output fits; a large model would multiply cost by an order of magnitude for no accuracy benefit once you have verified quality on a labelled sample.
2. Step (c) needs nuanced reasoning but only for 1,200 emails a day. A larger or reasoning model is justified here.
3. The decision: route. A small model classifies and extracts; only legal-risk items go to the larger model. If both are in the same Foundry project, the app calls two deployments.
4. Traffic is steady, but 40,000 requests a day is modest, so Global Standard (pay-per-token) is enough; Provisioned throughput would be over-engineering.

Why the alternatives are weaker. Sending everything to the largest model passes the quality test but fails cost. Using only the SLM risks poor legal summaries. Fine-tuning is premature before you have measured a prompt-only baseline.

3. Mapping requirements to Foundry capabilities

A frequent exam pattern: two services can do the job, and the question asks for the one with the least effort. A multimodal LLM can read a printed invoice, but when the requirement is "extract the vendor name, invoice date and line items with confidence scores and bounding boxes," a specialized extraction service is the better choice because it returns structured, validated output.

Worked example 2: invoices versus free-form reports

Scenario. A logistics firm receives (1) standard supplier invoices in many layouts and (2) long narrative incident reports from drivers. Requirements: structured fields from invoices with per-field confidence; plain-English summaries of reports.

Reasoning. Invoices have a known schema, so use an extraction analyzer (Content Understanding or Document Intelligence prebuilt invoice model) that returns fields and confidence. Reports have no schema and need language understanding, so use a language model prompt (or Azure AI Language summarization). Combining both is cheaper and more auditable than pushing everything through one large multimodal prompt.

4. Retrieval and indexing methods

Retrieval-augmented generation (RAG) quality depends more on retrieval than on the model. You need to understand the options in Azure AI Search.

Default recommendation: hybrid search with the semantic ranker for enterprise documents. Use pure keyword for identifier lookups, pure vector only when users search by meaning and identifiers do not matter.

4.1 Indexing choices

• Chunking. Split documents into passages (often a few hundred tokens) with some overlap. Too large and you waste context and dilute relevance; too small and you lose meaning. Respect headings and tables where possible.
• Embeddings. Use the same embedding model for indexing and querying. Changing the model means re-indexing. Vector dimensions affect storage and speed; some embedding models allow shortened dimensions.
• Metadata. Store fields such as department, language, sensitivity and date as filterable attributes so you can apply security trimming and freshness filters.
• Enrichment. Skillsets can run OCR, language detection, key phrase extraction, image captioning and chunking at ingestion time (covered in depth in the retrieval lesson).
• Refresh. Indexers pull from Blob Storage, SQL, Cosmos DB and other sources on a schedule or on demand; choose push (your code uploads documents) when you need near-real-time updates.

Worked example 3: choosing a retrieval method

Scenario. A manufacturer's technicians search a manual library. Queries look like "error E-4471 on press line 3" or "why does the hydraulic pump whine at start-up?" The first style contains exact codes; the second is descriptive.

Reasoning. A vector-only index may rank documents about "pump noise" above the page that contains the literal code E-4471, because embeddings blur rare tokens. Keyword-only misses "whine" if the manual says "squeal." Hybrid handles both, and the semantic ranker improves the order of the top results. Therefore: hybrid + semantic ranker, with metadata filters for the machine model.

5. Agent building blocks: memory, tools and knowledge

An agent is a model plus instructions, tools and state. Choose each deliberately.

5.1 Memory

Rules of thumb: do not stuff the entire history into the prompt, since cost and latency grow and relevant facts get diluted. Summarize or retrieve selectively. Treat stored memory as personal data: apply retention, consent and deletion policies.

5.2 Tools

5.3 Knowledge

Knowledge sources can be indexed (Azure AI Search, SharePoint, Blob) or queried live (web, APIs). Prefer indexed knowledge for stable internal content, because it gives predictable latency, relevance tuning and access control, and live tools for volatile data.

Worked example 4: designing an HR assistant

Scenario. An HR assistant must answer policy questions from 2,000 PDFs, remember each employee's preferred language, and look up leave balances in an HR system. Employees must see only documents for their own country.

Design.

1. Policies: Azure AI Search index (hybrid + semantic), with a country filter field. File search would not scale as cleanly and offers less control over security trimming.
2. Leave balances: a function tool with a strict JSON schema, calling the HR API using the user's identity or an on-behalf-of flow.
3. Language preference: long-term memory keyed by employee ID; the conversation itself uses a thread.
4. Model: a mid-size general model with tool calling. A reasoning model is unnecessary for lookup and answer tasks.
5. Guardrails: content filters and an instruction to refuse questions outside HR.

Common pitfalls

• Choosing the biggest model "to be safe." The exam favours right-sized choices.
• Assuming vector search replaces keyword search. Hybrid is the default for good reason.
• Using web grounding for confidential data requirements.
• Forgetting that changing the embedding model requires re-embedding the whole corpus.
• Putting secrets, or security trimming logic, inside the prompt instead of in the index filter or the tool code.
• Confusing a Foundry Tool (task API) with a deployed model (prompt API).

Exam tips

• Look for the discriminating phrase: "lowest latency," "data must stay in the EU," "structured fields with confidence," "multi-turn memory," "current news."
• When a question lists several valid options, choose the one needing the least custom code and the fewest moving parts.
• Distinguish recall (finding candidates: keyword, vector, hybrid) from precision (ordering candidates: semantic ranker).
• Remember: tools run in your code (function calling) or in the service (file search, code interpreter); the model only requests a function call.

Mini case study

Scenario. Fabrikam Legal wants a research assistant for 300,000 contracts in English, French and German. Lawyers search by clause meaning ("limitation of liability for data breach") and by contract number. Documents are confidential, must stay within the EU, and lawyers should only see matters they are staffed on. The assistant should also draft short summaries and compare two clauses side by side.

Proposed solution.

• Region and deployment: Data Zone (EU) deployments for chat and embedding models, so processing stays in the EU.
• Retrieval: Azure AI Search with a multilingual embedding model, hybrid queries and the semantic ranker; a matterId filter implements security trimming based on the caller's group claims.
• Model: a strong general model for drafting and clause comparison; a smaller model for classification of contract type at ingestion.
• Agent: Foundry Agent Service agent with the Azure AI Search tool and no web grounding.
• Memory: thread state only; no long-term personal memory is required.

Commentary. Keyword matching is required for contract numbers, and vectors for clause meaning, so hybrid is correct. Web grounding is excluded by confidentiality. A Global Standard deployment would violate residency, so Data Zone EU is chosen. Fine-tuning is not needed to start; measure retrieval quality first.

Key terms

• Foundry project: workspace containing agents, connections, evaluations and files, reached through a project endpoint.
• Deployment: a named, configured instance of a model you call.
• SLM: small language model, cheaper and faster, narrower capability.
• Embedding: numeric vector representing meaning.
• Hybrid search: keyword and vector retrieval merged into one ranked list.
• Semantic ranker: model-based re-ranking of top results.
• Function calling: model proposes a tool call; your code runs it.
• Security trimming: filtering results by what the caller is allowed to see.

Summary checklist

• I can choose between LLM, SLM, reasoning and multimodal models using explicit criteria.
• I can explain how deployment types affect residency, throughput and cost.
• I can map requirements to Foundry Tools versus a deployed model.
• I can pick keyword, vector, hybrid or semantic ranking for a scenario.
• I can design chunking, embeddings and metadata for an index.
• I can choose memory, tools and knowledge for an agent and justify them.

Sources

• AI-103 study guide (opens in a new tab)
• Azure AI Apps and Agents Developer Associate certification (opens in a new tab)
• Microsoft Foundry documentation (opens in a new tab)
• Azure AI Search documentation (opens in a new tab)