Skip to content
FR

Responsible AI, Security, Governance, and Compliance

Step 21 of 22

AI Agent Security, Governance, and Responsible AI for AB-100

AI agent security and governance connect identities, data permissions, tool controls, and responsible AI evidence. For AB-100, this lesson covers Design responsible AI, security, governance, risk management, and compliance within Deploy AI-powered business solutions (40–45% in the course outline before the announced November 16, 2026 update). You will assess risks across the complete data flow and select controls whose scope and enforcement can be verified.

Learning objectives

After this lesson you can:

  • Design security and governance for agents built with Copilot Studio, Microsoft 365 Copilot and Microsoft Foundry.
  • Design model security, including protection of model tuning.
  • Analyze solution and AI vulnerabilities, including prompt manipulation, and choose mitigations.
  • Review a solution against the Microsoft responsible AI principles.
  • Validate data residency and data movement compliance.
  • Design access controls on grounding data and model tuning, and audit trails for changes to models and data.

1. A layered view of risk

Agentic solutions add new risks on top of classic application risks. An architect reviews each layer:

Layer Typical risks Example controls
Identity and access Over-privileged agents, shared secrets, anonymous access Agent identities, least privilege, Conditional Access, managed identities
Data and grounding Oversharing, sensitive data in answers, stale content Sensitivity labels, permission-aware grounding, data loss prevention
Model Jailbreaks, harmful output, poisoned tuning data, model theft Content filters, Prompt Shields, protected tuning pipeline, private networking
Tools and actions Unintended actions, excessive autonomy, injection through tool results Narrow tool scopes, human approval for high-impact actions, input validation
Operations No traceability, undetected misuse, drift Audit logs, monitoring, alerting, incident response

The principle is defense in depth: no single control is trusted to stop every attack.

2. Design security for agents

Identity

  • Give each agent its own identity instead of a shared account. Microsoft Entra Agent ID represents agents as identities in Microsoft Entra, so they can be governed with Conditional Access, role-based access, access reviews and lifecycle workflows. Copilot Studio creates Entra Agent IDs for new agents after May 2026; older app-registration identities are being migrated. Manual migration is documented as preview and requires staged channel and connector validation. Identity governance does not automatically change the credentials used by every tool.
  • Prefer supported managed identities, certificates, or federated credentials over stored keys; use an approved secret store such as Azure Key Vault for remaining secrets. Check connector support instead of assuming every connection accepts the same authentication method.
  • Decide for every tool whether it runs with the end user's identity (delegated, so the user's permissions apply) or with the agent's or maker's identity (useful for unattended automation, but the target service must enforce the permitted business scope). Document this choice; it is a frequent source of privilege escalation.

Authentication and channels

  • Require user authentication (for example Microsoft Entra ID) for any agent that accesses internal data. Reserve unauthenticated agents for public, non-sensitive information.
  • Restrict which channels an agent is published to (Teams, Microsoft 365 Copilot, website, contact center) and who can use it.

Least privilege for tools and actions

  • Grant each connector, flow, API or MCP server the minimum scope it needs. A read-only question-answering agent should not hold write permissions.
  • Put human-in-the-loop approval in front of high-impact actions (payments, record deletion, external emails).
  • Validate tool inputs and outputs; treat tool results as untrusted data, not as instructions.
  • Limit autonomous agents with clear triggers, boundaries and kill switches.

Network and platform

  • For Foundry, use private endpoints, virtual network integration and disabled public access where required; use Azure RBAC on projects and resources.
  • In Power Platform, apply data policies (DLP) to control which connectors agents can use, and use environment-level controls such as IP firewall and customer-managed keys where needed.
  • Microsoft Defender for Cloud provides AI security posture recommendations, for example flagging agents with weak guardrails against indirect prompt injection.

3. Design governance for agents

Governance answers: who may build which agents, where, with which data, and how they are inventoried, reviewed and retired.

  • Inventory: keep a registry of agents with owner, purpose, data sources, tools, risk level and status. Microsoft Agent 365, where adopted and configured, provides a central control plane and registry. Keep Power Platform and Microsoft 365 policies in the design; onboarding to a registry does not by itself prove that every required policy is enforced.
  • Zoned governance: classify agents by risk. Personal productivity agents follow a light path; departmental agents need review; enterprise or customer-facing agents need security, privacy and responsible AI approval, managed environments and ALM pipelines.
  • Environment strategy: separate maker sandboxes from governed production environments; use Managed Environments in Power Platform for sharing limits, solution checker enforcement and usage insights.
  • Sharing controls: limit how widely makers can share agents and who can publish to Teams or Microsoft 365 Copilot.
  • Lifecycle: periodic access reviews, ownership transfer when people leave, and retirement of unused agents.
  • Policies and standards: approved models, approved connectors, prompt and instruction standards, and an escalation path. The AI Center of Excellence owns these standards.

4. Design model security

  • Protect the model endpoint: authenticate every call, restrict network access, apply quotas and rate limits to prevent abuse and cost attacks.
  • Content filtering: Review the supported Foundry guardrails for the selected model and deployment. Harmful-content categories, protected-material detection, and prompt-attack detection serve different purposes and can have different configuration requirements. Test thresholds against intended use rather than assuming one filter addresses all risks.
  • Protect model tuning: fine-tuning data is a high-value asset and an attack path. Restrict who can upload training data and start tuning jobs, validate and scan datasets, keep them versioned, and require approval before a tuned model is deployed. Poisoned or unvetted tuning data can embed harmful behavior.
  • Protect model artifacts: store custom models in controlled registries with access control, and track versions so a compromised or faulty model can be rolled back.
  • Version pinning and retirement: know which model version each agent uses and re-evaluate before switching.

5. Analyze vulnerabilities and mitigations, including prompt manipulation

Threat What happens Mitigations
Direct prompt injection (jailbreak) A user crafts input to override instructions or bypass safety rules Prompt Shields for user prompts, robust system instructions, content filters, refusal testing
Indirect prompt injection (cross-domain, XPIA) Malicious instructions hidden in a document, web page, email or tool result are followed by the agent Prompt Shields for documents, treat retrieved content as data, limit tool permissions, require approval for actions, restrict knowledge sources to trusted content
Sensitive data disclosure The agent reveals data the user should not see Permission-aware grounding, sensitivity labels, DLP, output filtering
Excessive agency The agent takes actions beyond its purpose Narrow tools, approvals, rate limits, explicit boundaries in instructions
Data or model poisoning Corrupted knowledge or tuning data changes behavior Content ownership, review before publishing sources, validated tuning datasets
Hallucination presented as fact Ungrounded answers mislead users Grounding, citations, groundedness evaluation, clear disclaimers
Denial of wallet Abuse drives token cost Authentication, quotas, throttling, monitoring of cost anomalies

Prompt Shields analyze supplied text for user-prompt and document-attack patterns. In Foundry, configured blocking can filter detected attacks; the Content Safety API returns detection results that the application must act on. Document-attack detection is off by default in the documented Foundry setup, and tool-response scanning depends on supported integrations. Detection does not establish factual accuracy, source provenance, or authorization. Test the protections in the actual Copilot Studio or Foundry path instead of assuming every retrieved document is automatically scanned and blocked.

Testing completes the picture: run red teaming before release and after major changes. The AI Red Teaming Agent in Foundry automates adversarial probing and reports an attack success rate, including agent-specific risks such as prohibited actions or sensitive data leakage. Attack Success Rate describes the attempted attacks and selected evaluators, not the probability of every future compromise. Review findings manually and add regression cases using the AI agent evaluation process.

6. Review adherence to responsible AI principles

Microsoft's Responsible AI Standard is built on six principles. A solution review asks concrete questions for each:

Principle Review questions
Fairness Do outcomes differ unfairly across groups? Has the training or grounding data been checked for bias?
Reliability and safety Is the agent evaluated against defined quality and safety thresholds? What happens when it fails?
Privacy and security Is personal data minimized, protected and used only for its purpose? Are access rights respected?
Inclusiveness Can people with different abilities, languages and contexts use it (accessibility, voice, localization)?
Transparency Do users know they are talking to AI, what it can and cannot do, and where answers come from (citations)?
Accountability Is there a named owner, human oversight for consequential decisions, and a way to contest or report outcomes?

Practical artifacts: an impact assessment at design time, documented intended uses and limitations (similar to a transparency note), evaluation results for quality and safety, a feedback and incident process, and human review for decisions that significantly affect people (credit, employment, health).

7. Validate data residency and movement compliance

  • Know where everything lives: the Power Platform or Foundry environment region, the location of each knowledge source, where conversation transcripts and logs are stored, and where the model runs.
  • Copilot Studio and Power Platform: inspect the selected model, harness, and feature’s processing terms. The Move data across regions control can require administrator consent, but the current guidance identifies models that do not support this restriction. Bing search, Microsoft 365 services, and EU flex routing also have separate data-flow implications. A cleared checkbox is not universal proof that all processing stays in the environment’s geography.
  • Foundry: check model deployment processing scope, stored data, and every tool separately. A resource’s region alone does not establish inference location; use the supported options described in Foundry deployment and extensibility.
  • Dynamics 365 and Microsoft 365 Copilot: check each app's documentation on data location and cross-geo processing for Copilot features before enabling them.
  • Transfers through tools: an agent that calls an external API or a third-party MCP server can move data outside the boundary even if the platform stays in-region. Include every tool in the data flow diagram.
  • Record the decision and evidence (region settings, contracts, data processing terms) for auditors.

8. Design access controls on grounding data and model tuning

  • Permission-aware grounding: supported user-authenticated Microsoft 365, SharePoint, and Dataverse retrieval uses the configured identity’s permissions. Verify that identity and check custom tools separately. Oversharing occurs when content is already accessible too widely; authentication alone does not correct those permissions. Run oversharing assessments (for example Microsoft Purview Data Security Posture Management data risk assessments) and fix permissions before broad rollout.
  • Sensitivity labels classify and can protect content in supported experiences. Microsoft 365 Copilot grounding guidance describes displaying the most restrictive reference label. Do not generalize that display behavior to every custom agent or tool. Purview DLP for Microsoft 365 Copilot can restrict supported labeled files from processing; Power Platform connector data policies govern a different boundary. Test the required control in the chosen workload.
  • Restrict knowledge sources to curated sites or libraries and exclude stale or confidential content; use restricted content discovery where appropriate.
  • Dataverse security: business units, security roles and column-level security limit what grounding queries can return.
  • Agent-level access: decide who can use each agent and whether its tools use user or agent credentials.
  • Model tuning access: separate the roles of data provider, tuning operator and approver; only approved, labeled datasets may be used, and only a small group can start tuning jobs or deploy tuned models.

9. Design audit trails for changes to models and data

Auditors and incident responders need to answer: who changed what, when, and what effect did it have?

  • Maker and admin activity: Copilot Studio audit events (creating, editing, publishing agents, changing settings) are available in Microsoft Purview audit. Check the documented event and channel coverage rather than assuming every runtime tool call and policy decision is included.
  • Interactions: Purview audit interaction events contain metadata and a transcript reference, not the full conversation text. Transcript access and the applicable eDiscovery, retention, and communication-compliance features need their own permissions, configuration, and coverage checks.
  • Azure resources: Azure Activity Log records control-plane changes to Foundry projects and model deployments; diagnostic logs go to Log Analytics.
  • Models: a model registry records each version, its training data version, evaluation results, approver and deployment history.
  • Data: version and lineage records show which dataset version grounded or trained which agent or model version; changes to knowledge sources follow the ALM process with approvals.
  • Configuration as code: solutions and infrastructure definitions in source control give a reviewable history through pull requests.
  • Retention and integrity: keep logs for the period required by policy, protect them from tampering, and restrict who can read them because they may contain sensitive data.

10. Worked example

An insurer deploys a claims agent in Copilot Studio that reads policy documents, looks up claims in Dataverse and can create a payment request.

  • Security: the agent has its own Entra Agent ID; users sign in with Entra ID; claim lookup runs with the user's permissions; the payment tool requires approval by a claims handler.
  • Prompt manipulation: uploaded customer documents are treated as untrusted; Prompt Shields check them for embedded instructions; red teaming tests injection attempts before go-live.
  • Grounding access: only the curated policy library is a knowledge source; sensitivity labels mark confidential reinsurance contracts, which a Purview DLP rule blocks from agent processing.
  • Residency: document the EU processing commitments of the selected model and harness, the cross-region and flex-routing settings, any search or Microsoft 365 dependencies, and the external fraud-scoring API’s processing and storage. Confirm every path rather than relying on environment location alone.
  • Responsible AI: an impact assessment documents that the agent never rejects a claim on its own; customers are told they are interacting with AI and can reach a human.
  • Governance and audit: the agent is in the enterprise zone with ALM pipelines and managed environments; Purview captures supported maker and interaction events; correlated application/tool logs record the required transaction evidence. Model and data versions are traceable through the agent ALM and release process.

Worked decision sequence: preserve the payment boundary

  1. Inventory the claims agent, its owner, each connection, the permitted claim records, and the payment-request API. Distinguish permission to create a request from permission to execute a payment.
  2. Test two authenticated users with different claim access. Verify retrieval, tool arguments, and backend denials; a successful login is only the start of the authorization check.
  3. Submit an approved test document containing a harmless instruction to ignore payment limits. Inspect whether the configured detection path recognizes it, whether the application blocks or removes it as intended, and whether backend limits remain effective if detection misses it.
  4. Require the handler to review the proposed payee, amount, and claim reference before the request proceeds. The API must validate the approved values and reject a changed amount or duplicate request.
  5. Trace the result across identity logs, supported Purview events, and application transaction records. Verify that the evidence contains no payment credentials or unnecessary personal details.

A longer prompt is insufficient because it does not change the connection’s permissions. A content filter is insufficient because a valid-looking payment request can still be unauthorized. A registry is insufficient because knowing that an agent exists does not constrain its actions. These controls complement enforceable service-side checks.

Mini case: a confidential label on a custom API response

A custom MCP tool returns a contract excerpt using a powerful service connection. The team assumes the agent will automatically inherit Microsoft 365 label display and per-user filtering. Is this assumption safe?

Solution: inspect the actual API authorization, identity, returned fields, and supported data-protection integration. Narrow the service connection and enforce caller-specific access before returning content. A label visible on the original document does not prove that a custom response is filtered or that the same label is shown in every host. Test an unauthorized user and retain the denial as release evidence.

Key terms

  • Delegated identity: a tool acts under the authorized user context supported by that integration.
  • Agent identity: the principal used to identify and govern the agent; tool connections still need inspection.
  • Indirect injection: instructions embedded in retrieved content that attempt to redirect behavior.
  • Data-flow boundary: where prompts, documents, results, logs, or tool arguments cross a processing or storage location.
  • Attack Success Rate: successful attempts divided by attempted attacks for a defined assessment.

Exam tips

  • Give agents their own identities (Entra Agent ID, managed identities) and least-privilege tools; add human approval for high-impact actions.
  • Indirect prompt injection comes from content the agent reads (documents, web pages, tool results); mitigate with Prompt Shields for documents, untrusted-content handling and limited tool permissions.
  • Verify the effective retrieval and tool identities; fix excessive permissions and use supported label/DLP controls for the chosen workload.
  • Verify cross-region model exceptions, flex routing, search, Microsoft 365 dependencies, and external tools; location and one admin toggle are insufficient evidence.
  • Know the six responsible AI principles and a concrete control for each.
  • Audit trails combine Purview audit, Azure activity and diagnostic logs, model registries, data lineage and source-controlled configuration.

Sources