Skip to content
FR

Design ALM for AI Business Solutions

Step 19 of 22

AI Agent ALM: Lifecycle Management for Agents, Data, and Models

AI agent ALM governs how agent definitions, grounding data, tools, and models move through development, testing, production, and retirement. For AB-100, this lesson covers Design the ALM process for AI-powered business solutions within Deploy AI-powered business solutions (40–45% in the course outline before the announced November 16, 2026 update). You will design environment-specific deployment, evaluation gates, and recovery procedures for Copilot Studio, Microsoft Foundry, and Dynamics 365.

Learning objectives

After this lesson you can:

  • Design an ALM process for the data used by AI models and agents.
  • Design an ALM process for Copilot Studio agents, connectors and actions.
  • Design an ALM process for Microsoft Foundry Agent Service.
  • Design an ALM process for custom AI models.
  • Design an ALM process for AI in Dynamics 365 apps for finance and supply chain and for customer experience and service.
  • Define an environment strategy that supports all of the above.

1. Why ALM is different for AI solutions

A traditional application changes when someone ships new code. An agent can change its behavior when any of these change: the instructions, the knowledge sources, the model version, a tool or connector, a policy, or the data. An architect therefore treats each of these as a versioned artifact and defines how it moves between environments.

Core principles:

  • Everything that affects behavior is versioned: agent definitions, prompts, topics, flows, evaluation sets, model versions, infrastructure definitions.
  • Separate environments for development, test (or UAT) and production, with different access rights and data.
  • Automate promotion with pipelines instead of manual copy and paste, so deployments are repeatable and auditable.
  • Gate every promotion with tests and evaluations (see the previous chapter) and approvals.
  • Plan recovery per artifact: retain earlier versions, but test whether recovery needs redeployment, a forward fix, endpoint routing, or a data restore. Reverting a definition does not undo completed business transactions.
  • Keep secrets and environment-specific values outside the artifact, set per environment.

2. Environment strategy

Environment Purpose Typical rules
Personal or sandbox Experiments by makers Low-risk, no business data, limited sharing
Development Build and unit test Unmanaged solutions, source control connected
Test or UAT Integration, evaluation and user acceptance Managed solutions, representative non-sensitive data
Production Real users and data Managed solutions, restricted maker access, monitoring

Design questions an architect answers:

  1. How many environments, and in which region, to respect data residency?
  2. Who may build, who may deploy, who may approve?
  3. How are data loss prevention (DLP) policies applied so that connectors allowed in development are not exposed in production by mistake?
  4. How are environment variables and connection references set so the same solution runs against different back ends per environment?
  5. How are different risk levels treated (a personal productivity agent versus an agent that changes financial records)? A zoned governance approach gives low-risk agents a light path and critical agents stronger controls.

3. ALM for data used by AI models and agents

Agents are only as good as their data. Data changes (new documents, updated products, schema changes) can alter answers without any change to the agent, so data needs its own lifecycle.

  • Data sources and ownership: name an owner for every knowledge source and grounding dataset, with a refresh schedule.
  • Separate data per environment: development and test use synthetic or masked data; production data does not flow downward. Do not copy personal data into non-production environments.
  • Schema and configuration as code: Dataverse tables, columns and relationships travel in solutions, not by hand-edited changes in production.
  • Quality gates: validate completeness, freshness, duplicates and access permissions before a new dataset or knowledge source is attached to an agent.
  • Versioning and lineage: record which dataset version trained or grounded which model or agent version, so a bad answer can be traced and a change reverted.
  • Retention and compliance: apply sensitivity labels, retention and regional rules consistently across environments.
  • Reindexing and drift: when sources change, retest representative questions; monitor for data drift after go-live.
  • Permissions: verify the identity and authorization used for each retrieval and tool call. Security roles, assignments, sharing, and connections need deliberate configuration and tests; not all are transported in a solution.

4. ALM for Copilot Studio agents, connectors and actions

Copilot Studio agents are built in Power Platform environments and use Power Platform's solution-based ALM.

Solutions

  • An agent, its topics, tools, flows, connection references and environment variables live in a solution, the container used to transport components between environments.
  • Build in development with unmanaged solutions. Deploy to downstream test, UAT, and production as managed build artifacts. Managed components are not edited directly inside the managed solution, although permitted unmanaged customizations can create layers in the target. Avoid those layers as a routine release practice. Uninstalling a managed solution can remove components and their data; it is not a safe generic rollback.
  • Use a custom solution with its own publisher rather than the default solution, so components are tracked and exported intentionally. Keep related components (agent, flows, connectors, tables) together or in layered solutions with clear dependencies.
  • Copilot Studio lets makers view, export and import solutions, choose a preferred environment and configure pipelines from within the app.

Pipelines and source control

  • Power Platform pipelines automate deployment of solutions from development to test to production, with deployment history and configurable governance such as delegated approvals. Configure the required gates; do not assume selecting a pipeline automatically creates every approval or evaluation. They suit teams that want a guided path.
  • For larger teams, combine Dataverse Git integration or an export to source control with Azure DevOps or GitHub pipelines (Power Platform build tools or CLI). This provides pull requests, reviews, branch strategies and automated checks.
  • Add automated evaluation (test sets) in the pipeline before promotion to production.

Connectors and actions

  • Connection references abstract the connection, and environment variables hold values such as URLs, IDs and thresholds, so the same managed solution works in each environment without editing.
  • Custom connectors (OpenAPI-based) and the APIs behind them have their own lifecycle: version the API, test it, and promote the connector in a solution. Authentication settings, such as the app registration used, differ per environment.
  • Actions and tools (flows, connectors, MCP servers, agent flows) are tested individually, then in the agent. A breaking change to an action's inputs or outputs must be coordinated with the agent that calls it.
  • Run-as identity: decide whether a flow runs with the user's identity or a service identity, and document it, because it changes between environments if connections are not managed.
  • DLP: confirm the production environment's DLP policy allows every connector used before deployment, otherwise the agent fails after import.

Release practices

  • Version the agent and its solution using the platform’s supported scheme (Power Platform solution versions use four numeric parts), with release notes. Copilot can help draft deployment notes in pipelines, but a human reviews them.
  • Use staged rollout: pilot group, then wider audience, with monitoring. Publish only when evaluations and approvals pass.
  • Retain previous build artifacts and rehearse recovery. Power Platform pipelines can redeploy a past successful version when the required configuration permits it, but Microsoft warns of irreversible data loss, including removed tables. Review schema differences, backups, and dependencies first. A forward corrective release may be safer than a downgrade; patches, updates, and upgrades also differ in whether they remove components.

What a solution does not transport automatically

An imported solution is not the whole running service. Copilot Studio ALM guidance identifies Application Insights settings, manual authentication, web-channel security, deployed channels, and sharing as post-deployment considerations. Record target values and owners, apply them through the supported configuration route, and run a smoke scenario with the intended identity before exposing the agent.

Solutions and pipelines do not carry Dataverse table records as ordinary solution components. Move approved configuration or knowledge content through a separate supported data process where required. Connection references identify the required connection, but do not turn a development user’s credentials into a production service identity.

5. ALM for Microsoft Foundry Agent Service

Foundry Agent Service supports portal-authored prompt agents as well as code-based agent development. Its lifecycle combines the agent definition, the model deployment, tools, and the Azure resources around them.

  • Agent versions: each saved version of an agent is immutable; a change creates a new version. This preserves the saved configuration for comparison and version-specific requests. It does not snapshot external knowledge, mutable tools, connections, or model-service availability.
  • Development to production: develop and iterate in a Foundry project, then publish the agent to a stable endpoint or share it through Microsoft Teams, Microsoft 365 Copilot or the agent registry. Publishing creates an agent application for integration; it does not certify that the chosen permissions, network controls, evaluations, capacity, and operational procedures meet production requirements.
  • Environments: use separate Foundry projects (and Azure subscriptions or resource groups) for development, test and production, each with its own model deployments, connections, networking and access control (Azure RBAC and managed identities).
  • Infrastructure as code: define projects, model deployments, connections and networking with Bicep, Terraform or Azure Developer CLI templates so each environment is reproducible.
  • Source control and CI/CD: store agent definitions, instructions, tool schemas, code for hosted agents and evaluation datasets in Git. Use pull requests and pipelines (GitHub Actions or Azure DevOps) for promotion. Foundry does not by itself provide a complete dev-to-test-to-prod promotion pipeline, so the architect designs one around these tools.
  • Evaluations in the pipeline: run built-in or custom evaluators (quality, groundedness, safety, tool-call accuracy) on a fixed dataset before promotion and compare against the previous version.
  • Model lifecycle: model deployments have versions and retirement dates. Plan to test and migrate to a newer model version before the old one is retired, and pin supported versions where predictability matters. Pinning does not override a provider retirement date.
  • Observability: tracing and monitoring (for example through Application Insights) in each environment support the feedback loop covered in the monitoring chapter.
  • Governance: register agents, apply content filters, keep secrets in an approved secret store such as Key Vault, and prefer managed identities for integrations that support them. Check each tool’s authentication requirements.

6. ALM for custom AI models

For custom models (AI Builder models, Azure Machine Learning models, fine-tuned models), use an MLOps approach.

Stage Lifecycle activity
Data preparation Versioned training, validation and hold-out datasets; data quality checks
Training Reproducible experiments; tracked parameters, code version and data version
Validation Criteria defined in advance, compared with a baseline; fairness and robustness checks
Registration Model registry with version, metadata, owner and approval status
Deployment Automated release to a managed endpoint per environment; staged or canary rollout
Monitoring Quality, latency, cost and data or prediction drift
Retraining and retirement Triggered by drift, new data or policy; old versions retired with a rollback window

Additional points:

  • Approvals: a model is promoted only after documented sign-off, including a responsible-AI review.
  • Traceability: keep the link between a prediction, the model version and the data used, for audit.
  • Consumers: agents and flows that call the model reference a stable endpoint or alias, so a model update does not require editing every consumer; contract changes (inputs and outputs) are versioned and tested.
  • Cost and capacity are managed per environment (for example quotas for model deployments).
  • A prebuilt model removes the training lifecycle but not the testing one: provider model updates can change behavior, so re-run evaluations.

7. ALM for AI in Dynamics 365

Dynamics 365 apps include AI features and agents with product-specific lifecycles. Customer-engagement apps use Dataverse solutions; finance and operations has its own application lifecycle plus integrated Power Platform components. Coordinate both rather than assuming one solution contains the entire ERP release.

Finance and supply chain

  • Dynamics 365 finance and supply chain management apps use their own environment topology (sandbox and production environments managed through Power Platform admin center and Lifecycle Services concepts) and deployable packages, together with Dataverse solutions for integrated components.
  • Test AI features (for example Copilot capabilities and agents for finance and supply chain processes) in a sandbox that has a recent copy of configuration, with masked data, before enabling them in production.
  • Check feature management, roles, and admin settings for the specific capability. Pilot where supported and document a fallback. Some features or service updates become mandatory, so do not assume every change can later be disabled.
  • Regression-test business-critical processes (invoicing, purchasing, inventory) after enabling an AI feature or after a service update, because one-version service updates arrive on a Microsoft schedule.
  • Custom extensions, agents and integrations are promoted through the standard release pipeline, with approval from process owners.

Customer experience and service

  • Customer Service, Sales and Contact Center apps are configured with Dataverse solutions. Transport supported solution-aware customizations through managed solutions. Inventory case/routing configuration, knowledge content, channels, sharing, and authentication separately; verify which items need configuration-data migration or post-deployment setup.
  • Promote knowledge articles and knowledge sources deliberately: content that grounds a service agent is part of the release, with an owner and review date.
  • Test channels and handoff (chat, voice, escalation to a human agent) in test environments with test queues before production.
  • Apply role-based access, privacy and regional settings consistently, and enable features first for a pilot team of agents.
  • Monitor adoption and quality (resolution, escalation, feedback) after release and feed findings into the backlog.

8. Putting it together: a release flow

  1. A maker or developer changes an agent in the development environment; the change is committed to source control (solution or Git).
  2. A pull request triggers automated checks: solution validation, prompt and instruction review, security scan.
  3. The pipeline deploys a managed build to test, sets environment variables and connection references, and runs the evaluation set and integration tests.
  4. Results are compared with thresholds and with the previous version; failures stop the release.
  5. Business owners perform UAT; compliance and security approve where required.
  6. The pipeline deploys to production with an approval, for a pilot group first.
  7. Monitoring tracks quality, safety, cost and drift; the feedback becomes backlog items.
  8. If a regression appears, follow the rehearsed recovery path, verify data integrity and outstanding transactions, and record the cause. Choose a forward fix or restoration if reverting the artifact would create additional harm.

9. Worked example

A bank builds a loan-assistant agent. In Copilot Studio it answers policy questions and calls a custom connector to a pricing API; in Foundry a pro-code agent summarizes credit files and uses a fine-tuned risk-explanation model.

  • Environments: three Power Platform environments and three Foundry projects (dev, test, prod) in the bank's required region; developers have no access to production data.
  • Copilot Studio: agent, flows and connection references are in one custom solution; development is unmanaged, other environments get managed builds through a pipeline with approvals. The pricing connector's endpoint is an environment variable.
  • Foundry: Bicep defines each project and model deployment; agent definitions and evaluation datasets are in Git; a GitHub Actions workflow runs evaluators on every pull request and blocks promotion if groundedness or safety falls below threshold.
  • Data: policy documents are a knowledge source with a named owner and monthly review; each new document set is re-evaluated in test before it is published to the production agent.
  • Custom model: registered with a version, validated against a baseline with a fairness review, deployed with a canary rollout and monitored for drift; retraining requires a new approval.
  • Recovery: retain the managed build and previous Foundry version, verify compatible model availability, and rehearse supported redeployment or endpoint routing. Separately plan data restoration and compensation for already-created transactions.

Step-by-step release evidence for the bank

  1. Create a release manifest linking the solution version, Foundry agent version, model deployment, pricing API contract, evaluation set, and approved policy-document revision. Store identifiers and configuration references, not credentials or customer credit files.
  2. Build once and promote the reviewed artifact. Record any environment-specific settings that are applied alongside it. Changing instructions after the test gate creates a new candidate and requires new evidence.
  3. In test, bind connections to test identities and verify both allowed and denied pricing requests. Confirm that the target endpoint is the test API before running write-capable scenarios.
  4. Run the AI agent evaluation and end-to-end testing process. Record blocking failures individually; a high average cannot compensate for unauthorized access or an incorrect business transaction.
  5. Configure production-only items through a controlled procedure: sharing, authentication, channels, telemetry, and connections. Run a restricted smoke scenario and confirm that telemetry reaches the intended monitoring resource with the approved content-logging settings.
  6. Pilot through supported audience or routing controls, with a named owner able to pause exposure. Use agent monitoring, telemetry, and feedback to compare the released candidate with the baseline.

The release record should answer what changed, who approved it, which evidence passed, and what the team will do if a dependency fails. A copied ZIP file alone cannot answer those questions. Review Foundry model deployment and Copilot extensibility when a release changes hosting or tool integration.

Mini case: a version rollback that is not a data rollback

A release adds a review-notes table and users enter notes into it. The agent’s new instructions are faulty, so the team proposes redeploying an older solution that lacks that table. Is retaining the old solution sufficient preparation?

Solution: inspect the supported redeployment behavior, schema delta, dependencies, and backup/restore plan before proceeding. Removing the table can destroy records. A forward release restoring the prior instruction behavior while preserving the schema may reduce disruption. If any external pricing or loan records were already changed, handle them through authorized business correction procedures; reverting the agent configuration does not reverse those effects.

Key terms

  • Managed solution: a downstream Power Platform build artifact with component layering and lifecycle behavior.
  • Connection reference: a solution component bound to a suitable target connection.
  • Immutable agent version: a saved configuration that changes only through creation of another version.
  • Release manifest: the recorded versions and dependencies that define one candidate.
  • Forward fix: a new approved release correcting behavior while retaining required data and compatibility.

Exam tips

  • In Power Platform ALM, build in development with unmanaged solutions and deploy managed solutions downstream; use pipelines for automated, approved promotion.
  • Use environment variables and connection references to avoid editing solutions per environment; check DLP policies in the target environment.
  • Foundry agent versions are immutable; combine Git, infrastructure as code and CI/CD with evaluations as release gates, and plan model-version retirement and external dependencies.
  • Data needs its own lifecycle: owners, masked data in non-production, quality checks, versioning, lineage and drift monitoring.
  • Custom models follow MLOps: versioned data, registry, approval, staged deployment, monitoring, retraining and retirement.
  • For Dynamics 365, test AI features in sandbox, enable them gradually by environment and role, and regression-test critical processes after service updates.
  • Always include rollback and human approval for high-risk agents.

Sources