Image and Video Generation for AI-103: GPT Image, Masks and Sora Jobs

Introduction

Image and video generation in Microsoft Foundry requires a supported model, correctly formatted media and an application that handles asynchronous work safely. This lesson opens AI-103's domain Implement computer vision solutions (10–15%). The official skills for the generative half of the domain are: generating images from text and from reference media, generating video, editing images with inpainting and mask-based changes, editing generated videos, and using the generation and editing controls that shape results. In practice you will deploy an image or video model in Microsoft Foundry, call it from Python, handle asynchronous video jobs, and apply the safety controls that visual generation requires.

After this lesson you will be able to choose between image and video models, write requests that use size, quality, count and format controls, build an edit request with an image and a mask, run a video generation job from submission to download, and plan the moderation and provenance steps that surround these calls.

Learning objectives

• Select a model family for text-to-image, image editing, text-to-video and video editing.
• Call the image generation and image edit APIs with the right parameters.
• Explain how a mask drives inpainting, and prepare a correct mask.
• Run an asynchronous video generation job and edit or remix a generated video.
• Apply content filters, rate limits, cost controls and provenance to generated media.
• Troubleshoot typical failures: blocked prompts, wrong size, weak edits and timeouts.

1. The model landscape

Foundry hosts several visual generation models. Names and availability change quickly and differ by region, so on the exam focus on capabilities rather than memorising a version list; always confirm availability in the model catalog and the documentation before design decisions.

Decision guide

• Need a picture from a description, or a modified picture → image model.
• Need motion, a clip from a prompt, a still image or an existing clip → video model.
• Need to analyse an existing image or video → multimodal model or Azure Content Understanding, not a generation model.
• Need exact brand logos, legible long text or legal accuracy → generation alone is risky; plan human review or composite the exact assets in code.

2. Generating images from text

Check Foundry deployment setup and managed-identity access before selecting the model endpoint.

2.1 The request

This image example uses the documented Azure OpenAI image API, an eligible GPT-image deployment, openai and azure-identity packages, and an identity with the required inference permission. Set AZURE_OPENAI_ENDPOINT to the resource endpoint and AZURE_IMAGE_DEPLOYMENT to the deployment name. Confirm region, access registration and model support before calling; lock the SDK versions used by your application.

import os
import base64
from pathlib import Path
from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
    credential, "https://cognitiveservices.azure.com/.default"
)
client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_version="2025-04-01-preview",
    azure_ad_token_provider=token_provider, timeout=120.0, max_retries=0,
)
result = client.images.generate(
    model=os.environ["AZURE_IMAGE_DEPLOYMENT"],
    prompt="An illustrated lighthouse at dawn, teal and orange palette, no text",
    size="1024x1024", quality="medium", output_format="png", n=1,
)
if not result.data or not result.data[0].b64_json:
    raise RuntimeError("No generated image was returned.")
Path("lighthouse.png").write_bytes(base64.b64decode(result.data[0].b64_json))

Points to notice: the model argument is the deployment name; authentication uses a Microsoft Entra token rather than a stored key; GPT-image models return base64 image data while legacy samples may show expiring URLs, so save the actual bytes with access controls rather than treating a URL as permanent storage. DefaultAzureCredential is a chain: verify the selected principal and explicitly select managed identity in production where appropriate. Close the client and credential at application shutdown.

2.2 Generation controls

2.3 Prompting for images

A prompt that reads like a creative brief gives the most control:

1. Subject: what is shown.
2. Setting and composition: foreground, background, viewpoint.
3. Style: photo, illustration, flat vector, watercolour.
4. Lighting and colour: palette and mood.
5. Constraints: "no text", "leave the top third empty for a headline".

Keep text-in-image short; long passages and small type are the most common artefact. For consistent brand output, generate backgrounds and add exact text, logos and legal lines in code afterwards.

Worked example 1: product banners at scale

Scenario. A shop needs 40 banner backgrounds per week, 1536×1024 landscape, with space on the left for a headline. Budget matters; designers pick favourites.

Step 1 – model and call. Use an image model deployment with the landscape size that the model supports, quality="low" for drafts, n=2 per prompt. Step 2 – prompt. "Wide product backdrop, soft pastel gradient, left third empty, no text, no logos." Step 3 – pipeline. Generate drafts at low quality, let designers choose, edit or refine the chosen reference at supported high quality, then review again; a fresh generation need not reproduce the selected draft, then add the headline in code. Step 4 – controls. Content filter on, enforced application usage limits plus a per-day budget alert, store prompt, deployment, timestamp and output hash for audit.

Why not always high quality with n=4? It multiplies cost and latency for images that will be discarded. Why not ask the model to draw the headline? Text rendering is less reliable than rendering it with a font in code.

3. Editing images: reference images, inpainting and masks

3.1 Image-to-image and reference media

The image edit operation takes one or more input images and a prompt describing the change. Typical uses: restyle a photo, combine several reference images (for example a product and a scene), extend or clean up a picture. Your prompt should state what to keep ("keep the product shape and label unchanged") as well as what to change.

3.2 Inpainting with a mask

Inpainting requests a localized edit; it does not guarantee that every other pixel stays unchanged. You send the original image, a mask and a prompt. The mask is an image of the same dimensions as the source in which the area to be edited is marked as transparent (alpha = 0) and the area to preserve is opaque. The model attempts to fill the marked area to match the prompt and surrounding context; inspect the whole result for drift.

with open("room.png", "rb") as image, open("sofa_mask.png", "rb") as mask:
    edit = client.images.edit(
        model=os.environ["AZURE_IMAGE_DEPLOYMENT"],
        image=image,
        mask=mask,                       # transparent where the edit should happen
        prompt="Replace the sofa with a green velvet sofa, same position and lighting",
        size="1024x1024", output_format="png", n=1,
    )
if not edit.data or not edit.data[0].b64_json:
    raise RuntimeError("No edited image was returned.")
Path("room-edited.png").write_bytes(base64.b64decode(edit.data[0].b64_json))

Mask checklist

• Same width and height as the source image.
• PNG with an alpha channel; the transparent area is what will change.
• Slightly larger than the object to be replaced so edges blend, but not so large that unrelated areas change.
• Masks guide the model; they are not pixel-perfect guarantees, so verify that preserved areas did not drift, especially faces and text.

For a strict unchanged-outside-region requirement, composite the approved generated region onto the original decoded image and save losslessly, then compare the decoded outside pixels. A prompt, mask or retry alone cannot guarantee exact preservation. This also changes the media artifact: preserve the original and record the transformation in its provenance.

Worked example 2: fixing a failed edit

Scenario. A developer masks a lamp in a living-room photo and asks for a "modern floor lamp". The result changes the sofa colour too and the lamp is placed in a different spot.

Diagnosis. (1) The mask was drawn with a soft, very large brush and covers part of the sofa. (2) The prompt did not say where the lamp should be or what to preserve. Fix. Redraw the mask tightly around the lamp with a small margin, export as PNG with transparency in the lamp region, and use the prompt "Replace only the lamp with a slim black floor lamp in the same position; keep everything else unchanged". Why not increase quality or n? More images do not fix a wrong mask. Why not remove the mask? Without a mask the model may re-render the whole picture.

4. Generating and editing video

Video generation models such as Sora create short clips from text, from a starting image, or from an existing video. Because rendering takes time, the API is asynchronous.

4.1 The job lifecycle

1. Submit a generation job with the prompt and settings (duration, resolution or size, and optional input image or video, depending on model).
2. Poll with a bounded deadline until the job is completed, failed or cancelled where supported. Persist the returned job ID; a local polling timeout does not cancel server work.
3. Download the video file from the completed job, and store it in your own storage; do not rely on the service to keep it indefinitely.

import time
from openai import OpenAI

# Uses the same authorized credential, but the v1 endpoint/client for video.
video_tokens = get_bearer_token_provider(credential, "https://ai.azure.com/.default")
video_client = OpenAI(
    base_url=os.environ["AZURE_OPENAI_ENDPOINT"].rstrip("/") + "/openai/v1/",
    api_key=video_tokens, timeout=60.0, max_retries=0,
)
job = video_client.videos.create(
    model=os.environ["AZURE_VIDEO_DEPLOYMENT"],
    prompt="A stylized animated aerial view of an imaginary harbour at sunrise",
    size="1280x720", seconds="8",
)
print("Persist this job ID before continuing:", job.id)
deadline = time.monotonic() + 20 * 60
while job.status in ("queued", "in_progress"):
    if time.monotonic() >= deadline:
        raise TimeoutError(f"Polling deadline reached; retain job ID {job.id} for reconciliation.")
    time.sleep(20)
    job = video_client.with_options(max_retries=2).videos.retrieve(job.id)
if job.status != "completed":
    raise RuntimeError(f"Video generation ended with status {job.status}; inspect the protected job record.")
video_client.with_options(max_retries=2).videos.download_content(job.id, variant="video").write_to_file("harbour.mp4")

Use an OpenAI Python SDK exposing the videos API and an eligible Sora 2 deployment. The eight-second, 1280x720 example is supported in the documented schema. Production workers also need bounded transient-error retries, durable status updates and reconciliation if submission times out; blindly repeating create can create another billable job. Automatic SDK retries are disabled for create in this example; idempotent status/download calls use a bounded retry allowance. Close the video client at shutdown.

4.2 Controls and limits

4.3 Editing generated videos

Video editing in this family is done by sending another generation request that references an earlier video and describes the change:

• Remix: take a completed video and apply a targeted edit ("same scene, change the weather to rain"). Keep edits small and specific; large changes behave like a new generation.
• Mask-based video changes: do not assume the image-mask contract also exists on the documented Sora 2 API. The current guide documents reference inputs and remix, not a pixel-mask video inpainting request. Use a separately supported editor/compositing workflow for a strict regional edit.
• Stitching: generate clips through supported calls and join them in an external video editor. Validate transitions; reusing a style description or reference does not guarantee temporal or identity consistency.

For consistency across clips, reuse the same descriptive "style block" in every prompt (camera, lens look, palette, character description) and keep reference frames.

Worked example 3: a marketing clip pipeline

Scenario. A company wants a 24-second product teaser made of three 8-second shots, reviewed by a human before publishing. Each generation job may take minutes. Web users must not wait on an open HTTP request.

Design.

1. A web API accepts the request, validates and filters the prompt, stores a job record and returns 202 Accepted with a job id.
2. A background worker (a queue-triggered function or container job) submits the three video jobs, using the same style block, and polls for completion with back-off. It uses managed identity with the correct endpoint audience and operation role; cap concurrent jobs to the deployment quota.
3. When each job completes the worker downloads the clip to Blob Storage and records deployment/model, supported parameters, request/job IDs, timestamps and authorized prompt/reference identifiers. Do not assume the video API supports seed.
4. A reviewer approves or asks for a remix of a specific shot ("same shot, brighter lighting").
5. A final step stitches the clips and attaches provenance metadata.

Why not hold the HTTP request open? Jobs outlive typical timeouts. Why not generate all 24 seconds in one job? A single 24-second request is not supported by the documented request table. Shot-level edits may reduce regenerated duration; compare actual model billing and review quality.

5. Safety, provenance and operations for generated media

Apply the Responsible AI guardrails and approval controls to the specific supported media path.

• Input and output filtering. Foundry content filters evaluate prompts and generated images for categories such as hate, sexual content, violence and self-harm with severity levels; blocked requests return an error you must handle gracefully, with a friendly message and no retry loop on the same prompt.
• Prompt attacks. Treat user text as untrusted. For app-generated prompts that include user content, consider Prompt Shields.
• Copyright and likeness. Do not ask for living people's likeness or protected characters unless your policy and the service terms allow it; document usage rights.
• Provenance. Supported model/output paths can attach signed Content Credentials (C2PA) or watermark signals. Verify the selected model and preserve the original artifact/manifest. Re-encoding or compositing may remove or invalidate a manifest; a visible disclosure label is useful but does not restore a signature. Provenance records origin, not truth, ownership or safety.
• Cost and quotas. Image pricing can involve input/output image tokens and quality/size; video is model-specific per-second billing. Enforce application spend/usage limits: Cost Management budget alerts do not stop generation. Honor retry headers and bounded backoff for 429, while reconciling uncertain creates to avoid duplicate jobs.
• Privacy. Reference images may contain personal data. Store them with access control and delete them according to retention rules.
• Human review. For public-facing media, add a review step and keep an audit trail of prompts and outputs.

For analysis rather than generation, continue with multimodal visual understanding and safety.

Comparison: which operation?

Common pitfalls

• Using the base model name instead of the deployment name in model.
• Supplying a mask with different dimensions from the image, or without transparency.
• Expecting a mask to protect faces and text perfectly.
• Keeping an expiring image URL instead of saving the file.
• Blocking a web request while a video job runs.
• Retrying a blocked prompt in a loop instead of returning a clear message.
• Requesting long legible text or exact logos from the model.
• Forgetting cost multipliers: n, quality and resolution all scale price.
• Stripping Content Credentials during post-processing.
• Using a generation model to analyse an image.

Exam tips

• "Change only the sofa" → inpainting with a mask. "Change the style of the entire photo" → edit without a mask.
• "Video takes minutes; users must not wait" → asynchronous job with polling or a queue and a background worker.
• "Edit an existing generated clip slightly" → supported remix by completed video ID; a masked video API must be verified separately.
• "Prompt is blocked" → content filter result; adjust the prompt or handle the error, do not disable safety.
• "Authenticate without keys" → Microsoft Entra ID with the endpoint-specific token audience and required operation role; image and v1 examples can use different documented scopes.
• "Record AI-generated origin" → verify supported Content Credentials/watermarks and add disclosure; a missing manifest does not prove human origin.

Mini case study: Fabrikam Furniture Studio

Fabrikam wants a web tool where sales staff upload a customer's room photo, mark a region (for example the sofa) and see the room with a catalogue sofa, plus a 6-second "walk-through" clip for the best result. Requirements: only the marked region may change, results in 20 seconds for images, video can be slower, no keys in code, unsafe prompts blocked, AI-generated outputs labelled, and spend capped.

Task. Propose the design.

Solution.

1. Image path: the browser sends the photo and region; the API creates a PNG mask with the same dimensions (transparent over the region, with a small margin), then calls the image edit operation with the photo, mask and prompt "Replace only the marked sofa with catalogue item X, same position and lighting; keep everything else unchanged". Output is saved to storage; the application composites only the approved edited region onto the original and verifies unchanged decoded pixels outside it. Store the result losslessly and recheck catalogue-product accuracy; the model alone cannot meet the exact-pixel requirement.

2. Video path: a "Create walk-through" button queues a job; a worker first verifies the model accepts the reference and content. Current Azure Sora 2 restrictions can exclude photorealistic room inputs; offer a compliant stylized preview or a separately rendered camera animation instead of promising a realistic walkthrough. Where generation is supported, request a supported duration (for example eight seconds), then trim externally to six seconds, poll with a deadline and store the result. Users never wait on an open request.

3. Identity: the API and worker use a managed identity with the minimum Foundry role; no keys are stored.

4. Safety: content filters on both deployments; user text goes through input validation and prompt-attack checks; blocked results show a neutral message.

5. Provenance: retain original Content Credentials where present, record compositing/trimming and verify the final provenance chain; add an "AI generated" caption. The 20-second image requirement is an application latency target, not a model guarantee. Benchmark the selected deployment and return progress through an asynchronous path if the target cannot be met; do not promise an exact product match or turnaround from the prompt alone.

6. Cost: medium quality for previews, high only on "Finalize", per-user daily limits, enforced application spend limits, budget alerts and bounded retry handling.

Key terms

• Deployment name: the name you give a deployed model, used in API calls.
• Inpainting: prompting a localized regeneration with a mask; exact preservation needs application controls.
• Mask: an image marking which pixels may change; transparent pixels are edited.
• Outpainting: extending an image beyond its original borders.
• Image-to-video: generating a clip from a still image.
• Remix: a targeted edit of a previously generated video.
• Asynchronous job: work that is submitted, polled and downloaded later.
• Content Credentials (C2PA): signed metadata recording that media is AI generated.
• Content filter: Foundry safety classifier on prompts and outputs.
• Style block: reusable prompt text that keeps clips visually consistent.

Summary checklist

• I can choose between image generation, image edit, video generation and a multimodal analysis model.
• I can set size, quality, count and format controls and explain their cost effect.
• I can prepare a correct mask and use inpainting.
• I can run a video job through create, poll and download without blocking users.
• I can remix a completed video and distinguish it from a separately supported masked editing operation.
• I can apply content filters, provenance, keyless authentication and cost limits.

Sources

• Provenance detection limits (opens in a new tab)

• Classic Content Credentials behavior (opens in a new tab)

• Microsoft Learn, AI-103 study guide: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-103 (opens in a new tab)

• Sora video generation overview: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/video-generation (opens in a new tab)

• Image generation with Foundry Models: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/dall-e (opens in a new tab)

• Generate images with Microsoft Foundry (learning module): https://learn.microsoft.com/en-us/training/modules/generate-images-azure-openai/ (opens in a new tab)

• Generate videos with Microsoft Foundry (learning module): https://learn.microsoft.com/en-us/training/modules/generate-video-with-foundry/ (opens in a new tab)

• Content filtering in Foundry: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/content-filter (opens in a new tab)

• Content provenance support for Foundry models: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/content-safety/provenance-disclosure (opens in a new tab)