Multimodal Generation
Generating images, video, and audio is a different engineering problem from understanding them. A generative-media product is an asset-processing pipeline of long-running, non-deterministic, expensive jobs, and most of the work is orchestrating that safely, proving provenance, and evaluating quality you cannot reproduce exactly. This chapter leads with those durable concerns and treats the specific models as a perishable snapshot. Every capability figure below is a reported or vendor claim unless tied to a primary source.
Scoping (read first): this topic is load-bearing for creative, media, marketing, game, and avatar products, and largely noise for backend services, data platforms, RAG, and text-only agents. The one piece that reaches any product touching user-uploaded or AI-generated media is provenance and safety, because the legal obligations attach to generating or distributing synthetic media, not to your domain. If your system never emits a pixel or a waveform, only that section applies.
Production Pipeline Patterns
This is the part that outlives any model.
Chaining is a DAG, not one call. The canonical creative pipeline is a graph of stages, often each a different model from a different vendor: prompt -> image (keyframes/style) -> image-to-video -> lip-sync -> voice/TTS -> music/SFX -> mux. The durable lesson is separation of concerns at the stage boundary, so any stage is swappable, the same chained-versus-single-model tradeoff the voice agents chapter draws. Node-graph tools (ComfyUI) codify this for images and video; treat the workflow graph as versioned code, not UI state.
Put every stage behind an adapter, because providers exit. OpenAI shut down the Sora 2 models and the Videos API on September 24, 2026, six months after a March notice, and its model page says there is no one-to-one replacement; ElevenLabs' release notes record ByteDance retiring Seedance 1.5 Pro on November 11, 2026 (Seedance 2.x is the replacement), Google shut down the Veo 3.1 preview IDs on the Gemini API on October 22, 2026, and OpenAI's original gpt-image-1 snapshot shut down on October 23, 2026. A pipeline that calls one video vendor directly has a hard outage on that date. One that calls a stage interface with two qualified providers has a config change and a regression run.
Async, queue, and long-job handling is mandatory. Image generation is seconds; video generation is tens of seconds to minutes per clip, so no synchronous request survives it. The universal pattern: submit returns a job ID immediately, work runs on a GPU worker pool, and the client learns completion via a webhook callback (with a polling fallback, because webhooks get dropped). Verify webhook signatures, and autoscale on queue depth, not CPU. Because a single video job costs real money and minutes, attach an idempotency key per logical request so a client retry or a redelivered webhook does not pay twice.
Cost and latency control, in rough order of leverage: cache on the full request fingerprint (prompt plus all parameters plus seed plus model version) so identical requests never re-bill; draft cheaply (low resolution, few steps, a small fast model) and re-render only approved drafts at full quality, the single biggest saver in video (Gemini Omni 1.2 Flash lists about $0.023 per second at 360p against about $0.104 at 1080p upsampled and $0.209 at 4K upsampled, a 9x spread on one model, and Runway's Seedance 2.5 Draft Mode, added October 2, sells exactly this: cheaper 480p drafts enhanced after generation); images have the same shape (FLUX 3 Image Klein bills $0.009 per 1k image against $0.048 for FLUX 3 Image, about 5x cheaper, and 17x at 4k, but BFL bills any custom size above 2k as 4k, a cost trap); batch where supported; pick the tier deliberately (turbo and distilled variants trade quality for large latency and cost wins, and vendors now ship the pair explicitly, such as OpenAI's fast gpt-image-2.5-flare and editing-focused gpt-image-2.5-sunburst, now joined by the 2.6 pair with a free complexity dial); and keep warm worker pools, because loading a multi-gigabyte checkpoint can take tens of seconds.
Server-held generation state changes the edit loop. Gemini Omni Flash (1.1 GA August 27, 2026; 1.2 GA November 5) edits video conversationally: each turn references the previous result by previous_interaction_id, so you do not re-upload the clip. That only works with storage turned on, which means the provider holds your intermediate assets, so check retention terms against your content policy. Storage is now the default even if you never use the feature: since October 26, 2026, new paid-tier Gemini API projects store generateContent, streamGenerateContent, and batchGenerateContent requests by default (store=true; existing projects keep their settings), so retention review belongs at project creation; opt out per request with store=false or in Logs and Datasets settings. It also does not replace your own manifest: the provider's interaction chain is not your provenance record.
Asset and prompt management. Treat prompts, negative prompts, seeds, model and version, and all conditioning inputs (control maps, reference images, adapter IDs and weights) as a structured, versioned record stored with every output. You cannot reproduce or debug a generation without the full parameter set, and providers silently change models behind a stable name. This manifest doubles as your provenance log.
Seeds and reproducibility, the hard truth. A seed makes a local, pinned run reproducible, but reproducibility degrades the moment you cross hardware or batch boundaries and is effectively absent on hosted APIs. The root causes are general, not generative-specific: floating-point arithmetic is non-associative, so different GPU kernels accumulate in different orders; batch size changes the kernel strategy (the most common source of numerical noise); and some GPU operations are nondeterministic unless explicitly forced. Practical stance: self-hosted with pinned hardware, fixed batch, fixed library versions, and a seed gives near-reproducible images; any hosted API, and especially video, should be treated as non-reproducible, so design QA around perceptual similarity, not exact match.
Provenance and Safety
The technology churns; the obligation to label and the inability to perfectly enforce it are durable. Two complementary layers, and both are removable.
C2PA / Content Credentials is an open standard that cryptographically binds provenance to an asset (a "nutrition label for media"). A manifest holds assertions about creation and edits, a signed claim, and references to source assets, with hard bindings (cryptographic hashes of the bytes, tamper-evident) and soft bindings (fingerprints or watermarks that survive re-encoding). "Durable Content Credentials" recover a stripped manifest by looking up a surviving watermark or content fingerprint in an external repository, which exists precisely because manifests get separated from assets by re-encoding and screenshots. Adoption is real and growing: major image and video generators now attach C2PA metadata and watermarks, some camera makers sign photos at capture (Apple's Reference Image on the iPhone 18 Pro, reported in September 2026, signs sensor data at capture), and platforms read credentials to apply AI labels. A cautionary example to teach: one camera maker added then suspended C2PA after a signing-key vulnerability, because the trust model is only as good as key custody.
Watermarking (SynthID-style) adds an imperceptible signal to generated media. Google now applies SynthID to every Gemini Omni video clip and to Gemini Live audio. The durable caveat for engineers: invisible watermarks are not robust against a motivated adversary. Peer-reviewed work shows a regeneration attack (add noise, denoise with a diffusion model) provably removes invisible image watermarks below a perturbation threshold while preserving quality, and text watermarks are degraded by paraphrase and back-translation. Audio is no better: 2026 preprints strip AudioSeal, WavMark, SilentCipher, and Perth with off-the-shelf speech enhancement or dedicated removal attacks. Watermarks deter casual misuse and enable platform labeling; they do not stop adversaries. Layer C2PA signed provenance, a watermark that survives metadata stripping, and classifier-based detection, and assume each can be defeated alone.
Safety filters apply input moderation (block disallowed prompts) and output moderation (NSFW and identity classifiers) before returning an asset. The high-severity classes are non-consensual intimate imagery, deepfakes of real people, voice cloning without consent, and copyright or likeness infringement; mitigations include prompt and region blocklists, known-face rejection, consent-gated likeness, rate limiting, and abuse logging tied to the provenance manifest.
The regulatory backdrop (this is the part that reaches non-media products that merely host media) turned concrete in the second half of 2026:
| Rule | Date | What it requires of a generator or platform |
|---|---|---|
| EU AI Act Article 50 | Applies since August 2, 2026; marking grace to December 2, 2026 only for systems placed on the market before August 2 | Providers mark generated output in a machine-readable, detectable way; deployers disclose deepfakes |
| EU Code of Practice on marking and labeling | Final June 10, 2026; voluntary compliance route | At least two marking layers (signed, tamper-evident metadata plus an imperceptible watermark), one layer for free-form text; text under 200 tokens exempt from watermarking; keep existing markings and prohibit removal in your terms; offer a free detection tool; watermark-detection interoperability due February 2, 2027 |
| EU Article 5 ban (Regulation 2026/1744) | December 2, 2026 | Prohibits systems that generate non-consensual intimate imagery of identifiable people or CSAM, including general-purpose generators where that output is reasonably foreseeable and the provider lacks adequate safeguards; prohibited-practice fines reach EUR 35M or 7% of turnover |
| California AI Transparency Act (SB 942 as amended by AB 853) | Operative August 2, 2026 | Latent disclosures in image, video, and audio, plus a free detection tool; capture devices first produced for sale from January 1, 2028 embed latent disclosures by default |
| California SB 1000 | Signed September 30, 2026, effective immediately | Removes the 1,000,000-monthly-user threshold, so smaller generators are now covered; latent disclosures must say whether AI created or altered the content; the detection tool becomes a "disclosure verification tool" and the manifest-disclosure option is no longer required |
| California AB 2713 | January 1, 2027 | Large platforms must detect and display provenance data, let users inspect it, and may not knowingly strip standards-compliant provenance or signatures |
| US TAKE IT DOWN Act (2025) and state likeness laws (Tennessee ELVIS Act and many sexually-explicit-deepfake statutes) | In force | Takedown duties for non-consensual intimate imagery; protection of voice and likeness |
The engineering consequences: watermark and sign every asset (one layer is not enough for the EU Code, and from January 1, 2027 California bars large platforms from knowingly stripping standards-compliant provenance data or signatures); keep red-team evidence that your image and video models refuse NCII before December 2, because a terms-of-service clause is not a "technical safety measure"; and if you redistribute user uploads, plan a provenance-inspection feature for 2027. See AI Governance and Compliance.
Training data provenance is now a procurement question, not only a legal one. Courts are separating whether training is lawful from how the data was acquired: music publishers sued Anthropic on August 28, 2026 over allegedly pirated acquisition of lyrics and sheet music (building on Bartz v. Anthropic, where training was held lawful but piracy-sourced acquisition was not, and which settled for $1.5B), while the US government filed a brief backing OpenAI's fair-use position in NYT v. OpenAI (reported September 2). Music is moving to licensed supply: Suno v6 (September 9) was trained on licensed data from Warner Music Group, BMG, and Believe rather than the data behind earlier versions, and Universal Music Group and ElevenLabs announced a multi-year agreement on September 10. A model's licensing and indemnity status belongs in your vendor review.
Evaluating Generative Quality
Two unavoidable facts: automated metrics are weak, and human preference is the real ground truth but expensive.
Automated metrics and why they are unreliable. FID, the long-standing image metric, contradicts human raters, mishandles distortions, and is biased and unstable at realistic sample sizes; CLIPScore measures text-image alignment but is gameable, and combining weak metrics does not make a strong one. FVD inherits FID's problems and is unstable at the small sample sizes typical of video. For audio, FAD is the FID analogue with the same caveats, and MOS (subjective 1-5) is ceiling-limited now that TTS approaches human quality. Use these as coarse signals, never as correctness.
Human preference is the gold standard. The field ranks image and video models with blind pairwise arenas scored by Elo or TrueSkill over many votes. Treat public arena standings as a coarse signal and validate on your own prompt distribution, the same discipline the benchmarks chapter prescribes for LLMs.
Regression testing for non-reproducible pipelines is the durable practice, because providers silently update models behind stable names. Build a versioned golden prompt set covering your real use cases and failure modes; on hosted APIs drop exact match and compare distributions of metric scores; use perceptual similarity (perceptual hashing, SSIM, CLIP-embedding distance to a baseline) rather than equality; add a VLM-as-judge on a rubric (prompt adherence, artifacts, brand safety) for what hashes cannot see; track judge and metric distributions over time and alert on drift, which often means the upstream provider changed the model; and pin to dated model versions wherever the API allows. A November 2026 example of why: xAI retired grok-imagine-image-quality and grok-imagine-image-pro and now serves both names with grok-imagine-image-2.0-low, the low-quality setting, at a lower per-image price and with no request change, so "pro" traffic silently became low-quality output under the old names; set quality explicitly and watch output-score drift. Remember that the judge drifts too: on September 25, 2026 OpenAI fixed an image-encoding bug in GPT-6 Sol and Luna under unchanged model IDs and told customers to rerun image evaluations, so a VLM judge on those models scored through a broken path for its first days. Gate releases on statistical significance, not raw deltas, and version prompts independently of models.
The Model Landscape
Kept light and dated, because it churns monthly and much of the circulating spec-sheet detail comes from low-quality sources.
- Image. OpenAI's GPT Image 2.5 (September 8, 2026) ships as a pair:
gpt-image-2.5-flarefor fast everyday generation andgpt-image-2.5-sunburstfor editing precision, at GPT Image 2's token rates ($5 per 1M text input, $8 per 1M image input, $30 per 1M image output) with newxhighandmaxquality settings. On November 4 OpenAI addedgpt-image-2.6-sunburstandgpt-image-2.6-flareat the same rates, with acomplexityfield (low, medium, high; default high; no separate fee), SVG and animated GIF output, and markup-guided edits that also work on 2.5; the pricing page lists Batch at half price for 2.5 but no Batch rate for 2.6, so batch-heavy pipelines should stay on 2.5 for now.gpt-image-1.5andgpt-image-1-miniretire December 1, 2026, and 2.6 is a same-price target. Midjourney V8.1 has been its default since June 11, 2026. The open-weight frontier is led by the FLUX family; Black Forest Labs launched FLUX 3 Image at the start of October 2026 and published open weights for FLUX 3 Image Klein (5.6B) on Hugging Face on October 13, gated behind a custom FLUX license; Klein reached the BFL API on October 15 as the draft tier priced above. The Stable Diffusion lineage remains the broad tooling base. The licensing gotcha to teach: open weights does not mean open use, since several FLUX variants (thedevline) are non-commercial and require a paid license for commercial work, while the distilledschnellvariant is permissively licensed (Apache-2.0); check the license of any new open release before building on it. Conditioning composes: ControlNet (structural control from pose, depth, edge, or segmentation maps), reference adapters (use an image as a prompt for style or likeness), inpainting and outpainting, and regional prompting; the production combo is an identity adapter plus structural control plus a text theme in one graph. LoRA is the default for style or subject personalization. - Video. Billing is per second of output, generation takes tens of seconds to minutes, and outputs are non-reproducible, which forces async jobs, draft-then-render, and hard per-user cost caps. A live example of how fast this section decays: OpenAI's Sora 2, a leader a year ago, is gone (the consumer Sora app closed April 26, 2026, and the API shut down September 24), and Google launched the Gemini Omni brand rather than a Veo 4. Verify a video model's status before building on it rather than trusting any snapshot, this chapter included. Strong contenders from ByteDance (Seedance 2.x, the replacement for the retiring 1.5 Pro), Kuaishou, Runway, and others sit alongside a fast-improving open-weight tier.
| Model | Status | Price per output second | Notes |
|---|---|---|---|
| Gemini Omni 1.2 Flash | GA November 5, 2026 | About $0.07 at 720p ($12.00 per 1M video output tokens at 5,792 tokens per second); about $0.023 at 360p, $0.104 at 1080p upsampled, $0.209 at 4K upsampled | Clips up to 15 s, conversational editing, up to 6 reference images; paid-tier rate limits doubled; SynthID on every clip |
| Gemini Omni 1.1 Flash | GA August 27, 2026; auto-routes to 1.2 Flash, at 1.2 rates, on December 3, 2026 (scheduled) | About $0.10 at 720p ($17.50 per 1M) | Pinned 1.1 pipelines get a forced model change: regression-test on 1.2 before December 3 |
| Veo 3.1 Standard / Fast / Lite | Gemini API preview IDs shut down October 22, 2026 (replacement gemini-omni-1.2-flash); on Vertex AI, veo-3.1-generate-001 and veo-3.1-fast-generate-001 are GA (retirement "May 4, 2027 or later") and Lite is Preview, us-central1 only | Check Vertex pricing | Native synchronized audio; Vertex IDs only from now on |
| FLUX 3 Video | BFL API preview since August 4, 2026 | $0.17 at HD, audio included | Text- or image-to-video |
| Sora 2 / Videos API | Shut down September 24, 2026 | - | No one-to-one OpenAI replacement; calls fail |
- Audio. Voice and TTS: ElevenLabs is the reference (Eleven v4, September 28, 2026, clones a voice from 10 seconds of audio and covers 90+ languages), and Gemini 3.8 Flash TTS went GA September 22 at promotional prices that double on January 1, 2027 (since November 4 it also remixes custom voices into a reusable
voice_ID, so voice-likeness consent applies to hyperscaler TTS too). Music: Google's Lyria 3.5 went GA in the Gemini API on September 3 at $0.08 per full song (Lyria 3 clips are $0.04 per 30 seconds), and licensed supply is replacing the litigation-era default (see above). Composition with video mirrors the voice chapter's split: native joint audio-video gives the best synchronization with the least control, while a cascaded approach (generate silent video, then add TTS and music and lip-sync separately) gives independent control and dubbing at the cost of effort, which is why most production dubbing pipelines stay cascaded. - Real-time avatars. Voice agents are gaining faces: Google's Gemini Live Avatar reportedly went GA in Gemini Enterprise on September 25, Meta announced (but has not shipped) a Muse Realtime Avatar model, and Tavus reports that 48% of participants in its own study believed its Griffin model was a real person after a one-minute call (vendor-reported). Every disclosure and deepfake rule above applies to a customer-facing avatar.
Interview Questions
Q: Design the backbone of a service that turns a script into a narrated, music-backed video. What are the hard parts?
Strong answer: The backbone is an asynchronous DAG of stages, each a swappable model behind its own adapter: prompt to keyframe images, image-to-video for motion, TTS for narration, a music model for score, and lip-sync to bind voice to video, then a mux step. Because video generation takes tens of seconds to minutes, every stage is a job: submit returns an ID, work runs on an autoscaling GPU worker pool scaled on queue depth, and completion fires a webhook with a polling fallback. The hard parts are cost, reliability, and vendor churn. Cost: cache on the full request fingerprint, draft on a cheap tier (Gemini Omni 1.2 Flash at 360p costs about a fifth of its 1080p rate per second) and only re-render approved drafts at full quality, and cap per-user spend, since each retry costs real money. Reliability: attach idempotency keys so a retried or redelivered job does not pay twice, and store a full manifest of prompt, seed, parameters, and model versions per asset, both for debugging and as the provenance record. Vendor churn: OpenAI shut down the Sora 2 API in September 2026 with no replacement, so I keep two qualified providers per expensive stage. Finally, compliance is part of the backbone: input and output moderation with documented NCII safeguards (the EU ban applies from December 2, 2026), plus both C2PA credentials and a watermark on every asset, because the EU Code of Practice expects two marking layers and California's AI Transparency Act requires latent disclosures in generated video and audio. Signed provenance also survives distribution better now that California's AB 2713 bars large platforms from knowingly stripping it from January 1, 2027.
Q: How do you regression-test a generative pipeline when outputs are not reproducible?
Strong answer: You give up exact match and test perceptually and statistically. I keep a versioned golden prompt set covering real use cases and known failure modes, and run it on every change. Where reproducibility holds, self-hosted with pinned hardware and seeds, I can use seed-locked checks; on hosted APIs I assume non-reproducibility, so I compare distributions of metric scores rather than single outputs and use perceptual similarity, perceptual hashes, SSIM, and CLIP-embedding distance to a baseline, plus a VLM-as-judge on a rubric for prompt adherence and artifacts. I track those distributions over time and alert on drift, because a sudden shift usually means the provider silently updated the model behind a stable name, so I also pin to dated model versions when the API allows, and I watch vendor changelogs for fixes that change my judge's behavior under the same ID. Releases gate on statistical significance, not raw deltas, and I version prompts independently of models so I can attribute a regression to the right change.
References
- C2PA specification (Content Credentials)
- "Invisible Image Watermarks Are Provably Removable Using Generative AI" (NeurIPS 2024, arXiv:2306.01953)
- Google DeepMind, SynthID
- "Rethinking FID: Towards a Better Evaluation Metric for Image Generation" (CVPR 2024) arXiv:2401.09603
- Black Forest Labs, FLUX and its licensing
- EU AI Act Article 50 (transparency for generated content), the Commission's Article 50 guidelines, and Regulation (EU) 2026/1744
- California, AB 2713 bill text
- Google, Gemini Omni video generation and Gemini API pricing
- OpenAI, deprecations, GPT Image 2.5 Sunburst, and the API changelog (GPT Image 2.6, November 4, 2026)
- Google, Gemini API deprecations and changelog; Black Forest Labs, release notes; xAI, release notes
Previous: Real-Time Voice Agents
From the AI System Design Guide by Om Bharatiya · MIT License ·view source