Generative AI
Text, image, video, audio and code generation — the model families, design patterns and how they are built.
- lessons
- 8
- steps
- 70
- time
- 93 min
Generative AI Foundations
Generative models learn the distribution of their training data and sample new content from it — one idea, five modalities, four architectures and one training recipe.
- What “generative” means: learn a distribution, then sample from it
- The five output modalities and the four core architectures
- How a Transformer's self-attention sees a whole sequence at once
- Why every modern model is pre-trained first and fine-tuned second
Language Model Families
How text generation works — next-token prediction — and the families built on it: encoder-only BERT, encoder-decoder T5, the decoder-only GPT line through ChatGPT, and open LLaMA.
- How a language model generates text one token at a time
- Encoder-only, decoder-only and encoder-decoder — and what each is good at
- What changed from GPT-3 to GPT-4 to ChatGPT
- When an open model like LLaMA beats a proprietary API
Prompting, Fine-Tuning and RLHF
Four levers for getting what you want from a pre-trained model — zero-shot instructions, few-shot examples and prompt patterns, fine-tuning on your data, and the three-stage RLHF pipeline.
- Zero-shot vs few-shot: instructions alone vs examples in the prompt
- Reusable prompt patterns: chain-of-thought, roles, structured output
- When fine-tuning beats prompting — and full vs LoRA
- The three stages of the RLHF pipeline
GANs, VAEs and Diffusion
The three architectures behind image generation: two networks in an adversarial game, a probabilistic latent space, and a model that learns to undo noise one step at a time.
- How a generator and a discriminator train each other — and why GANs are unstable
- How a VAE encodes images into a probabilistic latent space and samples new ones
- The forward (noising) and reverse (denoising) processes of a diffusion model
- How text conditioning and latent diffusion produce images from prompts
Image Generation: Diffusion, Text-to-Image and Editing
How DALL·E, Stable Diffusion and Midjourney turn a sentence into a picture by denoising random static, why latent space makes it fast, and how image-to-image edits work.
- How a diffusion model turns random noise into an image, one denoising step at a time
- How a text prompt steers generation through a CLIP text embedding
- Why Stable Diffusion runs in a compressed latent space
- How image-to-image editing reuses the same loop, and what the strength knob does
Video Generation: Diffusion Across Time
Video is many images that must agree. See how video diffusion denoises across space and time, and how the same machinery powers text-to-video, image-to-video, editing and prediction.
- Why temporal consistency is the core difficulty of video generation
- How video diffusion denoises within frames and across frames
- Three ways to produce a clip: keyframes + interpolation, sequential, whole sequence
- How image-to-video and video editing condition on existing footage
Audio and Speech Generation
From text-to-speech and WaveNet to voice cloning, music and sound effects: how models generate sound, and the consent safeguards voice cloning demands.
- The two-stage text-to-speech pipeline: acoustic model, then vocoder
- How WaveNet's dilated convolutions model both phonemes and prosody
- How voice cloning works, and the consent safeguards it needs
- Symbolic vs raw-audio music generation, and Jukebox's hierarchy
Code Generation: Completion, Explanation, Translation and Tests
How language models trained on code power completion tools like GitHub Copilot, and how the same model explains, translates and tests code, provided a human still reviews it.
- What a code LLM is and why it is still next-token prediction
- How code completion and GitHub Copilot assemble context from your editor
- Using the same model to explain code and translate it between languages
- Generating tests that probe edge cases