Skip to content
anythingai.space
Module 12

Generative AI

Text, image, video, audio and code generation — the model families, design patterns and how they are built.

lessons
8
steps
70
time
93 min
STAGE 1 · PRE-TRAINING (HUGE, ONCE)STAGE 2 · FINE-TUNING (SMALL, PER TASK)next tokenweightsUnlabeled dataweb · Wikipedia · codeFoundation modelgeneral knowledgeLabeled datatask examplesFine-tuneMedical modelCode model
01beginner·10 min·9 steps

Generative AI Foundations

Generative models learn the distribution of their training data and sample new content from it — one idea, five modalities, four architectures and one training recipe.

  • What “generative” means: learn a distribution, then sample from it
  • The five output modalities and the four core architectures
  • How a Transformer's self-attention sees a whole sequence at once
  • Why every modern model is pre-trained first and fine-tuned second
Start →
MASKED LANGUAGE MODELING: USE BOTH SIDESThebank[MASK]raiseditsratesBERTencoder-onlySentimenttask headEntitiestask headQ&A / searchtask head
02beginner·12 min·9 steps

Language Model Families

How text generation works — next-token prediction — and the families built on it: encoder-only BERT, encoder-decoder T5, the decoder-only GPT line through ChatGPT, and open LLaMA.

  • How a language model generates text one token at a time
  • Encoder-only, decoder-only and encoder-decoder — and what each is good at
  • What changed from GPT-3 to GPT-4 to ChatGPT
  • When an open model like LLaMA beats a proprietary API
Start →
1 · SFT2 · REWARD MODEL3 · RL OPTIMISATIONfine-tunesample outputstrainscorerewardDemonstrationshumans write answersSFT modelComparisonsA better than BReward modelpredicts preferencePolicystarts as SFT modelAligned model
03beginner·12 min·9 steps

Prompting, Fine-Tuning and RLHF

Four levers for getting what you want from a pre-trained model — zero-shot instructions, few-shot examples and prompt patterns, fine-tuning on your data, and the three-stage RLHF pipeline.

  • Zero-shot vs few-shot: instructions alone vs examples in the prompt
  • Reusable prompt patterns: chain-of-thought, roles, structured output
  • When fine-tuning beats prompting — and full vs LoRA
  • The three stages of the RLHF pipeline
Start →
conditionclean latentONE GENERATION = MANY DENOISING STEPS, EACH GUIDED BY THE PROMPTLatentnoiseblobsfox shapefur, snowdecodedenoising steps →Prompt“a red fox in snow”Text encoderDenoiserruns every step× NVAE decoderlatent → pixelsImageLatent noisestart here
04intermediate·13 min·9 steps

GANs, VAEs and Diffusion

The three architectures behind image generation: two networks in an adversarial game, a probabilistic latent space, and a model that learns to undo noise one step at a time.

  • How a generator and a discriminator train each other — and why GANs are unstable
  • How a VAE encodes images into a probabilistic latent space and samples new ones
  • The forward (noising) and reverse (denoising) processes of a diffusion model
  • How text conditioning and latent diffusion produce images from prompts
Start →
LATENT SPACE (SMALL)PIXEL SPACEtext embeddingRELATIVE GENERATION TIME (ILLUSTRATIVE)Pixel-space diffusion≈10×Latent diffusion1×Text encoderCLIPLatent noisecompressed spaceU-Netdenoises latents× NVAE decoderlatent → pixelsImage
05beginner·12 min·8 steps

Image Generation: Diffusion, Text-to-Image and Editing

How DALL·E, Stable Diffusion and Midjourney turn a sentence into a picture by denoising random static, why latent space makes it fast, and how image-to-image edits work.

  • How a diffusion model turns random noise into an image, one denoising step at a time
  • How a text prompt steers generation through a CLIP text embedding
  • Why Stable Diffusion runs in a compressed latent space
  • How image-to-image editing reuses the same loop, and what the strength knob does
Start →
HOW STRONGLY FRAMES INFORM EACH OTHER (ILLUSTRATIVE)f1f2f3f4f5f1f2f3f4f5Spatial: detail inside a frame.Temporal: agreement between frames.Noisy clipframes × pixelsVideo denoiserspace + time× N stepsVideo clipframes that agree
06intermediate·11 min·8 steps

Video Generation: Diffusion Across Time

Video is many images that must agree. See how video diffusion denoises across space and time, and how the same machinery powers text-to-video, image-to-video, editing and prediction.

  • Why temporal consistency is the core difficulty of video generation
  • How video diffusion denoises within frames and across frames
  • Three ways to produce a clip: keyframes + interpolation, sequential, whole sequence
  • How image-to-video and video editing condition on existing footage
Start →
PROSODY THE MODEL MUST GET RIGHTYoursoftordersteadyhasquickSHIPPEDstress ↑!pitch ↓Text“Order shipped!”Acoustic modeltext → featuresMel spectrogramacoustic featuresVocoderfeatures → waveformSpeechaudio waveform
07intermediate·12 min·9 steps

Audio and Speech Generation

From text-to-speech and WaveNet to voice cloning, music and sound effects: how models generate sound, and the consent safeguards voice cloning demands.

  • The two-stage text-to-speech pipeline: acoustic model, then vocoder
  • How WaveNet's dilated convolutions model both phonemes and prosody
  • How voice cloning works, and the consent safeguards it needs
  • Symbolic vs raw-audio music generation, and Jukebox's hierarchy
Start →
trainingNEXT-TOKEN PREDICTION, APPLIED TO CODEdefis_even(n):returnpredictedn % 2 == 0predictedSame architecture as text LLMs,specialised by dataCode repositoriesGitHub and moreCode LLMtransformer · code-trainedCodexStarCoderCodeLlamaMany languagesPython · JS · Java …
08beginner·11 min·9 steps

Code Generation: Completion, Explanation, Translation and Tests

How language models trained on code power completion tools like GitHub Copilot, and how the same model explains, translates and tests code, provided a human still reviews it.

  • What a code LLM is and why it is still next-token prediction
  • How code completion and GitHub Copilot assemble context from your editor
  • Using the same model to explain code and translate it between languages
  • Generating tests that probe edge cases
Start →
Next module · 13
Conversational AI