All lessons

Course

10th Grade AI: Deep Learning and Multimodal AI

Build and train neural networks in PyTorch for images, text, and sound.

Work through the lessons in order. By the end, you can compute a neuron's output, compare activations, and write a forward pass with NumPy. You can choose a loss, explain gradient descent and backpropagation, and write a full training and test loop in PyTorch. You can train a CNN to classify images, train a model to tell positive from negative reviews, explain how attention weighs other tokens, and classify short sound clips. You can reduce overfitting, fine-tune a pretrained model on new classes, group mistakes and find their cause, and present a wildlife monitor tested on unseen images.

How to use these lessons

  • Start at the lesson marked Do this. Go on when you can do the line under the title by yourself.
  • Play the Games first. Then take the Quizzes. Eight of ten correct passes. The Worksheets are on paper. The answers are at the bottom.
  • Each lesson builds on the one before it. Keep notes on every training run: the settings, the loss curve, and the test score.
  • The coding lessons use Python and PyTorch in a free online notebook with a GPU. Small models and small datasets are enough. Use only open datasets you are allowed to use, and never train on photos of people without their permission.

Each lesson has games, quizzes, and worksheets.

0 of 36

Neural network foundations

  1. 1NeuronsCompute a neuron's output from inputs, weights, and bias.
    • Write the weighted sum plus bias.
    • Compute it by hand for three inputs.
    • Compute it in code with a dot product.
  2. 2ActivationsCompare ReLU, sigmoid, and softmax.
    • Plot each activation.
    • Explain why a network needs a nonlinear activation.
    • Use softmax to turn scores into probabilities.
  3. 3Layers and networksStack layers into a network.
    • Write a layer as a matrix multiply.
    • Track the shape of the data through each layer.
    • Count a network's parameters.
  4. 4A small networkWrite a forward pass with NumPy.
    • Build a two-layer network in NumPy.
    • Run a batch of inputs through it.
    • Check the output shape.

Training neural networks

  1. 5LossChoose a loss for a task and compute it.
    • Use mean squared error for regression.
    • Use cross-entropy for classification.
    • Say what a falling loss means.
  2. 6Gradient descentExplain how gradients move weights downhill.
    • Find the slope of a loss curve.
    • Take steps with a learning rate.
    • See what happens when the rate is too big or too small.
  3. 7BackpropagationTrace how error flows back through a network.
    • Use the chain rule on a tiny network.
    • Compute one gradient by hand.
    • Check it against automatic differentiation.
  4. 8Train a networkTrain a small network and plot the loss.
    • Train with mini-batches.
    • Plot training and validation loss.
    • Stop when validation loss stops improving.

Deep learning frameworks

  1. 9TensorsCreate and reshape PyTorch tensors.
    • Create tensors and check their shape and type.
    • Reshape, stack, and slice them.
    • Move tensors to a GPU.
  2. 10ModulesBuild a model with torch.nn.
    • Subclass nn.Module.
    • Use Linear and activation layers.
    • Print the model and its parameter count.
  3. 11OptimizersTrain with SGD and Adam.
    • Use zero_grad, backward, and step.
    • Compare SGD and Adam on one task.
    • Use a learning-rate schedule.
  4. 12Training scriptWrite a full training and test loop.
    • Load data with a DataLoader.
    • Train, validate, and test.
    • Save the best model.

Computer vision

  1. 13Images as tensorsLoad images as tensors and normalize them.
    • Load an image as channels, height, and width.
    • Resize and normalize it.
    • Show a batch as a grid.
  2. 14ConvolutionsExplain how a filter finds edges.
    • Slide a 3×3 filter over an image by hand.
    • Apply edge filters in code.
    • Explain stride, padding, and pooling.
  3. 15CNNsBuild a convolutional neural network.
    • Stack convolution, activation, and pooling layers.
    • Flatten and add a classifier head.
    • Track shapes through the network.
  4. 16Image classifierTrain a CNN to classify images.
    • Train on a small image dataset.
    • Report test accuracy.
    • Show images it got wrong.

Natural language processing

  1. 17TokenizationTurn text into tokens and IDs.
    • Compare word and subword tokens.
    • Build a vocabulary.
    • Pad sequences to the same length.
  2. 18EmbeddingsExplain how embeddings place similar words close together.
    • Use an embedding layer.
    • Find the nearest words to a word.
    • Plot embeddings in two dimensions.
  3. 19Text modelsBuild a model that reads a sequence of tokens.
    • Average embeddings into one vector.
    • Use a simple recurrent layer.
    • Compare the two.
  4. 20Sentiment classifierTrain a model to tell positive from negative reviews.
    • Train on a review dataset.
    • Report accuracy, precision, and recall.
    • Find reviews that fool it, like sarcasm.

Transformers

  1. 21AttentionExplain how attention weighs other tokens.
    • Explain queries, keys, and values.
    • Compute attention weights for a short sentence.
    • Say why attention helps with long-range meaning.
  2. 22Transformer blocksName the parts of a transformer block.
    • Name multi-head attention, feed-forward, and normalization layers.
    • Explain residual connections.
    • Explain position encodings.
  3. 23Pretrained modelsUse a pretrained transformer for a task.
    • Load a pretrained model and tokenizer.
    • Classify text with it.
    • Compare it to your own sentiment model.
  4. 24Attention mapsRead an attention visualization.
    • Plot attention weights as a heat map.
    • Find which words a token attends to.
    • Say why attention is not a full explanation.

Audio and multimodal AI

  1. 25Sound as dataTurn audio into a waveform and a spectrogram.
    • Load audio and plot its waveform.
    • Explain sample rate.
    • Make a spectrogram.
  2. 26Audio classifierClassify short sound clips.
    • Treat spectrograms as images.
    • Train a CNN on them.
    • Test with background noise.
  3. 27Multimodal inputsExplain how a model combines images and text.
    • Explain separate encoders for each input.
    • Explain a shared embedding space.
    • Name three multimodal apps.
  4. 28Image and textMatch images to captions with a pretrained model.
    • Embed images and captions.
    • Rank captions for each image.
    • Search images with a text query.

Model improvement

  1. 29RegularizationReduce overfitting with dropout and weight decay.
    • Add dropout.
    • Add weight decay.
    • Compare validation curves.
  2. 30Data augmentationAdd flipped and cropped images to the training set.
    • Apply flips, crops, and color changes.
    • Augment only the training set.
    • Measure the change in test accuracy.
  3. 31Transfer learningFine-tune a pretrained model on new classes.
    • Freeze the pretrained layers.
    • Replace and train the head.
    • Unfreeze and fine-tune with a small learning rate.
  4. 32Error analysisGroup mistakes and find their cause.
    • Read a confusion matrix for many classes.
    • Sort errors into groups.
    • Pick the fix for the biggest group.

Deep learning project

  1. 33Wildlife dataPick and split an open wildlife image dataset.
    • Choose an open camera-trap or wildlife dataset.
    • Pick five to ten species.
    • Split by location so the test is fair.
  2. 34Fine-tune the modelFine-tune an image classifier on wildlife classes.
    • Fine-tune a pretrained model.
    • Track validation accuracy.
    • Save the best checkpoint.
  3. 35Demo interfaceBuild a page that shows predictions on uploaded images.
    • Accept an uploaded image.
    • Show the top three species and their scores.
    • Show “not sure” when confidence is low.
  4. 36Intelligent wildlife monitorPresent accuracy on unseen images and the main mistakes.
    • Report accuracy on unseen images.
    • Show the main mistakes and their causes.
    • Say what you would change next.