Deep Learning Frameworks

This guide covers TensorFlow/Keras and PyTorch for building deep learning models.

Deep learning curriculum map (this guide)

NumPy fundamentals and theory → Neural networks basics. Below: PyTorch and TensorFlow/Keras with runnable patterns.

• PyTorch tensors, autograd, and computational graph → PyTorch tensors, Autograd and graph
• Training pipeline and building networks in PyTorch → PyTorch workflow, Building models
• MLP-style classification/regression in PyTorch → PyTorch classification
• Regularization in frameworks (dropout, weight decay, batch norm) → TensorFlow/Keras sequential example (Dropout layers), Best practices
• Projects (MNIST, churn, house prices) → Intermediate DL projects

CNN and RNN curriculum map (PyTorch)

Theory and activations: Neural networks basics. CNN math and architectures → Computer vision. RNN, LSTM, GRU, Transformers → NLP guide.

• CNN in PyTorch (nn.Conv2d, pooling, training loop on batches) → PyTorch computer vision
• Image augmentation and pretrained CNNs → TorchVision / transfer in CV module, Transfer learning
• Sequence Dataset / DataLoader and padding → NLP: Sequence data and PyTorch

Table of Contents

• Deep learning curriculum map (this guide)
• CNN and RNN curriculum map (PyTorch)
• Introduction
• Autograd and the computational graph
• TensorFlow/Keras
  • Installation
  • Sequential API
  • Functional API
  • Model Subclassing
  • Data Preprocessing
  • Training
• PyTorch
  • Installation
  • Tensors
  • Building Models
  • Training Loop
• Framework Comparison
• Callbacks and Monitoring
• Model Saving and Loading
• Transfer Learning
• Practice Exercises
• PyTorch Fundamentals Deep Dive
  • Understanding Deep Learning and Why PyTorch?
  • PyTorch Tensors: The Foundation
• PyTorch Workflow: End-to-End
  • The PyTorch Workflow Pattern
  • Creating a Dataset with Linear Regression
  • Creating Training and Test Sets
  • Creating Your First PyTorch Model
  • Making Predictions
  • Training a Model with PyTorch
  • Testing Loop
  • Saving and Loading Models
• PyTorch Neural Network Classification
  • Introduction to Classification
  • Architecture of Classification Neural Network
  • Creating Classification Data
  • Building Classification Model
  • Using torch.nn.Sequential
  • Loss, Optimizer, and Evaluation Functions
  • From Logits to Probabilities to Labels
  • Training and Testing Loops for Classification
  • The Missing Piece: Non-Linearity
  • Multi-Class Classification
• PyTorch Computer Vision
  • Introduction to Computer Vision
  • What is a Convolutional Neural Network (CNN)?
  • TorchVision
  • Getting Computer Vision Dataset
  • Mini-Batches and DataLoaders
  • Training and Testing Loops for Batched Data
  • Running Experiments on GPU
  • Creating a CNN
  • Breaking Down nn.Conv2d and nn.MaxPool2d
  • Making Predictions on Random Test Samples
• PyTorch Custom Datasets
  • Introduction to Custom Datasets
  • Creating Custom Dataset Class
  • Data Augmentation
• TensorFlow Fundamentals Deep Dive
  • Deep Learning 101 with TensorFlow
  • TensorFlow Regression
  • TensorFlow Classification
  • TensorFlow Computer Vision
• TensorFlow NLP Fundamentals
  • Introduction to NLP with TensorFlow
  • Preparing Text Data with TensorFlow
  • Building NLP Models with TensorFlow
  • Transfer Learning for NLP with TensorFlow Hub
  • Using tf.data API for Efficient Text Processing
  • Evaluating NLP Models
• TensorFlow Time Series Fundamentals
  • Introduction to Time Series with TensorFlow
  • Preparing Time Series Data
  • Building Time Series Models
  • Time Series Evaluation Metrics
  • Advanced: N-BEATS Algorithm
  • Ensemble Models for Time Series
  • Prediction Intervals
• Advanced TensorFlow Features
  • tf.data API for Performance
  • Mixed Precision Training
  • Functional API vs Sequential API
  • Advanced Transfer Learning: Fine-Tuning
  • TensorBoard Integration
  • Model Evaluation and Debugging
  • TensorFlow Datasets (TFDS)
• Key Takeaways
• Best Practices
• Next Steps
• Comprehensive Learning Resources

Introduction

Why Use Frameworks?

Deep learning frameworks provide:

• Automatic Differentiation: No manual gradient computation
• GPU Acceleration: Easy GPU utilization
• Pre-built Layers: Ready-to-use components
• Optimization: Built-in optimizers and techniques
• Ecosystem: Pre-trained models, tools, and community

Framework Overview

TensorFlow/Keras:

• Widely used in industry training and serving stacks
• Great documentation and community
• TensorFlow Serving for deployment
• Keras: High-level API, easy to use

PyTorch:

• Research-friendly, flexible
• Dynamic computation graphs
• Pythonic, intuitive API
• Strong research community

Choosing a Framework:

• Keras: Beginners, rapid prototyping, production
• PyTorch: Research, custom architectures, flexibility
• Both: Learn both for maximum flexibility

TensorFlow/Keras

Installation

# Install TensorFlow
pip install tensorflow

# For GPU support (optional)
pip install tensorflow-gpu

# Verify installation
import tensorflow as tf
print(f"TensorFlow version: {tf.__version__}")
print(f"GPU available: {tf.config.list_physical_devices('GPU')}")

Sequential API

Simple linear stack of layers. Easiest way to build models.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# Sequential model
model = keras.Sequential([
    layers.Dense(128, activation='relu', input_shape=(784,), name='hidden1'),
    layers.Dropout(0.2, name='dropout1'),
    layers.Dense(64, activation='relu', name='hidden2'),
    layers.Dropout(0.2, name='dropout2'),
    layers.Dense(10, activation='softmax', name='output')
])

# Compile model
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# Model summary
model.summary()

# Visualize model architecture
keras.utils.plot_model(model, to_file='model.png', show_shapes=True)

Functional API

More flexible. Supports complex architectures (multi-input, multi-output, shared layers).

# Functional API (more flexible)
inputs = keras.Input(shape=(784,), name='input')
x = layers.Dense(128, activation='relu', name='hidden1')(inputs)
x = layers.Dropout(0.2, name='dropout1')(x)
x = layers.Dense(64, activation='relu', name='hidden2')(x)
x = layers.Dropout(0.2, name='dropout2')(x)
outputs = layers.Dense(10, activation='softmax', name='output')(x)

model = keras.Model(inputs=inputs, outputs=outputs, name='mlp_model')
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Multi-input example
input1 = keras.Input(shape=(784,), name='input1')
input2 = keras.Input(shape=(10,), name='input2')
x1 = layers.Dense(64, activation='relu')(input1)
x2 = layers.Dense(64, activation='relu')(input2)
merged = layers.concatenate([x1, x2])
outputs = layers.Dense(10, activation='softmax')(merged)
multi_input_model = keras.Model(inputs=[input1, input2], outputs=outputs)

Model Subclassing

Maximum flexibility. Define custom models by subclassing.

class MLP(keras.Model):
    def __init__(self, hidden_units=[128, 64], num_classes=10, dropout_rate=0.2):
        super().__init__()
        self.hidden_layers = []
        for units in hidden_units:
            self.hidden_layers.append(layers.Dense(units, activation='relu'))
            self.hidden_layers.append(layers.Dropout(dropout_rate))
        self.output_layer = layers.Dense(num_classes, activation='softmax')
    
    def call(self, inputs, training=False):
        x = inputs
        for layer in self.hidden_layers:
            x = layer(x, training=training)
        return self.output_layer(x)

# Create and compile
model = MLP(hidden_units=[128, 64], num_classes=10)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Functional API

# Functional API (more flexible)
inputs = keras.Input(shape=(784,))
x = layers.Dense(128, activation='relu')(inputs)
x = layers.Dropout(0.2)(x)
x = layers.Dense(64, activation='relu')(x)
outputs = layers.Dense(10, activation='softmax')(x)

model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Data Preprocessing

# Load data (MNIST example)
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Preprocess
x_train = x_train.reshape(60000, 784).astype('float32') / 255.0
x_test = x_test.reshape(10000, 784).astype('float32') / 255.0

# Alternative: Use ImageDataGenerator for augmentation
from tensorflow.keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=10,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.1,
    validation_split=0.2
)

# For image data (if using Conv2D)
# x_train = x_train.reshape(-1, 28, 28, 1)
# train_generator = datagen.flow(x_train, y_train, batch_size=32, subset='training')
# val_generator = datagen.flow(x_train, y_train, batch_size=32, subset='validation')

Training

# Train model
history = model.fit(
    x_train, y_train,
    batch_size=128,
    epochs=10,
    validation_split=0.2,
    verbose=1,
    callbacks=[
        keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True),
        keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=True, monitor='val_loss'),
        keras.callbacks.ReduceLROnPlateau(patience=2, factor=0.5, min_lr=1e-7)
    ]
)

# Plot training history
import matplotlib.pyplot as plt

plt.figure(figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.plot(history.history['loss'], label='Training Loss', linewidth=2)
plt.plot(history.history['val_loss'], label='Validation Loss', linewidth=2)
plt.xlabel('Epoch', fontsize=12)
plt.ylabel('Loss', fontsize=12)
plt.title('Model Loss', fontsize=14, fontweight='bold')
plt.legend()
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.plot(history.history['accuracy'], label='Training Accuracy', linewidth=2)
plt.plot(history.history['val_accuracy'], label='Validation Accuracy', linewidth=2)
plt.xlabel('Epoch', fontsize=12)
plt.ylabel('Accuracy', fontsize=12)
plt.title('Model Accuracy', fontsize=14, fontweight='bold')
plt.legend()
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Evaluate
test_loss, test_acc = model.evaluate(x_test, y_test, verbose=0)
print(f"Test Loss: {test_loss:.4f}")
print(f"Test Accuracy: {test_acc:.4f}")

# Make predictions
predicti>10])
predicted_classes = np.argmax(predictions, axis=1)
print(f"Predictions: {predicted_classes}")
print(f"True labels: {y_test[:10]}")

PyTorch

Installation

# Install PyTorch (CPU version)
pip install torch torchvision

# For GPU support (CUDA), visit: https://pytorch.org/get-started/locally/
# Example for CUDA 11.8:
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# Verify installation
import torch
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"CUDA device: {torch.cuda.get_device_name(0)}")

Tensors

PyTorch's fundamental data structure. Similar to NumPy arrays but with GPU support.

import torch
import numpy as np

# Create tensors
x = torch.tensor([1, 2, 3])
x = torch.tensor([[1, 2], [3, 4]], dtype=torch.float32)

# From NumPy
arr = np.array([1, 2, 3])
x = torch.from_numpy(arr)

# To NumPy
arr = x.numpy()

# GPU tensors
if torch.cuda.is_available():
    x_gpu = x.cuda()
    # or
    x_gpu = x.to('cuda')

# Operations
x = torch.randn(3, 4)
y = torch.randn(4, 5)
z = torch.matmul(x, y)  # Matrix multiplication

Building Models

import torch
import torch.nn as nn
import torch.nn.functional as F

# Method 1: Sequential
model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(64, 10)
)

# Method 2: Class-based (more flexible)
class NeuralNetwork(nn.Module):
    def __init__(self, input_size=784, hidden_sizes=[128, 64], num_classes=10, dropout=0.2):
        super().__init__()
        self.flatten = nn.Flatten()
        
        layers = []
        prev_size = input_size
        for hidden_size in hidden_sizes:
            layers.append(nn.Linear(prev_size, hidden_size))
            layers.append(nn.ReLU())
            layers.append(nn.Dropout(dropout))
            prev_size = hidden_size
        
        layers.append(nn.Linear(prev_size, num_classes))
        self.linear_stack = nn.Sequential(*layers)
    
    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_stack(x)
        return logits

model = NeuralNetwork(input_size=784, hidden_sizes=[128, 64], num_classes=10)
print(model)

# Count parameters
total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total parameters: {total_params:,}")
print(f"Trainable parameters: {trainable_params:,}")

Training Loop

from torch.utils.data import DataLoader, TensorDataset
import torch.optim as optim

# Prepare data
train_dataset = TensorDataset(torch.FloatTensor(x_train), torch.LongTensor(y_train))
val_dataset = TensorDataset(torch.FloatTensor(x_val), torch.LongTensor(y_val))

train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=128, shuffle=False)

# Move model to GPU if available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
print(f"Using device: {device}")

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=2)

# Training loop
num_epochs = 10
train_losses = []
val_losses = []
train_accs = []
val_accs = []

for epoch in range(num_epochs):
    # Training phase
    model.train()
    train_loss = 0.0
    train_correct = 0
    train_total = 0
    
    for batch_x, batch_y in train_loader:
        batch_x, batch_y = batch_x.to(device), batch_y.to(device)
        
        # Forward
        outputs = model(batch_x)
        loss = criterion(outputs, batch_y)
        
        # Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        # Statistics
        train_loss += loss.item()
        _, predicted = torch.max(outputs.data, 1)
        train_total += batch_y.size(0)
        train_correct += (predicted == batch_y).sum().item()
    
    # Validation phase
    model.eval()
    val_loss = 0.0
    val_correct = 0
    val_total = 0
    
    with torch.no_grad():
        for batch_x, batch_y in val_loader:
            batch_x, batch_y = batch_x.to(device), batch_y.to(device)
            
            outputs = model(batch_x)
            loss = criterion(outputs, batch_y)
            
            val_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            val_total += batch_y.size(0)
            val_correct += (predicted == batch_y).sum().item()
    
    # Calculate averages
    train_loss /= len(train_loader)
    val_loss /= len(val_loader)
    train_acc = 100 * train_correct / train_total
    val_acc = 100 * val_correct / val_total
    
    train_losses.append(train_loss)
    val_losses.append(val_loss)
    train_accs.append(train_acc)
    val_accs.append(val_acc)
    
    # Learning rate scheduling
    scheduler.step(val_loss)
    
    print(f"Epoch {epoch+1}/{num_epochs}:")
    print(f"  Train Loss: {train_loss:.4f}, Train Acc: {train_acc:.2f}%")
    print(f"  Val Loss: {val_loss:.4f}, Val Acc: {val_acc:.2f}%")
    print(f"  LR: {optimizer.param_groups[0]['lr']:.6f}")

# Plot training history
plt.figure(figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.plot(train_losses, label='Training Loss', linewidth=2)
plt.plot(val_losses, label='Validation Loss', linewidth=2)
plt.xlabel('Epoch', fontsize=12)
plt.ylabel('Loss', fontsize=12)
plt.title('Model Loss', fontsize=14, fontweight='bold')
plt.legend()
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.plot(train_accs, label='Training Accuracy', linewidth=2)
plt.plot(val_accs, label='Validation Accuracy', linewidth=2)
plt.xlabel('Epoch', fontsize=12)
plt.ylabel('Accuracy (%)', fontsize=12)
plt.title('Model Accuracy', fontsize=14, fontweight='bold')
plt.legend()
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Framework Comparison

Keras vs PyTorch

When to Use Each

Use Keras when:

• You're a beginner
• You need rapid prototyping
• You're building standard architectures
• You need production deployment
• You prefer high-level APIs

Use PyTorch when:

• You need maximum flexibility
• You're doing research
• You need dynamic computation graphs
• You prefer Pythonic code
• You need fine-grained control

Callbacks and Monitoring

Keras Callbacks

# Early Stopping
early_stopping = keras.callbacks.EarlyStopping(
    monitor='val_loss',
    patience=5,
    restore_best_weights=True,
    verbose=1
)

# Model Checkpointing
checkpoint = keras.callbacks.ModelCheckpoint(
    'best_model.h5',
    monitor='val_loss',
    save_best_only=True,
    verbose=1
)

# Learning Rate Reduction
reduce_lr = keras.callbacks.ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.5,
    patience=2,
    min_lr=1e-7,
    verbose=1
)

# TensorBoard
tensorboard = keras.callbacks.TensorBoard(
    log_dir='./logs',
    histogram_freq=1,
    write_graph=True
)

# CSV Logger
csv_logger = keras.callbacks.CSVLogger('training.log')

# Use in training
history = model.fit(
    x_train, y_train,
    epochs=50,
    validation_split=0.2,
    callbacks=[early_stopping, checkpoint, reduce_lr, tensorboard, csv_logger]
)

PyTorch Monitoring

from torch.utils.tensorboard import SummaryWriter

# TensorBoard writer
writer = SummaryWriter('runs/experiment_1')

# Log during training
for epoch in range(num_epochs):
    # ... training code ...
    
    writer.add_scalar('Loss/Train', train_loss, epoch)
    writer.add_scalar('Loss/Val', val_loss, epoch)
    writer.add_scalar('Accuracy/Train', train_acc, epoch)
    writer.add_scalar('Accuracy/Val', val_acc, epoch)
    writer.add_scalar('Learning_Rate', optimizer.param_groups[0]['lr'], epoch)

writer.close()

# View with: tensorboard --logdir=runs

Model Saving and Loading

Keras

# Save entire model (architecture + weights + optimizer state)
model.save('model.h5')
model.save('model.keras')  # New format in Keras 3

# Save only weights
model.save_weights('weights.h5')

# Save model as SavedModel (for TensorFlow Serving)
model.save('saved_model', save_format='tf')

# Load entire model
model = keras.models.load_model('model.h5')

# Load only weights (need to create model first)
model = create_model()  # Define architecture
model.load_weights('weights.h5')

# Load and compile
model = keras.models.load_model('model.h5', compile=False)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

PyTorch

# Save model state dict (recommended)
torch.save(model.state_dict(), 'model.pth')

# Save entire model (not recommended - includes file paths)
torch.save(model, 'model_full.pth')

# Save checkpoint (with optimizer state, epoch, etc.)
checkpoint = {
    'epoch': epoch,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': loss,
}
torch.save(checkpoint, 'checkpoint.pth')

# Load state dict
model = NeuralNetwork()  # Create model architecture
model.load_state_dict(torch.load('model.pth'))
model.eval()  # Set to evaluation mode

# Load checkpoint
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

Transfer Learning

Keras Transfer Learning

# Load pre-trained model (e.g., VGG16)
base_model = keras.applications.VGG16(
    weights='imagenet',
    include_top=False,
    input_shape=(224, 224, 3)
)

# Freeze base model
base_model.trainable = False

# Add custom classifier
inputs = keras.Input(shape=(224, 224, 3))
x = base_model(inputs, training=False)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(10, activation='softmax')(x)

model = keras.Model(inputs, outputs)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Fine-tuning: Unfreeze some layers
base_model.trainable = True
for layer in base_model.layers[:-4]:
    layer.trainable = False

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-5),  # Lower LR for fine-tuning
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

PyTorch Transfer Learning

import torchvision.models as models

# Load pre-trained model
model = models.resnet18(pretrained=True)

# Freeze parameters
for param in model.parameters():
    param.requires_grad = False

# Replace classifier
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 10)  # 10 classes

# Fine-tuning: Unfreeze last few layers
for param in list(model.parameters())[-10:]:
    param.requires_grad = True

# Use different learning rates
optimizer = optim.Adam([
    {'params': model.fc.parameters(), 'lr': 1e-3},
    {'params': [p for p in model.parameters() if p.requires_grad and p not in model.fc.parameters()], 'lr': 1e-5}
])

Practice Exercises

Exercise 1: MNIST with Keras

Task: Build and train CNN for MNIST classification.

Solution:

# Load and preprocess data
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.reshape(-1, 28, 28, 1).astype('float32') / 255.0
x_test = x_test.reshape(-1, 28, 28, 1).astype('float32') / 255.0

# Build CNN
model = keras.Sequential([
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5, validation_split=0.2, batch_size=128)
test_loss, test_acc = model.evaluate(x_test, y_test)
print(f"Test accuracy: {test_acc:.4f}")

Exercise 2: Same Model in PyTorch

Task: Build the same CNN in PyTorch.

Solution:

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.c>1, 32, 3)
        self.c>32, 64, 3)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 6 * 6, 64)
        self.fc2 = nn.Linear(64, 10)
        self.dropout = nn.Dropout(0.2)
    
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 64 * 6 * 6)
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

model = CNN()
# ... training code ...

Autograd and the computational graph

PyTorch records operations on tensors that have requires_grad=True, building a dynamic computational graph. Calling .backward() runs reverse-mode automatic differentiation (generalization of backprop) and deposits gradients in .grad.

import torch

# Scalar graph: y = x^2 + 3x  →  dy/dx = 2x + 3
x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x
y.backward()
print("x:", x.item(), "y:", y.item(), "dy/dx:", x.grad.item())

# Vector loss on a tiny linear layer (pattern used in real training loops)
W = torch.tensor([[0.5, -0.2]], requires_grad=True)  # 1x2
b = torch.tensor([0.1], requires_grad=True)
inp = torch.tensor([[1.0, -1.0]])  # batch 1, two features
logits = inp @ W.T + b
target = torch.tensor([[1.0]])
loss = ((logits - target) ** 2).mean()
loss.backward()
print("dL/dW:", W.grad, "dL/db:", b.grad)

Training loop pattern: loss.backward() then optimizer.step() then optimizer.zero_grad() (or zero_grad before forward) so gradients do not accumulate across iterations.

PyTorch Fundamentals Deep Dive

Understanding Deep Learning and Why PyTorch?

What is Deep Learning?

• Subset of machine learning using neural networks with multiple layers
• Learns hierarchical representations from data
• Excels at pattern recognition in complex data

Why Use Machine/Deep Learning?

• Automatic Feature Learning: No manual feature engineering
• Scalability: Handles large, complex datasets
• Performance: Competitive results in many domains when tuned well
• Flexibility: Adapts to various problem types

The Number One Rule of ML:

Data > Algorithms > Models

Having good, representative data is more important than choosing the best algorithm or model architecture.

Machine Learning vs Deep Learning:

What Can Deep Learning Be Used For?

• Computer Vision: Image classification, object detection, segmentation
• Natural Language Processing: Translation, sentiment analysis, chatbots
• Speech Recognition: Voice assistants, transcription
• Recommendation Systems: Product recommendations, content filtering
• Time Series: Forecasting, anomaly detection
• Generative AI: Text, image, code generation

What is/Why PyTorch?

• Pythonic: Feels like NumPy, intuitive API
• Dynamic Graphs: Build and modify graphs on-the-fly
• Research-Friendly: Easy to experiment and prototype
• Strong Community: Widely used in research and industry
• GPU Acceleration: Seamless CUDA integration
• Ecosystem: torchvision, torchaudio, torchtext

PyTorch Tensors: The Foundation

What are Tensors? Tensors are multi-dimensional arrays. The fundamental data structure in PyTorch.

import torch
import numpy as np

# Creating tensors
# Scalar (0D tensor)
scalar = torch.tensor(42)
print(f"Scalar: {scalar}, Shape: {scalar.shape}")

# Vector (1D tensor)
vector = torch.tensor([1, 2, 3, 4])
print(f"Vector: {vector}, Shape: {vector.shape}")

# Matrix (2D tensor)
matrix = torch.tensor([[1, 2], [3, 4]])
print(f"Matrix: {matrix}, Shape: {matrix.shape}")

# 3D tensor
tensor_3d = torch.tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
print(f"3D Tensor: {tensor_3d}, Shape: {tensor_3d.shape}")

# Random tensors
random_tensor = torch.rand(3, 4)  # Uniform [0, 1)
normal_tensor = torch.randn(3, 4)  # Normal distribution
zeros = torch.zeros(3, 4)
>3, 4)

Tensor Data Types:

# Different data types
float32_tensor = torch.tensor([1.0, 2.0], dtype=torch.float32)
int64_tensor = torch.tensor([1, 2], dtype=torch.int64)
bool_tensor = torch.tensor([True, False], dtype=torch.bool)

# Check data type
print(f"Data type: {float32_tensor.dtype}")
print(f"Device: {float32_tensor.device}")  # cpu or cuda
print(f"Shape: {float32_tensor.shape}")
print(f"Requires grad: {float32_tensor.requires_grad}")  # For autograd

Tensor Attributes:

tensor = torch.randn(3, 4)

print(f"Shape: {tensor.shape}")  # or tensor.size()
print(f"Number of dimensions: {tensor.ndim}")
print(f"Number of elements: {tensor.numel()}")
print(f"Data type: {tensor.dtype}")
print(f"Device: {tensor.device}")
print(f"Layout: {tensor.layout}")  # Usually 'strided'

Manipulating Tensors:

# Reshaping
x = torch.randn(4, 4)
x_reshaped = x.reshape(2, 8)  # or x.view(2, 8)
x_flattened = x.flatten()  # Flatten to 1D

# Stacking
a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])
stacked = torch.stack([a, b])  # Shape: (2, 3)

# Concatenation
c b], dim=0)  # Shape: (6,)

# Squeezing and Unsqueezing
x = torch.randn(1, 3, 1, 4)
x_squeezed = x.squeeze()  # Remove dims of size 1 -> (3, 4)
x_unsqueezed = x.unsqueeze(0)  # Add dimension at index 0

# Permuting (transposing)
x = torch.randn(2, 3, 4)
x_permuted = x.permute(2, 0, 1)  # New shape: (4, 2, 3)

Matrix Multiplication:

# Matrix multiplication
x = torch.randn(3, 4)
y = torch.randn(4, 5)
z = torch.matmul(x, y)  # or x @ y
print(f"Result shape: {z.shape}")  # (3, 5)

# Element-wise operations
a = torch.tensor([[1, 2], [3, 4]])
b = torch.tensor([[5, 6], [7, 8]])
element_wise = a * b  # Element-wise multiplication
element_wise_sum = a + b  # Element-wise addition

Finding Min, Max, Mean, Sum:

x = torch.tensor([[1, 2, 3], [4, 5, 6]])

# Min/Max
print(f"Min: {x.min()}")
print(f"Max: {x.max()}")
print(f"Min along dim 0: {x.min(dim=0)}")  # Returns (values, indices)
print(f"Argmin: {x.argmin()}")  # Index of minimum
print(f"Argmax: {x.argmax()}")  # Index of maximum

# Mean/Sum
print(f"Mean: {x.mean()}")
print(f"Sum: {x.sum()}")
print(f"Mean along dim 1: {x.mean(dim=1)}")
print(f"Sum along dim 0: {x.sum(dim=0)}")

PyTorch and NumPy Interoperability:

# NumPy to PyTorch
numpy_array = np.array([1, 2, 3, 4])
torch_tensor = torch.from_numpy(numpy_array)  # Shares memory

# PyTorch to NumPy
torch_tensor = torch.tensor([1, 2, 3, 4])
numpy_array = torch_tensor.numpy()  # Shares memory if on CPU

# Note: If tensor is on GPU, need to move to CPU first
# numpy_array = torch_tensor.cpu().numpy()

Reproducibility:

# Set random seed for reproducibility
torch.manual_seed(42)
np.random.seed(42)

# For CUDA
if torch.cuda.is_available():
    torch.cuda.manual_seed(42)
    torch.cuda.manual_seed_all(42)

# Make operations deterministic (may slow down)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Accessing GPU and Device-Agnostic Code:

# Check GPU availability
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"Using device: {device}")

# Move tensor to device
x = torch.randn(3, 4)
x_gpu = x.to(device)  # or x.cuda() if CUDA available

# Device-agnostic code (best practice)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = NeuralNetwork().to(device)
data = data.to(device)

# Check CUDA device properties
if torch.cuda.is_available():
    print(f"CUDA Device: {torch.cuda.get_device_name(0)}")
    print(f"CUDA Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.2f} GB")

PyTorch Workflow: End-to-End

The PyTorch Workflow Pattern

Standard Workflow:

1. Prepare Data: Load and preprocess
2. Create Model: Define architecture
3. Setup Loss & Optimizer: Choose loss function and optimizer
4. Training Loop: Forward pass, backward pass, update weights
5. Evaluation: Test on validation/test set
6. Save/Load Model: Persist trained models

Creating a Dataset with Linear Regression

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

# Create synthetic linear regression dataset
weight = 0.7  # True weight
bias = 0.3     # True bias

# Generate data
X = torch.randn(100, 1)
y = weight * X + bias + torch.randn(100, 1) * 0.1  # Add noise

# Visualize
plt.scatter(X, y, alpha=0.6)
plt.xlabel('X')
plt.ylabel('y')
plt.title('Linear Regression Dataset')
plt.show()

Creating Training and Test Sets

from torch.utils.data import TensorDataset, DataLoader, random_split

# Create dataset
dataset = TensorDataset(X, y)

# Split into train and test (80/20)
train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train_dataset, test_dataset = random_split(dataset, [train_size, test_size])

# Create DataLoaders
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)

print(f"Train batches: {len(train_loader)}")
print(f"Test batches: {len(test_loader)}")

Creating Your First PyTorch Model

# Method 1: Using nn.Sequential
model_sequential = nn.Sequential(
    nn.Linear(in_features=1, out_features=1)
)

# Method 2: Using nn.Module (recommended)
class LinearRegressionModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear_layer = nn.Linear(in_features=1, out_features=1)
    
    def forward(self, x):
        return self.linear_layer(x)

model = LinearRegressionModel()
print(model)
print(f"Model parameters: {list(model.parameters())}")

Important Model Building Classes:

# nn.Module: Base class for all neural network modules
# nn.Linear: Fully connected layer
# nn.Conv2d: 2D convolution
# nn.ReLU: ReLU activation
# nn.Sequential: Container for sequential layers
# nn.ModuleList: List of modules
# nn.ModuleDict: Dictionary of modules

Checking Model Internals:

# Model state dict
print("Model state dict:")
for name, param in model.named_parameters():
    print(f"{name}: {param.data}")

# Model structure
print("\nModel structure:")
print(model)

# Count parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"\nTotal parameters: {total_params}")

Making Predictions

# Set model to evaluation mode
model.eval()

# Make predictions (no gradient tracking)
with torch.no_grad():
    sample_input = torch.tensor([[0.5]])
    prediction = model(sample_input)
    print(f"Prediction for input 0.5: {prediction.item()}")

Training a Model with PyTorch

# Setup
loss_fn = nn.MSELoss()  # Mean Squared Error
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Training loop
num_epochs = 100
train_losses = []

for epoch in range(num_epochs):
    model.train()  # Set to training mode
    epoch_loss = 0.0
    
    for batch_X, batch_y in train_loader:
        # Forward pass
        predicti>
        loss = loss_fn(predictions, batch_y)
        
        # Backward pass
        optimizer.zero_grad()  # Zero gradients
        loss.backward()         # Compute gradients
        optimizer.step()        # Update weights
        
        epoch_loss += loss.item()
    
    avg_loss = epoch_loss / len(train_loader)
    train_losses.append(avg_loss)
    
    if (epoch + 1) % 10 == 0:
        print(f"Epoch {epoch+1}/{num_epochs}, Loss: {avg_loss:.4f}")

# Plot training loss
plt.plot(train_losses)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Training Loss')
plt.show()

Testing Loop

# Evaluation
model.eval()
test_loss = 0.0

with torch.no_grad():
    for batch_X, batch_y in test_loader:
        predicti>
        loss = loss_fn(predictions, batch_y)
        test_loss += loss.item()

avg_test_loss = test_loss / len(test_loader)
print(f"Test Loss: {avg_test_loss:.4f}")

# Visualize predictions
model.eval()
with torch.no_grad():
    y_pred = model(X)

plt.figure(figsize=(10, 6))
plt.scatter(X, y, alpha=0.6, label='True')
plt.plot(X, y_pred, 'r-', label='Predicted', linewidth=2)
plt.xlabel('X')
plt.ylabel('y')
plt.legend()
plt.title('Model Predictions')
plt.show()

Saving and Loading Models

# Save model state dict (recommended)
torch.save(model.state_dict(), 'linear_model.pth')

# Save entire model (not recommended for production)
torch.save(model, 'linear_model_full.pth')

# Load model
# Method 1: Load state dict
loaded_model = LinearRegressionModel()
loaded_model.load_state_dict(torch.load('linear_model.pth'))
loaded_model.eval()

# Method 2: Load entire model
loaded_model_full = torch.load('linear_model_full.pth')
loaded_model_full.eval()

# Verify loaded model works
with torch.no_grad():
    test_input = torch.tensor([[0.5]])
    prediction = loaded_model(test_input)
    print(f"Loaded model prediction: {prediction.item()}")

PyTorch Neural Network Classification

Introduction to Classification

Classification vs Regression:

• Regression: Predict continuous values (e.g., house price)
• Classification: Predict discrete categories (e.g., cat vs dog)

Classification Input and Outputs:

• Input: Features (e.g., image pixels, text embeddings)
• Output: Class probabilities or class labels
• Binary Classification: 2 classes (e.g., spam/not spam)
• Multi-class Classification: Multiple classes (e.g., digits 0-9)

Architecture of Classification Neural Network

class ClassificationModel(nn.Module):
    def __init__(self, input_features, hidden_units, output_features):
        super().__init__()
        self.layer_stack = nn.Sequential(
            nn.Linear(input_features, hidden_units),
            nn.ReLU(),
            nn.Linear(hidden_units, hidden_units),
            nn.ReLU(),
            nn.Linear(hidden_units, output_features)
        )
    
    def forward(self, x):
        return self.layer_stack(x)

# For binary classification
binary_model = ClassificationModel(input_features=2, hidden_units=10, output_features=1)

# For multi-class classification
multiclass_model = ClassificationModel(input_features=784, hidden_units=128, output_features=10)

Creating Classification Data

from sklearn.datasets import make_circles
import torch

# Create binary classification dataset (circles)
X, y = make_circles(n_samples=1000, noise=0.03, random_state=42)

# Convert to tensors
X = torch.from_numpy(X).type(torch.float)
y = torch.from_numpy(y).type(torch.float)

# Reshape y for binary classification
y = y.unsqueeze(1)  # Shape: (1000, 1)

# Split into train and test
from torch.utils.data import TensorDataset, DataLoader, random_split
dataset = TensorDataset(X, y)
train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train_dataset, test_dataset = random_split(dataset, [train_size, test_size])

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)

Building Classification Model

class BinaryClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 10)
        self.layer2 = nn.Linear(10, 10)
        self.layer3 = nn.Linear(10, 1)
        self.relu = nn.ReLU()
    
    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.relu(self.layer2(x))
        x = self.layer3(x)  # No activation - raw logits
        return x

model = BinaryClassifier()

Using torch.nn.Sequential

# Simpler way using Sequential
model = nn.Sequential(
    nn.Linear(2, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 1)
)

Loss, Optimizer, and Evaluation Functions

# Loss function for binary classification
loss_fn = nn.BCEWithLogitsLoss()  # Binary Cross Entropy with Logits
# Note: Use this when output doesn't have sigmoid activation

# Optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# Evaluation function
def accuracy_fn(y_true, y_pred):
    """Calculate accuracy"""
    correct = torch.eq(y_true, y_pred).sum().item()
    acc = (correct / len(y_pred)) * 100
    return acc

From Logits to Probabilities to Labels

# Model outputs logits (raw predictions)
logits = model(X)

# Convert logits to probabilities (for binary classification)
probabilities = torch.sigmoid(logits)

# Convert probabilities to prediction labels
predicti  # 0 or 1

# For multi-class classification
logits = model(X)  # Shape: (batch_size, num_classes)
probabilities = torch.softmax(logits, dim=1)  # Shape: (batch_size, num_classes)
predicti dim=1)  # Shape: (batch_size,)

Training and Testing Loops for Classification

# Training loop
num_epochs = 100
train_losses = []
train_accs = []

for epoch in range(num_epochs):
    model.train()
    epoch_loss = 0.0
    epoch_correct = 0
    epoch_total = 0
    
    for batch_X, batch_y in train_loader:
        # Forward pass
        logits = model(batch_X)
        loss = loss_fn(logits, batch_y)
        
        # Calculate accuracy
        probabilities = torch.sigmoid(logits)
        predicti>
        correct = (predicti batch_y).sum().item()
        epoch_correct += correct
        epoch_total += len(batch_y)
        
        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        epoch_loss += loss.item()
    
    avg_loss = epoch_loss / len(train_loader)
    avg_acc = (epoch_correct / epoch_total) * 100
    train_losses.append(avg_loss)
    train_accs.append(avg_acc)
    
    if (epoch + 1) % 10 == 0:
        print(f"Epoch {epoch+1}/{num_epochs}, Loss: {avg_loss:.4f}, Acc: {avg_acc:.2f}%")

# Testing loop
model.eval()
test_loss = 0.0
test_correct = 0
test_total = 0

with torch.no_grad():
    for batch_X, batch_y in test_loader:
        logits = model(batch_X)
        loss = loss_fn(logits, batch_y)
        
        probabilities = torch.sigmoid(logits)
        predicti>
        correct = (predicti batch_y).sum().item()
        
        test_loss += loss.item()
        test_correct += correct
        test_total += len(batch_y)

avg_test_loss = test_loss / len(test_loader)
avg_test_acc = (test_correct / test_total) * 100
print(f"\nTest Loss: {avg_test_loss:.4f}, Test Acc: {avg_test_acc:.2f}%")

The Missing Piece: Non-Linearity

Why Non-Linearity?

• Without activation functions, multiple layers = single layer (linear transformation)
• Non-linearity enables learning complex patterns
• ReLU is most common for hidden layers

# Model without non-linearity (won't learn non-linear patterns)
linear_model = nn.Sequential(
    nn.Linear(2, 10),
    nn.Linear(10, 10),
    nn.Linear(10, 1)
)

# Model with non-linearity (can learn non-linear patterns)
n>
    nn.Linear(2, 10),
    nn.ReLU(),  # Non-linearity!
    nn.Linear(10, 10),
    nn.ReLU(),  # Non-linearity!
    nn.Linear(10, 1)
)

Multi-Class Classification

from torchvision import datasets
from torchvision.transforms import ToTensor

# Load MNIST dataset
train_data = datasets.MNIST(
    root='data',
    train=True,
    download=True,
    transform=ToTensor()
)

test_data = datasets.MNIST(
    root='data',
    train=False,
    download=True,
    transform=ToTensor()
)

# Create DataLoaders
train_loader = DataLoader(train_data, batch_size=32, shuffle=True)
test_loader = DataLoader(test_data, batch_size=32, shuffle=False)

# Multi-class classification model
class MNISTClassifier(nn.Module):
    def __init__(self, input_shape, hidden_units, output_shape):
        super().__init__()
        self.flatten = nn.Flatten()
        self.layer_stack = nn.Sequential(
            nn.Linear(input_shape, hidden_units),
            nn.ReLU(),
            nn.Linear(hidden_units, hidden_units),
            nn.ReLU(),
            nn.Linear(hidden_units, output_shape)
        )
    
    def forward(self, x):
        x = self.flatten(x)
        return self.layer_stack(x)

model = MNISTClassifier(input_shape=28*28, hidden_units=128, output_shape=10)

# Loss for multi-class classification
loss_fn = nn.CrossEntropyLoss()  # Includes softmax
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# Training loop (similar to binary, but different loss)
for epoch in range(num_epochs):
    model.train()
    for batch_X, batch_y in train_loader:
        logits = model(batch_X)
        loss = loss_fn(logits, batch_y)  # batch_y is class indices, not one-hot
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

PyTorch Computer Vision

Introduction to Computer Vision

What is Computer Vision?

• Field of AI that enables machines to interpret visual information
• Tasks: Image classification, object detection, segmentation, etc.

Computer Vision Input and Outputs:

• Input: Images (tensors of shape [batch, channels, height, width])
• Output: Predictions (class labels, bounding boxes, masks, etc.)

What is a Convolutional Neural Network (CNN)?

Why CNNs for Images?

• Translation Invariance: Detect features anywhere in image
• Parameter Sharing: Same filters across image (fewer parameters)
• Local Patterns: Detect edges, shapes, textures hierarchically

CNN Architecture:

Input Image → Conv Layers → Pooling → Conv Layers → Pooling → Fully Connected → Output

TorchVision

import torchvision
from torchvision import datasets, transforms

# TorchVision provides:
# - Pre-trained models
# - Common datasets
# - Image transformations
# - Utilities for computer vision

Getting Computer Vision Dataset

from torchvision import datasets
from torchvision.transforms import ToTensor

# Fashion-MNIST dataset
train_data = datasets.FashionMNIST(
    root='data',
    train=True,
    download=True,
    transform=ToTensor(),  # Convert PIL Image to tensor
    target_transform=None
)

test_data = datasets.FashionMNIST(
    root='data',
    train=False,
    download=True,
    transform=ToTensor()
)

# Visualize sample
import matplotlib.pyplot as plt
figure = plt.figure(figsize=(8, 8))
cols, rows = 4, 4
for i in range(1, cols * rows + 1):
    sample_idx = torch.randint(len(train_data), size=(1,)).item()
    img, label = train_data[sample_idx]
    figure.add_subplot(rows, cols, i)
    plt.title(f"Label: {label}")
    plt.axis("off")
    plt.imshow(img.squeeze(), cmap="gray")
plt.show()

Mini-Batches and DataLoaders

from torch.utils.data import DataLoader

# Create DataLoaders
BATCH_SIZE = 32
train_loader = DataLoader(
    train_data,
    batch_size=BATCH_SIZE,
    shuffle=True,  # Shuffle training data
    num_workers=0  # Number of subprocesses for data loading
)

test_loader = DataLoader(
    test_data,
    batch_size=BATCH_SIZE,
    shuffle=False,  # Don't shuffle test data
    num_workers=0
)

# Check batch shape
for batch_X, batch_y in train_loader:
    print(f"Batch shape: {batch_X.shape}")  # [batch_size, channels, height, width]
    print(f"Labels shape: {batch_y.shape}")  # [batch_size]
    break

Training and Testing Loops for Batched Data

def train_step(model, dataloader, loss_fn, optimizer, device):
    """Training step for one epoch"""
    model.train()
    train_loss, train_acc = 0, 0
    
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(device), y.to(device)
        
        # Forward pass
        y_pred = model(X)
        loss = loss_fn(y_pred, y)
        train_loss += loss.item()
        
        # Calculate accuracy
        y_pred_class = torch.argmax(torch.softmax(y_pred, dim=1), dim=1)
        train_acc += (y_pred_class == y).sum().item() / len(y_pred)
        
        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
    train_loss /= len(dataloader)
    train_acc /= len(dataloader)
    return train_loss, train_acc

def test_step(model, dataloader, loss_fn, device):
    """Testing step"""
    model.eval()
    test_loss, test_acc = 0, 0
    
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(device), y.to(device)
            
            y_pred = model(X)
            loss = loss_fn(y_pred, y)
            test_loss += loss.item()
            
            y_pred_class = torch.argmax(torch.softmax(y_pred, dim=1), dim=1)
            test_acc += (y_pred_class == y).sum().item() / len(y_pred)
    
    test_loss /= len(dataloader)
    test_acc /= len(dataloader)
    return test_loss, test_acc

Running Experiments on GPU

# Setup device
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Move model to device
model = model.to(device)

# Training loop with GPU
for epoch in range(num_epochs):
    train_loss, train_acc = train_step(model, train_loader, loss_fn, optimizer, device)
    test_loss, test_acc = test_step(model, test_loader, loss_fn, device)
    
    if (epoch + 1) % 5 == 0:
        print(f"Epoch {epoch+1}/{num_epochs}")
        print(f"Train Loss: {train_loss:.4f} | Train Acc: {train_acc:.2f}%")
        print(f"Test Loss: {test_loss:.4f} | Test Acc: {test_acc:.2f}%")

Creating a CNN

class CNN(nn.Module):
    def __init__(self, input_shape, hidden_units, output_shape):
        super().__init__()
        self.c>
            nn.Conv2d(in_channels=input_shape, out_channels=hidden_units, 
                     kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(in_channels=hidden_units, out_channels=hidden_units,
                     kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)
        )
        self.c>
            nn.Conv2d(hidden_units, hidden_units, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(hidden_units, hidden_units, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(hidden_units * 7 * 7, output_shape)  # 7*7 from pooling
        )
    
    def forward(self, x):
        x = self.conv_block_1(x)
        x = self.conv_block_2(x)
        x = self.classifier(x)
        return x

model = CNN(input_shape=1, hidden_units=10, output_shape=10)

Breaking Down nn.Conv2d and nn.MaxPool2d

# Conv2d parameters
c>
    in_channels=1,      # Input channels (grayscale=1, RGB=3)
    out_channels=10,    # Number of filters
    kernel_size=3,      # Filter size (3x3)
    stride=1,           # Step size
    padding=1           # Padding to maintain size
)

# MaxPool2d parameters
pool_layer = nn.MaxPool2d(
    kernel_size=2,      # Pooling window size
    stride=2            # Step size (usually same as kernel_size)
)

# Example: Understanding output shapes
x = torch.randn(1, 1, 28, 28)  # [batch, channels, height, width]
print(f"Input shape: {x.shape}")

# After Conv2d
x = conv_layer(x)
print(f"After Conv2d: {x.shape}")  # [1, 10, 28, 28] (with padding=1)

# After MaxPool2d
x = pool_layer(x)
print(f"After MaxPool2d: {x.shape}")  # [1, 10, 14, 14]

Making Predictions on Random Test Samples

def make_predictions(model, data, device):
    """Make predictions on random samples"""
    model.eval()
    image, label = data[torch.randint(0, len(data), size=(1,)).item()]
    
    with torch.no_grad():
        image = image.unsqueeze(dim=0).to(device)
        pred_logit = model(image)
        pred_prob = torch.softmax(pred_logit, dim=1)
        pred_label = torch.argmax(pred_prob, dim=1)
    
    return image, label, pred_label, pred_prob

# Make predictions
image, true_label, pred_label, pred_prob = make_predictions(model, test_data, device)
print(f"True label: {true_label}")
print(f"Predicted label: {pred_label.item()}")
print(f"Prediction probability: {pred_prob.max().item():.2%}")

PyTorch Custom Datasets

Introduction to Custom Datasets

Why Custom Datasets?

• Your own data (images, text, etc.)
• Specific format or structure
• Need custom preprocessing

Creating Custom Dataset Class

from torch.utils.data import Dataset
import os
from PIL import Image

class CustomImageDataset(Dataset):
    def __init__(self, image_dir, transform=None, target_transform=None):
        self.image_dir = image_dir
        self.image_paths = [os.path.join(image_dir, f) for f in os.listdir(image_dir) 
                           if f.endswith(('.png', '.jpg', '.jpeg'))]
        self.transform = transform
        self.target_transform = target_transform
    
    def __len__(self):
        return len(self.image_paths)
    
    def __getitem__(self, idx):
        image_path = self.image_paths[idx]
        image = Image.open(image_path).convert('RGB')
        label = self._get_label(image_path)  # Implement based on your needs
        
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        
        return image, label
    
    def _get_label(self, image_path):
        # Example: Extract label from filename or directory
        # Implement based on your data structure
        return 0

Data Augmentation

from torchvision import transforms

# Training transforms (with augmentation)
train_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomRotation(degrees=15),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                        std=[0.229, 0.224, 0.225])  # ImageNet stats
])

# Test transforms (no augmentation)
test_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                        std=[0.229, 0.224, 0.225])
])

# Apply to dataset
train_data = CustomImageDataset(image_dir='train', transform=train_transform)
test_data = CustomImageDataset(image_dir='test', transform=test_transform)

TensorFlow Fundamentals Deep Dive

Deep Learning 101 with TensorFlow

Why TensorFlow?

• Deployment: Common choice for exporting and serving models
• Scalability: Handles large-scale production systems
• Ecosystem: TensorFlow Serving, TensorFlow Lite, TensorFlow.js
• Keras Integration: High-level API for easy model building
• Strong Community: Extensive documentation and support

TensorFlow Tensors:

import tensorflow as tf

# Creating tensors
scalar = tf.constant(42)
vector = tf.constant([1, 2, 3, 4])
matrix = tf.constant([[1, 2], [3, 4]])
tensor_3d = tf.constant([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])

# Random tensors
random_tensor = tf.random.normal(shape=(3, 4))  # Normal distribution
uniform_tensor = tf.random.uniform(shape=(3, 4), minval=0, maxval=1)

# Zeros and ones
zeros = tf.zeros(shape=(3, 4))
>shape=(3, 4))

Tensor Variables:

# tf.Variable: Mutable tensors (for model parameters)
weights = tf.Variable(tf.random.normal(shape=(10, 5)))
bias = tf.Variable(tf.zeros(shape=(5,)))

# Update variable
weights.assign_add(tf.random.normal(shape=(10, 5)) * 0.1)

Tensor Operations:

# Matrix multiplication
x = tf.random.normal(shape=(3, 4))
y = tf.random.normal(shape=(4, 5))
z = tf.matmul(x, y)  # or x @ y

# Element-wise operations
a = tf.constant([[1, 2], [3, 4]])
b = tf.constant([[5, 6], [7, 8]])
element_wise = a * b  # Element-wise multiplication
element_wise_sum = a + b

# Aggregation
x = tf.constant([[1, 2, 3], [4, 5, 6]])
print(tf.reduce_mean(x))  # Mean of all elements
print(tf.reduce_sum(x, axis=0))  # Sum along axis 0
print(tf.reduce_max(x, axis=1))  # Max along axis 1

Tensor Attributes:

tensor = tf.random.normal(shape=(3, 4))
print(f"Shape: {tensor.shape}")
print(f"Rank: {tf.rank(tensor)}")  # Number of dimensions
print(f"Size: {tf.size(tensor)}")  # Total number of elements
print(f"Dtype: {tensor.dtype}")

TensorFlow and NumPy:

import numpy as np

# NumPy to TensorFlow
numpy_array = np.array([1, 2, 3, 4])
tf_tensor = tf.constant(numpy_array)

# TensorFlow to NumPy
tf_tensor = tf.constant([1, 2, 3, 4])
numpy_array = tf_tensor.numpy()

GPU Acceleration:

# Check GPU availability
print(f"GPU Available: {tf.config.list_physical_devices('GPU')}")

# Use GPU if available
if tf.config.list_physical_devices('GPU'):
    with tf.device('/GPU:0'):
        # GPU operations
        x = tf.random.normal(shape=(1000, 1000))
        y = tf.random.normal(shape=(1000, 1000))
        z = x @ y

TensorFlow Regression

Regression Inputs and Outputs:

• Input: Features (e.g., house size, bedrooms)
• Output: Continuous value (e.g., house price)

Regression Architecture:

from tensorflow import keras
from tensorflow.keras import layers

# Simple regression model
model = keras.Sequential([
    layers.Dense(100, activation='relu', input_shape=(10,)),  # 10 features
    layers.Dense(50, activation='relu'),
    layers.Dense(1)  # Single output (regression)
])

model.compile(
    optimizer='adam',
    loss='mse',  # Mean Squared Error for regression
    metrics=['mae']  # Mean Absolute Error
)

Improving Regression Models:

# Method 1: Add more layers
model = keras.Sequential([
    layers.Dense(100, activation='relu', input_shape=(10,)),
    layers.Dense(100, activation='relu'),
    layers.Dense(50, activation='relu'),
    layers.Dense(1)
])

# Method 2: Change learning rate
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.01),
    loss='mse',
    metrics=['mae']
)

# Method 3: Feature scaling (normalize inputs)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Evaluating Regression Models:

# Visualize predictions
import matplotlib.pyplot as plt

predicti>
plt.figure(figsize=(10, 6))
plt.scatter(y_test, predictions, alpha=0.6)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--', lw=2)
plt.xlabel('True Values')
plt.ylabel('Predictions')
plt.title('Regression Predictions')
plt.show()

# Metrics
train_loss, train_mae = model.evaluate(X_train, y_train, verbose=0)
test_loss, test_mae = model.evaluate(X_test, y_test, verbose=0)
print(f"Train MAE: {train_mae:.4f}, Test MAE: {test_mae:.4f}")

TensorFlow Classification

Classification Inputs and Outputs:

• Input: Features (e.g., image pixels)
• Output: Class probabilities or labels

Classification Architecture:

# Binary classification
binary_model = keras.Sequential([
    layers.Dense(100, activation='relu', input_shape=(10,)),
    layers.Dense(50, activation='relu'),
    layers.Dense(1, activation='sigmoid')  # Binary classification
])

binary_model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# Multi-class classification
multiclass_model = keras.Sequential([
    layers.Dense(100, activation='relu', input_shape=(784,)),
    layers.Dense(50, activation='relu'),
    layers.Dense(10, activation='softmax')  # 10 classes
])

multiclass_model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',  # For integer labels
    metrics=['accuracy']
)

Non-Linearity in Neural Networks:

# Model without non-linearity (linear)
linear_model = keras.Sequential([
    layers.Dense(10, input_shape=(2,)),
    layers.Dense(10),
    layers.Dense(1, activation='sigmoid')
])

# Model with non-linearity (can learn non-linear patterns)
n>
    layers.Dense(10, activation='relu', input_shape=(2,)),  # ReLU activation
    layers.Dense(10, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])

Tuning Learning Rate:

# Learning rate callback
lr_scheduler = keras.callbacks.LearningRateScheduler(
    lambda epoch: 1e-4 * 10**(epoch / 20)
)

# Find ideal learning rate
history = model.fit(
    X_train, y_train,
    epochs=100,
    validation_data=(X_val, y_val),
    callbacks=[lr_scheduler]
)

# Plot learning rate vs loss
lrs = 1e-4 * (10 ** (np.arange(100) / 20))
plt.semilogx(lrs, history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding Ideal Learning Rate')
plt.show()

Classification Evaluation:

from sklearn.metrics import confusion_matrix, classification_report

# Make predictions
y_pred_probs = model.predict(X_test)
y_pred = tf.argmax(y_pred_probs, axis=1)

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)

# Classification report
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

# Accuracy
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"\nTest Accuracy: {test_acc:.4f}")

TensorFlow Computer Vision

CNNs for Image Classification:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Data augmentation
train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    zoom_range=0.2
)

test_datagen = ImageDataGenerator(rescale=1./255)

# Load data
train_generator = train_datagen.flow_from_directory(
    'data/train',
    target_size=(224, 224),
    batch_size=32,
    class_mode='categorical'
)

test_generator = test_datagen.flow_from_directory(
    'data/test',
    target_size=(224, 224),
    batch_size=32,
    class_mode='categorical'
)

Building CNN:

# CNN model
model = keras.Sequential([
    # Conv Block 1
    layers.Conv2D(32, 3, activation='relu', input_shape=(224, 224, 3)),
    layers.MaxPooling2D(2),
    
    # Conv Block 2
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(2),
    
    # Conv Block 3
    layers.Conv2D(128, 3, activation='relu'),
    layers.MaxPooling2D(2),
    
    # Classifier
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

Transfer Learning with TensorFlow:

from tensorflow.keras.applications import ResNet50

# Load pre-trained model
base_model = ResNet50(
    weights='imagenet',
    include_top=False,
    input_shape=(224, 224, 3)
)

# Freeze base model
base_model.trainable = False

# Add custom classifier
model = keras.Sequential([
    base_model,
    layers.GlobalAveragePooling2D(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

Saving and Loading Models:

# Save model
model.save('model.h5')  # HDF5 format
model.save('model.keras')  # Keras 3 format
model.save('saved_model')  # SavedModel format (for TensorFlow Serving)

# Load model
loaded_model = keras.models.load_model('model.h5')

# Save only weights
model.save_weights('weights.h5')

# Load weights (need to create model first)
new_model = create_model()
new_model.load_weights('weights.h5')

TensorFlow NLP Fundamentals

Introduction to NLP with TensorFlow

What is NLP? Natural Language Processing enables machines to understand, interpret, and generate human language.

NLP Inputs and Outputs:

• Input: Text sequences (sentences, documents, tweets)
• Output: Predictions (sentiment, classification, translation, generation)
• Sequence Problems: Order matters (unlike images where spatial structure matters)

Typical NLP Architecture:

Text → Tokenization → Embedding → RNN/LSTM/Transformer → Output

Preparing Text Data with TensorFlow

TextVectorization Layer:

from tensorflow.keras.layers import TextVectorization
import tensorflow as tf

# Create TextVectorization layer
text_vectorizer = TextVectorization(
    max_tokens=10000,  # Maximum vocabulary size
    output_sequence_length=250,  # Pad/truncate to this length
    output_mode='int'  # Return integer sequences
)

# Adapt to training data
text_vectorizer.adapt(train_texts)

# Convert text to numbers
text_vectorized = text_vectorizer(train_texts)
print(f"Vocabulary size: {text_vectorizer.vocabulary_size()}")

Creating Embeddings:

from tensorflow.keras.layers import Embedding

# Embedding layer
embedding = Embedding(
    input_dim=10000,  # Vocabulary size
    output_dim=128,   # Embedding dimension
    input_length=250  # Sequence length
)

# Use in model
model = keras.Sequential([
    text_vectorizer,
    embedding,
    # ... rest of model
])

Building NLP Models with TensorFlow

Model 0: Baseline (Dense Layers Only):

# Simple baseline model
model_0 = keras.Sequential([
    text_vectorizer,
    embedding,
    layers.GlobalAveragePooling1D(),  # Average embeddings
    layers.Dense(64, activation='relu'),
    layers.Dense(1, activation='sigmoid')  # Binary classification
])

model_0.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)