Deep Learning — TechForge

7+Architectures

1000+Layers (GPT)

GPURequired

2012Modern Era

00

What is Deep Learning?

Deep Learning = neural networks with many layers that automatically learn hierarchical representations from raw data. No manual feature engineering needed. The core loop: Input → Hidden Layers → Output → Loss → Backprop → Update → Repeat.

Input (pixels/text/audio) → [Layer 1: edges] → [Layer 2: shapes] → [Layer N: faces] → Output

01

The Neuron — Basic Computing Unit

Each neuron receives inputs, multiplies by learned weights, adds a bias, passes through a nonlinear activation function. Inspired loosely by biological neurons but far simpler. Single neuron = linear classifier; stack them and they approximate anything.

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b) = activation(Wᵀx + b)

02

Input → Hidden → Output Layers

Input layer: Raw features (784 pixels for MNIST, 512 tokens for text). No computation — just passes data forward. Hidden layers: Learn increasingly abstract representations. More layers = more abstraction. Output layer: Task-specific — sigmoid for binary, softmax for multiclass, linear for regression.

Dense(784) → Dense(256,ReLU) → Dense(128,ReLU) → Dense(10,Softmax)

03

Universal Approximation Theorem

A single hidden layer with enough neurons can approximate any continuous function to arbitrary precision. This mathematical guarantee is why neural networks are so powerful — in theory, they can learn any mapping. In practice, depth is far more efficient than width.

∀ε > 0, ∃NN: |f(x) − NN(x)| < ε · depth wins over width in practice

04

Deep vs Shallow — Why Depth Matters

Shallow networks need exponentially more neurons to match deep ones. Deep networks learn compositional features hierarchically — pixels → edges → shapes → objects → faces. This matches the structure of natural data and is exponentially more parameter-efficient.

Layer 1: edges & gradients → Layer 2: textures → Layer 3: parts → Layer N: objects

Core DL Architectures

ANN / MLP

Fully Connected

Every neuron connects to every next-layer neuron. Good for structured tabular data and simple pattern recognition.

Tabular data · churn · pricing · regression

CNN

Convolutional

Sliding filters detect local spatial patterns. Translation invariant. Hierarchical feature learning from pixels to objects.

Images · video · medical imaging · OCR

RNN / LSTM

Recurrent

Maintains hidden state across time steps. LSTM gates solve vanishing gradient. Processes sequences left-to-right.

Time series · text (legacy) · speech

Transformer

Attention

Self-attention: every token attends to every other. Fully parallelizable. No sequential bottleneck. Powers all modern LLMs.

NLP · vision · audio · multimodal

GAN

Generative

Generator vs Discriminator adversarial game. Generator fools discriminator with increasingly realistic outputs.

Image synthesis · deepfakes · style transfer

Autoencoder

Self-Supervised

Compresses input to bottleneck latent space, reconstructs it. Learns representations without labels. VAE adds probabilistic latent space.

Anomaly detection · compression · generation

Why Deep Learning dominates unstructured data: Images, audio, and text have spatial/temporal structure that CNNs and Transformers exploit natively. For tabular data, XGBoost still often beats neural networks — always try both.

00

The 3-Step Learning Loop

Every neural network training run repeats this loop thousands or millions of times: 1. Forward pass — make a prediction. 2. Compute loss — measure how wrong it is. 3. Backprop — compute gradients and update weights to reduce loss.

Predict → Measure Loss → Backprop Gradients → Update Weights → Repeat

01

Forward Pass

Input flows through the network layer by layer. Each layer applies: weighted sum → activation → next layer. Final layer produces prediction ŷ. Loss function measures error: gap between ŷ and true label y.

ŷ = forward(x) · L = CrossEntropy(ŷ, y) · lower L = better prediction

02

Backpropagation & Chain Rule

Compute gradient of loss with respect to every weight using the chain rule. Flow gradients backwards from output to input. Each weight receives a gradient — the direction and magnitude to adjust to reduce loss. This is the most important algorithm in modern AI.

∂L/∂w = ∂L/∂ŷ · ∂ŷ/∂h · ∂h/∂w (chain rule all the way back)

03

Gradient Descent & Optimizers

SGD: vanilla — stable but slow. SGD + Momentum: accumulates velocity, escapes local minima faster. Adam: adaptive learning rates per parameter + momentum — the default for most DL tasks. AdamW: Adam + weight decay — standard for Transformers.

w ← w − α·∂L/∂w · Adam: m̂/(√v̂+ε) · AdamW adds λw decay

04

Epochs, Batches & Iterations

Epoch: one full pass through all training data. Batch size: samples processed before one weight update. Iteration: one forward + backward + update = one batch. Mini-batches (32–512) give stable gradients + efficient GPU utilization. Powers of 2 for GPU efficiency.

1000 samples, batch=100 → 1 epoch = 10 iterations · typical: 32, 64, 128, 256

05

Learning Rate & Schedulers

Too high → loss diverges or oscillates. Too low → extremely slow convergence. Warmup + cosine decay is standard for Transformers. ReduceLROnPlateau: decay when val_loss stagnates. Cyclic LR: periodically vary rate to escape local minima.

CosineAnnealingLR(optimizer, T_max=100) · typical lr: 1e-3 (Adam), 0.1 (SGD)

06

Vanishing & Exploding Gradients

Vanishing: gradients shrink exponentially in deep networks with sigmoid/tanh. Layers near input stop learning. Fixed by: ReLU activations, skip connections (ResNet), proper weight init (Xavier/He). Exploding: gradients blow up — fixed by gradient clipping.

clip_grad_norm_(params, max_norm=1.0) · He init for ReLU, Xavier for tanh

07

Regularization Techniques

Dropout: randomly zero neurons during training (p=0.1–0.5) — forces redundant representations. L2 / Weight Decay: penalize large weights. Batch Normalization: normalize activations per batch — dramatically stabilizes and accelerates training. Early Stopping: halt when val_loss stops improving (patience=5–10).

Dropout(p=0.3) → BatchNorm → EarlyStopping(patience=7) → weight_decay=1e-4

Loss Functions by Task

MSE

Mean Squared Error

Regression. Penalises large errors heavily. (ŷ − y)² averaged. Standard for continuous prediction.

BCE

Binary Cross-Entropy

Binary classification (0 or 1 output). −[y·log(ŷ) + (1−y)·log(1−ŷ)]. Use with sigmoid output.

CCE

Categorical Cross-Entropy

Multi-class classification. −Σy·log(ŷ). Use with softmax output. The standard classification loss.

Huber

Huber Loss

Regression with outliers. MSE for small errors, MAE for large — robust blend. Use when target has outliers.

Training checklist: Start with lr=1e-3 + Adam. Use BatchNorm before activation. Add Dropout after large Dense layers. Always plot train vs val loss — the gap reveals overfitting instantly. Use early stopping in production.

00

Why Activations Matter

Without activation functions, stacking layers collapses to a single linear transformation — no matter how deep. Activations introduce nonlinearity, enabling the network to learn complex, curved decision boundaries and hierarchical patterns.

No activations: N layers = 1 linear layer · With activations: can approximate any function

Activation Function Reference

Sigmoid

σ(x) = 1/(1+e⁻ˣ)

Output: (0, 1)

Binary output layer only. Suffers vanishing gradient — avoid in hidden layers.

Tanh

tanh(x) = (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ)

Output: (−1, 1)

Zero-centred (better than sigmoid). Still vanishes in very deep networks.

ReLU

max(0, x)

Output: [0, ∞)

Default for hidden layers. Fast, no vanishing for x>0. Can "die" (neurons stuck at 0).

Leaky ReLU

max(αx, x), α=0.01

Output: (−∞, ∞)

Fixes dying ReLU — small negative slope keeps neurons active. Robust choice.

GELU

x·Φ(x)

Smooth, unbounded

Used in BERT, GPT, Transformers. Smooth approximation of ReLU. Better gradient flow.

Softmax

eˣⁱ / Σⱼ eˣʲ

Output: (0,1), sums=1

Multi-class output only. Converts raw logits to probability distribution.

Quick Selection Guide

Hidden

Hidden Layers

Use ReLU by default. GELU for Transformers. Leaky ReLU if seeing dead neurons. Never Sigmoid/Tanh in hidden layers of deep nets.

Binary

Binary Output

Sigmoid — outputs probability (0–1). Loss: binary cross-entropy. Threshold at 0.5 by default (adjust for precision/recall tradeoff).

Multi

Multi-class Output

Softmax — outputs class probability distribution summing to 1. Loss: categorical cross-entropy. One output per class.

Regression

Regression Output

Linear (no activation) — unbounded output. Loss: MSE or Huber. Can also use ReLU if output must be non-negative.

Rule of thumb: Start every hidden layer with ReLU + BatchNorm. Switch to GELU if building attention-based models. The output activation is always determined by your task — never apply softmax in hidden layers.

01

The Convolution Operation

A small filter (kernel) slides across the input, computing a dot product at each position to produce a feature map. Learns to detect local patterns: edges, textures, gradients. Key property: shared weights across positions → translation invariance (a cat is a cat wherever it is in the image).

output[i,j] = Σₘ Σₙ kernel[m,n] · input[i+m, j+n] · one kernel = one feature map

02

Padding & Stride

Padding='same': add zeros around border to preserve spatial size. Padding='valid': no padding — output smaller than input. Stride: step size of the filter. Stride=2 halves the spatial dimensions without a pooling layer.

Output size = (W − K + 2P)/S + 1 · W=input, K=kernel, P=pad, S=stride

03

Pooling — Spatial Downsampling

Max pooling: keeps the most prominent feature in each window — the standard choice. Average pooling: takes mean — smoother. Global Average Pooling: collapses each feature map to a single number — replaces Flatten+Dense, greatly reduces parameters.

MaxPool2d(2,2) → halves H and W · AdaptiveAvgPool2d(1) → [B, C, 1, 1]

04

CNN Architecture Pattern

Stack [Conv → BatchNorm → ReLU → Pool] blocks. Early blocks: low-level features (edges, textures). Later blocks: high-level semantics (eyes, wheels). Flatten or Global Avg Pool → Dense head for classification.

[Conv→BN→ReLU→Pool] × N → GlobalAvgPool → Dense → Softmax

05

Landmark CNN Architectures

LeNet (1998): first practical CNN for handwritten digits. AlexNet (2012): won ImageNet by 10%, launched the DL era. VGG: deep stacks of 3×3 convs, simple but heavy. ResNet (2015): skip connections allow 50–152 layers without degradation. EfficientNet: compound scaling of all dimensions optimally. ViT: pure Transformer on image patches — beats CNNs at scale.

ResNet skip: output = F(x) + x · prevents vanishing, enables very deep networks

06

Transfer Learning — Practical Power

Load a pretrained CNN (ResNet, EfficientNet) trained on ImageNet (1.2M images, 1000 classes). Freeze early layers (generic edge detectors). Replace classification head. Fine-tune last few layers on your task. Works with only hundreds of images — the most practical DL technique for most projects.

model = resnet50(pretrained=True) · model.fc = nn.Linear(2048, n_classes)

Real talk: In production, you almost never train a CNN from scratch. Always start with a pretrained backbone (ResNet-50, EfficientNet-B4, ViT-B/16) and fine-tune. Training from scratch requires millions of images and weeks of GPU time.

01

Recurrent Neural Network (RNN)

Processes sequences by maintaining a hidden state hₜ that carries information from previous time steps. At each step: new input xₜ + previous hidden state hₜ₋₁ → new state. This "memory" enables modeling temporal dependencies in text, audio, and time series.

hₜ = tanh(Wₕ·hₜ₋₁ + Wₓ·xₜ + b) · same weights used at every step

02

The Vanishing Gradient Problem

Gradients decay exponentially through time steps during backprop-through-time. With 100 steps, the gradient from step 1 is multiplied by Wₕ 100 times. If |Wₕ| < 1, gradient → 0; if |Wₕ| > 1, gradient explodes. RNNs can't remember what happened 50+ steps ago.

gradient ∝ Wₕᵗ → 0 as t → ∞ (if |Wₕ| < 1) · makes long-range memory impossible

03

LSTM — Long Short-Term Memory

Solves vanishing gradient with a cell state cₜ that acts as a memory highway. Three gates control information flow: Forget gate fₜ: what to erase from memory. Input gate iₜ: what new information to store. Output gate oₜ: what to output. Gradients can flow unchanged through the cell state.

cₜ = fₜ⊙cₜ₋₁ + iₜ⊙c̃ₜ · hₜ = oₜ⊙tanh(cₜ) · all gates use sigmoid (0–1)

04

GRU — Gated Recurrent Unit

Simplified LSTM — merges cell state and hidden state, uses 2 gates instead of 3. Reset gate: how much past to forget. Update gate: how much to update state. ~33% fewer parameters than LSTM. Similar performance, faster training. Often preferred when compute is limited.

hₜ = (1−zₜ)⊙hₜ₋₁ + zₜ⊙h̃ₜ · simpler than LSTM, comparable performance

05

Bidirectional RNNs

Run one LSTM left-to-right and another right-to-left. Concatenate both hidden states. This lets the model use both past and future context — crucial for NLP understanding tasks (BERT does this implicitly with attention, not recurrence).

BiLSTM: hₜ = [→hₜ ; ←hₜ] · doubles hidden size · used in NER, QA

06

Status: Largely Replaced by Transformers

Transformers parallelise over sequence length — LSTMs are sequential and can't be parallelised during training. For NLP, Transformers (BERT, GPT) dominate since 2018. LSTMs still competitive for: streaming time series (lower latency), small datasets, edge deployment (fewer parameters).

LSTM still wins: streaming · resource-constrained · structured time series

01

Self-Attention — The Core Idea

Every token attends to every other token simultaneously. In "The bank by the river" — "bank" attends strongly to "river" (financial meaning suppressed). No sequential processing — all positions computed in parallel. This is why Transformers are both more powerful and faster to train than RNNs.

Attention(Q,K,V) = softmax(QKᵀ / √d_k) · V · √d_k prevents vanishing softmax gradients

02

Query, Key, Value — The Retrieval Analogy

Each token produces three vectors via learned matrices: Q (Query): "what am I looking for?" K (Key): "what do I offer to others?" V (Value): "what information do I contain?" Attention score = Q·K similarity. Output = weighted sum of Vs (high-scoring tokens contribute more).

Q = X·Wq K = X·Wk V = X·Wv · all Wq, Wk, Wv are learned

03

Multi-Head Attention

Run attention H times in parallel with different weight matrices. Each head learns to focus on a different aspect — one head may track syntax, another semantics, another coreference. Concatenate all head outputs, project to original dimension. GPT-3 uses 96 heads.

MultiHead(Q,K,V) = Concat(head₁,...,headₕ)·Wₒ · each headᵢ = Attention(QWᵢq, KWᵢk, VWᵢv)

04

Positional Encoding

Attention is permutation-invariant — it doesn't know word order. Fix: add a position signal to each token embedding before the first attention layer. Original paper uses sinusoidal encoding. Modern models (BERT, GPT) use learned positional embeddings. RoPE (Rotary Position Embedding) is now standard for long contexts.

PE(pos,2i) = sin(pos/10000^(2i/d)) · modern: learned or RoPE

05

Transformer Block Architecture

One Transformer block = Multi-Head Attention → Add&Norm → Feed-Forward → Add&Norm. Residual connections (Add) prevent degradation and allow gradient flow in very deep stacks. LayerNorm stabilizes training. FFN is typically 4× wider than model dim.

x = x + MHA(LN(x)) · x = x + FFN(LN(x)) · stack N of these (GPT-3: 96 blocks)

06

Encoder vs Decoder Transformers

Encoder-only (BERT): bidirectional attention, sees all tokens simultaneously. Pre-trained with MLM. Best for understanding: classification, NER, QA. Decoder-only (GPT, Claude, Llama): causal mask — only attends left-to-right. Pre-trained with next-token prediction. Best for generation. Encoder-Decoder (T5, BART): translation, summarization.

BERT: [MASK] prediction · GPT: predict next token · T5: text-to-text everything

BERT

Encoder · 110M params

Bidirectional. MLM + NSP pretraining. Dominates classification, NER, QA tasks. Fine-tune on your task in minutes.

GPT-4

Decoder · ~1.8T params

Causal autoregressive. Powers ChatGPT. In-context learning — no fine-tuning needed for many tasks.

T5

Enc-Dec · 11B params

Text-to-text framework — treats every NLP task as generating text from text. Translation, summarization, QA.

ViT

Vision Transformer

Split image into 16×16 patches, treat as sequence tokens. Beats CNNs on large-scale vision with enough data.

01

GAN — Generative Adversarial Network

Two networks in competition: Generator G: takes random noise z, produces fake data (images). Discriminator D: tells real from fake. G trains to fool D; D trains to not be fooled. At equilibrium, G produces data indistinguishable from real. Training is notoriously unstable — mode collapse is a common failure.

G objective: max log D(G(z)) · D objective: max [log D(x) + log(1−D(G(z)))]

02

VAE — Variational Autoencoder

Encoder maps input x to a distribution (μ, σ) in latent space rather than a single point. Decoder samples z ~ N(μ, σ²) and reconstructs x. The KL divergence term forces the latent space to be smooth and continuous — enabling interpolation and controlled generation.

z ~ N(μ(x), σ²(x)) · Loss = Recon + β·KL(N(μ,σ²) ‖ N(0,1))

03

Diffusion Models — The New SOTA

Forward process: Gradually add Gaussian noise to data over T steps until pure noise. Reverse process: Train a U-Net to predict and remove noise one step at a time. At inference: start from pure noise and iteratively denoise. Produces stunning image quality — DALL-E 2, Midjourney, Stable Diffusion all use this approach.

Forward: x₀→xₜ add noise · Reverse: xₜ→x₀ denoise with ε-prediction network

04

Conditional Generation & Guidance

Condition generation on text, class, or other signals. Classifier-Free Guidance (CFG): trains both conditioned and unconditioned models, combines at inference — the standard for text-to-image. Higher CFG scale → more faithful to prompt, less diverse. Stable Diffusion uses CFG ≈ 7.5.

ε̂ = (1+w)·ε_cond − w·ε_uncond · w=CFG scale, typical 5–15

Generative Model Comparison

Model	Quality	Speed	Stability	Diversity	Best For
GAN	⭐⭐⭐⭐	Fast	Hard to train	Medium	Real-time synthesis, style transfer
VAE	⭐⭐⭐	Fast	Stable	High	Latent interpolation, anomaly detection
Diffusion	⭐⭐⭐⭐⭐	Slow inference	Very stable	Very high	Text-to-image (DALL-E, SD, Midjourney)
Flow-based	⭐⭐⭐	Medium	Stable	High	Exact likelihood, audio (WaveGlow)

python — PyTorch CNN Training Loop (MNIST)

import torch
import torch.nn as nn
import torch.optim as optim

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1), nn.Flatten()
        )
        self.classifier = nn.Sequential(
            nn.Dropout(0.3), nn.Linear(64, 10)
        )
    def forward(self, x):
        return self.classifier(self.features(x))

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model     = CNN().to(device)
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss()
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)

for epoch in range(10):
    model.train()
    for imgs, labels in train_loader:
        imgs, labels = imgs.to(device), labels.to(device)
        optimizer.zero_grad()
        loss = criterion(model(imgs), labels)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
    scheduler.step()
    # eval loop
    model.eval()
    with torch.no_grad():
        correct = sum((model(x.to(device)).argmax(1) == y.to(device)).sum().item()
                      for x, y in val_loader)
    print(f"Epoch {epoch+1} · Val acc: {correct/len(val_loader.dataset):.3f}")

python — Transfer Learning with ResNet-50

import torchvision.models as models

# Load pretrained backbone
model = models.resnet50(weights='IMAGENET1K_V2')

# Freeze all layers
for p in model.parameters():
    p.requires_grad = False

# Replace classification head for your task
model.fc = nn.Sequential(
    nn.Linear(2048, 512), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(512, num_classes)
)

# Only train the new head
optimizer = optim.Adam(model.fc.parameters(), lr=1e-3)
# After a few epochs, unfreeze and fine-tune everything at lower lr
for p in model.parameters():
    p.requires_grad = True
optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=1e-4)

python — Keras Sequential (Quick Prototyping)

from tensorflow import keras

model = keras.Sequential([
    keras.layers.Dense(256, activation='relu', input_shape=(20,)),
    keras.layers.BatchNormalization(),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
cb = [keras.callbacks.EarlyStopping(patience=7, restore_best_weights=True),
      keras.callbacks.ReduceLROnPlateau(patience=3)]
model.fit(X_train, y_train, epochs=100, batch_size=64,
          validation_split=0.2, callbacks=cb)

Use this guide to pick the right architecture for your problem. When in doubt: start with a pretrained Transformer for text/multimodal, pretrained CNN for images, XGBoost for tabular.

Architecture Selection Table

Architecture	Best Data	Parallelise?	Long Context	Params (typical)	Start With
MLP / ANN	Tabular	Yes	No	1K–10M	`nn.Linear`
CNN	Images, audio spectrograms	Yes	Limited	1M–100M	ResNet-50, EfficientNet
RNN / LSTM	Short sequences	No (sequential)	Poor	1M–50M	`nn.LSTM`
Transformer	Text, images, audio, video	Yes	Excellent	100M–1T+	BERT, GPT-2, ViT
GAN	Images, audio	Yes	N/A	5M–500M	StyleGAN2, DCGAN
Diffusion	Images, video, audio	Yes	N/A	100M–3B	Stable Diffusion

Optimizer Comparison

SGD

+ Momentum

Best final accuracy for CNNs with careful tuning. Slow to converge. lr=0.1 with cosine decay. Rarely the default anymore.

Adam

Adaptive Moments

Adaptive lr per parameter + momentum. Fast convergence. Default for most tasks. lr=1e-3. Can generalize slightly worse than SGD on images.

AdamW

Adam + Weight Decay

Decouples weight decay from gradient update. Standard for Transformers. lr=1e-4 to 3e-4. Always prefer over Adam for LLMs.

Lion

EvoLved Sign

Sign-based update. More memory-efficient than Adam. Good for large models. Use weight decay=0.1 (10× higher than usual).

Regularization Techniques Summary

Dropout

Random Zeroing

p=0.1–0.5. Applied during training only. Forces redundant representations. Essential for Dense layers. Less used after BatchNorm became standard.

BatchNorm

Batch Normalisation

Normalises activations per mini-batch. Dramatically stabilises training, allows higher lr. Place before activation. Standard in CNNs.

LayerNorm

Layer Normalisation

Normalises across features (not batch). Standard in Transformers. Works at batch_size=1. Replace BatchNorm when using attention.

MixUp/CutMix

Data Augmentation

Blend pairs of training images (MixUp) or cut-paste patches (CutMix). State-of-the-art image regularization. Easy gains with 2 lines of code.