# Practice Problems Day 12
### Author:Introduction to Machine Learning
University of Redlands - DATA 301
Prof: Joanna Bieri joanna_bieri@redlands.edu
Class Website
These are practice problems. They are not graded and you do not turn them in. They are here so you can practice today’s code before the weekly homework. Weekly Homework 6 assumes you have done them. Plan on about 45 minutes.
Solutions are in HW-day12-solutions.ipynb in this folder. Try every problem yourself first, then check your work.
Reading: Geron chapter 11, Batch Normalization, Faster Optimizers and Learning Rate Scheduling.
GOALS:
- Switch a network from plain SGD to Adam.
- Add a learning rate schedule to a training run.
- Explain training mode and prediction mode in words.
How much of this should you write yourself? The starting cell is given, it is the network from the Day 11 practice. Problem 1 changes one line. Problem 2 lists the steps. Problem 3 is words only. The Optional exploration at the end is extra, only if you are curious.
Save your work. Commit and push when you are done, so you have your code for the weekly homework and the exam:
git add .
git commit -m "Day 12 practice"
git push# RUN THIS. The California housing data from the notes, split, scaled, and turned into tensors.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
housing = pd.read_csv("data/california_housing.csv")
X = housing.drop(columns=["MedHouseVal"])
y = housing["MedHouseVal"]
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.25, random_state=42)
preprocessing = Pipeline([("scale", StandardScaler())]) # the preprocessing pipeline, fit on the training rows only
X_train_scaled = preprocessing.fit_transform(X_train)
X_valid_scaled = preprocessing.transform(X_valid)
X_train_t = torch.tensor(X_train_scaled, dtype=torch.float32)
X_valid_t = torch.tensor(X_valid_scaled, dtype=torch.float32)
y_train_t = torch.tensor(y_train.values, dtype=torch.float32).reshape(-1, 1)
y_valid_t = torch.tensor(y_valid.values, dtype=torch.float32).reshape(-1, 1)
train_data = TensorDataset(X_train_t, y_train_t) # pairs the features with the labels for the DataLoader
print("training rows:", X_train_t.shape[0], " validation rows:", X_valid_t.shape[0])# RUN THIS. The network from the Day 11 practice,
# 4 hidden layers, Kaiming, plain SGD.
torch.manual_seed(42)
model = nn.Sequential(
nn.Linear(8, 40), # 8 features in, 40 neurons
nn.ReLU(),
nn.Linear(40, 40),
nn.ReLU(),
nn.Linear(40, 40),
nn.ReLU(),
nn.Linear(40, 40),
nn.ReLU(),
nn.Linear(40, 1)) # 1 prediction out
# the starting weights
for layer in model:
if isinstance(layer, nn.Linear): # only the Linear layers have weights
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
nn.init.zeros_(layer.bias)
# the batches, the loss and the optimizer
train_loader = DataLoader(train_data, batch_size=32, shuffle=True)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001) # the optimizer
# the training loop
n_epochs = 20
for epoch in range(n_epochs):
model.train() # training mode
for X_batch, y_batch in train_loader:
optimizer.zero_grad() # clear the old gradients
y_pred = model(X_batch) # forward pass
loss = loss_fn(y_pred, y_batch) # how far off are we?
loss.backward() # backward pass
optimizer.step() # one step, once per BATCH
# after the first epoch and every 5 epochs, check the validation RMSE
if epoch == 0 or (epoch + 1) % 5 == 0:
model.eval() # prediction mode
with torch.no_grad():
rmse = torch.sqrt(loss_fn(model(X_valid_t), y_valid_t)).item()
print("epoch", epoch + 1, " valid RMSE:", round(rmse, 4))Problem 1: Switch to Adam
What to copy, what to write: Copy the cell above into the cell below. Change only the optimizer line.
1a. Change the optimizer to Adam with a learning rate of 0.001 and run it. How does it compare to plain SGD after 1 epoch and after 20 epochs?
1b. In your own words, why can Adam do better than plain SGD with the same learning rate? Use the word “adaptive”.
# 1a: your copy, with AdamYour answers here.
Problem 2: Add a learning rate schedule (steps outlined)
What to copy, what to write: Copy your Adam cell from Problem 1 and add the schedule yourself. It is the same pattern as the “real life” cell at the end of the notes.
2a. Add a StepLR schedule with step_size=10 and gamma=0.1 to your Adam cell, and train for 20 epochs.
2b. Compare the validation RMSE at epochs 15 and 20, with and without the schedule. Did the schedule help here?
2c. Where does scheduler.step() go in the loop, and where does optimizer.step() go? Why are they in different places?
# Step 1: copy your Adam cell from Problem 1
# Step 2: right after the optimizer line, make the scheduler:
# torch.optim.lr_scheduler.StepLR(optimizer, step_size=..., gamma=...)
# Step 3: inside the epoch loop, AFTER the batch loop, add scheduler.step()Your answers here.
Problem 3: In words
3a. A classmate added batch normalization to their network. Their validation RMSE comes out a little different every time they change the batch size they use for the validation rows, even though the model has not changed. What did they forget, and why does it cause this?
3b. A friend’s network with 10 hidden layers gives a loss of nan when they train it with plain SGD. Name three things from Day 11 and Day 12 they could try, in the order you would try them, and say why each one could help.
Your answers here.
Optional exploration
Only if you are curious. Nothing here is needed for the weekly homework.
A. Go back to the starting cell and change the optimizer to SGD with momentum=0.9 and a learning rate of 0.0001. Compare it to plain SGD with a learning rate of 0.001. Why are they so close?
B. Train your Adam cell with a learning rate of 0.01 instead of 0.001. Is the bigger learning rate better here?
# your code hereYour answers here.