Introduction to Machine Learning

# Weekly Homework 6: Diamond Prices

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website

Due Sunday 11/1 at 11:59pm. This covers Day 11 and Day 12.

Can we predict the price of a diamond from its size and quality? This week you write the whole process in one cell, just like the “What this looks like in real life” cells at the end of the Day 11 and Day 12 notes: the network, Kaiming initialization, the Adam optimizer, a learning rate schedule and the training loop. The comments in each code cell outline the steps for you.

GOALS:

Explain the problems in a dataset that has no missing values on paper.
Build a linear regression baseline with a preprocessing pipeline.
Write the whole training process for a deep network in one cell, with and without a learning rate schedule, and compare.
Open the test set once and explain what the model can and can not do.

What you can copy. Copy freely from the Day 8, 10, 11 and 12 notes and the practice problems, and change the names. The “real life” cells at the end of the Day 11 and Day 12 notes are your best guide for Part 3. Every sentence you write is yours.

The data. data/diamonds.csv. One row per diamond, almost 54,000 diamonds. This is the diamonds dataset that comes with the seaborn and ggplot2 plotting libraries. It is real data, and it has some mistakes in it.

column	what it means
`carat`	the weight of the diamond
`cut`	the quality of the cut: Fair, Good, Very Good, Premium, Ideal
`color`	the color grade, from J (worst) to D (best)
`clarity`	how clear the diamond is: I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best)
`depth`, `table`	two measurements of the diamond’s shape, in percent
`x`, `y`, `z`	the length, width and depth of the diamond in millimeters
`price`	the price in US dollars. This is what we predict.

How long does it take? Plan on about two hours. All of the code together runs in a few minutes.

How to turn this in. Your repository on GitHub is your submission. There is nothing to hand in on Canvas.

The finished work is on your main branch.
It is pushed to GitHub before the deadline.
Your name is on the Author line at the top of this notebook.

git add .
git commit -m "Weekly homework 6"
git push

I grade from whatever is on GitHub at the deadline. If it is not pushed, I cannot see it.

Part 1: Find the problems

The code in this part is given. Run it and read the output.

1a. isna() says there are 0 missing values. But look at the smallest values of x, y and z. How many rows have an x, y or z that is exactly 0? What do you think a 0 means in these columns?

1b. Look at the rows where y or z is bigger than 20. Compare their x, y, z and carat to a typical diamond. Why do you think these are mistakes?

The last cell drops both kinds of problem rows and makes our label, price_k, the price in thousands of dollars. (Remember Weekly Homework 5: big labels make big gradients.)

# RUN THIS. Load the data.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

diamonds = pd.read_csv("data/diamonds.csv")

print("rows and columns:", diamonds.shape)
diamonds.head()

# RUN THIS. 1a: missing values, and the smallest and biggest value in each number column
print("missing values:", diamonds.isna().sum().sum())
print(diamonds[["carat", "depth", "table", "x", "y", "z", "price"]].describe().round(2))

# rows where a length is exactly 0
zero_rows = (diamonds["x"] == 0) | (diamonds["y"] == 0) | (diamonds["z"] == 0)      # | means "or"
print()
print("rows with an x, y or z of 0:", zero_rows.sum())

# RUN THIS. 1b: rows where y or z is bigger than 20 millimeters
too_big = (diamonds["y"] > 20) | (diamonds["z"] > 20)
print("rows with y or z bigger than 20:", too_big.sum())
diamonds[too_big]

# RUN THIS. Drop both kinds of problem rows, then make the label
diamonds = diamonds[~zero_rows & ~too_big].copy()      # ~ means "not", so we keep the rows that are not a problem
print("rows left:", diamonds.shape[0])

diamonds["price_k"] = diamonds["price"] / 1000         # the price in thousands of dollars
print(diamonds["price_k"].describe().round(3))

Your answers here.

Part 2: Split, preprocess, and a baseline

The split is given: training (60 percent), validation (20 percent) and test (20 percent). Then you write the preprocessing and the baseline.

Our baseline is a linear regression. It is the number our network has to beat. Put the preprocessing and a LinearRegression together in one Pipeline, fit it on the training rows, and report its validation RMSE.

2a. What is the linear regression’s validation RMSE? Say it in dollars (the label is in thousands of dollars). Then look at the prices in the training set. Is that a big error for a diamond at the 25th percentile of price? For one at the 75th percentile?

# RUN THIS. The imports and the split.
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LinearRegression
from sklearn.metrics import root_mean_squared_error

X = diamonds.drop(columns=["price", "price_k"])          # the features: every column except the prices
y = diamonds["price_k"]                                    # the label: the price in thousands of dollars

X_train_full, X_test, y_train_full, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full, test_size=0.25, random_state=42)
print("training:", X_train.shape[0], "  validation:", X_valid.shape[0], "  test:", X_test.shape[0])

# 2a: the preprocessing and the linear regression baseline

# Step 1: two lists of column names. The number columns are carat, depth, table, x, y and z.
#         The text columns are cut, color and clarity.

# Step 2: a ColumnTransformer called preprocessing.
#         StandardScaler() on the number columns,
#         OneHotEncoder(handle_unknown="ignore", sparse_output=False) on the text columns.

# Step 3: a Pipeline called lin_pipeline with two steps: the preprocessing, then LinearRegression()

# Step 4: fit lin_pipeline on the training rows (X_train, y_train)

# Step 5: rmse_lin = the validation RMSE. Use root_mean_squared_error(y_valid, lin_pipeline.predict(X_valid)). Print it.

# Step 6: print y_train.describe() to see the prices in the training set

Your answers here.

Part 3: Train a Neural Net

Now the network. You can write it the way we did at the end of the Day 11 and Day 12 notes: everything in one cell, with no helper functions. The comments outline every step. If you want to write or use helpers that is okay too, just comment each line so it is clear you understand what is going on.

NOTE - Just like last week, a PyTorch model can not go inside a scikit-learn Pipeline. So the first step sends the rows through your preprocessing yourself: fit_transform on the training rows, transform only on the validation rows.

3a. Create a network with 5 hidden layers of 50 neurons with ReLU, Kaiming initialization, the Adam optimizer with a learning rate of 0.001, and a StepLR schedule with step_size=10 and gamma=0.1, trained for 20 epochs. How many columns come out of the preprocessing? How does the network’s validation RMSE compare to the linear regression baseline?

3b. Copy your cell into the next cell and take out the schedule, so it is Adam alone. Then run the plot cell. Look at the last five epochs of each. Which model would you keep, and why?

3c. For the model you chose in 3b, compute the RMSE on the training rows and compare it to its validation RMSE. What does the gap tell you?

# 3a: the whole process in one cell, Adam with a learning rate schedule
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

# Step 1: send the rows through your preprocessing:
#         X_train_prep = preprocessing.fit_transform(X_train)    (fit on the training rows only)
#         X_valid_prep = preprocessing.transform(X_valid)        (transform ONLY)
#         Print X_train_prep.shape. How many columns came out?

# Step 2: float32 tensors X_train_t, X_valid_t, y_train_t, y_valid_t.
#         The labels need .values and .reshape(-1, 1)

# Step 3: torch.manual_seed(42), then schedule_model = nn.Sequential(...)
#         5 hidden layers of 50 neurons, each followed by nn.ReLU(), and one output.
#         The first nn.Linear needs as many inputs as X_train_prep has columns.

# Step 4: Kaiming initialization. Loop over the layers. For every nn.Linear use
#         nn.init.kaiming_normal_ on the weight (nonlinearity="relu") and nn.init.zeros_ on the bias.

# Step 5: the DataLoader, TensorDataset(X_train_t, y_train_t) with batch_size=64 and shuffle=True,
#         and loss_fn = nn.MSELoss()

# Step 6: the optimizer, torch.optim.Adam with lr=0.001,
#         and the scheduler, torch.optim.lr_scheduler.StepLR with step_size=10 and gamma=0.1

# Step 7: the training loop for 20 epochs. Start with an empty list valid_rmse_schedule = []
#         For every epoch:
#           schedule_model.train()
#           the five steps for every batch: zero_grad, predict, loss, backward, optimizer.step()
#           scheduler.step()   (once per EPOCH, after the batches)
#           schedule_model.eval(), then the validation RMSE inside torch.no_grad()
#           append the RMSE to valid_rmse_schedule
#           print it after the first epoch and every 5 epochs

# 3b: copy your 3a cell here and take out the schedule
#     Change the model name to adam_model and the list name to valid_rmse_adam.
#     Delete the scheduler line in Step 6 and the scheduler.step() line in the loop.

# RUN THIS after 3a and 3b. The two Adam learning curves and the last five epochs of each.
epochs = range(1, 21)
plt.figure(figsize=(6, 4))
plt.plot(epochs, valid_rmse_adam, label="Adam")
plt.plot(epochs, valid_rmse_schedule, label="Adam with a schedule")
plt.axhline(rmse_lin, color="gray", linestyle="--", label="linear regression")
plt.xlabel("epoch")
plt.ylabel("validation RMSE (thousands of dollars)")
plt.legend()
plt.grid()
plt.show()

print("last five epochs, Adam:              ", np.round(valid_rmse_adam[15:], 4))
print("last five epochs, Adam with schedule:", np.round(valid_rmse_schedule[15:], 4))

# 3c: the training RMSE and the validation RMSE of the model you chose
# Step 1: put the model in prediction mode with .eval()
# Step 2: inside torch.no_grad(), compute the RMSE on X_train_t, y_train_t and on X_valid_t, y_valid_t
#         (the square root of loss_fn(...), then .item())
# Step 3: print both

Your answers here.

Part 4: The test set, once

BEWARE! The test rows go through the same preprocessing, with transform only:

X_test_prep = preprocessing.transform(X_test)     # transform ONLY, never fit_transform

4a. Compute your chosen network’s test RMSE, one time. Compare it to its validation RMSE, and to the linear regression’s RMSE on the same test rows. What do you notice?

4b. An RMSE is one number for all of the diamonds. The second cell (given) splits it into diamonds under $1,000 and diamonds that cost $10,000 or more. What do you notice?

4c. Write three or four sentences to the owner of a jewelry store who wants to use your model to set prices. Say what the model does, how far off it typically is for cheap diamonds and for expensive ones, and one reason to be careful with it. No jargon: no “RMSE”, no “Adam”.

# 4a: the test set, once
# Step 1: X_test_prep with preprocessing.transform (transform ONLY), then the tensors X_test_t and y_test_t
# Step 2: your chosen model in .eval() mode, and inside torch.no_grad():
#         y_pred_test = the model's predictions on X_test_t, and test_rmse = the RMSE against y_test_t
# Step 3: rmse_lin_test = root_mean_squared_error(y_test, lin_pipeline.predict(X_test))
# Step 4: print test_rmse, the model's validation RMSE, and rmse_lin_test

# RUN THIS after 4a. 4b: the test RMSE for cheap diamonds and for expensive diamonds
cheap = (y_test_t < 1.0).flatten()               # under $1,000 (the labels are in thousands)
expensive = (y_test_t >= 10.0).flatten()         # $10,000 or more

errors = (y_pred_test - y_test_t).flatten()
rmse_cheap = torch.sqrt((errors[cheap] ** 2).mean()).item()
rmse_expensive = torch.sqrt((errors[expensive] ** 2).mean()).item()

print("diamonds under $1,000:    ", cheap.sum().item(), "diamonds, average price", round(y_test_t.flatten()[cheap].mean().item(), 3), " RMSE:", round(rmse_cheap, 4))
print("diamonds $10,000 or more: ", expensive.sum().item(), "diamonds, average price", round(y_test_t.flatten()[expensive].mean().item(), 3), " RMSE:", round(rmse_expensive, 4))

Your answers here.