Introduction to Machine Learning

The Full Pipeline: Preprocessing, Search, Model Selection

The Full Pipeline

Important Information

• Email: joanna_bieri@redlands.edu
• Office Hours take place in Duke 209, Office Hours Schedule
• Class Website
• Syllabus

How to use these notes

Two kinds of box show up in these notes.

Blue Q boxes are questions for you to answer by hand, in a notebook, with a pen. Not because I am old fashioned. Writing something down by hand is slow, and slow is the point: it is very hard to write an explanation you do not actually understand. You are welcome to use AI in this class for the mechanics of code, but these boxes are the part where you do the thinking yourself. Bring your written notes to class, I will ask to see them.

Green You Try boxes are optional code for you to work through. Nothing is collected and nothing is graded. They are there because you will learn more from changing a number and rerunning than from watching me do it.

Every code cell begins with a tag that tells you what to do with it.

• # RUN THIS. Setup, loading data, a plot. Copy it, run it, move on. You do not need to be able to write it from memory.
• # LEARN TO WRITE THIS. The pattern of the day. The homework will ask you for it, and so will the exam. Type it out yourself at least once rather than pasting it.
• # DEMO ONLY. Fake data or a contrived experiment that exists to show one idea. You would never write this for a real project and you do not need to be able to.

Short answers to the Q boxes are in drop down boxes at the very bottom. Write yours first.

Reading: Geron chapter 2, all of it. It is the end to end project chapter, and today is the same shape: clean the data, handle the text columns, scale, search for settings, and score once at the end.

Everything since Day 2 has been a piece of one machine. Today we put the pieces together.

Here is the problem we are fixing. In Weekly Homework 3 you filled in missing values, turned text columns into numbers with get_dummies, scaled the features, split, fitted a model, and scored it. That worked, but it had two problems.

1. Every one of those steps had to be repeated by hand on the validation set, and then again on the test set, in the same order, with the same settings. Miss one and your numbers are wrong.
2. It is very easy to do the cleaning before the split, which is the leak from Day 4.

Today’s answer is a pipeline: one object that holds the cleaning, the encoding, the scaling, and the model. You fit it once. It does the steps in order, learns them on training rows only, and repeats them exactly on any new data you hand it. Then we let the computer search for good settings, and open the test set once.

New data today, because our wine has no missing values and no text columns. We are using the Titanic passenger list, which has both.

Load the data and a Look at the Mess

# RUN THIS. The passenger list.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

titanic = pd.read_csv("data/titanic.csv")

print("rows and columns:", titanic.shape)
print("share who survived:", round(titanic["survived"].mean(), 3))
print()
print("missing values per column:")
print(titanic.isna().sum()[titanic.isna().sum() > 0])

rows and columns: (891, 15)
share who survived: 0.384

missing values per column:
age            177
embarked         2
deck           688
embark_town      2
dtype: int64

titanic['alive'].value_counts()

alive
no     549
yes    342
Name: count, dtype: int64

This data set has 891 passengers, 38 percent of whom survived. Three columns have NAs in them: age is missing for 177 people, deck for 688, and embarked for 2.

NOTE: This command is doing a lot of work

titanic.isna().sum()[titanic.isna().sum() > 0]

but if you unpack it a bit we are:

• Going to each entry in the data to see if it is NA
• Adding these up along the columns where T=1 and F=0
• Then in [] only show the results where there are NAs so we don’t see a bunch of zeros.

Before we build anything, we have to choose columns to use as our input data. Remember two of the choices from before:

• alive is the word “yes” or “no”, and it is exactly the survived column in different words. A model given alive scores 100 percent and has learned nothing. Drop it.
• deck is missing for 688 of 891 passengers. Filling in 77 percent of a column is inventing data. Drop it.
• class, embark_town and adult_male repeat pclass, embarked and who. Keep one of each.

That leaves nine columns worth using, and they come in two kinds.

# LEARN TO WRITE THIS. Two kinds of column, and they need different treatment.

# NUMERICAL DATA
number_columns = ["age", "sibsp", "parch", "fare"]  # sibsp and parch: siblings and parents aboard
# CATEGORICAL DATA
text_columns = ["pclass", "sex", "embarked", "who", "alone"]

X = titanic[number_columns + text_columns]
y = titanic["survived"]

# Look at the data
display(X.head(3))
print()
# Check for missing data again
print("missing values in the columns we kept:")
print(X.isna().sum()[X.isna().sum() > 0])

	age	sibsp	parch	fare	pclass	sex	embarked	who	alone
0	22.0	1	0	7.2500	3	male	S	man	False
1	38.0	1	0	71.2833	1	female	C	woman	False
2	26.0	0	0	7.9250	3	female	S	woman	True

missing values in the columns we kept:
age         177
embarked      2
dtype: int64

Q1. Write this one out by hand

a. alive says “yes” or “no” and survived says 1 or 0, for the same passenger. What would happen to your accuracy if you left alive in, and why is that number worthless?

b. deck is missing for 688 of 891 passengers. What could go wrong if you filled those in with the most common deck and used the column anyway?

c. pclass is written as 1, 2, 3. It is already a number, so why did we put it in the text list instead of the number list? (Hint: is a third class ticket three times a first class ticket?)

Do the Split!

Here we did a little bit of looking at the data, but mostly just value counts and checking for NAs. Before we do anything else we must do our test train split!

# LEARN TO WRITE THIS. The same two splits as every day since Day 5.
from sklearn.model_selection import train_test_split

X_train_full, X_test, y_train_full, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)

X_train, X_valid, y_train, y_valid = train_test_split(
    X_train_full, y_train_full, test_size=0.25, stratify=y_train_full, random_state=42)

print("training rows:  ", len(X_train))
print("validation rows:", len(X_valid))
print("test rows:      ", len(X_test), "  (closed until the end)")

training rows:   501
validation rows: 167
test rows:       223   (closed until the end)

---

Why Not Just Clean It By Hand?

Everything that you do by hand had to be done carefully, using only information from the training data, but then applied exactly the same way to the validation and at the very end the testing data. Say you want to do something simple like fill missing ages with the average age. You would have to first calculate the average age based on only the training data, remember that number for the future of the model, fill missing values in the training data, the validation data, and the testing data. If you forget just one, everything is messed up. Or if you calculate that average using information outside the training data you have leakage. This is an easy mistake to make. Here are some tools we can use:

Pipeline - chains a sequence of things into one object that behaves like a single model. Calling .fit() fits each step in order on the training data only, and .predict() runs new data through the same chain, which is what prevents data leakage from preprocessing steps.

SimpleImputer - fills missing values (NaN) using a specified strategy: mean, median, most-frequent, or a constant. Fit on training data (learns the fill value), then applied to both train and test with that same learned value.

OneHotEncoder - converts a categorical column into multiple binary (0/1) columns, one per category. We used get_dummies() before, this does the same thing.

StandardScaler - rescales each numeric feature to zero mean and unit variance: z = \frac{x - \mu}{\sigma}. Matters for distance-based or gradient-based models (SVM, KNN, linear/logistic regression, neural nets); tree-based models (RF, GBR, XGBoost) are invariant to it and don’t need it.

---

Different Columns, Different Treatment

Numbers and text cannot be cleaned the same way.

• A missing age should be filled with a number, say the median age.
• A missing port should be filled with a category, say the most common port.
• Numbers should be put on one scale, as they were on Day 5.
• Text has to become numbers first, one column per category, which is what get_dummies did. The pipeline version is called OneHotEncoder.

ColumnTransformer - applies different preprocessing steps to different subsets of columns in a single object, e.g. scaling numeric columns while one-hot encoding categorical ones. Outputs are put back together and ready for the model

# LEARN TO WRITE THIS. One path for numbers, one for text.
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# Create a pipeline for the numerical data
number_path = Pipeline([
    ("fill", SimpleImputer(strategy="median")),     # missing age becomes the median age
    ("scale", StandardScaler())])

# Create a pipeline for the categorical data
text_path = Pipeline([
    ("fill", SimpleImputer(strategy="most_frequent")),      # missing port becomes the most common port
    ("dummy", OneHotEncoder(handle_unknown="ignore"))])     # one 0/1 column per category. More on the flag below

# Create a preprocessing step that tracks both pipelines above
# we defined number_columns and text_columns above
preprocessing = ColumnTransformer([
    ("num", number_path, number_columns),      
    ("cat", text_path, text_columns)])

# Do all of the preprocessing of the data in a single step
preprocessing.fit(X_train)       # learns the medians, the most common values, and the category lists

print("columns in: ", X_train.shape[1])
print("columns out:", preprocessing.transform(X_train).shape[1])
print()

columns in:  9
columns out: 17

Nine columns in, seventeen out. The four numerical columns stay as four columns, filled in and scaled. The five text columns turn into thirteen 0/1 columns, one per category in the training data.

Notice that preprocessing.fit(X_train) is where the learning happens. The median age, the most common port, and the list of categories all come from the training rows and nowhere else.

About handle_unknown="ignore". Next year a passenger list might contain a port your training data never had. Without this flag the encoder raises an error and your program stops. With it, the unknown category becomes all zeros and the model still gives an answer. In a live system that is the difference between a prediction and a 3am phone call from your boss about a crashing model.

Q2. Write this one out by hand

a. A passenger has no age recorded. Walk through what happens to that row as it goes down the number path.

b. Why does sex need OneHotEncoder but fare does not?

c. SimpleImputer(strategy="median") learned one number from the training rows. Which rows should it be used on afterward, and which should it never be recomputed on?

---

The Whole Thing As One Object

We can add one more pipeline. Above we had the preprocessing step as a set of pipelines. Now we put that in with a model and we have everything in one workflow!

# LEARN TO WRITE THIS. Preprocessing and model, one object.
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, average_precision_score

full_pipeline = Pipeline([
    ("prep", preprocessing),
    ("model", LogisticRegression(max_iter=5000))])

full_pipeline.fit(X_train, y_train)      # fills, scales, encodes, and fits the model, in that order

y_prob = full_pipeline.predict_proba(X_valid)[:, 1]     # does the same steps to the validation rows

print("validation accuracy:        ", round(accuracy_score(y_valid, (y_prob >= 0.5)), 3))
print("validation average precision:", round(average_precision_score(y_valid, y_prob), 3))

validation accuracy:         0.838
validation average precision: 0.873

We only have to call fit once, predict_proba once, and there is no chance of forgetting a step or leaking a median from the validation set.

Trying different models

Swapping the model is now a one line change, which makes comparing models easy. Here we will compare three different models all within a single for loop and we don’t lose any of our preprocessing steps we decided above.

# LEARN TO WRITE THIS. Same preprocessing, three models.
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier

models = [("logistic regression", LogisticRegression(max_iter=5000)),
          ("random forest", RandomForestClassifier(n_estimators=200, random_state=42)),
          ("boosting", XGBClassifier(random_state=42))]

for name, model in models:
    pipe = Pipeline([("prep", preprocessing), ("model", model)])
    pipe.fit(X_train, y_train)
    ap = average_precision_score(y_valid, pipe.predict_proba(X_valid)[:, 1])
    print(name, "  validation average precision:", round(ap, 3))

logistic regression   validation average precision: 0.873
random forest   validation average precision: 0.857
boosting   validation average precision: 0.885

Great we got three numbers! Do we just say boosting is best?!?!?

from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold

# Cross Validation
cv = RepeatedStratifiedKFold(
    n_splits=5,        # 5 folds
    n_repeats=4,       # done 4 times with different shuffles, so 20 fits per model
    random_state=42)

for name, model in models:
    pipe = Pipeline([("prep", preprocessing), ("model", model)])
    scores = cross_val_score(pipe, X_train_full, y_train_full, cv=cv, scoring="average_precision")
    print(name)
    print("   mean average precision:", round(scores.mean(), 3))
    print("   standard deviation:    ", round(scores.std(), 3))
    print("   worst and best fold:   ", round(scores.min(), 3), "to", round(scores.max(), 3))

logistic regression
   mean average precision: 0.846
   standard deviation:     0.035
   worst and best fold:    0.769 to 0.897
random forest
   mean average precision: 0.816
   standard deviation:     0.05
   worst and best fold:    0.715 to 0.897
boosting
   mean average precision: 0.838
   standard deviation:     0.041
   worst and best fold:    0.759 to 0.892

Now we can evaluate each model and compare it to the noise in the cross validation.

Q3. Write this one out by hand

a. Day 4 said any step with a .fit() has to live inside the pipeline. Name the three steps inside ours that have a .fit(), and say what each one learns.

b. When cross_val_score runs on a pipeline, the whole pipeline is refitted on each fold. Why does that matter for the imputer and the scaler?

c. What single line would you change to try decision tree instead of logistic regression?

---

Letting the Computer Search

GridSearch

Every model has hyperparameters and settings that we have been picking by hand. It is nice to have a systematic way to search for the best values. GridSearchCV is a method that tries every combination you list, with cross validation, and keeps the best.

One thing to be careful about is how to get the new hyperparameters to reach inside the pipeline! Here is the outline:

• You define the model and just leave the parameter out: model = RandomForestClassifier(random_state=42)
• This is put inside your pipeline
• Then for a parameter to be set you call model__n_estimators = 100 notice it starts with the name you defined for the model and then there are TWO UNDERSCORES and then the parameter name.

Below you will see this done as a dictionary of possible values:

grid = {"model__n_estimators": [100, 300],    
        "model__max_depth": [None, 5, 10],        
        "model__min_samples_leaf": [1, 5]} 

# LEARN TO WRITE THIS. Try every combination, with cross validation.
from sklearn.model_selection import GridSearchCV, StratifiedKFold


# Define the pipeline the Random Forest is named "model"
search_pipeline = Pipeline([("prep", preprocessing),
                            ("model", RandomForestClassifier(random_state=42))])

# Create the grid of values you want to try using the name above and the parameters it can take
grid = {"model__n_estimators": [100, 300],        # two choices
        "model__max_depth": [None, 5, 10],        # three choices
        "model__min_samples_leaf": [1, 5]}        # two choices

# Call the grid search
grid_search = GridSearchCV(
    search_pipeline, # send in the pipeline
    grid, # send in the grid
    cv=StratifiedKFold(5, shuffle=True, random_state=42), # tell it the type of cross validation
    scoring="average_precision")

grid_search.fit(X_train_full, y_train_full)     # training and validation together here 
                                                # Remember the folds do the validating for you

print("combinations tried:", len(grid_search.cv_results_["params"]))
print("best settings:", grid_search.best_params_)
print("best cross validated average precision:", round(grid_search.best_score_, 3))

combinations tried: 12
best settings: {'model__max_depth': None, 'model__min_samples_leaf': 5, 'model__n_estimators': 300}
best cross validated average precision: 0.85

Twelve combinations, five folds each, so sixty fits. It takes a few seconds. The winner uses 300 trees with at least 5 passengers in every leaf, and scores 0.850 across the folds.

Two things about that fit line are worth noticing

1. We fitted on X_train_full, the training and validation rows together. When the search does its own cross validation, the folds are the validation. Keeping a separate validation set as well would just be holding data back for no reason.
2. The old validation set is now inside the search, so we cannot use it to score the winner. The next honest number has to come from the test set.

GridSearchCV() also saves the best model it found, including your whole pipeline. You can use this later when you are ready to score your model on the test set!

grid_search.best_estimator_

Pipeline(steps=[('prep',
                 ColumnTransformer(transformers=[('num',
                                                  Pipeline(steps=[('fill',
                                                                   SimpleImputer(strategy='median')),
                                                                  ('scale',
                                                                   StandardScaler())]),
                                                  ['age', 'sibsp', 'parch',
                                                   'fare']),
                                                 ('cat',
                                                  Pipeline(steps=[('fill',
                                                                   SimpleImputer(strategy='most_frequent')),
                                                                  ('dummy',
                                                                   OneHotEncoder(handle_unknown='ignore'))]),
                                                  ['pclass', 'sex', 'embarked',
                                                   'who', 'alone'])])),
                ('model',
                 RandomForestClassifier(min_samples_leaf=5, n_estimators=300,
                                        random_state=42))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.