Introduction to Machine Learning

# Weekly Homework 2: Wine Quality

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website

Due Sunday 9/13 at 11:59pm. This covers Day 3 and Day 4.

GOALS:

  1. Load and inspect a real file that is not as clean as it looks.
  2. Use regularization on data where the features are genuinely related to each other.
  3. Do cross validation properly, with pipelines, so nothing leaks.
  4. Find a real leak in real data and measure what it costs you.
  5. Say what your numbers actually mean.

How to turn this in. Your repository on GitHub is your submission. There is no Pull Request to open any more.

Manage your git however you like. Use branches if you want them, or commit straight to main if you do not. What I need is only this:

  1. The finished work is on your main branch.
  2. It is pushed to GitHub before the deadline.
  3. Your name is on the Author line at the top of this notebook.
git add .
git commit -m "Weekly homework 2"
git push

If you did the work on a branch, merge it into main and push before the deadline:

git checkout main
git merge my-branch-name
git push

I grade from whatever is on GitHub at the deadline. If it is not pushed, I cannot see it.

What I am grading. Working code with no writing is not a pass. Every question that asks “why” or “what does this mean” needs real sentences. That is the part I actually care about.

This is a full mini project, so unlike the daily notes there are no optional parts here. Answer everything, in order. Several questions ask you to commit to a guess before you run the code, and those are there on purpose, so do not skip ahead and backfill them.

The data

data/winequality-red.csv, 1599 Portuguese red wines. Eleven chemical measurements per wine, plus a quality score from 0 to 10 given by wine tasters.

The question: can we predict a wine’s quality score from its chemistry?

This is real data from a real study, which means it has real problems in it. Finding them is Part 1.

Part 1: Load it and look at it

1a. Load the file. It will not work the first time. Look at what you get, figure out why, and fix it. Say in one sentence what was wrong. (Hint: open the raw file in a text editor and look at what separates the columns.)

1b. How many rows and how many columns? What are the column names?

1c. Are there any missing values?

1d. Now check for duplicate rows, with .duplicated().sum(). How many are there? That is a lot. Write two or three sentences on what you think they are: did somebody measure the same wine twice, or are there genuinely two different wines with identical readings on all eleven measurements? Which explanation do you believe, and why?

1e. Look at .describe(). Compare the typical size of density to the typical size of total sulfur dioxide. Roughly what factor separates them? Given Day 3, what does that tell you that you must do before regularizing?

1f. Plot a histogram of quality. Is it balanced? Which scores are rare?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# your code here

Your answers here.

Part 2: Split first

Before you explore any relationships, split off a test set. Day 4 was clear about why: if you look at everything first, your later choices are shaped by what you saw.

2a. Decide what to do about the duplicates you found in 1d, and do it before you split. Whatever you decide, defend it in two or three sentences. There is a defensible argument either way, and I am grading the argument, not the choice.

2b. Split into a training set and a test set, 75/25. Use stratify= on the quality score so both sets have a similar mix of quality levels.

2c. Why did stratifying matter here specifically? Point at something from your 1f histogram.

2d. Print the shape of both sets, and check that the proportion of each quality score really is similar in the two.

Now put the test set away and do not touch it again until Part 6.

from sklearn.model_selection import train_test_split

# your code here

Your answers here.

Part 3: A baseline, then regularization

Never report a model score without something to compare it to.

3a. What RMSE would you get by ignoring the chemistry entirely and predicting the mean quality for every wine? Compute it with 5-fold cross validation on the training set. This is the number every model has to beat.

3b. Build a pipeline of StandardScaler and LinearRegression. Get its 5-fold cross validation RMSE on the training set. Print all five fold scores, not just the mean.

3c. How much better is it than the baseline? Is the improvement bigger or smaller than the spread across your five folds? Say what that comparison means.

3d. Now Ridge and Lasso. Use GridSearchCV to choose alpha for each, over at least 20 values with np.logspace. Report the best alpha and CV RMSE for each.

3e. Do Ridge and Lasso beat plain LinearRegression here? Day 3 said regularizing almost always helps. Did it help much this time? Suggest a reason based on how many features and how many rows you have.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.dummy import DummyRegressor
from sklearn.model_selection import cross_val_score, GridSearchCV

# your code here

Your answers here.

Part 4: Which chemistry actually matters

4a. Take your tuned Lasso and print its coefficients next to the feature names. Which features did it zero out?

4b. Now sweep alpha by hand over about four values and print how many features survive at each. Find an alpha that keeps only about three features. Which three?

4c. Compare those three to the correlation of each feature with quality, which you can get with .corr(). Do they agree?

4d. A winemaker reads your result and says “so if I add more sulphates, my wine will score higher.” What is wrong with that conclusion? Two or three sentences.

4e. From Day 3: two of your eleven features are strongly related to each other. Find a pair with correlation above 0.6. If Lasso kept one and dropped the other, what should you not conclude about the dropped one?

# your code here

Your answers here.

Part 5: The leak that is already in your data

Back to those duplicate rows. This part shows you what they cost.

5a. Before you run anything, guess. If you leave the duplicates in and split randomly, some identical wines will land in both the training and the test set. Out of 400 test rows, how many do you think will be exact copies of a training row? Write a number down.

5b. Now measure it. Using the original data with duplicates still in, split it randomly and count how many test rows appear identically in the training set.

5c. Fit the same model two ways and compare the test RMSE:

  • on the data with duplicates left in
  • on the deduplicated data

Average over at least 5 different random_state values for the split, because Day 4 taught you that one split proves nothing.

5d. Which one scores better? By how much? Which number is the honest one?

5e. Explain in your own words why duplicate rows across a split are a form of leakage. Connect it to the patient scans example from the Day 4 notes.

5f. Suppose these duplicates are genuinely different wines that happen to have identical chemistry, rather than the same wine recorded twice. Does that change your answer to 5e? Think carefully, this one is not obvious.

from sklearn.metrics import root_mean_squared_error
from sklearn.preprocessing import PolynomialFeatures

# your code here

Your answers here.

Part 6: The test set, once

6a. Take your single best model from Part 3, refit it on the whole training set, and score it on the test set you have not touched since Part 2. Report the RMSE.

6b. Compare it to the cross validation RMSE you got for that model in Part 3. Is it better or worse? Is the difference bigger than the spread across your folds?

6c. Quality scores run from 3 to 8 in this data. Write one sentence a winemaker would understand about how accurate your model is, using your test RMSE and that scale.

6d. Would you use this model to decide the price of a bottle? Answer yes or no and defend it. Consider both how accurate it is and what happens when it is wrong.

6e. One last one, and it previews Day 5. Instead of predicting the score, suppose we only wanted to flag good wines, meaning quality 7 or above. Work out what fraction of the wines that is. Then say what accuracy you would get from a model that ignores the chemistry completely and always answers “not good”. What does that tell you about using accuracy to judge a model like this?

# your code here

Your answers here.