Introduction to Data Science

University of Redlands - DATA 101

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website: data101.joannabieri.com

# Day 4 - In Class Challenge - CDC Health Data - SOLUTIONS

These are one set of answers, not the only set. Groups will pick different plot types and that is fine as long as the plot actually answers the question. The written answers below are what I would hope to hear when a group presents.

Numbers quoted below come from the 60 person sample. If you swap in a different sample they will change.

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import plotly.express as px
from plotly.subplots import make_subplots
import plotly.io as pio
pio.renderers.default = 'notebook_connected'
file_location = 'https://joannabieri.com/introdatascience/data/cdc.samp.csv'
DF = pd.read_csv(file_location)
display(DF)
genhlth exerany hlthplan smoke100 height weight wtdesire age gender
0 very good 1 1 1 75 265 225 45 m
1 excellent 1 1 0 72 150 150 24 m
2 excellent 1 1 1 69 137 150 47 m
3 good 1 1 1 66 159 125 26 f
4 very good 1 1 0 63 145 125 33 f
5 very good 1 1 1 64 125 120 28 f
6 excellent 1 1 1 72 175 175 34 m
7 very good 1 1 1 60 104 104 42 f
8 good 1 1 1 65 140 140 50 f
9 very good 1 1 1 70 188 170 57 m
10 very good 1 1 0 67 170 155 57 m
11 good 0 0 1 65 120 130 19 f
12 good 1 1 0 73 290 220 42 m
13 fair 1 0 1 66 170 160 43 m
14 good 1 1 0 65 120 120 60 f
15 very good 0 1 0 64 140 110 34 f
16 fair 1 1 1 63 164 144 48 f
17 good 1 1 1 69 170 170 37 m
18 good 0 1 0 62 170 150 74 f
19 good 1 1 0 73 198 190 52 m
20 fair 1 1 0 62 130 130 61 f
21 good 0 1 1 66 300 150 68 f
22 very good 1 1 1 74 192 185 24 m
23 good 1 0 0 72 175 175 24 m
24 good 0 1 1 68 195 170 59 f
25 excellent 1 1 0 70 155 155 49 m
26 good 0 0 0 67 400 200 37 f
27 excellent 0 1 0 68 145 140 47 m
28 very good 1 1 1 62 145 120 56 f
29 good 0 1 1 63 165 140 74 f
30 excellent 1 1 1 68 140 140 28 m
31 very good 1 1 0 73 220 200 24 m
32 excellent 1 1 1 71 200 185 68 m
33 excellent 1 0 1 64 120 110 33 f
34 very good 1 1 1 65 180 140 43 f
35 fair 0 0 1 64 135 135 49 f
36 excellent 1 1 0 69 170 150 37 f
37 excellent 1 1 0 63 151 135 51 f
38 good 1 1 1 67 180 140 52 f
39 very good 1 1 1 71 195 180 50 m
40 fair 1 0 0 63 120 120 47 f
41 good 1 1 0 68 175 175 38 m
42 excellent 0 1 0 62 115 115 43 f
43 very good 1 1 1 66 160 140 35 f
44 excellent 1 1 1 72 200 188 50 m
45 good 1 0 1 70 154 154 31 m
46 very good 0 1 0 62 210 180 77 f
47 very good 0 1 0 62 135 135 69 f
48 very good 1 1 0 66 148 140 23 f
49 excellent 1 0 0 72 230 225 25 m
50 fair 1 1 1 60 192 147 48 f
51 excellent 1 1 1 74 200 200 51 m
52 poor 0 1 1 72 210 190 64 m
53 excellent 1 1 1 70 199 180 35 m
54 excellent 0 1 0 65 160 140 30 m
55 good 1 1 0 61 132 108 55 f
56 very good 1 1 0 72 165 160 19 m
57 fair 0 0 0 62 160 140 32 f
58 excellent 1 1 0 73 175 190 28 m
59 very good 1 1 0 69 165 165 40 m
## Warm Up

Q1 Observations and variables.

print(DF.shape)
print('Observations (rows):', len(DF))
print('Variables (columns):', len(DF.columns))
(60, 9)
Observations (rows): 60
Variables (columns): 9

60 observations (people) and 9 variables.

Q2 Numerical vs categorical.

Numerical: * height (inches), weight (pounds), wtdesire (pounds), age (years). Height, weight and age are really continuous quantities, but they were recorded as whole numbers so they look discrete in the data. Either answer is fine as long as the group can say why.

Categorical: * genhlth is ordinal: poor, fair, good, very good, excellent has a natural order. * gender is nominal. * exerany, hlthplan, smoke100 are the tricky ones. They are stored as 0 and 1 but they are yes/no questions, so they are categorical (nominal). The 1 does not mean “one of something”. If you average them you get a proportion, which is useful, but you should not treat them like weight.

A quick way to see this is that describe() happily computes a mean of smoke100 (about 0.52) which just means 52% of the sample has smoked 100 cigarettes.

DF.describe()
exerany hlthplan smoke100 height weight wtdesire age
count 60.000000 60.000000 60.000000 60.000000 60.000000 60.000000 60.000000
mean 0.750000 0.833333 0.516667 67.183333 173.300000 155.083333 43.766667
std 0.436667 0.375823 0.503939 4.151747 49.034854 29.896597 14.813378
min 0.000000 0.000000 0.000000 60.000000 104.000000 104.000000 19.000000
25% 0.750000 1.000000 0.000000 63.750000 143.750000 135.000000 32.750000
50% 1.000000 1.000000 1.000000 67.000000 165.000000 150.000000 43.000000
75% 1.000000 1.000000 1.000000 71.000000 192.750000 176.250000 52.000000
max 1.000000 1.000000 1.000000 75.000000 400.000000 225.000000 77.000000

Q3 Counts of genhlth, in order.

counts = DF['genhlth'].value_counts()
display(counts)

health_order = ['poor', 'fair', 'good', 'very good', 'excellent']

fig = px.bar(DF,
             x='genhlth',
             category_orders={'genhlth': health_order},
             color_discrete_sequence=['lightseagreen'])
fig.update_traces(dict(marker_line_width=0))
fig.update_layout(title='How people rate their own health',
                  title_x=0.5,
                  xaxis_title='Self rated general health',
                  yaxis_title='Number of people')
fig.show()
genhlth
very good    18
excellent    17
good         17
fair          7
poor          1
Name: count, dtype: int64

Very good 18, excellent 17, good 17, fair 7, poor 1. Once the bars are in order the shape is lopsided toward the good end. Almost everyone says they are at least “good”, only 8 of 60 say fair or poor. People tend to rate their own health well (or people in poor health are less likely to answer a phone survey, which is worth mentioning).

## Challenge 1 - How much weight do people want to lose?

Q4 Scatter plot of weight vs desired weight.

fig = px.scatter(DF,
                 x='weight',
                 y='wtdesire',
                 color='gender',
                 hover_data='age',
                 title='Desired weight vs current weight')
fig.update_traces(marker=dict(size=10))
fig.update_layout(title_x=0.5,
                  xaxis_title='Current weight (pounds)',
                  yaxis_title='Desired weight (pounds)')
fig.show()

Q5 A point on the line wtdesire = weight is a person who is happy with their weight. A point above the line wants to gain weight. A point below the line wants to lose weight. Almost everyone is below the line, most people want to lose weight.

Q6 Find the people who want to gain weight with a filter.

gainers = DF[DF['wtdesire'] > DF['weight']]
display(gainers)
print('Number who want to gain weight:', len(gainers))
print(gainers['gender'].value_counts())
genhlth exerany hlthplan smoke100 height weight wtdesire age gender
2 excellent 1 1 1 69 137 150 47 m
11 good 0 0 1 65 120 130 19 f
58 excellent 1 1 0 73 175 190 28 m 
Number who want to gain weight: 3
gender
m    2
f    1
Name: count, dtype: int64

Only 3 people out of 60 want to gain weight, 2 men and 1 woman. All three only want to add 10 to 15 pounds, and none of them is heavy to start with.

Q7 The outlier and the mean.

# The point way off by itself
display(DF[DF['weight'] > 300])

mean_all = DF['weight'].mean()
mean_no_outlier = DF[DF['weight'] < 300]['weight'].mean()
median_all = DF['weight'].median()

print(f'Mean weight, everyone:          {mean_all:.1f}')
print(f'Mean weight, without outlier:   {mean_no_outlier:.1f}')
print(f'Median weight, everyone:        {median_all:.1f}')
genhlth exerany hlthplan smoke100 height weight wtdesire age gender
26 good 0 0 0 67 400 200 37 f 
Mean weight, everyone:          173.3
Mean weight, without outlier:   167.2
Median weight, everyone:        165.0

The outlier is a 37 year old woman, 67 inches tall, 400 pounds, who wants to weigh 200. One person out of 60 pulls the mean from about 167 up to about 173 pounds. The median (165) barely notices her. For a “typical” person the median is the better description here, because the mean is being dragged around by a single observation.

Q8 New column and compare by gender.

DF['want_to_lose'] = DF['weight'] - DF['wtdesire']

fig = px.box(DF,
             x='want_to_lose',
             y='gender',
             color='gender',
             title='Pounds people want to lose, by gender')
fig.update_layout(title_x=0.5,
                  xaxis_title='Current weight minus desired weight (pounds)',
                  yaxis_title='Gender')
fig.show()
print('Women')
display(DF[DF['gender'] == 'f']['want_to_lose'].describe())
print('Men')
display(DF[DF['gender'] == 'm']['want_to_lose'].describe())
Women
Men
count     31.000000
mean      26.354839
std       42.640003
min      -10.000000
25%        0.000000
50%       20.000000
75%       27.500000
max      200.000000
Name: want_to_lose, dtype: float64
count    29.000000
mean      9.517241
std      16.101510
min     -15.000000
25%       0.000000
50%       5.000000
75%      15.000000
max      70.000000
Name: want_to_lose, dtype: float64

Women in this sample want to lose more: median 20 pounds vs 5 pounds for men, mean about 26 vs about 10. The women’s mean is pulled up by the 400 pound outlier (she alone wants to lose 200), so the medians are the fairer comparison. Even so, the difference between the groups (about 15 pounds at the median) is smaller than the spread inside each group (the women’s box runs from 0 to about 28), so there is a lot of overlap.

## Challenge 2 - Does how people feel match the numbers?

Q9 BMI column.

DF['bmi'] = 703 * DF['weight'] / DF['height']**2
display(DF[['gender', 'height', 'weight', 'bmi', 'genhlth']])
DF['bmi'].describe()
gender height weight bmi genhlth
0 m 75 265 33.119111 very good
1 m 72 150 20.341435 excellent
2 m 69 137 20.229154 excellent
3 f 66 159 25.660468 good
4 f 63 145 25.682792 very good
5 f 64 125 21.453857 very good
6 m 72 175 23.731674 excellent
7 f 60 104 20.308889 very good
8 f 65 140 23.294675 good
9 m 70 188 26.972245 very good
10 m 67 170 26.622856 very good
11 f 65 120 19.966864 good
12 m 73 290 38.256709 good
13 m 66 170 27.435721 fair
14 f 65 120 19.966864 good
15 f 64 140 24.028320 very good
16 f 63 164 29.048123 fair
17 m 69 170 25.101869 good
18 f 62 170 31.090010 good
19 m 73 198 26.120098 good
20 f 62 130 23.774714 fair
21 f 66 300 48.415978 good
22 m 74 192 24.648649 very good
23 m 72 175 23.731674 good
24 f 68 195 29.646410 good
25 m 70 155 22.237755 excellent
26 f 67 400 62.642014 good
27 m 68 145 22.044766 excellent
28 f 62 145 26.517950 very good
29 f 63 165 29.225246 good
30 m 68 140 21.284602 excellent
31 m 73 220 29.022331 very good
32 m 71 200 27.891291 excellent
33 f 64 120 20.595703 excellent
34 f 65 180 29.950296 very good
35 f 64 135 23.170166 fair
36 f 69 170 25.101869 excellent
37 f 63 151 26.745528 excellent
38 f 67 180 28.188906 good
39 m 71 195 27.194009 very good
40 f 63 120 21.254724 fair
41 m 68 175 26.605753 good
42 f 62 115 21.031478 excellent
43 f 66 160 25.821855 very good
44 m 72 200 27.121914 excellent
45 m 70 154 22.094286 good
46 f 62 210 38.405307 very good
47 f 62 135 24.689126 very good
48 f 66 148 23.885216 very good
49 m 72 230 31.190201 excellent
50 f 60 192 37.493333 fair
51 m 74 200 25.675676 excellent
52 m 72 210 28.478009 poor
53 m 70 199 28.550408 excellent
54 m 65 160 26.622485 excellent
55 f 61 132 24.938457 good
56 m 72 165 22.375579 very good
57 f 62 160 29.261186 fair
58 m 73 175 23.085945 excellent
59 m 69 165 24.363579 very good 
count    60.000000
mean     26.890102
std       6.936902
min      19.966864
25%      23.149111
50%      25.679234
75%      28.496109
max      62.642014
Name: bmi, dtype: float64

Q10 BMI by self rated health, in order.

fig = px.box(DF,
             x='genhlth',
             y='bmi',
             color='genhlth',
             category_orders={'genhlth': health_order},
             title='BMI by self rated health')
fig.update_layout(title_x=0.5,
                  xaxis_title='Self rated general health',
                  yaxis_title='BMI',
                  showlegend=False)
fig.show()

# Check the medians with a table
DF.groupby('genhlth')['bmi'].median().reindex(health_order)
genhlth
poor         28.478009
fair         27.435721
good         26.120098
very good    25.752323
excellent    23.731674
Name: bmi, dtype: float64

Yes, mostly. The median BMI goes down steadily as the rating goes up: poor about 28.5, fair about 27.4, good about 26.1, very good about 25.8, excellent about 23.7. Where it does not line up: the “poor” group is a single person so that box is not really a box, and “very good” contains the 400 pound outlier with a BMI over 60, which shows a high BMI does not stop someone rating their own health well. The boxes overlap a lot, so BMI does not determine how people feel, it just trends with it.

Q11 BMI by exercise.

# First try - the legend says 0 and 1
fig = px.histogram(DF, x='bmi', color='exerany', nbins=15, opacity=0.7)
fig.show()
DF['exercise'] = DF['exerany'].replace({0: 'no', 1: 'yes'})

fig = px.histogram(DF,
                   x='bmi',
                   color='exercise',
                   nbins=15,
                   opacity=0.7,
                   barmode='overlay',
                   title='BMI for people who did and did not exercise last month')
fig.update_layout(title_x=0.5,
                  xaxis_title='BMI',
                  yaxis_title='Number of people',
                  legend_title='Exercised last month')
fig.show()

DF.groupby('exercise')['bmi'].median()
exercise
no     28.478009
yes    25.660468
Name: bmi, dtype: float64

The annoying part is that the legend reads 0 and 1 and the reader has to go look up which is which. After the replace the legend says yes and no.

The exercisers sit a little lower, median BMI about 25.7 vs about 28.5 for the non exercisers, but there are only 15 non exercisers so their histogram is very bumpy. The two groups overlap a lot. barmode='overlay' is optional, the default stacked histogram also works but is harder to compare.

## Challenge 3 - Smoking and health, and being a skeptic

Q12 Bar plot of health rating colored by smoking.

DF['smoker'] = DF['smoke100'].replace({0: 'no', 1: 'yes'})

fig = px.bar(DF,
             x='genhlth',
             color='smoker',
             category_orders={'genhlth': health_order},
             barmode='group',
             title='Self rated health for smokers and non smokers')
fig.update_traces(dict(marker_line_width=0))
fig.update_layout(title_x=0.5,
                  xaxis_title='Self rated general health',
                  yaxis_title='Number of people',
                  legend_title='Smoked 100+ cigarettes')
fig.show()

It is hard to say from the bars. The only “poor” is a smoker and the smokers have one fewer “excellent”, so a group might say smokers rate their health a little worse. The stacked version (the default) is even harder to read.

Q13 Compare within each group.

print('Smokers')
display(DF[DF['smoker'] == 'yes']['genhlth'].value_counts())
print('Non smokers')
display(DF[DF['smoker'] == 'no']['genhlth'].value_counts())
Smokers
Non smokers
genhlth
very good    9
good         9
excellent    8
fair         4
poor         1
Name: count, dtype: int64
genhlth
excellent    9
very good    9
good         8
fair         3
Name: count, dtype: int64
# Optional - as a fraction of each group, which is the fair comparison
DF.groupby('smoker')['genhlth'].value_counts(normalize=True).unstack().reindex(columns=health_order).round(2)
genhlth poor fair good very good excellent
smoker
no NaN 0.10 0.28 0.31 0.31
yes 0.03 0.13 0.29 0.29 0.26

There are 31 smokers and 29 non smokers, so the groups are close in size here and the bar plot was not lying much. As fractions: about 26% of smokers say excellent vs 31% of non smokers, and 13% vs 10% say fair. The one “poor” is a smoker. So smokers rate their health slightly worse, but the difference is one or two people in each category. A group that said “no real difference” is right, and a group that said “smokers a little worse” is also right, as long as they say how small it is.

The habit to take away: whenever you color a bar plot by a group, check whether the groups are the same size before you compare the heights.

Q14 Being a skeptic. Any two of these (or others) are good:

  • Tiny sample. 60 people, and the difference is one or two people per category. If we picked a different 60 people the answer could easily flip.
  • Self reported. Both smoking and health are what people said on the phone, not measured. People tend to say they weigh a little less and feel a little better than they really do.
  • Other differences between the groups. Smokers in this sample might be older, exercise less, or have different jobs. Any of those could explain a health difference without smoking causing it. (We will call these confounding variables later in the semester.)
  • Who answers the phone. People in poor health are less likely to be surveyed at all, which is one reason “poor” has only one person.
  • Direction. Even if there is a link, this data cannot say smoking caused the lower rating. That takes a very different kind of study.
## Challenge (optional) - Age

All three questions have a real signal. Here is one plot for each.

fig = px.box(DF,
             x='age',
             y='genhlth',
             color='genhlth',
             category_orders={'genhlth': health_order},
             title='Age by self rated health')
fig.update_layout(title_x=0.5, xaxis_title='Age (years)', yaxis_title='Self rated health', showlegend=False)
fig.show()

DF.groupby('genhlth')['age'].median().reindex(health_order)
genhlth
poor         64.0
fair         48.0
good         50.0
very good    41.0
excellent    37.0
Name: age, dtype: float64

Older people rate their health lower. Median age is about 37 for excellent, 41 very good, 50 good, 48 fair, and the one poor is 64.

fig = px.scatter(DF,
                 x='age',
                 y='want_to_lose',
                 color='gender',
                 title='Pounds people want to lose vs age')
fig.update_layout(title_x=0.5, xaxis_title='Age (years)', yaxis_title='Pounds to lose')
fig.show()

No clear trend with age once you ignore the 400 pound outlier. The gender difference from Q8 is a lot easier to see than any age pattern.

DF['health_plan'] = DF['hlthplan'].replace({0: 'no', 1: 'yes'})

fig = px.box(DF,
             x='age',
             y='health_plan',
             color='health_plan',
             title='Age of people with and without a health plan')
fig.update_layout(title_x=0.5, xaxis_title='Age (years)', yaxis_title='Has a health plan', showlegend=False)
fig.show()

DF.groupby('health_plan')['age'].median()
health_plan
no     32.5
yes    47.0
Name: age, dtype: float64

The 10 people without a health plan are younger, median age about 32 vs about 47 for people with one. Only 10 people though!

## FREE FORM

No solution here, this is whatever the group came up with. When they present, listen for: a real title and labeled axes with units, one sentence saying what the plot shows, and one honest reason they might be wrong (sample size and self reporting are the easy ones).