Intermediate Data Science

Welcome Back

Author

Joanna Bieri
DATA201

Welcome to Intermediate Data Science!

Important Information

Computer Set Up

Before we can do much data science we need to make sure everyone has a stable computer set up for this semester!

We will be using Python and Jupyter Lab as our foundation for programming. You should have these already from DATA 101. If your setup still works, you are done here - open Jupyter Lab and go on to the next section.

Two things changed over the summer, so read on if yours is not working.

If Anaconda is asking you about a license, or what company you work for: you did not break anything. Anaconda changed the way they handle sign ups and the form is confusing, especially since none of us work for a company. It is still free for us. Here is the three minute fix: Sort out your Anaconda account

If you are starting fresh, or your setup is broken: DATA 101 moved to Miniforge this year. It does the same job as Anaconda without asking you to make an account. Follow How to set up your computer

You do not need to reinstall anything if what you have is working.

Software

Let’s start by checking what versions of the software we have:

  • Python 3.13.5
  • conda 24.11.3
# If you need to update:
!conda update -n base conda -y
!conda update --all -y
!python --version
Python 3.11.7
!conda --version
conda 24.11.3

Version Control

We also need to make sure we have git and jupyterlab-git installed.

First make sure you have a GitHub Account if not Sign Up For GitHub Account!. Log into your GitHub. Then check that you have git installed on your computer.

!git --version
git version 2.43.0

We will probably use jupyterlab-git to clone/push/pull from GitHub. If you see this icon on the left side of your jupyter, then you are good to go!

Git Icon

If you don’t see it you might need to run the following code and restart your Jupyter Lab.

!conda config --add channels conda-forge
!conda install -c conda-forge -y git
!conda install -c conda-forge -y jupyterlab-git

!conda update -y pexpect

Set up Workflow

This semester you will work in two different repositories, and they do two very different jobs. Getting this straight today will save you a lot of confusion later!

1. Your sandbox (sandbox- and then your GitHub username)

This is your own private copy of everything I post. Lecture notebooks, You Try cells, data files, all of it. Work through the notes here, run the cells, change things, break things, try stuff out. Nothing in your sandbox is graded and nobody else works in it. It is yours.

2. Your team’s work repo (team- and then your team name)

This is where your team’s homework goes. It has one folder for each week, Week01 through Week05, and nothing else in it. No lecture notes, no data, just your team’s work. Teams reshuffle after Exam 1, and you get a fresh team repo then.

You cannot commit directly to main in this repo. Every piece of work goes on a branch and gets merged with a Pull Request. That is on purpose, and it is how most data science teams actually work.

Why two repos? When you scribble on my lecture notebook in your sandbox, and then I post an updated version of that same notebook, git has to stop and ask you which version you want. In your sandbox that does not matter, nothing there is graded. If it happened in your team’s repo it would be your whole team’s problem, right before a deadline. Keeping them separate means it never comes up.

Step 1 - Make a folder for this class

Using the Jupyter File Browser, make a folder (directory) that will hold all of your work for this class. Name it something you will actually be able to find later!

Step 2 - Clone your sandbox and connect it to the class repo

In a terminal window, inside the folder you just made:

git clone <your sandbox url>
cd sandbox-<your github username>
git remote add upstream https://github.com/Redlands-DATA201/FALL26.git

That last line is a one time step. upstream is the class repo, and it is how you pull in new material that I post after today.

Step 3 - Install the packages you will need all semester

Your sandbox includes a requirements.txt listing every package we will use. Install them all at once, right now, so you never have to think about it again:

pip install -r requirements.txt

If a later assignment tells you to install something new, it means the package genuinely was not on this list.

Step 4 - Turn on clean notebook diffs

nbstripout strips notebook outputs before they get committed, so that git only sees your actual code changes and not the fact that you re-ran a cell. It needs one activation step per person, per clone:

nbstripout --install

Run this in your sandbox now, and again in your team’s repo after Step 5.

Step 5 - Clone your team’s work repo

Teams are assigned in class today. Once you have your team’s repo address:

git clone <your team repo url>
cd team-<your team name>
nbstripout --install

There is no upstream to add here. This repo holds only your team’s work, so there is nothing to sync and nothing to collide with.

Step 6 - Make your branch and practice the Pull Request cycle, right now

You get one branch in your team’s repo and you keep it all semester. Name it after yourself, so you never have to think about branch names again. Inside your team’s repo:

git checkout -b your_branch_name

Make a new Markdown or Text file inside the Week01 folder and type something in it, like “Hello from <your name>!” Save it. Then:

git add Week01
git commit -m 'testing the workflow'
git push -u origin your_branch_name

In jupyter-git:

triple check you are in your TEAM's repo, not your sandbox
add the untracked file using the + sign, stage it, write a commit message, and push on the up arrow cloud

Then on GitHub: open a Pull Request into main, have a teammate look at it, and merge it.

Your branch does not get deleted. It is yours for the whole semester, and you will use it again next week.

Troubleshooting: “no upstream branch” error. A brand new branch has to be told which remote branch it belongs to the first time you push it. If you see fatal: The current branch ... has no upstream branch, the -u in the command above is the fix. If the jupyter-git push button fails this way, drop into a terminal and run the command once. After that the button works normally for that branch.

Weekly Workflow

Here is the rhythm for the rest of the semester:

1. Pull the new material into your sandbox. Whenever I post new notes:

git checkout main
git fetch upstream
git merge upstream/main

2. Work through the notes and the You Try cells in your sandbox. Change whatever you want, it is your copy.

If git complains about a conflict in one of my files, that just means you changed a file that I also changed. Nothing in your sandbox is graded, so the easy fix is to take my version and move on:

git checkout upstream/main -- <the file it named>

3. Do the homework with your team, in your team’s repo, in that week’s folder.

Start every week by bringing your branch up to date with whatever your teammates merged last week. This is the one habit that keeps Pull Requests from getting painful:

git checkout main
git pull origin main
git checkout your_branch_name
git merge main

Then work in that week’s folder, commit, push, open a Pull Request, have a teammate review it, and merge. You cannot commit to main directly, and you should not try to.

4. Submit your short individual self-reflection on Canvas each week. What you contributed, what you learned, and what was hard.

Prerequsites

This class assumes that you have had a course in Introductory Data Science and Introductory Programming (Python is really helpful). This means that you are familiar with things like:

  • Importing modules
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import plotly.express as px
from plotly.subplots import make_subplots
import plotly.io as pio
pio.renderers.default = 'notebook_connected'

  • Reading data into a computer

      DF = pd.read_csv()

  • Interacting with data in a data frame

      DF.columns
  DF.shape
  DF.dtypes
  DF.describe()

  • Masking data frames to focus on important columns

      focal_column = "names"
  mask = DF[focal_column]=='Joanna_Bieri'
  DF_joanna = DF[mask].copy()

  • Finding value counts

      DF[focal_column].value_counts()

  • Grouping and sorting data.

      colummns = ['names','money']
  DF[colummns].groupby('names').sum().sort_values('money',ascending=False)

  • Using operations or Applying Lambdas

      DF['money'].mean()
  DF['money'].sum()
  DF['first_names'] = DF['names'].apply(lambda x: str(x).split('_')[0])

  • Boolean statements and FOR loops

      for n in DF['first_names']:
      if n == 'Joanna':
          print(f'{n} is the best!')
      else:
          print('Still waiting for Joanna!')

  • Writing functions

       def add_two_numbers(a,b):
      print(f'adding {a} and {b}')
      return a+b

I have lots of videos and assignments that can help you to review this material!

Data 101 - Videos Playlist

Data 101 - Archive Website 2024

Intro to Python - Videos Playlist

Intro to Python - Website