Intermediate Data Science

Welcome Back

Author

Joanna Bieri
DATA201

Welcome to Intermediate Data Science!

Important Information

Computer Set Up

Before we can do much data science we need to make sure everyone has a stable computer set up for this semester!

We will be using Python and Jupyter Lab as our foundation for programming. You should have these already from DATA 101. If your setup still works, you are done here - open Jupyter Lab and go on to the next section.

Two things changed over the summer, so read on if yours is not working.

If Anaconda is asking you about a license, or what company you work for: you did not break anything. Anaconda changed the way they handle sign ups and the form is confusing, especially since none of us work for a company. It is still free for us. Here is the three minute fix: Sort out your Anaconda account

If you are starting fresh, or your setup is broken: DATA 101 moved to Miniforge this year. It does the same job as Anaconda without asking you to make an account. Follow How to set up your computer

You do not need to reinstall anything if what you have is working.

Software

Let’s start by checking what versions of the software we have:

  • Python 3.13.5
  • conda 24.11.3
# If you need to update:
!conda update -n base conda -y
!conda update --all -y
!python --version
Python 3.11.7
!conda --version
conda 24.11.3

Version Control

We also need to make sure we have git and jupyterlab-git installed.

First make sure you have a GitHub Account if not Sign Up For GitHub Account!. Log into your GitHub. Then check that you have git installed on your computer.

!git --version
git version 2.43.0

We will probably use jupyterlab-git to clone/push/pull from GitHub. If you see this icon on the left side of your jupyter, then you are good to go!

Git Icon

If you don’t see it you might need to run the following code and restart your Jupyter Lab.

!conda config --add channels conda-forge
!conda install -c conda-forge -y git
!conda install -c conda-forge -y jupyterlab-git

!conda update -y pexpect

Set up Workflow

1. Create a Directory for your work in this class

Using the Jupyter File Browser, make a folder (directory) that will contain all of your work for this class. Make sure to name it something good so that it is easily findable!

2. Once your team is announced today, clone your team’s repo

Teams are assigned in class today. Each team gets its own repo – something like Redlands-DATA201/hw1-team-a – that I create as a direct duplicate of our class materials repo, Redlands-DATA201/FALL26. That means your team’s repo already contains everything posted there so far – you never need to clone FALL26 yourself.

Get your team’s repo URL, then:

In a terminal window:

git clone <your-team-repo-url>
cd <your-team-repo-folder-name>
git remote add upstream https://github.com/Redlands-DATA201/FALL26.git

In jupyter-git:

click on the jupyter-git icon
select clone repo
enter your team's repo address

Just clone it directly, no need to fork first. Adding upstream is a one-time setup step – it’s how you’ll pull in new material I post after today.

3. Install the packages you’ll need all semester

Your team’s repo includes a requirements.txt listing every package we’ll use this semester. Install them all at once, right now, so you never have to think about it again:

In a terminal window (inside your team’s repo folder):

pip install -r requirements.txt

If a later assignment tells you to install something new, it means the package genuinely wasn’t on this list – otherwise, if you already ran this, you’re covered.

4. Turn on clean notebook diffs

Your team’s requirements.txt includes nbstripout, which strips notebook outputs from every commit automatically – so .ipynb “merge conflicts” almost always come from real code differences, not just someone re-running a cell. It needs one extra activation step, once per person, per clone (this part doesn’t happen automatically):

In a terminal window (inside your team’s repo folder):

nbstripout --install

Run this once, right now. Every teammate needs to run it in their own clone.

5. Do a first test push with your team, right now

This is the exact cycle you’ll use for every piece of work all semester – let’s do it once together so it’s not new when it counts:

In a terminal window:

git checkout -b test-push-<your-name>

Then create a new Markdown or Text file (New Launcher) and type something in it, like “Hello from <your name>!” Save it.

git add <your file name>
git commit -m 'testing the workflow'
git push -u origin test-push-<your-name>

In jupyter-git:

triple check you're in your TEAM's repo, not the class repo (you can't write to that one)
add the untracked file using the + sign, stage it, write a commit message, and push on the up arrow cloud

Troubleshooting: “no upstream branch” error. A brand-new branch has to be told which remote branch to track the first time you push it. If you see fatal: The current branch ... has no upstream branch:

  • Terminal: run git push -u origin test-push-<your-name> (the -u is the fix – notice it’s included in the command above).
  • jupyter-git UI: clicking the push cloud icon on a brand-new branch may show a dialog asking you to confirm the remote branch name, or may just fail with the same error. If it fails, switch to a terminal window and run the command above once – after that, the push button will work normally for this branch.

Then on GitHub: open a Pull Request into your team’s main, ask a teammate to review and approve it, then merge and delete the branch.

6. Clean up your local branch

Merging on GitHub deletes the remote branch, but your local copy is still sitting on your computer. Every teammate who worked on that branch should clean it up locally too, once it’s merged:

In a terminal window:

git checkout main
git pull origin main
git branch -d test-push-<your-name>

git branch -d only deletes a branch git can confirm is fully merged – that’s a safety check, not a bug, so if it refuses, something didn’t merge the way you expected.

In jupyter-git:

switch to the "Branches" panel, click over to `main`, then hover over the old branch name and use the delete option

Do this after every merged Pull Request, all semester – otherwise your branch list just keeps growing.

This exact cycle – branch, commit, push, Pull Request, review, merge, clean up – is what you’ll do for every homework from here on. Full quick reference: see “Your Team Git Workflow” on the last page of the syllabus.

Weekly Workflow

Once your team’s repo is set up, here’s the rhythm for the rest of the semester:

  1. Whenever I announce new material, one teammate pulls it into your team’s repo: fetch upstream, merge it into main, push – see the syllabus’s Git Workflow page for the exact commands.
  2. Everyone else just pulls the update from your team’s repo like normal.
  3. Do the assignment on a branch, following the branch → review → merge cycle above.
  4. Submit a short individual self-reflection on Canvas each week – see the syllabus’s Classwork section for details.

Prerequsites

This class assumes that you have had a course in Introductory Data Science and Introductory Programming (Python is really helpful). This means that you are familiar with things like:

  • Importing modules
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import plotly.express as px
from plotly.subplots import make_subplots
import plotly.io as pio
pio.renderers.default = 'notebook_connected'

  • Reading data into a computer

      DF = pd.read_csv()

  • Interacting with data in a data frame

      DF.columns
  DF.shape
  DF.dtypes
  DF.describe()

  • Masking data frames to focus on important columns

      focal_column = "names"
  mask = DF[focal_column]=='Joanna_Bieri'
  DF_joanna = DF[mask].copy()

  • Finding value counts

      DF[focal_column].value_counts()

  • Grouping and sorting data.

      colummns = ['names','money']
  DF[colummns].groupby('names').sum().sort_values('money',ascending=False)

  • Using operations or Applying Lambdas

      DF['money'].mean()
  DF['money'].sum()
  DF['first_names'] = DF['names'].apply(lambda x: str(x).split('_')[0])

  • Boolean statements and FOR loops

      for n in DF['first_names']:
      if n == 'Joanna':
          print(f'{n} is the best!')
      else:
          print('Still waiting for Joanna!')

  • Writing functions

       def add_two_numbers(a,b):
      print(f'adding {a} and {b}')
      return a+b

I have lots of videos and assignments that can help you to review this material!

Data 101 - Videos Playlist

Data 101 - Archive Website 2024

Intro to Python - Videos Playlist

Intro to Python - Website