{ "cells": [ { "cell_type": "markdown", "id": "33738fde-205d-4e95-b885-807e97a8adf6", "metadata": {}, "source": [ "## Introduction to Data Science\n", "\n", "#### University of Redlands - DATA 101\n", "#### Prof: Joanna Bieri [joanna_bieri@redlands.edu](mailto:joanna_bieri@redlands.edu)\n", "#### [Class Website: data101.joannabieri.com](https://data101.joannabieri.com)\n", "\n", "---------------------------------------\n", "# Homework Day 12\n", "---------------------------------------\n", "\n", "GOALS:\n", "\n", "1. Talk about Ethics of Web Scraping\n", "2. Practice web scraping.\n", "\n", "----------------------------------------------------------\n", "\n", "This homework has **3 questions**, **1 exercise**, **1 essay response** and **1 optional challenge problem**." ] }, { "cell_type": "markdown", "id": "2bf78cd1", "metadata": {}, "source": [ "## Important Information\n", "\n", "- Email: [joanna_bieri@redlands.edu](mailto:joanna_bieri@redlands.edu)\n", "- Office Hours: Duke 209 Click Here for Joanna's Schedule\n", "\n", "## Announcements\n", "\n", "**Day 12 and Day 13 homework are due together Sunday 11/1 11:59pm.** (Moved from 10/25: Thursday 10/22 is a Catch-Up Day.)\n", "\n", "**Data Ethics starts Tuesday 10/27 (Day 13).** Come to class Tuesday prepared to discuss your ethics reading with your classmates!\n", "\n", "**Final Project Proposals are due Sunday 11/8.** See the Final Project link on the class website.\n", "\n", "------------------------------\n", "---------------------\n", "\n", "### Web Scraping Ethical Issues\n", "\n", "There are some things to be aware of before you start scraping data from the web. \n", "\n", "- Some data is private or protected. Just because you have access to a websites data doesn't mean you are allowed to scrape it. For example, when you log into Facebook or another social media site, you are granted special access to data about your connected people. It is unethical to use that access to scrape their private data!\n", "\n", "- Some websites have rules against scraping and will cut off service to users who are clearly scraping data. How do they know? Webscrapers access the website very differently than regular users. If the site has a policy about scraping data then you should follow it and/or contact them about getting the data if you have a true academic interest in the data.\n", "\n", "- The line between web scraping and plagiarism can be very blurry. Make sure that you are citing where your data comes from AND not just reproducing the data exactly. Always cite the source of your data and make sure you are doing something new with it.\n", "\n", "- Ethics are different depending on if you are using the data for a personal project (eg. you just want to check scores for your favorite team daily and print the stuff you care about) vs if you are using the project for your business or website (eg. publishing information to drive clicks to your site/video/account or making money from the data you collect). In the latter case it is EXTRA important to respect the original owner of the data. Drive web traffic back to their site, check with them about using their data, etc.\n", "\n", "**The Ethical Scraper** (from https://towardsdatascience.com/ethics-in-web-scraping-b96b18136f01):\n", "\n", "I, the web scraper will live by the following principles:\n", "\n", "- If you have a public API that provides the data I’m looking for, I’ll use it and avoid scraping all together.\n", "- I will always provide a User Agent string that makes my intentions clear and provides a way for you to contact me with questions or concerns.\n", "- I will request data at a reasonable rate. I will strive to never be confused for a DDoS attack.\n", "- I will only save the data I absolutely need from your page. If all I need it OpenGraph meta-data, that’s all I’ll keep.\n", "- I will respect any content I do keep. I’ll never pass it off as my own.\n", "- I will look for ways to return value to you. Maybe I can drive some (real) traffic to your site or credit you in an article or post.\n", "- I will respond in a timely fashion to your outreach and work with you towards a resolution.\n", "- I will scrape for the purpose of creating new value from the data, not to duplicate it." ] }, { "cell_type": "code", "execution_count": 1, "id": "b608007e", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:07.413800Z", "iopub.status.busy": "2026-10-07T20:48:07.413700Z", "iopub.status.idle": "2026-10-07T20:48:08.474004Z", "shell.execute_reply": "2026-10-07T20:48:08.473493Z" } }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "import matplotlib.pyplot as plt\n", "import plotly.express as px\n", "from plotly.subplots import make_subplots\n", "import plotly.io as pio\n", "pio.renderers.default = 'notebook_connected'\n", "\n", "# This stops a few warning messages from showing\n", "pd.options.mode.chained_assignment = None \n", "import warnings\n", "warnings.simplefilter(action='ignore', category=FutureWarning)" ] }, { "cell_type": "code", "execution_count": null, "id": "cd38c98b-cc3d-4150-973e-b7bc883f0b74", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "3eec300d", "metadata": {}, "source": [ "## Using pandas to get table data.\n", "\n", "**Optional** - try using pandas to scrape a new site. See if you get errors or are able to find tables on that site.\n" ] }, { "cell_type": "markdown", "id": "9faebc13", "metadata": {}, "source": [ "---\n", "---\n", "\n", "## Using Beautiful Soup to get HTML code\n", "\n", "### How to get data from static sites:\n", "\n", "You already have the packages bs4 and requests. You installed them with `pip install -r requirements.txt` during the Day 1 setup, so there is nothing to install here." ] }, { "cell_type": "code", "execution_count": 2, "id": "1688f055", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:08.476217Z", "iopub.status.busy": "2026-10-07T20:48:08.476124Z", "iopub.status.idle": "2026-10-07T20:48:08.514710Z", "shell.execute_reply": "2026-10-07T20:48:08.514234Z" } }, "outputs": [], "source": [ "import requests\n", "from bs4 import BeautifulSoup" ] }, { "cell_type": "code", "execution_count": 3, "id": "aad9d963", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:08.516301Z", "iopub.status.busy": "2026-10-07T20:48:08.516169Z", "iopub.status.idle": "2026-10-07T20:48:08.517900Z", "shell.execute_reply": "2026-10-07T20:48:08.517610Z" } }, "outputs": [], "source": [ "website = 'https://www.scrapethissite.com/pages/simple/'" ] }, { "cell_type": "code", "execution_count": 4, "id": "08ea4457", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:08.518985Z", "iopub.status.busy": "2026-10-07T20:48:08.518919Z", "iopub.status.idle": "2026-10-07T20:48:09.258159Z", "shell.execute_reply": "2026-10-07T20:48:09.257069Z" } }, "outputs": [], "source": [ "raw_code = requests.get(website)\n", "html_doc = raw_code.text\n", "soup = BeautifulSoup(html_doc, 'html.parser')" ] }, { "cell_type": "markdown", "id": "dd104b88", "metadata": {}, "source": [ "#### Lets see what is in soup\n", "\n", "**Q1** Uncomment this line and run the cell. You will see a TON of text!" ] }, { "cell_type": "code", "execution_count": 5, "id": "98582e0c", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:09.260070Z", "iopub.status.busy": "2026-10-07T20:48:09.260001Z", "iopub.status.idle": "2026-10-07T20:48:09.262099Z", "shell.execute_reply": "2026-10-07T20:48:09.261438Z" } }, "outputs": [], "source": [ "#soup" ] }, { "cell_type": "markdown", "id": "8a248cb1", "metadata": {}, "source": [ "### Extracting data from HTML:\n", "\n", "We will use the **soup.find_all()** function.\n", "\n", "Here is the simplified function signature:\n", "\n", " soup.find_all(name=None,attrs={})\n", " \n", "You can type soup.find_all? and run it to see all the information about additional arguments and advanced processes.\n", "\n", "**Here is how we will mostly use it**, but there are much more advanced things you can do:\n", "\n", " soup.find_all( , )\n", " \n", "The **.find_all()** function searches through the information in soup to match and return only sections that match the info. Here are some important types you might search for:\n", "\n", "- 'h2' - this is a heading\n", "- 'div' - this divides a block of information\n", "- 'span' - this divides inline information\n", "- 'a' - this specifies a hyperlink\n", "- 'li' - this is a list item\n", "\n", "- class_= - many things have the class label (notice the underscore!)\n", "- string= - you can also search by strings.\n", "\n", "\n", "### Using Developer tools:\n", "\n", "To figure out what data to extract I suggest you use developer tools on the website to find what you need. \n", "\n", "**Q2** Navigate to the website and use developer tools to explore the html code.\n", "\n", "[Scrape This Site - Website](https://www.scrapethissite.com/pages/simple/)\n", "\n", "I really like Brave Browser or Google Chrome for this, but most browsers will have **More Tools/Developer Tools** where you can see the code.\n", "\n", "### Search for all the country names\n", "\n", "The names of the country are inside \n", " \n", "

\n", "\n", "So lets search for this:" ] }, { "cell_type": "code", "execution_count": 6, "id": "6129c2d5-5e75-4b22-8d94-87ea473528a2", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:09.263293Z", "iopub.status.busy": "2026-10-07T20:48:09.263239Z", "iopub.status.idle": "2026-10-07T20:48:09.267092Z", "shell.execute_reply": "2026-10-07T20:48:09.266728Z" } }, "outputs": [], "source": [ "result = soup.find_all('h3',class_=\"country-name\")\n", "#result" ] }, { "cell_type": "code", "execution_count": 7, "id": "bf0a0301-a7c3-4e79-a236-cce7547991aa", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:09.268326Z", "iopub.status.busy": "2026-10-07T20:48:09.268273Z", "iopub.status.idle": "2026-10-07T20:48:09.271928Z", "shell.execute_reply": "2026-10-07T20:48:09.271618Z" } }, "outputs": [ { "data": { "text/plain": [ "

\n", "\n", " Andorra\n", "

" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result[0]" ] }, { "cell_type": "code", "execution_count": 8, "id": "95ad57dd-31e2-43e6-aa43-0b832af3aade", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:09.273055Z", "iopub.status.busy": "2026-10-07T20:48:09.273005Z", "iopub.status.idle": "2026-10-07T20:48:09.275343Z", "shell.execute_reply": "2026-10-07T20:48:09.274997Z" } }, "outputs": [ { "data": { "text/plain": [ "'Andorra'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result[0].text.lstrip().rstrip()" ] }, { "cell_type": "code", "execution_count": 9, "id": "aa7c4663-5aca-4899-95a6-9d83c310fe65", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:09.276353Z", "iopub.status.busy": "2026-10-07T20:48:09.276306Z", "iopub.status.idle": "2026-10-07T20:48:09.281912Z", "shell.execute_reply": "2026-10-07T20:48:09.281648Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
country
0Andorra
1United Arab Emirates
2Afghanistan
3Antigua and Barbuda
4Anguilla
......
245Yemen
246Mayotte
247South Africa
248Zambia
249Zimbabwe
\n", "

250 rows × 1 columns

\n", "
" ], "text/plain": [ " country\n", "0 Andorra\n", "1 United Arab Emirates\n", "2 Afghanistan\n", "3 Antigua and Barbuda\n", "4 Anguilla\n", ".. ...\n", "245 Yemen\n", "246 Mayotte\n", "247 South Africa\n", "248 Zambia\n", "249 Zimbabwe\n", "\n", "[250 rows x 1 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a data frame and add the country names\n", "DF = pd.DataFrame()\n", "DF['country']=result\n", "DF['country'] = DF['country'].apply(lambda x: x.text.lstrip().rstrip())\n", "DF" ] }, { "cell_type": "markdown", "id": "e89e9aa7-957a-4c03-9c2e-0909dcc9bb59", "metadata": {}, "source": [ "### Your turn to try this!\n", "\n", "**Q3** Now it's your turn. See if you can write code that gets the capital, population and area information into the data frame. See if you can make your example match what I get below, including having the correct data types. Population should be an int and area should be a float.\n", "\n", "Your goal is to get a data frame that looks like the one in lecture with columns for country, capital, population, and area!" ] }, { "cell_type": "markdown", "id": "176e1115-9454-41ae-92fb-c65426fb2fd4", "metadata": {}, "source": [ "### Where can you get more practice\n", "\n", "Here is a website dedicated to allowing students to practice webscraping:\n", "\n", "[www.scrapethissite.com](https://www.scrapethissite.com/pages/)\n", "\n", " website=https://www.scrapethissite.com/pages/\n", "-----------------\n", "### Exercise 1:\n", "\n", "We are going to scrape the site above to get the list of all links. \n", "\n", "Open the website and look at the developer tools.\n", "\n", "Here is our goal: Make a pandas data frame that contains three columns: \n", "\n", "* \"site_name\" - which contains just the words of the link\n", "* \"link\" - which contains just the website part of the link\n", "* \"description\" - which contains the words below the link\n", "\n", "When looking for links you can use\n", "\n", " result = soup.find_all('a')\n", " result[0].text # Get the text associated with the link\n", " result[0].get('href') # Get the link location\n", "\n", "\n", "\n", "See if you can do this without looking at my code in the notes! What would your plan have to be?\n", "\n", "**See the lecture notes for hints and the solution!**\n" ] }, { "cell_type": "markdown", "id": "ac2190a2-a49f-491c-9fa1-65557cd9a6ff", "metadata": {}, "source": [ "## Essay Response\n", "\n", "Give a brief report on what you are reading and what you have learned so far. If you chose your own articles please insert a link to the articles. If you chose a book, please give the book title and author.\n", "\n", "1. What drew you to the book/articles\n", "2. What have you learned so far.\n", "\n", "eg. In the first chapter of the book the author talks about..... or My first article discusses..... or I disagree with what the author is saying so far.... or I really like what I am learning about because...." ] }, { "cell_type": "code", "execution_count": null, "id": "c4013d5a-a6a9-4881-80e1-c52d12e59c71", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "ccdc8ae8-0880-4abb-a9ef-7d497210ee34", "metadata": {}, "source": [ "### Challenge Problem\n", "\n", "Here is another website to scrape. See if you can create a data frame that looks like the one in the lecture notes. Notice that you can only scrape the first page with the first link. \n", "\n", "If you want to try scraping the other pages you have to notice how the website updates its address for each page. Then write a for loop to loop through how ever many pages you want to scrape. Do the same set of operations for each page keep adding data to your data frame.\n", "\n", "Make a histogram of your final data." ] }, { "cell_type": "code", "execution_count": 10, "id": "267722fb-a33b-4969-82eb-26c94edd0bb6", "metadata": { "execution": { "iopub.execute_input": "2026-10-07T20:48:09.283062Z", "iopub.status.busy": "2026-10-07T20:48:09.283010Z", "iopub.status.idle": "2026-10-07T20:48:09.606686Z", "shell.execute_reply": "2026-10-07T20:48:09.606104Z" } }, "outputs": [], "source": [ "website='https://books.toscrape.com/index.html'\n", "raw_code = requests.get(website)\n", "html_doc = raw_code.text\n", "soup = BeautifulSoup(html_doc, 'html.parser')" ] }, { "cell_type": "markdown", "id": "09b86c1d-5c13-4178-9685-c09922f4ec2d", "metadata": {}, "source": [ "**Try to scrape the name, the link to the book, and the prices! I decided to put the name and link information into a single column and then break that apart**" ] }, { "cell_type": "code", "execution_count": null, "id": "429be6b6-4fdc-41e4-a481-c778ac9bc9ca", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "08613fec-b6db-44cd-b093-69e43704cef0", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.15" } }, "nbformat": 4, "nbformat_minor": 5 }