Web Scraping

Overview

We can consume web sites with our eyes, ears and fingers. But how about getting your computer to consume it for you?

If a service exposes an API then you're good to go. But what if they haven't? That's where web scraping comes in.

Feedback Discussion

• 'Less review more preview'
  • some push back, helps retain concepts
  • less time
  • seeing different approaches
  • instructor lead code review, fork repo, lead it
    • code review volunteers
• Emphasis on python code
• less emphasis with whiteboarding
• More structure to lectures
  • Concepts to paper (ide)
  • smaller whys and whats of code
• Roadmap/big picture 'Why'
  • 'Why _ does what it does'
• Sense of intimidation

Class Outline

Learning Objectives

Students will be able to

Describe and Define

• Why to scrape a web site
• Legal perspectives on web scraping
• Challenges of web scraping

Execute

• Use requests and beautiful soup libraries
• Inspect a web pages content to determine an effective scraping method
• Execute web scraper from CLI