Web Scraping — Extract Data from Web Pages

Install: pip install requests beautifulsoup4

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")

Finding elements:

# By tag
title = soup.find("h1").text
all_links = soup.find_all("a")

# By class
card = soup.find("div", class_="product-card")

# By id
header = soup.find(id="main-header")

# CSS selector
prices = soup.select(".price")
first_nav_link = soup.select_one("nav a")

Extracting data:

for link in soup.find_all("a"):
    href = link.get("href")
    text = link.text.strip()
    print(f"{text}: {href}")

Legal & ethical: Check robots.txt before scraping. Don't overload servers. Many sites have APIs — use those instead. Some sites prohibit scraping in their ToS.

Reference:

Wikipedia: Web Scraping

https://en.wikipedia.org/wiki/Web_scraping