5.3 Web Scraping Pipelines

5.3 Web Scraping Pipelines

Web scraping is the process of programmatically retrieving information from web pages and extracting useful data from their HTML structure.

In Python, a common static-HTML scraping workflow uses:

• requests to send HTTP requests and retrieve web pages.
• BeautifulSoup to parse HTML and locate elements.
• Python data structures to clean and organize extracted data.
• pandas when the final result needs tabular analysis.

1. What Is Static HTML?

A static page contains the required information in the HTML response received from the server.

For such pages, requests can retrieve the HTML and BeautifulSoup can parse it without opening a browser.

2. Installing the Required Libraries

pip install requests beautifulsoup4

If you also want to work with extracted tabular data:

pip install pandas

In a Conda environment:

conda install requests beautifulsoup4 pandas

3. Importing the Libraries

import requests

from bs4 import BeautifulSoup

The package is installed as beautifulsoup4, while the Python class is imported from the bs4 module.

4. Sending Your First HTTP Request

import requests

url = "https://example.com"

response = requests.get(url)

print(response.status_code)

The response object contains the information returned by the server.

5. Understanding HTTP Status Codes

6. Using raise_for_status()

import requests

url = "https://example.com"

response = requests.get(url)

response.raise_for_status()

print(response.text)

raise_for_status() raises an exception for unsuccessful HTTP status codes.

7. Reading the HTML Response

response = requests.get(
    "https://example.com"
)

html = response.text

print(html)

response.text provides the response body decoded as text.

For raw bytes, use:

content = response.content

8. Parsing HTML with BeautifulSoup

from bs4 import BeautifulSoup

soup = BeautifulSoup(
    response.text,
    "html.parser"
)

The parser converts the HTML text into a structure that can be searched and navigated using Python.

9. Finding the First Matching Element

Consider this HTML:

<h1>Python Web Scraping</h1>

Extract the first <h1> element:

heading = soup.find("h1")

print(heading)

To obtain only its text:

print(
    heading.get_text()
)

10. Finding Multiple Elements with find_all()

Suppose a page contains:

<p>First paragraph</p>
<p>Second paragraph</p>
<p>Third paragraph</p>

paragraphs = soup.find_all("p")

for paragraph in paragraphs:

    print(
        paragraph.get_text(strip=True)
    )

11. Searching by HTML Tag

links = soup.find_all("a")

images = soup.find_all("img")

headings = soup.find_all("h2")

tables = soup.find_all("table")

HTML tag names are one of the simplest ways to begin extracting content.

12. Searching by CSS Class

Suppose the HTML contains:

<div class="product">
    <h2>Laptop</h2>
</div>

Find the element using its class:

product = soup.find(
    "div",
    class_="product"
)

13. Searching by HTML ID

<div id="main-content">
    ...
</div>

content = soup.find(
    id="main-content"
)

14. Extracting Clean Text

HTML elements contain tags as well as text. Use get_text() to extract the human-readable content.

element = soup.find("h1")

text = element.get_text(
    strip=True
)

print(text)

strip=True removes unnecessary whitespace around the extracted text.

15. Extracting HTML Attributes

HTML elements often contain useful attributes.

<a
    href="https://example.com"
    class="learn-link"
>
    Learn
</a>

Access an attribute using dictionary-style syntax:

link = soup.find("a")

print(
    link["href"]
)

print(
    link["class"]
)

16. Safely Reading an Attribute with get()

href = link.get("href")

print(href)

get() returns None when the attribute is absent instead of immediately raising a KeyError.

17. Extracting Links from a Page

links = soup.find_all("a")

for link in links:

    text = link.get_text(
        strip=True
    )

    href = link.get("href")

    print(text, href)

This is one of the most common web scraping tasks.

18. Extracting Image URLs

images = soup.find_all("img")

for image in images:

    src = image.get("src")

    print(src)

Other useful attributes may include alt, width, and height.

19. Using CSS Selectors

BeautifulSoup supports CSS-style selectors through select() and select_one().

heading = soup.select_one(
    "h1"
)

Select all matching elements:

items = soup.select(
    ".product"
)

20. Common CSS Selectors

21. Extracting Nested Content

Consider:

<article class="course">

    <h2>Python</h2>

    <p>Learn Python programming.</p>

</article>

course = soup.select_one(
    "article.course"
)

title = course.select_one(
    "h2"
).get_text(strip=True)

description = course.select_one(
    "p"
).get_text(strip=True)

print(title)
print(description)

22. Scraping HTML Tables

Example HTML:

<table>

    <tr>
        <th>Name</th>
        <th>Score</th>
    </tr>

    <tr>
        <td>Alex</td>
        <td>95</td>
    </tr>

</table>

Extract rows:

table = soup.find("table")

rows = table.find_all("tr")

for row in rows:

    cells = row.find_all(
        ["th", "td"]
    )

    values = [
        cell.get_text(strip=True)
        for cell in cells
    ]

    print(values)

23. Extracting Tables with Pandas

When the page contains standard HTML tables, pandas can sometimes simplify extraction.

import pandas as pd

tables = pd.read_html(
    "https://example.com"
)

print(
    tables[0]
)

24. Sending Request Headers

HTTP requests can include headers such as User-Agent.

import requests

headers = {
    "User-Agent":
        "Mozilla/5.0"
}

response = requests.get(
    "https://example.com",
    headers=headers
)

response.raise_for_status()

Headers are part of the HTTP protocol and may affect how a server responds.

25. Always Use a Timeout

response = requests.get(
    "https://example.com",
    timeout=10
)

A timeout prevents the program from waiting indefinitely for a server response.

26. Handling Request Errors

import requests

url = "https://example.com"

try:

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    print("Request successful.")

except requests.Timeout:

    print(
        "The request timed out."
    )

except requests.HTTPError as error:

    print(
        "HTTP error:",
        error
    )

except requests.RequestException as error:

    print(
        "Request failed:",
        error
    )

27. Using a requests.Session

A session can persist certain settings, such as headers and cookies, across multiple requests.

import requests

session = requests.Session()

session.headers.update({
    "User-Agent": "Mozilla/5.0"
})

response = session.get(
    "https://example.com",
    timeout=10
)

response.raise_for_status()

Sessions can also reuse connections, which can be useful when making multiple requests to the same host.

28. Handling Relative URLs

Scraped links are often relative:

/courses/python.html

Convert them into absolute URLs with urllib.parse.urljoin().

from urllib.parse import urljoin

base_url = (
    "https://example.com"
)

relative_url = (
    "/courses/python.html"
)

absolute_url = urljoin(
    base_url,
    relative_url
)

print(absolute_url)

29. Building a Link Extraction Pipeline

import requests

from bs4 import BeautifulSoup

from urllib.parse import urljoin


url = "https://example.com"

response = requests.get(
    url,
    timeout=10
)

response.raise_for_status()


soup = BeautifulSoup(
    response.text,
    "html.parser"
)


for link in soup.select("a[href]"):

    text = link.get_text(
        " ",
        strip=True
    )

    href = link.get("href")

    absolute_url = urljoin(
        url,
        href
    )

    print(
        text,
        absolute_url
    )

30. Cleaning Scraped Text

Web pages frequently contain extra whitespace and formatting.

text = element.get_text(
    " ",
    strip=True
)

Further cleaning can be performed using standard Python string methods.

clean_text = (
    text
    .replace("\n", " ")
    .strip()
)

31. Normalizing Whitespace

import re

clean_text = re.sub(
    r"\s+",
    " ",
    text
).strip()

This converts repeated whitespace characters into a single space.

32. Extracting Structured Records

Suppose a page contains:

<div class="course">

    <h2>Python for Data Science</h2>

    <p class="duration">
        20 hours
    </p>

</div>

courses = []

for item in soup.select(
    ".course"
):

    title = item.select_one(
        "h2"
    ).get_text(strip=True)

    duration = item.select_one(
        ".duration"
    ).get_text(strip=True)

    courses.append({
        "title": title,
        "duration": duration
    })

print(courses)

This converts unstructured HTML into Python dictionaries.

33. Converting Scraped Data into a DataFrame

import pandas as pd

df = pd.DataFrame(
    courses
)

print(df)

Once the data is in a DataFrame, the cleaning and transformation techniques from Module 3 can be applied.

34. Saving Scraped Data

df.to_csv(
    "courses.csv",
    index=False
)

Other useful formats include:

df.to_excel(
    "courses.xlsx",
    index=False
)

df.to_json(
    "courses.json",
    orient="records",
    indent=2
)

35. Scraping Multiple Pages

Many websites divide results into multiple pages.

A simple pattern is:

import requests

from bs4 import BeautifulSoup


for page_number in range(1, 6):

    url = (
        f"https://example.com/"
        f"products?page={page_number}"
    )

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    products = soup.select(
        ".product"
    )

    for product in products:

        print(
            product.get_text(
                " ",
                strip=True
            )
        )

36. Following Pagination Links

Instead of constructing URLs manually, you can extract the next-page link.

next_link = soup.select_one(
    "a.next[href]"
)

if next_link:

    next_url = urljoin(
        url,
        next_link["href"]
    )

    print(next_url)

37. Multi-Page Scraping Workflow

current_url = (
    "https://example.com/products"
)

while current_url:

    response = requests.get(
        current_url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    # Extract records here

    next_link = soup.select_one(
        "a.next[href]"
    )

    if next_link:

        current_url = urljoin(
            current_url,
            next_link["href"]
        )

    else:

        current_url = None

38. Rate Limiting and Request Delays

Sending requests too quickly can overload a server and may violate site rules.

import time

time.sleep(2)

In a multi-page scraper, a delay can be placed between requests when appropriate.

39. Understanding robots.txt

Many websites publish a robots.txt file describing crawling preferences for automated agents.

It can often be found at:

https://example.com/robots.txt

Scrapers should review the site's published crawling rules and terms before collecting data.

40. Responsible Web Scraping

Technical capability does not automatically mean that data should be collected.

• Respect website terms and published policies.
• Check applicable crawling guidance.
• Avoid excessive request rates.
• Collect only the information required for the legitimate task.
• Avoid collecting sensitive personal information unnecessarily.
• Do not attempt to bypass authentication, access controls, or technical restrictions.
• Prefer official APIs when they are available and appropriate.
• Identify and handle data provenance appropriately.

41. API vs Web Scraping

If a website provides an official API for the information you need, the API is often the better engineering choice.

42. When requests + BeautifulSoup Is Not Enough

A page may display information in the browser that is not present in the initial HTML response.

This can happen when JavaScript obtains the data after page load.

Before using browser automation, inspect whether the site exposes the required data through an accessible API or embedded data source.

43. Complete Basic Web Scraper

import requests

from bs4 import BeautifulSoup


url = "https://example.com"


try:

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    heading = soup.find("h1")

    if heading:

        print(
            heading.get_text(
                " ",
                strip=True
            )
        )

except requests.RequestException as error:

    print(
        "Request failed:",
        error
    )

44. Mini Project — Product Data Scraper

Assume the page contains product cards structured like:

<div class="product-card">

    <h2 class="product-name">
        Laptop Pro
    </h2>

    <span class="price">
        $999
    </span>

</div>

Extract the product names and prices.

import requests

from bs4 import BeautifulSoup

import pandas as pd


url = "https://example.com/products"


response = requests.get(
    url,
    timeout=10
)

response.raise_for_status()


soup = BeautifulSoup(
    response.text,
    "html.parser"
)


products = []


for card in soup.select(
    ".product-card"
):

    name_element = card.select_one(
        ".product-name"
    )

    price_element = card.select_one(
        ".price"
    )

    name = (
        name_element.get_text(
            " ",
            strip=True
        )
        if name_element
        else None
    )

    price = (
        price_element.get_text(
            " ",
            strip=True
        )
        if price_element
        else None
    )

    products.append({
        "name": name,
        "price": price
    })


df = pd.DataFrame(
    products
)


df.to_csv(
    "products.csv",
    index=False
)

print(df)

45. Validating Scraped Data

Scraping is not complete simply because HTML was downloaded. Extracted values should be validated.

if not name:
    print(
        "Missing product name."
    )

if not price:
    print(
        "Missing product price."
    )

Check for:

• Missing fields.
• Unexpected formats.
• Duplicate records.
• Invalid URLs.
• Changed HTML structures.
• Unexpected error pages.

46. Removing Duplicate Scraped Records

df = df.drop_duplicates()

When a particular field identifies a record:

df = df.drop_duplicates(
    subset=["name"]
)

47. Cleaning Scraped Prices

Web pages often represent prices as strings.

df["price"] = (
    df["price"]
    .str.replace("$", "", regex=False)
    .str.replace(",", "", regex=False)
)

df["price"] = pd.to_numeric(
    df["price"],
    errors="coerce"
)

This converts values such as $1,299 into numeric data suitable for analysis.

48. Logging a Scraping Pipeline

import logging

logging.basicConfig(
    level=logging.INFO,
    format=(
        "%(asctime)s - "
        "%(levelname)s - "
        "%(message)s"
    )
)

logging.info(
    "Scraping started."
)

logging.info(
    "Page downloaded."
)

logging.info(
    "Records extracted."
)

Logging becomes particularly useful when a scraper processes many pages.

49. Handling Temporary Failures

Real-world network requests can fail temporarily. A production pipeline may use controlled retries with backoff.

import time
import requests


for attempt in range(3):

    try:

        response = requests.get(
            url,
            timeout=10
        )

        response.raise_for_status()

        break

    except requests.RequestException:

        if attempt == 2:

            raise

        time.sleep(
            2 ** attempt
        )

50. Designing a Maintainable Scraper

Avoid putting the entire scraper into one large block of code. Separate responsibilities into functions.

def fetch_page(url):
    ...


def parse_products(html):
    ...


def clean_data(records):
    ...


def save_data(records):
    ...


def main():
    ...


if __name__ == "__main__":
    main()

This structure makes the scraper easier to test, debug, and maintain when the website changes.

51. A Reusable Scraping Architecture

URL
 ↓
fetch_page()
 ↓
parse_html()
 ↓
extract_records()
 ↓
clean_data()
 ↓
validate_data()
 ↓
save_data()
 ↓
log_result()

This separation is an important step from writing a simple scraper toward building a reliable data pipeline.

52. Important Concepts to Remember

53. Web Scraping Interview Questions

54. Examination Questions — MCQs

55. Practical Examination Questions

56. Debugging a Scraper

When extraction fails, inspect the actual HTML before changing your Python code.

print(
    response.status_code
)

print(
    response.url
)

print(
    response.text[:1000]
)

Also inspect:

• Whether the URL is correct.
• Whether the request succeeded.
• Whether the desired element exists in the response HTML.
• Whether the selector matches the current structure.
• Whether the content is generated dynamically.
• Whether the server returned an error or alternate page.

57. Common Web Scraping Errors

58. Web Scraping Best Practices

1. Use timeouts for network requests.
2. Check HTTP status codes.
3. Use raise_for_status() where appropriate.
4. Use CSS selectors that accurately describe the required data.
5. Keep fetching, parsing, cleaning, and saving as separate stages.
6. Validate extracted data.
7. Handle missing HTML elements gracefully.
8. Avoid unnecessary requests.
9. Use controlled request delays where appropriate.
10. Prefer official APIs when available.
11. Do not bypass authentication or access controls.
12. Protect sensitive collected information.
13. Log important processing events.
14. Expect website HTML structures to change.

59. Expert Tips

60. Capstone Project — Web Data Collection Pipeline

Build a reusable Python application that collects publicly accessible data from a static HTML website and prepares it for analysis.

Required Features

1. Accept a starting URL.
2. Download the page with requests.
3. Use a timeout and error handling.
4. Parse HTML using BeautifulSoup.
5. Extract structured records using CSS selectors.
6. Convert relative URLs to absolute URLs.
7. Follow permitted pagination links.
8. Clean whitespace and malformed values.
9. Remove duplicate records.
10. Validate required fields.
11. Store results in a pandas DataFrame.
12. Export the final dataset to CSV and Excel.
13. Log the number of pages processed and records collected.

61. Self-Assessment Checklist

Before moving to the next topic, make sure you can:

• ☐ Explain what web scraping is.
• ☐ Explain the difference between static and dynamic content.
• ☐ Install and import requests.
• ☐ Send an HTTP GET request.
• ☐ Read the HTTP status code.
• ☐ Use raise_for_status().
• ☐ Set a request timeout.
• ☐ Parse HTML using BeautifulSoup.
• ☐ Use find() and find_all().
• ☐ Use select() and select_one().
• ☐ Extract text from elements.
• ☐ Extract HTML attributes.
• ☐ Extract links and images.
• ☐ Handle relative URLs.
• ☐ Scrape HTML tables.
• ☐ Extract structured records.
• ☐ Convert scraped records into a DataFrame.
• ☐ Save scraped data to CSV and Excel.
• ☐ Handle pagination.
• ☐ Add delays and controlled retries.
• ☐ Handle request and parsing errors.
• ☐ Validate scraped data.
• ☐ Explain responsible web scraping.
• ☐ Decide when an API is preferable to HTML scraping.