Web scraping is one of the most practical superpowers you can add to your Python toolkit. Whether you're collecting product prices, extracting news headlines, or gathering data for analysis, scraping lets you automate what would otherwise be manual copy-pase drudgery.

In this post, we’ll explore how to scrape a web page using:

• 🐍 Python
• 🌐 Selenium (for dynamic websites)
• 🍲 Beautiful Soup (for parsing HTML)

We’ll explain when and why to use each tool, and build a complete working example.

Why Use Selenium and Beautiful Soup?

Before jumping into code, let’s clarify the roles of each tool.

Beautiful Soup

BeautifulSoup (from the bs4 library) is used to:

• Parse HTML
• Navigate and search the DOM
• Extract data cleanly

However, it cannot execute JavaScript.

Selenium

Selenium automates a real browser like Chrome or Firefox by loading JavaScript-heavy websites, clicking buttons, scrolling pages, filling forms, or waiting for content to load.

Modern websites, like those built with React, Angular, or Vue, often load content dynamically via JavaScript. In those cases, requests and BeautifulSoup won’t work because the HTML returned doesn’t contain the rendered content.

That’s where Selenium comes in.

Installation

First, install the required packages:

pip install selenium beautifulsoup4

You’ll also need a WebDriver. For Chrome you can download ChromeDriver that matches your Chrome version or use webdriver-manager:

pip install webdriver-manager

Example: Scraping Quotes from a Web Page

We’ll scrape articles from a test site:

👉 https://www.tawandam.co.zw/blog

This website loads the articles using JavaScript — perfect for Selenium.

Step 1: Set Up Selenium

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import time

What’s happening here?

• webdriver controls the browser
• Service manages the ChromeDriver
• By helps locate elements
• ChromeDriverManager() automatically downloads and manages ChromeDriver

Step 2: Launch the Browser and Load the Page

# Set up Chrome driver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

# Open the webpage
url = "https://www.tawandam.co.zw/blog"
driver.get(url)

# Wait for JavaScript to load
time.sleep(3)

Why time.sleep()?

The page loads content dynamically. Without waiting, the HTML might not contain the quotes yet.

⚠️ In production, use WebDriverWait instead of sleep() (more on that shortly).

Step 3: Pass the Page Source to Beautiful Soup

Now that Selenium has rendered the page, we extract the HTML:

from bs4 import BeautifulSoup

# Get rendered HTML
html = driver.page_source

# Parse with BeautifulSoup
soup = BeautifulSoup(html, "html.parser")

Now we can use Beautiful Soup to navigate the DOM.

Step 4: Extract the Data

Inspecting the page shows that each quote is inside an anchor element:

<a class="relative rounded-[0.5rem] text-text-color-light dark:text-text-color-dark no-underline transition-[ease] duration-[0.1s] text-[1.2rem] text-center border-[0.1rem] border-solid border-link-color-light dark:border-link-color-dark pt-[1rem] px-[0.5rem] pb-0 shadow-md shadow-[#0000001a] hover:shadow-xl hover:shadow-link-color-light dark:hover:shadow-link-color-dark cursor-pointer min-h-[50vh]">.

articles = soup.find_all("a", class_="relative")

for article in articles:
    text = article.find("h4", class_="capitalize").get_text()
    date = quote.find("div", class_="text-code-color-light dark:text-code-color-dark text-center mb-2 my-[1rem]").get_text()
    print(f"{text} - {date}")

What’s happening?

• find_all() locates all quote containers
• Inside each quote:
  • h4.capitalize → article title
  • div → the publish date of the article
• get_text() extracts clean text

Step 5: Close the Browser

Always close the browser when done:

driver.quit()

Better Practice: Using WebDriverWait

Instead of time.sleep(), use explicit waits:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

wait.until(
    EC.presence_of_element_located((By.CLASS_NAME, "relative"))
)

This is much more reliable because it waits until at least one article appears.

Full Working Script

Here’s the complete script:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup

# Set up driver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

url = "https://www.tawandam.co.zw/blog"
driver.get(url)

# Wait for quotes to load
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CLASS_NAME, "relative")))

# Get rendered HTML
html = driver.page_source
soup = BeautifulSoup(html, "html.parser")

# Extract data
articles = soup.find_all("a", class_="relative")

for article in articles:
    text = article.find("h4", class_="capitalize").get_text()
    date = quote.find("div", class_="text-code-color-light dark:text-code-color-dark text-center mb-2 my-[1rem]").get_text()
    print(f"{text} - {date}")

driver.quit()

When Should You Use This Approach?

Use Selenium and Beautiful Soup when:

• ✅ The site loads content via JavaScript
• ✅ You need to click buttons or scroll
• ✅ The site requires login
• ✅ You need to simulate user behavior

Use requests and Beautiful Soup only when:

• ✅ The HTML contains all data on initial load
• ✅ No JavaScript rendering is required

⚠️Important: Scrape Responsibly:

Before scraping:

• Check the robots.txt of the website.
• Read the website’s terms of service.
• Don’t overload servers.
• Use rate limiting.
• Respect privacy and legal boundaries.
• Ethical scraping matters ✊.

Final Thoughts

Using Selenium and Beautiful Soup gives you the best of both worlds. Selenium renders and interacts while Beautiful Soup parses and extracts. Together, they can handle nearly any scraping task — from simple blogs to complex JavaScript-heavy dashboards.

If you're building data pipelines, research tools, or automation systems, mastering this combo is absolutely worth it.