5.4 Browser Automation with Selenium

5.4 Browser Automation with Selenium

Browser automation is the process of controlling a web browser programmatically. Instead of manually opening pages, clicking buttons, filling forms, and navigating between screens, Python can perform these actions automatically.

Selenium is widely used for automating browsers and testing web applications.

1. Why Use Selenium?

Selenium is particularly useful when a web page depends on JavaScript and browser interaction rather than simply returning all required information in the initial HTML.

• Automate repetitive browser tasks.
• Test web applications.
• Navigate multi-page workflows.
• Fill authorized forms.
• Click buttons and links.
• Interact with dropdowns and checkboxes.
• Handle dynamically generated page content.
• Capture screenshots.
• Read browser cookies for authorized sessions.
• Validate web application behavior.

2. Selenium vs requests + BeautifulSoup

3. Installing Selenium

pip install selenium

Verify the installation:

python -c "import selenium; print(selenium.__version__)"

4. Your First Selenium Program

from selenium import webdriver

driver = webdriver.Chrome()

driver.get(
    "https://example.com"
)

print(
    driver.title
)

driver.quit()

Selenium launches Chrome, opens the requested page, reads the title, and then closes the browser.

5. Understanding WebDriver

WebDriver provides the programming interface through which Selenium controls a browser.

Modern Selenium versions can generally manage the required driver components automatically when supported by the environment.

6. Running Chrome in Headless Mode

Headless mode runs the browser without displaying its graphical window.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()

options.add_argument(
    "--headless=new"
)

driver = webdriver.Chrome(
    options=options
)

driver.get(
    "https://example.com"
)

print(
    driver.title
)

driver.quit()

Headless execution is particularly useful for servers and automated pipelines.

7. Browser Window Management

driver.maximize_window()

Set a specific browser size:

driver.set_window_size(
    1280,
    800
)

8. Browser Navigation

driver.get(
    "https://example.com"
)

Back

driver.back()

Forward

driver.forward()

Refresh

driver.refresh()

9. Reading Page Information

print(
    driver.title
)

print(
    driver.current_url
)

Selenium also provides access to the current page source:

html = driver.page_source

print(
    html[:500]
)

10. Locating Web Elements

Selenium must first locate an element before interacting with it.

Common locator strategies include:

11. Finding an Element

from selenium.webdriver.common.by import By

element = driver.find_element(
    By.ID,
    "username"
)

Selenium returns a WebElement representing the matching element.

12. Finding Elements with CSS Selectors

element = driver.find_element(
    By.CSS_SELECTOR,
    ".product"
)

Multiple elements can be located with:

elements = driver.find_elements(
    By.CSS_SELECTOR,
    ".product"
)

13. Using XPath

element = driver.find_element(
    By.XPATH,
    "//button"
)

XPath can express relationships and conditions that are sometimes difficult to represent with a simple CSS selector.

Example

element = driver.find_element(
    By.XPATH,
    "//input[@name='username']"
)

14. Reading Element Text

heading = driver.find_element(
    By.TAG_NAME,
    "h1"
)

print(
    heading.text
)

The text property provides the element's visible text as exposed through Selenium.

15. Reading Element Attributes

link = driver.find_element(
    By.CSS_SELECTOR,
    "a"
)

href = link.get_attribute(
    "href"
)

print(href)

Other examples:

print(
    link.get_attribute("class")
)

print(
    link.get_attribute("id")
)

16. Simulating a Click

button = driver.find_element(
    By.ID,
    "submit"
)

button.click()

Selenium sends a browser interaction to the located element.

17. Filling a Form

from selenium.webdriver.common.by import By

username = driver.find_element(
    By.ID,
    "username"
)

password = driver.find_element(
    By.ID,
    "password"
)

username.send_keys(
    "student@example.com"
)

password.send_keys(
    "example-password"
)

18. Clearing an Input Field

field.clear()

field.send_keys(
    "New value"
)

19. Submitting a Form

A form may be submitted by clicking its button:

submit = driver.find_element(
    By.CSS_SELECTOR,
    "button[type='submit']"
)

submit.click()

In many modern applications, clicking the visible submit control is preferable to directly manipulating the DOM because it follows the application's normal interaction flow.

20. Simulating Keyboard Input

from selenium.webdriver.common.keys import Keys

search = driver.find_element(
    By.NAME,
    "q"
)

search.send_keys(
    "Python"
)

search.send_keys(
    Keys.ENTER
)

21. Working with Select Dropdowns

Selenium provides the Select helper for standard HTML <select> elements.

from selenium.webdriver.support.ui import Select

dropdown = Select(
    driver.find_element(
        By.ID,
        "country"
    )
)

dropdown.select_by_visible_text(
    "Canada"
)

Select by value:

dropdown.select_by_value(
    "ca"
)

Select by index:

dropdown.select_by_index(
    2
)

22. Working with Checkboxes

checkbox = driver.find_element(
    By.ID,
    "terms"
)

if not checkbox.is_selected():

    checkbox.click()

Check the current state using is_selected().

23. Working with Radio Buttons

radio = driver.find_element(
    By.ID,
    "option1"
)

if not radio.is_selected():

    radio.click()

24. Checking Element State

print(
    element.is_displayed()
)

print(
    element.is_enabled()
)

print(
    element.is_selected()
)

25. Implicit Wait

An implicit wait tells Selenium to wait for a specified amount of time when searching for elements.

driver.implicitly_wait(
    10
)

This can be useful for simple scripts, but explicit waits provide more precise control for complex applications.

26. Explicit Waits

Modern web applications frequently load elements asynchronously. Instead of using arbitrary delays, wait for a meaningful condition.

from selenium.webdriver.support.ui import WebDriverWait

wait = WebDriverWait(
    driver,
    10
)

27. Expected Conditions

from selenium.webdriver.support import expected_conditions as EC

button = wait.until(
    EC.element_to_be_clickable(
        (
            By.ID,
            "submit"
        )
    )
)

button.click()

Selenium waits until the specified condition is satisfied or the timeout is reached.

28. Waiting for an Element to Exist

element = wait.until(
    EC.presence_of_element_located(
        (
            By.ID,
            "result"
        )
    )
)

Presence means the element exists in the DOM. It does not necessarily mean that it is visible or ready for interaction.

29. Waiting for Visibility

element = wait.until(
    EC.visibility_of_element_located(
        (
            By.ID,
            "result"
        )
    )
)

30. Waiting Until a Button Is Clickable

button = wait.until(
    EC.element_to_be_clickable(
        (
            By.CSS_SELECTOR,
            "button.submit"
        )
    )
)

button.click()

31. Waiting for a URL Change

wait.until(
    EC.url_contains(
        "/dashboard"
    )
)

This is useful after an interaction that navigates to another page.

32. Waiting for a Page Title

wait.until(
    EC.title_contains(
        "Dashboard"
    )
)

33. WebDriverWait vs time.sleep()

34. Understanding Browser Cookies

Cookies are small pieces of data stored by the browser for a website. They may be used for preferences, session management, analytics, and other purposes.

Selenium can inspect and manage cookies within an authorized browser session.

35. Reading Cookies

cookies = driver.get_cookies()

for cookie in cookies:

    print(
        cookie["name"],
        cookie["value"]
    )

36. Adding a Cookie in an Authorized Session

Selenium can add cookies for the current domain when the operation is authorized.

driver.get(
    "https://example.com"
)

driver.add_cookie({
    "name": "theme",
    "value": "dark"
})

driver.refresh()

37. Deleting Cookies

Delete one cookie:

driver.delete_cookie(
    "theme"
)

Delete all cookies:

driver.delete_all_cookies()

38. Automating Authorized Login Workflows

Selenium can automate login forms for applications where you have permission to access the account and automation is allowed.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get(
    "https://example.com/login"
)

wait = WebDriverWait(
    driver,
    10
)

username = wait.until(
    EC.visibility_of_element_located(
        (By.ID, "username")
    )
)

password = wait.until(
    EC.visibility_of_element_located(
        (By.ID, "password")
    )
)

username.send_keys(
    "authorized-user"
)

password.send_keys(
    "PASSWORD_FROM_SECURE_STORAGE"
)

submit = wait.until(
    EC.element_to_be_clickable(
        (By.CSS_SELECTOR, "button[type='submit']")
    )
)

submit.click()

39. Never Hard-Code Credentials

Avoid:

password = "MyRealPassword123"

For local development, environment variables can be used:

import os

username = os.environ.get(
    "APP_USERNAME"
)

password = os.environ.get(
    "APP_PASSWORD"
)

Production systems should use an appropriate secret-management solution rather than storing credentials directly in source code.

40. Login, MFA and Security Controls

Some applications require additional security steps such as multi-factor authentication, CAPTCHA, device verification, or security keys.

Selenium should not be used to defeat or circumvent these controls.

For legitimate automation, use an application-supported method, such as:

• Authorized test accounts.
• Test environments.
• Official APIs.
• Service accounts where supported.
• Application-provided automation mechanisms.
• Human-assisted authentication when required by policy.

41. Handling JavaScript Alerts

Websites may display JavaScript alert dialogs.

alert = driver.switch_to.alert

print(
    alert.text
)

alert.accept()

To dismiss the alert:

alert.dismiss()

42. Working with Iframes

An iframe contains another document inside the current page. Selenium must switch into the frame before interacting with its contents.

frame = driver.find_element(
    By.CSS_SELECTOR,
    "iframe"
)

driver.switch_to.frame(
    frame
)

Return to the main document:

driver.switch_to.default_content()

43. Handling Multiple Browser Windows or Tabs

original_window = (
    driver.current_window_handle
)

handles = (
    driver.window_handles
)

for handle in handles:

    if handle != original_window:

        driver.switch_to.window(
            handle
        )

Selenium identifies browser windows and tabs through window handles.

44. Taking Screenshots

driver.save_screenshot(
    "page.png"
)

Screenshots are useful for debugging failed automation steps.

45. Taking an Element Screenshot

element = driver.find_element(
    By.CSS_SELECTOR,
    ".result"
)

element.screenshot(
    "result.png"
)

46. Executing JavaScript

Selenium can execute JavaScript in the current browser context.

title = driver.execute_script(
    "return document.title;"
)

print(title)

47. Scrolling a Page

driver.execute_script(
    "window.scrollTo(0, document.body.scrollHeight);"
)

A more interaction-oriented approach is to scroll an element into view:

driver.execute_script(
    "arguments[0].scrollIntoView();",
    element
)

48. Advanced Mouse and Keyboard Actions

Selenium's ActionChains API supports more complex interactions.

from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(
    driver
)

actions.move_to_element(
    element
).click().perform()

49. Mouse Hover

from selenium.webdriver.common.action_chains import ActionChains

menu = driver.find_element(
    By.ID,
    "menu"
)

ActionChains(
    driver
).move_to_element(
    menu
).perform()

Hover interactions are common in menus and interactive interfaces.

50. Selenium + BeautifulSoup

Selenium can render a page and execute JavaScript, after which the resulting page source can be parsed with BeautifulSoup.

from bs4 import BeautifulSoup

html = driver.page_source

soup = BeautifulSoup(
    html,
    "html.parser"
)

headings = soup.find_all(
    "h2"
)

for heading in headings:

    print(
        heading.get_text(
            strip=True
        )
    )

51. Common Selenium Exceptions

52. Selenium Error Handling

from selenium.common.exceptions import (
    TimeoutException,
    NoSuchElementException
)

try:

    button = wait.until(
        EC.element_to_be_clickable(
            (By.ID, "submit")
        )
    )

    button.click()

except TimeoutException:

    print(
        "Button was not ready."
    )

except NoSuchElementException:

    print(
        "Button was not found."
    )

53. Always Close the Browser

Use quit() when the entire automation session is finished.

driver.quit()

A robust pattern is:

driver = webdriver.Chrome()

try:

    driver.get(
        "https://example.com"
    )

    # Automation steps

finally:

    driver.quit()

54. Page Object Model (POM)

The Page Object Model is a design pattern used to organize Selenium automation code.

Instead of placing selectors and browser operations throughout test scripts, page-specific behavior can be encapsulated in classes.

from selenium.webdriver.common.by import By


class LoginPage:

    USERNAME = (
        By.ID,
        "username"
    )

    PASSWORD = (
        By.ID,
        "password"
    )

    SUBMIT = (
        By.CSS_SELECTOR,
        "button[type='submit']"
    )

    def __init__(self, driver):

        self.driver = driver

    def login(
        self,
        username,
        password
    ):

        self.driver.find_element(
            *self.USERNAME
        ).send_keys(username)

        self.driver.find_element(
            *self.PASSWORD
        ).send_keys(password)

        self.driver.find_element(
            *self.SUBMIT
        ).click()

POM reduces duplication and makes large Selenium projects easier to maintain.

55. Data-Driven Browser Automation

Selenium can be combined with pandas to execute an authorized workflow for multiple records.

import pandas as pd

df = pd.read_csv(
    "test_data.csv"
)

for _, row in df.iterrows():

    print(
        row["username"]
    )

In testing, this pattern is useful for executing the same workflow against multiple test cases.

56. Selenium for Automated Testing

Selenium is especially valuable in quality assurance because it can reproduce user interactions in a real browser.

Example Test Flow

1. Open the application.
2. Navigate to the login page.
3. Enter test credentials.
4. Submit the form.
5. Wait for the dashboard.
6. Verify the expected heading.
7. Capture a screenshot if the test fails.
8. Close the browser.

57. Verifying Results

heading = wait.until(
    EC.visibility_of_element_located(
        (
            By.TAG_NAME,
            "h1"
        )
    )
)

assert (
    heading.text
    == "Dashboard"
)

Assertions allow automation scripts to verify expected behavior instead of merely performing actions.

58. Mini Project — Automated Search Workflow

Build a Selenium script that:

1. Opens an authorized test website.
2. Finds a search field.
3. Enters a keyword.
4. Submits the search.
5. Waits for the results.
6. Extracts the result headings.
7. Saves a screenshot.
8. Closes the browser.

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.common.keys import Keys

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Chrome()

wait = WebDriverWait(
    driver,
    10
)


try:

    driver.get(
        "https://example.com"
    )

    search = wait.until(
        EC.visibility_of_element_located(
            (
                By.NAME,
                "q"
            )
        )
    )

    search.send_keys(
        "Python"
    )

    search.send_keys(
        Keys.ENTER
    )

    wait.until(
        EC.presence_of_element_located(
            (
                By.CSS_SELECTOR,
                ".results"
            )
        )
    )

    headings = driver.find_elements(
        By.CSS_SELECTOR,
        ".results h2"
    )

    for heading in headings:

        print(
            heading.text
        )

    driver.save_screenshot(
        "search_results.png"
    )

finally:

    driver.quit()

59. Mini Project — Dynamic Content Extraction

Create an authorized test workflow in which a button loads content dynamically.

button = wait.until(
    EC.element_to_be_clickable(
        (
            By.ID,
            "load-data"
        )
    )
)

button.click()

result = wait.until(
    EC.visibility_of_element_located(
        (
            By.ID,
            "result"
        )
    )
)

print(
    result.text
)

The important concept is that the script waits for the result rather than assuming that it appears immediately after the click.

60. Mini Project — Authorized Login Test

For a test environment or an application where automation is explicitly permitted, automate a normal login workflow.

import os

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC


username = os.environ.get(
    "TEST_USERNAME"
)

password = os.environ.get(
    "TEST_PASSWORD"
)


driver = webdriver.Chrome()

wait = WebDriverWait(
    driver,
    10
)


try:

    driver.get(
        "https://example.com/login"
    )

    username_field = wait.until(
        EC.visibility_of_element_located(
            (
                By.ID,
                "username"
            )
        )
    )

    password_field = wait.until(
        EC.visibility_of_element_located(
            (
                By.ID,
                "password"
            )
        )
    )

    username_field.send_keys(
        username
    )

    password_field.send_keys(
        password
    )

    submit = wait.until(
        EC.element_to_be_clickable(
            (
                By.CSS_SELECTOR,
                "button[type='submit']"
            )
        )
    )

    submit.click()

    wait.until(
        EC.url_contains(
            "/dashboard"
        )
    )

    print(
        "Login test completed."
    )

finally:

    driver.quit()

61. Mini Project — Inspecting an Authorized Session

driver.get(
    "https://example.com"
)

cookies = driver.get_cookies()

for cookie in cookies:

    print(
        cookie["name"]
    )

In production systems, do not print cookie values because session tokens may be sensitive.

62. Mini Project — Headless Automation

from selenium import webdriver

from selenium.webdriver.chrome.options import Options


options = Options()

options.add_argument(
    "--headless=new"
)

driver = webdriver.Chrome(
    options=options
)


try:

    driver.get(
        "https://example.com"
    )

    print(
        driver.title
    )

finally:

    driver.quit()

63. Production-Style Selenium Architecture

Configuration
      ↓
Create WebDriver
      ↓
Navigate
      ↓
Locate Elements
      ↓
Wait for Conditions
      ↓
Perform Actions
      ↓
Validate Results
      ↓
Capture Logs / Screenshots
      ↓
Store Results
      ↓
Quit Driver

Separating these stages makes automation easier to maintain and troubleshoot.

64. Common Selenium Mistakes

65. Selenium Interview Questions

element.click()

element.send_keys(
    "text"
)

66. Examination Questions — MCQs

67. Practical Examination Questions

68. Debugging Selenium Scripts

When an automation script fails, inspect the browser state and determine exactly which step failed.

print(
    driver.current_url
)

print(
    driver.title
)

driver.save_screenshot(
    "debug.png"
)

Also inspect:

• Whether the expected page loaded.
• Whether the locator still matches the HTML.
• Whether the element is inside an iframe.
• Whether the element is dynamically generated.
• Whether an overlay is blocking the element.
• Whether the expected condition is correct.
• Whether the page changed after an interaction.

69. Choosing Robust Locators

Prefer selectors that are stable and meaningful.

70. Responsible Browser Automation

Selenium is a powerful automation tool, but browser automation should be performed only within an authorized context.

• Automate applications you own or are authorized to test.
• Use dedicated test accounts where possible.
• Respect application terms and automation policies.
• Do not bypass authentication or access controls.
• Do not defeat CAPTCHA or MFA.
• Do not harvest session cookies or credentials.
• Avoid excessive automated requests.
• Protect test credentials and session information.
• Prefer official APIs for data access when appropriate.

71. Selenium Best Practices

1. Use explicit waits for dynamic interfaces.
2. Use stable locators.
3. Keep selectors centralized in larger projects.
4. Separate page objects from test logic.
5. Never hard-code production credentials.
6. Capture screenshots when useful for diagnosing failures.
7. Always close the WebDriver session.
8. Keep browser automation focused on genuine user workflows.
9. Use headless mode for appropriate server-side workloads.
10. Prefer lighter HTTP/API approaches when browser execution is unnecessary.
11. Do not automate around security controls.

72. Selenium Quick Reference Cheat Sheet

73. Final Challenge — End-to-End Selenium Automation

Build a Selenium automation project for an authorized test website.

Required Workflow

1. Launch Chrome in headless or normal mode.
2. Navigate to the test application.
3. Locate a form using robust selectors.
4. Enter test data.
5. Select a dropdown option.
6. Select a checkbox.
7. Submit the form.
8. Wait for the resulting content.
9. Verify the expected result.
10. Capture a screenshot.
11. Record the result in a log.
12. Close the browser safely.

74. Self-Assessment Checklist

Before moving to the next topic, make sure you can:

• ☐ Explain browser automation.
• ☐ Explain when Selenium should be used.
• ☐ Install Selenium.
• ☐ Launch a browser with WebDriver.
• ☐ Navigate between pages.
• ☐ Read page titles and URLs.
• ☐ Locate elements using IDs.
• ☐ Locate elements using CSS selectors.
• ☐ Use XPath.
• ☐ Click buttons.
• ☐ Fill input fields.
• ☐ Work with dropdowns.
• ☐ Work with checkboxes and radio buttons.
• ☐ Use keyboard actions.
• ☐ Use explicit waits.
• ☐ Handle dynamic content.
• ☐ Work with authorized browser cookies.
• ☐ Handle iframes.
• ☐ Handle multiple tabs or windows.
• ☐ Take screenshots.
• ☐ Handle Selenium exceptions.
• ☐ Use Page Object Model.
• ☐ Run Selenium in headless mode.
• ☐ Combine Selenium with BeautifulSoup.
• ☐ Protect credentials and session data.
• ☐ Explain why security controls must not be bypassed.