Web scraping in Python is easier to plan when you separate the work into stages: request a page, parse its HTML, follow links across a site, or render a page in a browser. A single package does not need to do all four.
The right choice depends on what the site returns and how much of the workflow you need to manage. For a static page, an HTTP client and an HTML parser are often enough. For a crawl or JavaScript-heavy site, a framework or browser automation tool may be a better fit.
Python scraping tools at a glance
| Tool | What it does | Choose it when |
|---|---|---|
| Requests | Synchronous HTTP client | You need to fetch ordinary HTTP responses |
| HTTPX | Synchronous and asynchronous HTTP client | You need an async client or HTTP/2 support |
| Beautiful Soup | HTML and XML parser | You want a beginner-friendly way to navigate parsed documents |
| lxml | HTML and XML parser | You need XPath or a parser that fits an existing lxml workflow |
| Scrapy | Crawling and extraction framework | You need to collect data from many pages with a managed crawl |
| Playwright for Python | Browser automation | The required content or interaction is available only in a browser |
| Selenium WebDriver | Browser automation | You already use WebDriver or need its browser-automation ecosystem |
Requests and HTTPX fetch responses; Beautiful Soup and lxml parse documents. Scrapy coordinates a crawl. Playwright and Selenium control a browser. They overlap in places, but they solve different problems.
Fetch pages with Requests or HTTPX
Requests is a straightforward choice for fetching a page that returns the HTML you need in its HTTP response. It handles common HTTP tasks such as query parameters, headers, cookies, and sessions. It does not execute JavaScript or parse the returned HTML for you.
Set a timeout and check the response before parsing it:
import requests
from bs4 import BeautifulSoup
url = "https://example.com/"
response = requests.get(url, timeout=20)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for heading in soup.select("h2"):
print(heading.get_text(" ", strip=True))
This combines two small tools: Requests gets the response, and Beautiful Soup turns its HTML into a document you can query. The MDN guide to HTTP methods explains methods such as GET and POST.
HTTPX provides both synchronous and asynchronous APIs, with optional HTTP/2 support. Consider it when your application already uses async/await, or when you need its HTTP features. Async requests can help a network-bound workflow handle multiple requests efficiently, but they do not make every scrape faster: respect the target site's limits and keep concurrency controlled.
Choose one HTTP client for a workflow unless you have a concrete reason to combine them. Neither library renders a page in a browser.
Parse HTML with Beautiful Soup or lxml
Beautiful Soup is a convenient interface for searching and navigating an HTML or XML document. It can work with Python's built-in html.parser or other parsers such as lxml. It does not make network requests or schedule a crawl, so pair it with an HTTP client when the source is a web page.
The HTML structure you receive can differ from the visual page; MDN's overview of document and website structure is a useful primer.
lxml is another HTML and XML parsing option. It supports XPath and can fit projects that already use the lxml ecosystem. Use it directly when its parsing and query interfaces suit your code, or use it as Beautiful Soup's underlying parser. CSS-selector support through lxml.cssselect requires the cssselect package.
For a small one-off extraction, start with whichever parser makes the selectors easiest to understand. If a page changes, test your selectors against representative responses rather than assuming a selector will remain stable.
Crawl multiple pages with Scrapy
Scrapy is a framework for crawling sites and extracting structured items. It includes request scheduling, concurrent downloads, selectors, retry and pipeline components, and feed exports. Those features are useful when a job must follow links or collect records across many pages; they are more machinery than a single-page task usually needs.
Scrapy also does not turn every response into a fully rendered browser page. You can use it for ordinary responses and reserve browser automation for pages that genuinely depend on client-side rendering or interaction. Review the framework's settings and the target site's access rules before increasing crawl concurrency.
Use a browser only when the page requires one
If important content appears only after JavaScript runs or after a user interaction, a normal HTTP request may not contain it. First inspect the response and check whether the site offers an official API or structured data. If browser rendering is actually necessary, Playwright for Python automates Chromium, Firefox, and WebKit browsers. It can interact with page elements and inspect the rendered page, but it has more setup and resource overhead than fetching HTML directly.
Selenium WebDriver is another browser automation option, especially if your project already uses WebDriver or its browser-testing ecosystem. Neither Selenium nor Playwright is a lightweight replacement for Requests: launching and managing browsers costs more than making direct HTTP requests.
If you also work in JavaScript, the existing Puppeteer web-scraping guide covers a browser automation option outside the Python ecosystem. See our article on using JavaScript for frontend and backend development for broader context.
Choose a tool combination
- One static page: Requests plus Beautiful Soup is a simple starting point.
- Async application: HTTPX plus a parser such as Beautiful Soup or lxml keeps the network and parsing roles separate.
- Many pages and structured items: Scrapy can manage a crawl, extraction, and output workflow.
- JavaScript-rendered content or interactions: Try an official API or direct response first; use Playwright or Selenium only when browser execution is needed.
- Existing parser or browser-test project: Prefer a tool that fits the project's current dependencies and maintenance practices.
These tools are not an official ranking. Start with the least complex approach that can reliably obtain the data you are authorized to collect, then add a framework or browser only when a specific requirement calls for it.
Scrape responsibly and make jobs reliable
Before collecting data, check the site's terms and access policies, use an official API when available, and avoid collecting personal or sensitive information without a lawful basis. Follow reasonable request rates, identify your client with an appropriate User-Agent, and use timeouts and bounded retries. Do not bypass authentication, access controls, CAPTCHAs, or rate limits.
Check a site's robots.txt instructions as part of planning. The Robots Exclusion Protocol (RFC 9309) describes how crawlers should interpret those rules; it also makes clear that robots rules are not access authorization. A permitted path in robots.txt is not a substitute for the site's terms or applicable law.
For more on planning a resilient scraper, see our guide to web-scraping challenges and best practices.
Which Python scraping library should you start with?
Start with Requests and Beautiful Soup when you need a simple static-page extraction. Choose HTTPX when its async or HTTP/2 support fits your application, lxml when its parser interfaces suit your project, and Scrapy when you need a managed multi-page crawl. Reach for Playwright or Selenium only when a browser is part of the requirement.
Keeping fetching, parsing, crawling, and browser automation as separate decisions makes a scraper easier to maintain—and helps you avoid running a browser when a regular HTTP request is enough.
Cover photo by Christina Morillo via Pexels.
