Python Web Scraping: A Broke Founder's Guide (2026)

The Night I Scraped 40,000 Listings Nobody Asked For

In 2023, right around the time I was neck-deep in Clickly, I got a second idea. A directory. You know the kind — a beautiful, SEO-optimized site listing every tool in some niche, with filters and ratings and a slick UI. I’d read that directories were a cheat code for organic traffic. Google loves a big, structured, content-rich domain. I was sold.

There was just one problem. A directory needs data. Thousands of rows of it. And I had exactly zero.

So I did what every engineer does at 1 AM with more ambition than sense: I decided to scrape it. Free data, sitting on the public internet, waiting for me to vacuum it up into my shiny Postgres table. What could go wrong?

Everything, obviously. But I learned Python web scraping properly in the process — the real toolkit, the ways you get blocked, and the uncomfortable legal questions most tutorials skip entirely. This is that guide. Not a spec sheet. The version I wish someone had handed me before I burned a weekend.

---

The Only Three Setups You Actually Need

Here’s the thing about scraping tutorials: they list twenty libraries and let you drown. You don’t need twenty. You need to correctly answer one question — is the data in the HTML, or does JavaScript build it after the page loads? — and then pick from three setups.

1. Static HTML → requests + BeautifulSoup. The data is already in the page source. This is 90% of the web, still, in 2026.
2. JavaScript-rendered → Playwright. The page loads, then JS fetches data and paints it in. You need a real browser.
3. Actual scale → Scrapy. Thousands of pages, link-following, retries, exports. A framework, not a script.

That’s it. Everything else is a variation on those three. Let me break down each one the way I actually use it.

---

requests + BeautifulSoup: Start Here, Always

This is the boring, beautiful default. requests fetches the page. BeautifulSoup parses the HTML so you can pluck out the bits you want with CSS selectors. Four lines and you have data:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com/tools", timeout=10).text
soup = BeautifulSoup(html, "html.parser")
names = [el.text.strip() for el in soup.select("h2.tool-name")]

That’s genuinely most of what I did for the directory. Before you write a single line, though, do one thing: open the page, right-click, View Source, and Ctrl-F for a piece of the data you want. If you can see it in the raw source, you’re in static-land — requests is all you need, and you should stop reading about headless browsers immediately.

Two upgrades worth knowing. First, if you’re hammering a lot of pages, swap requests for httpx and go async — same feel, but you can fetch dozens of URLs concurrently instead of one at a time. Second, for parsing speed on huge documents, tell BeautifulSoup to use lxml as its backend (BeautifulSoup(html, "lxml")). It’s noticeably faster than the built-in parser and worth the one extra dependency.

Do not reach for a browser until this setup provably fails. I’ve watched people spin up Selenium to scrape a site that served everything in plain HTML. That’s like renting a moving truck to carry a sandwich.

---

When the Page Is JavaScript: Playwright (not Selenium)

Sometimes you View Source and the data just… isn’t there. You see an empty <div id="app"> and a pile of script tags. That’s a single-page app — React, Vue, whatever — fetching data after load. requests grabs the shell before the data arrives, so you get nothing.

Now you need a real browser that runs the JavaScript. For years that meant Selenium. In 2026, start with Playwright instead.

I say that having used both. Playwright’s killer feature is auto-waiting — it waits for elements to actually exist before interacting, which kills an entire genus of flaky bug where your scraper works on your fast laptop and dies on a slow server. Selenium makes you sprinkle sleep() and explicit waits everywhere like salt on fries, and it’s still flaky. Playwright’s async support also means real throughput without duct tape.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/app")
    page.wait_for_selector("h2.tool-name")
    names = [el.inner_text() for el in page.query_selector_all("h2.tool-name")]
    browser.close()

When would I still touch Selenium? If I inherited an existing Selenium codebase, or needed some ancient WebDriver-specific integration. For anything new — Playwright. Just remember the cost: a headless browser eats 10–50x the RAM and CPU of a plain HTTP request. Use it only for the pages that truly need it, and scrape everything else with requests. On the directory, maybe a fifth of my sources were JS-heavy. I did not run Playwright for the other four-fifths.

---

When It’s Real Scale: Scrapy

Here’s where the Broken Engineer in me went to war. I didn’t have users. I didn’t even have a launched directory. And I sat there building a full Scrapy pipeline — spiders, item pipelines, middlewares, automatic retries, a rotating user-agent layer — for a dataset I could’ve collected with a for loop over a weekend.

Was it overkill? For that project, absolutely. But Scrapy is genuinely the right tool once scraping becomes a crawler rather than a script — when you’re following links across thousands of pages, need built-in retry and throttling, want to export straight to JSON or a database, and care about doing it politely and concurrently. It handles the async plumbing, deduplication, and rate control that you’d otherwise reinvent badly.

My honest rule: if it fits in one file and finishes in one sitting, it’s a requests script. If it’s a persistent, scheduled, multi-thousand-page operation you’ll maintain for months, it’s Scrapy. Don’t build the cathedral to store one sandwich. (Yes, I clearly needed to hear that more than once.)

---

Getting Blocked: The Order That Actually Matters

Scrape anything real and you’ll meet the wall: 403s, CAPTCHAs, empty responses, IP bans. Every guide screams “USE PROXIES.” That’s backwards. Proxies are the last lever, not the first, because they cost money and you’re broke. Pull them in this order:

1. Set a real User-Agent and headers. The single biggest reason you get blocked is that your request screams “I am a Python script” via the default python-requests/2.x user-agent. Send a normal browser User-Agent and an Accept-Language header. Free. Fixes a shocking number of blocks.
2. Slow down. A human doesn’t hit 200 pages a second. Add a delay between requests (a second or two, randomized). You’re not just avoiding bans — you’re being a decent guest on someone else’s server. This alone got me through most of the directory sources.
3. Rotate — then proxy. If you’re still blocked, rotate user-agents, then reach for proxies. This is where the wallet opens: datacenter proxies start around $0.50 per IP, and residential proxies run roughly $3–$15 per GB depending on the provider (Bright Data is the big name). That adds up fast on a large scrape.
4. Or just pay someone to handle it. Services like ScrapingBee take a URL and hand back rendered HTML with proxies and CAPTCHA-solving baked in. Pricey per request, but sometimes cheaper than the days you’d spend fighting an anti-bot system yourself.

One hard line I won’t cross, and neither should you: don’t build tooling to defeat CAPTCHAs or bot-detection to get behind a login. That’s not “clever scraping” anymore. That’s the exact behavior that turns a legal gray area into a lawsuit — which brings me to the part everyone skips.

---

The Part Every Tutorial Skips: Is This Even Legal?

I am not a lawyer, and this isn’t legal advice. But you need the lay of the land before you point a scraper at anything, because the tutorials that show you requests.get() in ten seconds never mention that you can get sued.

Here’s the 2026 shape of it, in the US:

• Public data is broadly fair game. In hiQ v. LinkedIn, the Ninth Circuit held that scraping publicly accessible data doesn’t violate the Computer Fraud and Abuse Act — you can’t “access without authorization” something that requires no authorization. In January 2024, Meta v. Bright Data reinforced it: a federal judge found Meta’s terms only bind users who are logged in, so logged-off scraping of public pages was allowed.
• But it’s not a free pass. hiQ won the CFAA fight and still lost the war — the case later turned on breach-of-contract, and hiQ ended up with a $500,000 judgment and a permanent injunction against scraping LinkedIn. The lesson: logging in and violating Terms of Service is a different, more dangerous game than reading a public page.
• robots.txt is etiquette, not law. Ignoring it won’t land you in jail, but honoring it is best practice and keeps you on the right side of “reasonable.” Read it. Respect the Crawl-delay.
• Personal data is its own minefield. Even public personal data is regulated under GDPR in the EU and CCPA in California. “It was public” is not a defense for hoovering up names and emails. The EU is the strict end of the spectrum; the US is the permissive end.

My practical filter: public, non-personal, aggregate data with the site’s robots.txt respected and a sane request rate? Reasonable. Data behind a login, personal data, or anything where you’re bypassing a block? Stop and think hard, because you’ve left the safe zone.

---

Running It Cheap: Cron, Fly, and a Postgres Table

Once the scraper works, you don’t leave it running on your laptop. But you also don’t spin up a Kubernetes cluster (I’ve made that mistake in another life). Scraping is a perfect fit for cheap, scheduled infra.

I put the scraper in a small script, containerize it, and run it as a scheduled job on Railway or Fly.io — a cron that wakes up, scrapes, writes rows, and goes back to sleep. A cron job that runs for two minutes a day costs basically nothing. The scraped data lands in PostgreSQL, which for me means a free Neon database that scales to zero when idle. Store the raw rows, dedupe on a unique key, and let your app read from Postgres — never scrape live on a user’s request. Total infra bill for a modest scraper: a rounding error.

That’s the whole point of learning these tools. Your competitor pays for an enterprise data feed; you run a $0 cron job that does 80% of the same thing. Knowledge is the discount.

---

The Hard Truth: Scraping Is a Fragile Foundation

Now the part I actually most want you to hear.

Scraping is a phenomenal tool and a terrible moat. The day I felt clever about my 40,000 scraped listings, I also quietly built a business whose entire value depended on someone else’s website not changing its HTML. And websites change their HTML constantly. They redesign. They add anti-bot walls. They send a cease-and-desist. Every one of those is a Tuesday, and every one of them can nuke your product overnight.

If your whole business is “I scrape X and repackage it,” you don’t have a business. You have a countdown timer on someone else’s server.

Scraping is great for seeding — populating a directory’s first version, doing one-time research, monitoring a handful of competitor prices, bootstrapping a dataset you’ll then own and grow. It’s a bad bet as the load-bearing wall of the whole thing. I know because I leaned on it and watched a source site redesign and take half my directory down with it.

Use it to get moving. Don’t marry it. The moat was never the data — it was distribution, the thing I keep relearning the hard way.

---

The directory never launched, of course. Same story as every project I over-engineer into the ground. But I walked away knowing Python scraping cold — when a four-line requests script is enough, when to bring the Playwright hammer, when Scrapy earns its keep, and exactly how to not get blocked or sued along the way. That knowledge has paid for itself a dozen times since, on projects that actually shipped.

So scrape. Start with requests and BeautifulSoup, respect the robots.txt, run it cheap on a cron, dump it in Postgres. Just don’t build your entire future on data you don’t own.

This is the Broken Engineer Guide. I over-engineer everything, fail at business, and share the scars so you can skip a few. Go scrape something — responsibly.