Scraping Google SERP results in 2026: what works and what doesn't

Scraping Google search results pages (SERPs) is one of the most-attempted and most-frustrating scraping tasks in 2026. Google has invested heavily in making it expensive: aggressive rate limiting, a rotating fingerprint surface, captcha walls, and SERPs that increasingly use JavaScript to render the actually-interesting parts.

Below is what actually works in 2026, what doesn't, and the honest tradeoffs. Spoiler: direct Google scraping is rarely the right answer; the alternatives are usually better.

What people want from SERP data#

Five common use cases:

SEO rank tracking: where does my site rank for queries X, Y, Z?
Competitor research: what content ranks for queries my customers search?
Featured snippet analysis: which queries trigger featured snippets, and who owns them?
Keyword research: what does Google autocomplete suggest for these seed terms?
News and trend monitoring: what's surfacing for query X today vs last week?

Each has a different "best path" in 2026. Don't generalize from "I need to scrape Google". The right tool depends on which use case.

The direct-scraping option (and why it usually loses)#

Plain HTTP fetch of https://www.google.com/search?q=... returns a page that's been hostile to scrapers since at least 2010. In 2026 it's worse:

Multiple request -> immediate captcha
Rotating IP -> still captcha after ~10 requests
Residential IP -> ~50 requests before captcha
Headless browser without stealth -> blocked at the TLS handshake
Stealth headless + residential proxy -> works, costs ~$0.005-0.01 per query, fragile

You can do it. People do do it. The blended cost ends up around $0.01-0.03 per successful SERP fetch when you account for bypass infrastructure, captcha solving, and failed attempts. At any meaningful volume, it's worse than the dedicated SERP APIs that handle the bypass plumbing for you.

The dedicated SERP API option (the usual right answer)#

Several services exist whose entire product is "give me Google results, structured":

SerpAPI, Serper, DataForSEO, ScraperAPI, Bright Data SERP API, and a long tail of others
Pricing is around $0.001-$0.005 per result page depending on volume tier
Latency is 1-5 seconds per query
They handle the bypass, captcha solving, IP rotation, and HTML parsing
They return structured JSON with positions, titles, snippets, sitelinks, featured snippets, knowledge panels, related searches

For most of the SERP use cases above, this is the right answer. Per-query cost is lower than rolling your own bypass; structure is more reliable than parsing the HTML yourself; reliability is higher.

The catch: you're locked into Google's results as Google sees them, with whatever localization and personalization the API provider configures. Some of the better APIs let you specify location, language, device (mobile vs desktop), and search type (web, news, images, shopping). Some don't.

The Google-as-a-tool option (often forgotten)#

If your use case is "find pages that match a topic," you don't actually need SERPs. You need URLs that are about the topic. Google has alternatives that are friendlier to programmatic use:

Programmable Search Engine (PSE)#

Google's officially-supported way to query its index programmatically. 100 free queries/day, $5 per 1,000 above that. Returns up to 10 results per query with title, snippet, link.

Tradeoffs: not the full Google index (you configure which sites to search), can't access featured snippets / knowledge panels / etc., expensive at scale.

For "find pages on these specific sites about query X," PSE is better than scraping.

Sitemaps + your own indexing#

If you're tracking content from a specific set of sites (competitors, news outlets, blogs), don't search Google for it. Pull each site's sitemap, ingest the new URLs, extract them yourself with your own tooling.

This is what Runo's /crawl endpoint is built for. Seed at the sitemap, follow the listing pattern, extract structured data from each page. You get richer data than Google would surface and you don't depend on Google's ranking decisions.

Use case 1: SEO rank tracking#

What people want: "for query X, what position does my domain appear at, in geography Y?"

Best path: dedicated SERP API. SerpAPI / Serper / DataForSEO. Per-query cost around $0.002. Track 1,000 keywords daily for ~$60/month.

Roll your own only if you have an existing scraping team and the volume justifies it (>1M queries/month). The build is significantly more complex than people think because Google's ranking surface depends on so many signals (location, browser, device, prior search history, time of day) that consistency requires careful infrastructure.

Use case 2: Competitor content research#

What people want: "for queries my customers search, what content is ranking?"

Best path: SERP API to get the URLs, then a scraping API like Runo to extract structured data from each ranking page. The SERP gives you positions; Runo gives you the content, structure, schema, headings, word count, structured data markup.

Schema for a competitor content audit:

[
  { "field": "title",         "type": "string",        "example": "How to choose a CRM in 2026" },
  { "field": "h1",            "type": "string",        "example": "Choosing a CRM..." },
  { "field": "wordCount",     "type": "integer",       "example": 2400 },
  { "field": "publishedDate", "type": "date",          "example": "2026-03-15" },
  { "field": "author",        "type": "string",        "example": "Sarah Chen" },
  { "field": "headings",      "type": "array<string>", "example": ["Introduction", "Top picks"] },
  { "field": "hasVideo",      "type": "boolean",       "example": false },
  { "field": "hasFAQ",        "type": "boolean",       "example": true }
]

Run this against the top 10 results for each tracked query. You get a structural picture of what Google rewards in your space (avg word count, common headings, FAQ presence, etc.). Cheaper than enterprise tools that bundle this into a $500/month seat.

Use case 3: Featured snippet analysis#

What people want: "which queries in my space trigger featured snippets, and who's winning them?"

Best path: SERP API. They return the snippet content explicitly. SerpAPI exposes answer_box; DataForSEO exposes featured_snippet.

Once you have the data, the analysis is:

For each tracked query, is there a featured snippet?
If yes, what's the source domain and the answer text?
Which of your competitors win snippets in your space?
What does the snippet structure look like (paragraph, list, table)?

A snippet-winning content strategy emerges from this: write content shaped like the snippet structure for queries where snippets exist. The data drives the strategy.

Use case 4: Keyword research / autocomplete#

What people want: "for seed term X, what variations does Google suggest?"

Best path: Google Suggest API directly. It's effectively undocumented but stable and returns autocomplete suggestions in JSON. No bypass needed.

import httpx

resp = httpx.get(
    "https://suggestqueries.google.com/complete/search",
    params={"client": "firefox", "q": "best crm for"}
)
print(resp.json())
# ['best crm for', ['best crm for small business', 'best crm for startups', ...]]

Iterate this with seed expansions ("best CRM for a", "best CRM for b", etc.) to build a keyword corpus. Free and reliable. There's a long tail of keyword research tools that do nothing more than wrap this with a UI.

For search-volume data, you need a paid source (Google Keyword Planner via API, Ahrefs, Semrush). Volume isn't in the autocomplete response.

Use case 5: News and trend monitoring#

What people want: "what's surfacing for query X today vs last week?"

Best path depends on what "surfacing" means.

For Google News specifically: Google News RSS feeds (https://news.google.com/rss/search?q=...) return structured news results without bypass. Free, reliable, but rate-limited and missing some signals.
For "trending topics": Google Trends has an unofficial API (pytrends library wraps it) that returns trend data. Brittle but workable.
For monitoring across many news sites: a scraping API like Runo against the news sites' own pages. You get full article text and structured fields, not just headlines.

If you want article-level structured data from news sites, scraping the source sites is better than scraping Google News. The fields are richer (full body, byline, photos, related articles) and you avoid the Google bypass plumbing entirely.

Avoiding the dead-end paths#

A few approaches that come up and rarely work:

"I'll just use Selenium with a residential proxy and rotate user agents"#

This was a working strategy in 2018. In 2026 it gets you blocked within 50 queries. Google's anti-bot is far more sophisticated than user-agent rotation.

"I'll use the Google Custom Search API"#

Google deprecated parts of this surface and the parts that remain (Programmable Search Engine) are limited to configured sites, not the full index. PSE is fine for what it is; it's not "Google search results."

"I'll scrape Google scholar / books / images / shopping separately"#

Each surface has its own anti-bot measures. The verticals are usually better served by dedicated APIs (Crossref for academic, e-commerce platforms directly for shopping).

"I'll use the gemini API to do search"#

Gemini has tool-calling for Google Search, but the tool returns model-summarized results, not raw SERPs. Useful for AI features in your product; not useful for SEO rank tracking or competitor analysis.

When direct scraping makes sense#

Two narrow cases:

Very low volume, ~100 queries/day for an internal one-off project. The setup cost is low and you don't need long-term reliability.
You're the bypass-as-a-service company, in which case scraping Google is your product and you have a team for it.

For everyone else, the dedicated SERP API path is faster, cheaper, more reliable, and lets you spend your engineering time on the parts of your product that differentiate.

Combining SERP with extraction#

The most common end-to-end pattern: SERP API for URL discovery + scraping API for content extraction. SerpAPI gives you the top 10 URLs for a query; Runo gives you the structured content from each URL.

import httpx

# 1. Get SERP URLs
serp = httpx.get(
    "https://serpapi.com/search",
    params={"q": "best crm software", "api_key": SERP_KEY},
).json()
urls = [r["link"] for r in serp["organic_results"][:10]]

# 2. Extract structured data from each
results = httpx.post(
    "https://api.scrapewithruno.com/v1/batch",
    json={
        "urls": urls,
        "schema": [
            {"field": "title", "type": "string", "example": "Best CRMs"},
            {"field": "h1", "type": "string", "example": "Best CRMs"},
            {"field": "wordCount", "type": "integer", "example": 2400},
        ],
    },
    headers={"X-API-Key": RUNO_KEY},
).json()

Per-query cost: ~$0.002 SERP + 10 × $0.001 extraction = ~$0.012 per fully-analyzed query. For 1,000 queries/day that's $360/month for a content-research surface that would cost $2K-$5K/month from an enterprise SEO tool.

TL;DR#

Direct scraping of Google SERPs in 2026 is technically possible but expensive and fragile. Bypass + captcha + IP rotation pushes per-query cost to $0.01-$0.03 with high failure rates.
Dedicated SERP APIs (SerpAPI, Serper, DataForSEO) charge $0.001-$0.005 per query, return structured JSON, and are the right answer for most use cases.
Google Suggest API is free and reliable for autocomplete-based keyword research; no bypass needed.
Google News RSS feeds work for news monitoring without bypass.
For SEO rank tracking, competitor content research, featured snippet analysis: SERP API + scraping API combination beats enterprise tools on cost.
For "find content on specific sites about topic X," skip Google entirely. Use sitemaps and crawl the sources directly with Runo's /crawl endpoint.
Don't try to roll your own Google bypass unless your business is bypass-as-a-service. The economics don't work.