Map Extract

When a scraper is blocked, the response is telling you something specific: a CAPTCHA or "unusual traffic" page means your pattern looks automated, a 429 means you are going too fast, a 403 means the server refuses this request, and a 200 with no data usually means the content never loaded. Identify the symptom first, then fix the cause, which is usually rate, concurrency, retries or session state. If the site has clearly said no, the right fix is to stop and use another route.

This page is the hub for scraping problems on this blog, written for people running their own lead collection rather than engineers building a crawler from scratch.

Symptom to cause to fix

SymptomWhat it usually meansLikely causePolite fix
CAPTCHA, "Our systems have detected unusual traffic"The site suspects automation from your networkToo many requests, too regular a pattern, shared IP or VPNStop the run, solve it yourself if you are a person browsing, resume much slower or later
HTTP 429 Too Many RequestsRate limitingRequests per minute too high for this hostWait at least the Retry-After time, lower concurrency, cap retries
HTTP 403 ForbiddenThe server understood and refusesBot protection, geographic block, a path you are not allowed to fetchCheck robots.txt and terms; do not disguise the request; skip the site or ask for access
HTTP 503 Service UnavailableServer overloaded or in maintenanceYour load, or theirsBack off; honour Retry-After if sent
200 OK but no dataContent not present in what you savedJavaScript rendering, consent banner, layout change, soft block pageCompare with a normal browser; render the page; record as failed, not zero
Login page instead of contentContent requires an accountLogged-out access restrictedUse only what is public, or the platform's official route
TimeoutsNo response in timeSlow site, too many parallel requests, network issueRaise the timeout modestly, reduce concurrency, retry a bounded number of times
Redirect to an unrelated site or a "domain for sale" pageThe domain changed hands or expiredParked or lapsed domainMark the site parked or dead; see the parked domain guide
DNS failure (NXDOMAIN)The domain does not resolveExpired or mistyped domainMark dead; do not retry in a loop
Results in the wrong country or languageThe site localised to your IPVPN, proxy or IP geolocationSearch with explicit location, check your network location

Two common symptoms have their own pages: the Google unusual traffic CAPTCHA (which also covers 429 handling) and parked domains and dead sites.

Read the status code properly

HTTP status codes are defined in RFC 9110. A few matter most when collecting:

  • 403 Forbidden: "the server understood the request but refuses to fulfill it." Retrying the same request will not change that answer.
  • 404 Not Found and 410 Gone: the page is not there. 410 says access is "no longer available" and the condition is likely permanent. Remove the URL from future runs.
  • 429 Too Many Requests: defined in RFC 6585 as the user having sent "too many requests in a given amount of time". The response "MAY include a Retry-After header indicating how long to wait".
  • 503 Service Unavailable: the server is "currently unable to handle the request". Often temporary.
  • 301 and 308 are permanent redirects; 302 and 307 are temporary. A permanent redirect to a different domain is a data point (the business moved or was bought), not just a hop to follow.

Retry-After can be a number of seconds or an HTTP date. Respect whichever is sent, per host, and do not let other workers keep hitting the same host meanwhile.

Rate, concurrency and retries

Most blocks start here. Three numbers decide how hard you hit a site:

  1. Concurrency per host. Five parallel workers each fetching one page per second is five requests per second to that site. For a small business website on shared hosting, that can be more than its real visitors produce in an hour.
  2. Delay between requests. A fixed interval looks mechanical. A modest random variation around a sensible delay is more natural and, more importantly, kinder to the server.
  3. Retries. A page that fails three times with three retries each is nine extra requests. Retries that ignore Retry-After turn a slowdown into a block.

A sensible baseline for lead collection:

  • Limit parallel requests to any one host to a small number.
  • Add a delay between pages on the same site, longer on large platforms.
  • Use exponential backoff for failures and a hard cap on retries per page.
  • Treat 429 and 503 with Retry-After as a cooldown for the whole host.
  • Never re-fetch pages you already have; keep a record of what was collected.
  • Collect only the pages you need. A contact page and a homepage rarely need a full-site crawl.

Robots.txt and site terms

RFC 9309 standardised the Robots Exclusion Protocol in 2022. The points that matter for a lead crawler:

  • The file lives at /robots.txt at the top level of the host.
  • If fetching it returns a 4xx status, the file is "unavailable" and the crawler "MAY access any resources on the server".
  • If it returns a 5xx or the server is unreachable, the crawler "MUST assume complete disallow".
  • Crawlers should follow at least five redirects for robots.txt, and should not use a cached copy for more than 24 hours.
  • When rules conflict, the most specific (longest) match wins; an equivalent allow beats a disallow.
  • The rules "are not a form of access authorization". They are a request, not a lock.

Terms of service are separate from robots.txt and often stricter. Google's Terms of Service, for example, prohibit "using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages", and list "bypassing our systems or protective measures" as abuse. Read the terms of each platform you collect from. To test which paths a robots file allows before you start, paste it into the free robots.txt tester.

Session, profile and account problems

When collection runs in a browser, state carried between pages causes its own failures:

  • Cookie consent banners can hide the content until accepted or rejected. A saved page full of consent text is not a result.
  • Expired sessions turn content pages into login pages. Logged-out access to social platforms is often restricted; that is a platform decision, not a bug to work around.
  • One profile for everything mixes cookies, languages and locations across platforms. Separate browser profiles per platform keep a problem on one from spilling into another.
  • Security challenges such as "confirm it is you" or two-factor prompts must be handled by the account holder. Automating them, or creating extra accounts to spread the load, breaches most platforms' terms and is the fastest route to suspension.

Empty and partial results

A run that "worked" but saved nothing useful is more dangerous than a run that failed loudly, because it looks like an answer. Before you accept zero results:

  1. Open the same URL in a normal browser on the same network.
  2. Compare the visible content with the saved HTML or fields.
  3. If the browser shows content the tool missed, the page probably builds its content with JavaScript, so use a tool that renders the page in a browser before reading it.
  4. If the browser shows a challenge or block page, the site is limiting you; treat it as blocked.
  5. Record the outcome as "failed" or "blocked", never as "no businesses found". A block is not evidence that a business does not exist.

Long runs that stall partway have a different set of causes (memory, sleep mode, network drops). Check those before assuming the site blocked you.

Notes by type of site

Blocks look different depending on what you are collecting from.

  • Google Search and Google Maps. Expect the "unusual traffic" page rather than a status code. It is triggered by volume and pattern from your IP address, and it applies to everyone on a shared network or VPN. Large Maps runs split into many searches are the classic trigger. Pace them, and treat repeated challenges as a stop signal.
  • Social platforms. Logged-out access is deliberately limited, and login walls appear quickly. What is public without an account is what you can collect; anything more needs the platform's official route, and automated actions on accounts carry the risk of restriction or suspension.
  • Small business websites. These rarely have sophisticated bot protection, but they often sit on cheap shared hosting that buckles under parallel requests. A 503 or a timeout here is usually load, not hostility. One or two requests at a time, homepage and contact page first, is enough.
  • Sites behind a security service. A challenge page ("checking your browser") from a content delivery network means the site owner has chosen to filter automated traffic. Respect it: record the site as blocked and use the business's map listing or social profile instead.
  • Directories and review platforms. These usually have explicit terms on automated collection and often rate limits per IP. Read the terms first; many offer business or partner access.

Pre-flight checklist for a new run

  • Robots.txt read for each new host, and disallowed paths excluded.
  • Terms checked for each platform in the run.
  • Parallel requests per host set low; delays set.
  • Retries capped, with backoff, and Retry-After honoured.
  • Only the pages you need are in scope.
  • Progress saved per item, so a pause or stop loses nothing.
  • Blocks, CAPTCHAs and empty pages recorded as outcomes.

Why "buy proxies" is usually the wrong answer

Most of the pages ranking for this problem sell proxies, so their answer is always more IP addresses. For someone building a lead list, that is usually wrong:

  • It ignores the message. The site has told you, in a standard way, that you are too fast or not welcome on that path.
  • It can breach terms that forbid bypassing protective measures.
  • It multiplies the load a small site receives rather than reducing it.
  • It hides the real fault. If the cause is a retry storm or a consent banner, new IPs change nothing.

Slowing down, collecting less and using official routes solve the majority of blocks for lead collection.

When to stop

Stop the run, rather than adjusting and retrying, when:

  • You see a CAPTCHA or "unusual traffic" page more than once after slowing down.
  • A site returns 403 for every page after you have checked robots.txt and your rate.
  • The platform's terms forbid the collection you are doing.
  • Content needs a login and the account's terms do not allow automated access.
  • Your account receives a warning, restriction or security challenge.
  • The platform offers an API, export or data licence that covers what you need.

Stopping is not failure. A partial list from a polite run is worth more than a complete one that got your network or account blocked.

A worked example (fictional)

A fictional agency crawls 400 websites from a Google Maps list of garages, five at a time, with three retries per page and no delay. Twenty minutes in, about a third of the sites on one shared hosting provider start returning 429 with Retry-After: 120, and the retries keep hitting them. By the end, those sites return 403.

What should change:

  1. Group sites by host or hosting provider and limit parallel requests to each group, not just overall.
  2. Treat the first 429 as a 120-second cooldown for that host.
  3. Cut retries to one or two, with backoff.
  4. Crawl only the homepage and contact page on the first pass.
  5. Re-run the 403 sites a day later at a slow rate. If they still refuse, record them as blocked and move on.

How Map Extract helps

Map Extract builds these safeguards into its runs, and gives you the controls to slow down or stop.

  • The run centre lets you start, pause, resume and stop prepared runs, and shows which sources or sites need attention.
  • The website data extractor reads robots.txt and skips disallowed pages (pages that look like privacy or terms pages are still fetched), uses a 12 second request timeout by default and a capped number of parallel pages, applies Retry-After cooldowns, retries a bounded number of times and tracks consecutive blocks. A site that keeps blocking is quarantined, and only you can choose to retry it.
  • Blocked, CAPTCHA, parked and dead outcomes are recorded as outcomes, not as data, so a block is never saved as "no results".
  • When a CAPTCHA is detected, work pauses so you can handle it manually; it is not counted as a result.
  • Browser profile management keeps a separate saved profile per platform and project, and security challenges stay manual.

What it does not do: it does not guarantee uninterrupted collection, and its safeguards cannot prevent a platform from restricting your network or account. Third-party platforms may restrict data collection or automated actions. Before running a workflow, check the platform's rules and make sure you have permission for the actions you configure.

Public business and social results the app collects are uploaded automatically to Map Extract's servers while it is licensed and running; the privacy notice explains this.

Key takeaways

  • Diagnose by symptom: CAPTCHA, 429, 403, empty 200 and timeouts have different causes.
  • Most blocks come from concurrency per host, missing delays and unbounded retries.
  • Honour Retry-After, read robots.txt per RFC 9309, and read the terms.
  • Record blocks as blocks, never as zero results.
  • When a site keeps saying no after you slow down, stop and look for an official route.

Frequently asked questions

Why does my scraper work for a while and then get blocked?

Most sites measure request patterns over time. A run that starts politely often speeds up as more workers start, retries pile on top of failures, or the same pages are fetched again. Once the rate or pattern crosses the site's threshold, it answers with 429, 403 or a CAPTCHA. Lower concurrency per site, add delays, cap retries and stop re-fetching pages you already have.

Should I use proxies when my scraper is blocked?

Rotating proxies to get around a block treats the symptom and ignores the message. A 403, 429 or CAPTCHA means the site has asked you to slow down or stop, and many platforms' terms forbid bypassing protective measures. First reduce your rate, honour Retry-After, check robots.txt and the terms, and ask whether an official API or data export exists.

What does HTTP 429 Too Many Requests mean for scraping?

RFC 6585 defines 429 as the user having sent too many requests in a given amount of time, which is rate limiting. The response may include a Retry-After header saying how long to wait. Pause requests to that host for at least that long, reduce how many requests you send at once, and cap retries so failed pages do not multiply the load.

Does robots.txt make scraping legal or illegal?

Neither on its own. RFC 9309 says robots.txt rules are not a form of access authorisation; they are instructions crawlers are asked to follow. Ignoring them is impolite and can breach a site's terms, and some terms forbid automated access regardless of robots.txt. Data protection law applies separately to any personal data you collect. This is information, not legal advice.

Why does my scraper return empty results with no error?

A 200 response with no data usually means the content is built by JavaScript after the page loads, a cookie consent banner is covering it, the site's layout has changed, or the site is serving a soft block page. Open the same URL in a normal browser, compare what you see with what your tool saved, and record the run as failed rather than as zero results.

Sources

  1. RFC 9309: Robots Exclusion Protocol
  2. RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
  3. RFC 9110: HTTP Semantics
  4. Unusual traffic from your computer network, Google Search Help
  5. Google Terms of Service

Written by the Map Extract editorial team. Found an error? Email [email protected] and we will check and correct it. How we research guides: editorial policy.

See it working on your own searches.

Start with one keyword, one location and a small limit. Review the records, then decide on a licence.

Top