Guide
Blocked? The site is telling you something.
"Too many requests". "Access denied". "Just a moment...". A captcha in place of page eight. Being blocked is one of the most common scraping complaints, and most of the time it is the site saying the requests came too fast, or from somewhere it does not trust.
Why sites block scrapers
Too fast
Dozens of pages a minute, at even intervals, look like a machine because they are one. Rate limits answer with 429 "too many requests".
From a server, not a person
Cloud scrapers run from data-centre addresses that many sites refuse outright, or meet with a check page on every visit.
Without your session
A server-side tool is not signed in and has none of your cookies, so the site treats it as a stranger.
On a site that forbids it
Some sites do not allow automated collection at all, whatever the speed.
How Magic Scraper avoids it
By not looking like a robot hammering the site.
It runs in your own browser
Pages are read in your browser, from your connection, with your session, the way you would read them by hand. There is no server of ours in between and no data-centre address.
It paces itself
A random pause between pages, longer for detail pages. Gentle is the slowest and the least likely to be blocked, Normal suits most sites, and Fast is for sites you own.
It backs off
A 429 or 503 answer makes it wait, as long as the site's Retry-After asks, up to four minutes, and try again, saying so on screen.
It stops at a human check
A "verify you are human" page stops the run and is put in front of you. It never tries to get past one.
It uses no proxies, rotates nothing and solves no captchas. If a site keeps refusing, the honest answer is that it does not want to be collected this way.
If you are still blocked
Choose Gentle pacing, crawl fewer pages at a time, and avoid running the same list over and over. Check the site's terms: many offer an export, an API or a data feed that is quicker and allowed.
Some sites, LinkedIn among them, do not allow automated collection, and no tool you can trust should promise to get round that. The guide to sites that don't work has more.
Reasonable questions
Why does my scraper get a 429 too many requests error?
The site limits how fast it answers one visitor, and the scraper went over. Slowing down, and waiting as long as the site's Retry-After asks before trying again, fixes most of them. Magic Scraper does both.
Will Magic Scraper solve captchas?
No. It stops at a human check and shows you the page; you can answer it yourself and carry on, or leave the site alone.
Does it use proxies or rotate IP addresses?
No. It runs in your own browser from your own connection, which is also why sites block it far less than server-based scrapers.
Which pacing should I choose?
Normal for most sites, Gentle for any site that has pushed back, and Fast only for sites you own or that allow it.
Is scraping a site that blocks me legal?
It depends on the site's terms and your country's law. If a site keeps blocking automated access, treat that as an answer. The guide to scraping business directories goes into more detail.