Guide
It got to page twelve, then stopped. Now what?
Not the first page: the run got going, collected a few hundred rows, and then stopped, or closed with no message, or came back with gaps. In forum threads about every scraper this comes up again and again. There are a handful of causes, and a crawl that says which one it hit is most of the fix.
Why a crawl stops partway
The site asked it to slow down
A 429 "too many requests" or 503 answer means the site is rate-limiting you. Retrying straight away makes it worse; waiting, for as long as the site asks, usually works.
A "verify you are human" page appeared
A check page in place of page 13 has no rows on it. A tool that reads it as data stops with "nothing found", or worse, carries on and saves the check page's text as a row.
The pages started repeating
Some sites send page one again, or the same last page, once you go past the end. A crawl that cannot tell keeps going for ever, or adds the same rows twice.
The tab closed, crashed or navigated away
The rows of most extension crawls live in the page. Close the tab at row 4,000 and they are gone.
You pressed Stop, then Start
Many tools start again from page one, find nothing new on the pages they already have, and decide the list is finished.
What Magic Scraper does about each
And what you can do.
It says why it stopped
Every crawl ends with its reason on screen: no further Next link, the last page, Load more gone, the site stopped answering, a human check, or Stop. A run that stopped short never says it finished.
It waits when a site pushes back
On a 429 or 503 it waits and retries, honouring the site's own Retry-After up to four minutes, and says so while it waits. Choose Gentle pacing for sites that keep pushing back.
It hands you a human check
The run stops and the page with the check is opened in front of you. It never tries to get past it. Clear it yourself and press Start: the crawl carries on from the page it never read.
It notices repeats
Two pages in a row that add no new rows end the crawl, with that reason, instead of looping.
It keeps the rows if the tab dies
Rows are kept as the crawl goes. Open the extension on the same site again and it offers to recover the run, on the page it reached.
Start after Stop carries on next update
Stop, then Start, picks up at the page after the last one read, and so does Start after the max-pages limit you set.
Rows collected before a stop are kept either way. Nothing is charged per page or per row, so a run that stopped costs nothing to finish.
Rows missing from the middle
When a sheet has the first pages and the last ones but gaps between, the usual cause is a page that loaded slowly: the tool read it before its new rows had appeared. Magic Scraper waits for new rows after each click or scroll before reading them, and a page that fails to load is retried, never silently skipped.
If rows are still missing, compare the row count with what the site says it has. The missing data guide covers the other causes.
Reasonable questions
Why does my scraper stop after a few pages?
Most often the site started rate-limiting it, showed a human check, began repeating pages past the end, or the tab closed. A scraper that says why it stopped tells you which, and Magic Scraper does on every run.
Can I continue a crawl instead of starting over?
Yes. After a human check or the max-pages limit, pressing Start carries on from the page it never read. In the next update the same goes for Stop and for a site that stopped answering.
I closed the tab during a crawl. Are the rows gone?
No. They are kept as the crawl goes. Open Magic Scraper on the same site and it offers to recover the run.
Will it get past a captcha for me?
No, and no honest tool should. It stops, puts the page in front of you, and carries on once you have answered it yourself.
Is there a limit on pages or rows?
No. Crawling is free, with no page or row limit.