Guide
Asked for 261 rows, got 16,000?
One user on a scraper forum asked for 261 links and got more than 16,000 rows, the same links over and over. Others find the export has rows the preview did not, or is missing ones it did. Duplicates are rarely the site's doing: they come from how the tool walks the pages.
Where duplicates come from
The last page is read again and again
Past the end of a list, many sites send the last page, or page one, again. A tool that only stops when Next disappears reads it until something gives.
Infinite scroll is read whole each time
A scrolling list keeps every row on the page, so each pass reads the earlier rows again. Unless the tool remembers what it has, they are added every time.
A selector matches more than one thing
Each record holds several elements that match, so each record turns into several rows.
One row opens many pages
Following every link on a detail page multiplies rows on purpose. A tool that repeats the parent row when it runs out of pages makes copies that look like data.
How Magic Scraper keeps each row once
Without setting anything up.
Rows already collected are not added again
Each new page's rows are checked against the ones already held, column by column, so a page read twice, or the top of a scrolling list read again, adds nothing.
Repeating pages end the crawl
Two pages in a row with nothing new, or a Next link that leads back to a page already read, end it, and it says so.
A record is one row
Rows are found as repeated blocks on the page, the way you see them, and each block is read once, whatever it holds.
Drill-down out of pages says so once next update
When a row’s page budget runs out, one row says it stopped, instead of a copy of the parent for every link left over.
Drop a row you do not want with the × beside it. Rows that are identical in every column are treated as one, so dropping it drops its copies too.
When the export does not match the preview
The preview draws the first 100 rows of a large run, and says so: “Showing the first 100 of 2,000 rows. Exports include all of them.” Every export, CSV, Excel, JSON or copy, writes the same table you see, with your dropped columns, renamed columns and filter applied, and the status line says how many rows went into the file.
If the page behind the popup has been closed or reloaded, it no longer holds the rows. The next update refuses to export the 100-row preview as if it were the run next update, and the rows of a crawl that stopped can be brought back with Recover run.
Reasonable questions
Why does my scraper export duplicate rows?
Usually because it read the last page, or the top of an infinite-scroll list, again and again without remembering what it already had. Magic Scraper checks each page's rows against the ones it holds and adds only new ones.
Why is my export different from the preview?
Large runs show only the first 100 rows in the preview; the export has all of them. Magic Scraper says this under the table, and the export applies exactly the columns, renames and filter you see.
Can I remove duplicates in Excel instead?
Yes: Data, Remove Duplicates. But it is better not to collect them, because a crawl reading the same page again is also a crawl that has stopped making progress.
Does it remove rows that only look alike?
No. Only rows identical in every column are treated as the same. Two products with the same name and different prices stay two rows.