New in v1.22
Some sites are read by name.
Most pages are read by finding the list on them. A few popular ones are worth more care than that: Magic Scraper knows how they are built, and gives you each post, product or paper with its facts in their own columns — the score, the author, the price, the DOI — and keeps going page after page.
What "read by name" means
On these sites the extension does not guess which block is the list or what to call its columns. It recognises the site from its markup — not its address, so every Amazon country site and both Reddit designs work — and fills a fixed set of columns from the site's own fields.
Nothing else changes: open the page, click the icon, export. No account, no API key, and no server of ours in between. If a site changes its markup, the extension falls back to reading the page the ordinary way.
Video: Scrape Reddit, Amazon, PubMed and arXiv to a spreadsheet, free (Magic Scraper 1.22). It plays from YouTube when you press play.
What each one gives
Posts: title, subreddit, author, score, comments, time, flair, post type, link, permalink, promoted. Comments: author, text, score, time, reply depth and parent, so a thread can be rebuilt. A thread opens on its comments, with the post one Another table away. Works on the current design and on old.reddit.com; old Reddit pages through next, the current feed by infinite scroll.
Amazon search results
Title, price, list price, coupon, rating, review count, “bought in past month”, sponsored, Prime, delivery, ASIN, the product's own link without the search tracking, and the largest image. Sponsored results are kept and marked. Pages through Amazon's own pager.
Hacker News
Rank, title, link, site, points, author, age, the exact time posted, comment count and the discussion link — the line under each title included. Pages through More.
PubMed
PMID, title, authors, journal, year, DOI, publication type, the snippet, whether there is free full text, and the paper's page. Pages through Show more results.
arXiv
Listings and search results: arXiv ID, title, every author in one cell, abstract, subjects, comments, submission date, and the abstract and PDF links. Highlighted search words no longer split a title in pieces.
Wikipedia tables, Goodreads lists and most shops need no special reading: the ordinary detection already names their columns. Want another site read by name? Say which.
Using it for research
It is built for the size of sample a person collects while looking: a few subreddits' threads, a few hundred papers, a category of products. For a dataset of millions, use the site's own data access — Reddit's Data API, NCBI's E-utilities for PubMed, arXiv's API — which exist for exactly that.
Usernames and comments are personal data. Ethics boards generally expect usernames to be
replaced and quotations paraphrased, since a quoted comment can be searched back to its
author. Whether you may collect from a site at all is between you and that site: its
terms, its robots.txt, and data-protection law. The pacing setting is gentle
by default, and a “verify you're human” check stops the run and hands you the
tab rather than pressing on.
Reasonable questions
Is it free?
Yes. Reading these sites, crawling their pages and exporting are free with no limit. Drill-down — opening each row's own page, such as every paper's abstract page or every product page — is the paid feature, and free runs it on the first three rows.
Do I need an account or an API key?
No. It reads the page you are looking at, in your own browser, signed in or not as you already are. Nothing is sent to us; there is no server to send it to.
Does it work on amazon.co.uk, amazon.de and the rest?
Yes. The site is recognised by how its results are built, which is the same in every country.
What happens when one of these sites changes its design?
The extension falls back to reading the page the ordinary way, which usually still finds the list. Tell us and the site reading is updated in the next release.