About this practice page
One table out of five, every page of it
Customer portals, supplier accounts, admin panels and analytics dashboards put several tables on
one screen — orders, invoices, shipments, messages — and give each its own pager. The job is
almost always one of them: every invoice for the accountant, every shipment for a claim. This page is
that screen. The task it sets is specific: all 40 invoices across their four pages, and not one row
from the other tables.
Invoices chosen and crawled: 40 rows from four pages, and none from the orders table whose pager comes first. Why scrapers page the wrong table
Choosing the invoices is the easy half. The trap is the Next button. The orders table above them
has a pager worded and styled exactly like the invoices’ pager — same “Next ›”,
same class — and it comes first on the page. A scraper that looks for “the Next button on the
page” finds the orders’ one, opens orders page two, where the invoices are back at page one, and
stops with ten invoices and a message about repeated rows. The other tables add their own twists:
shipments are paged by a JavaScript button that has no address, returns by a Show more button, and
messages have no pager at all — so a scraper that borrows a neighbour’s Next for them is wrong
too.
In Magic Scraper
It opens on the invoices, the biggest table here; for another, use Wrong list? →
Another table until the right one is outlined, or Pick manually and click it. Then
Start crawling . It looks for the Next that belongs to the table you chose — the one in the
part of the page around that table, not another table’s — so the invoices come back as 40 rows
from four addresses, the shipments as 15 rows from three clicks, the returns as 12 after one
Show more , and the messages as their 5 rows with nothing to page. If a site ever fools it,
Pick “Next” manually on the Crawl tab settles it: click the right table’s Next.
To collect all five tables, each fully paged, do them one after another the same way and export
each one: five tables, five files.
In Python
Scope everything to the table’s own section: find the table there, and the Next there too.
import requests, pandas as pd
from io import StringIO
from urllib.parse import urljoin
from bs4 import BeautifulSoup
url, pages = "https://magicscraper.app/demo/account", []
while url:
soup = BeautifulSoup(requests.get(url).content, "html.parser")
section = soup.select_one("section#invoices") # this table's own part of the page
pages.append(pd.read_html(StringIO(str(section.table)))[0])
nxt = section.select_one(".pager a.next") # its Next, not the orders' one
url = urljoin(url, nxt["href"]) if nxt else None
df = pd.concat(pages, ignore_index=True)
df.shape # (40, 5)Search the whole page instead — soup.select_one(".pager a.next") — and you can
watch the trap happen. The loop wanders through the orders’ pages, reads invoices page one three
times and page two once, and ends with 40 rows of which only 20 are different invoices. The row count
looks right; the invoice numbers give it away.
A correct result
Invoices: 40 rows and 5 columns, INV-2026-1301 to INV-2026-1340, no order numbers among them.
Orders: 18. Shipments: 15. Returns: 12. Messages: 5. Check that the invoice numbers are all
different, not only that there are 40 of them: repeated numbers mean another table’s pager was
followed.
Try it yourself: get Magic Scraper free and open it on this page.