Wholesale price scraping

About this practice page

Volume pricing that lives on each product’s page

Trade and wholesale shops, print shops, packaging and hardware suppliers all price by quantity: one price for one to nine, a lower one for ten to forty-nine, lower again for fifty or more. The list page shows only “from $1.80”. The breaks themselves are in a small table on each product’s own page, next to its specifications. If you are comparing suppliers, those breaks are the whole point.

Why this is two jobs

First, every product link has to be opened. Second, each product has several price breaks, so one product is not one row. Put the breaks side by side as columns and you get “Quantity 1, Unit price 1, Quantity 2…”, which cannot be sorted or compared. What you want is one row per product per break, with the product’s details repeated on each.

In Magic Scraper

Turn on drill-down and preview the first product. It shows the specifications (SKU, Finish, Ships in) and notices the price-break table. Tick Every row of its table — one row each and run it: the four products become 12 rows, each with the product, its specifications, and one Quantity, Unit price and Discount. Reading every product page is a Pro feature; the free version reads the first three so you can see the result.

In Python

url = "https://magicscraper.app/demo/wholesale-prices"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
rows = []
for link in soup.select("article.product h3 a"):
    page = BeautifulSoup(requests.get(urljoin(url, link["href"])).content, "html.parser")
    specs = {tr.th.get_text(strip=True): tr.td.get_text(strip=True) for tr in page.select("table.specs tr")}
    for tr in page.select("table.tiers tbody tr"):
        qty, price, off = [td.get_text(strip=True) for td in tr.select("td")]
        rows.append({"product": link.get_text(strip=True), **specs,
                     "quantity": qty, "unit_price": price, "discount": off})
len(rows)   # 12

A correct result

12 rows: three breaks for each of four products. The steel hinge runs from $2.40 for 1–9 to $1.80 for 50 or more; the first break of each product has an em dash for its discount, which means none, not missing. To compare at a given order size, filter on the quantity range that contains it.

Try it yourself: get Magic Scraper free and open it on this page.