Store locator scraping

About this practice page

A store locator split under letters

“Where to buy” pages, dealer lists, member directories and lists of branches are often grouped like this: a heading for each letter, region or county, and a short list under each. It reads well. For a spreadsheet it is awkward, because the heading is information too — the region a dealer is in, the county a branch serves — and it is written once above its group rather than on every line.

What goes wrong

There are five separate lists here, one under each letter, with three, four, two, three and two shops. A scraper that takes “the biggest list” gets the four under B and stops. One that gathers every shop without its heading gets fourteen rows and loses the grouping, which on a real locator is usually the region you wanted to filter by.

In Magic Scraper

It sees that the five lists are the same kind of thing, joins them into one table and adds the heading each shop sat under as a Section column: 14 rows with Section, Shop Name, Town Name and Tel. Nothing to set.

In Python

Walk the groups and carry each heading down to its rows:

soup = BeautifulSoup(requests.get("https://magicscraper.app/demo/stockists").content, "html.parser")
rows = [{"section": sec.h2.get_text(strip=True),
         "name": li.select_one(".shop-name").get_text(strip=True),
         "town": li.select_one(".town-name").get_text(strip=True),
         "tel":  li.select_one(".tel").get_text(strip=True)}
        for sec in soup.select("section.town") for li in sec.select("li")]
len(rows)   # 14

Using a list like this

Phone numbers come through as written, with their spaces, so a spreadsheet treats them as text and keeps the leading zero — which is what you want. On a real locator, the shop’s own page often has the address and opening hours; drill-down adds those as columns, the same way the business directory practice page does.

A correct result

14 rows in four columns, the first being Section A, Ashby Home & Hearth, Ashby, 01530 555 014, and the last Section E, Ember Kitchenware, Exeter.

Try it yourself: get Magic Scraper free and open it on this page.