Magic Scraper Desktop: API and webhooks

For Magic Scraper Desktop on Windows and Linux · set it up on the app's Integrations page

MagicScraper can send results to other apps (webhooks) and let other apps run it (local API). The local API can also be reached from the internet through a Cloudflare Tunnel. Set all three up on the Integrations page.

The API and the tunnel work while MagicScraper is running. To keep them up with the window closed, turn on Settings → Run in the background: MagicScraper then keeps running in the tray, and can start with Windows. Webhooks are sent by every run, including scheduled monitor runs, which happen in the background even when MagicScraper is not running.

Webhooks

When an event happens, MagicScraper sends an HTTP POST with a JSON body to each webhook that wants it.

Headers

A delivery succeeds when the receiver answers with a 2xx status. Failures are retried after 5 and 30 seconds. 4xx answers other than 408 and 429 are not retried. The last 200 deliveries are listed on the Integrations page.

run.finished

{
  "id": "evt_3f9c1a0b7e2d4c6a8b1e0f2d",
  "event": "run.finished",
  "created_at": "2026-10-11T08:30:00.123+00:00",
  "app": { "name": "MagicScraper", "version": "0.1.0" },
  "run": {
    "id": "run_1a2b3c4d5e6f7a8b",
    "source": "app",
    "start_urls": ["https://magicscraper.app/demo/catalogue.html"],
    "ok": true,
    "message": "Finished: 54 rows",
    "row_count": 54
  },
  "columns": ["Name", "Link", "Price"],
  "rows": [{ "Name": "Cast Iron Mug Set", "Link": "https://…/item1.html", "Price": "$239.00" }],
  "rows_truncated": false
}

source is app (the Scrape page), api or test (the Send a test button). At most 10,000 rows are sent. If a run has more, rows_truncated is true; use the API or an export to get the rest.

monitor.changed, monitor.failed, monitor.run

{
  "id": "evt_…",
  "event": "monitor.changed",
  "created_at": "…",
  "app": { "name": "MagicScraper", "version": "0.1.0" },
  "monitor": { "id": "m1760171400000", "name": "Competitor prices", "start_urls": ["https://…"] },
  "run": { "id": "…", "started_at": 1760171400, "finished_at": 1760171431, "ok": true, "message": "",
           "rows": 54, "added": 1, "removed": 0, "changed": 2, "baseline": false },
  "changes": {
    "added":   [{ "Name": "New product", "Price": "$19.00" }],
    "removed": [],
    "changed": [{ "key": "https://…/item3.html",
                  "before": { "Price": "$64.00" }, "after": { "Price": "$59.00" },
                  "fields": ["Price"] }],
    "unchanged": 51
  }
}

Tick Include every row in monitor events to also receive columns, rows and rows_truncated.

Checking the signature

Each webhook has a signing secret (whsec_…). The signature is the hex HMAC-SHA256 of <t>.<raw request body> with that secret. Compare it with v1, and reject old timestamps to stop replays.

import hashlib, hmac, time

def verify(secret: str, header: str, body: bytes, tolerance=300) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    if abs(time.time() - int(parts["t"])) > tolerance:
        return False
    expected = hmac.new(secret.encode(), f"{parts['t']}.".encode() + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts["v1"])

import crypto from "node:crypto";

function verify(secret, header, rawBody, tolerance = 300) {
  const parts = Object.fromEntries(header.split(",").map((p) => p.split("=")));
  if (Math.abs(Date.now() / 1000 - Number(parts.t)) > tolerance) return false;
  const expected = crypto.createHmac("sha256", secret).update(`${parts.t}.`).update(rawBody).digest("hex");
  return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(parts.v1));
}

Local API

• Base address: http://127.0.0.1:8787. The port is set on the Integrations page. The API only listens on this PC; use the tunnel to reach it from elsewhere.
• Authentication: send the API key (msk_…) with every request, as Authorization: Bearer <key> or X-API-Key: <key>. Without a valid key the API answers 401.
• Answers are JSON. Errors look like {"error": "…"}.
• Unlicensed copies stop each run at the trial row limit, as in the app.

GET /v1/status

{ "version": "0.1.0", "licensed": true, "row_limit": null, "scrape_enabled": true, "running_scrapes": 0 }

POST /v1/scrape

Scrapes one or more pages and answers with the rows.

curl -X POST http://127.0.0.1:8787/v1/scrape \
  -H "Authorization: Bearer msk_…" -H "Content-Type: application/json" \
  -d '{"url": "https://magicscraper.app/demo/catalogue.html", "max_items": 100}'

Body fields:

Any other job setting from a job file saved in the app can be given too. API runs never use saved logins. Answer:

{ "id": "run_…", "status": "done", "start_urls": ["…"], "started_at": 1760171400, "finished_at": 1760171409,
  "error": "", "row_count": 54, "columns": ["Name", "Link", "Price"], "rows": [ … ] }

status is running, done or failed. Two scrapes can run at once; a third gets 429. The Allow scraping any web address switch turns this endpoint off (403).

GET /v1/runs/{id}

The status and rows of a run started with POST /v1/scrape (?format=csv for CSV). Answers 202 while it is running. The last 20 runs are kept until the app closes.

GET /v1/monitors

The saved monitors with their latest run.

POST /v1/monitors/{id}/run

Runs a monitor now, with its own settings (including its saved login). It waits for the run (pass ?wait=false to not wait, ?timeout= in seconds) and answers like /runs/latest below. The run also sends the monitor webhooks.

GET /v1/monitors/{id}/runs/latest

{ "run": { …summary… }, "changes": { "added": [], "removed": [], "changed": [], "unchanged": 54 },
  "columns": [ … ], "rows": [ … ] }

?rows=false leaves out rows. ?format=csv answers with the rows as CSV.

Remote access (Cloudflare Tunnel)

The tunnel connects the local API to Cloudflare's network, so it can be reached at a public https address without opening ports on the router. The app runs Cloudflare's cloudflared, which is shipped in the tools folder.

• Quick tunnel: no account needed. The address (https://<words>.trycloudflare.com) changes every time the tunnel starts. Cloudflare provides quick tunnels for testing, with no uptime guarantee.
• My Cloudflare tunnel: a permanent address on your own domain.
• In the Cloudflare dashboard, go to Zero Trust → Networks → Tunnels → Create a tunnel → Cloudflared.
• Copy the token.
• Add a public hostname (e.g. scraper.example.com) with service http://localhost:8787.
• Paste the token and hostname into MagicScraper.

You can add Cloudflare Access in front for extra sign-in protection.

Anyone who has the public address and the API key can run scrapes on the PC. Keep the key secret and make a new one if it leaks.