Download any dataset with one HTTP call

1. 1

   Create a download

   Both body fields are optional. maxAgeHours (default 24) is how old the data may be. If the dataset was refreshed within that window, you are charged the cached rate immediately and the download is ready. If it is older, the scraper is re-run with its configured settings, the download is refreshing, and you are charged the live rate once that run completes. The API answers 202 with the download, whose id you keep.

   Terminal

   curl -X POST "https://api.pandorlabs.com/v1/datasets/cryptopunks-leaderboard/downloads" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{"maxAgeHours":24,"format":"csv"}'

2. 2

   Poll the download

   Fetch the download until status is ready. A cached download is ready on the first response; a refresh takes as long as the scraper run. If the refresh fails, the status is failed and nothing is charged.

   Terminal

   curl "https://api.pandorlabs.com/v1/downloads/$DOWNLOAD_ID" \  -H "Authorization: Bearer $PANDORLABS_KEY"

3. 3

   Fetch the file

   The file is streamed as CSV or JSONL, whichever format you asked for. Once paid, it can be fetched any number of times until expiresAt, 24 hours after the download became ready.

   Terminal

   curl "https://api.pandorlabs.com/v1/downloads/$DOWNLOAD_ID/file" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -o dataset.csv

The whole flow in one script

DOWNLOAD_ID=$(curl -s -X POST "https://api.pandorlabs.com/v1/datasets/cryptopunks-leaderboard/downloads" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{"maxAgeHours":24,"format":"csv"}' | jq -r '.id')while true; do  STATUS=$(curl -s "https://api.pandorlabs.com/v1/downloads/$DOWNLOAD_ID" \    -H "Authorization: Bearer $PANDORLABS_KEY" | jq -r '.status')  [ "$STATUS" != "refreshing" ] && break  sleep 5donecurl -s "https://api.pandorlabs.com/v1/downloads/$DOWNLOAD_ID/file" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -o dataset.csv

1. 1

   Start a run

   Send the scraper's parameters as input. Every parameter is optional; anything you leave out uses its documented default. The API answers 202 straight away with the run, whose id you keep.

   curlNode.jsPython

   curl -X POST "https://api.pandorlabs.com/v1/scrapers/cryptopunks-leaderboard/runs" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{    "input": {      "maxHolders": 100,      "enrichWithOpenSea": false,      "maxEnriched": 100,      "enrichmentDelayMs": 500    }  }'

2. 2

   Poll the run

   Runs are asynchronous and take from seconds to minutes. Fetch the run until status is completed, failed or cancelled. At that point it is billed at the live rate for the rows it collected, and billing.status becomes paid.

   Terminal

   curl "https://api.pandorlabs.com/v1/runs/$RUN_ID" \  -H "Authorization: Bearer $PANDORLABS_KEY"

3. 3

   Read the rows

   Rows are readable once the run has finished and is paid; before that the endpoint answers 409 (still queued or running) or 402 (payment required). They are paginated: keep requesting the next page while hasNext is true.

   Terminal

   curl "https://api.pandorlabs.com/v1/runs/$RUN_ID/rows?page=1&limit=100" \  -H "Authorization: Bearer $PANDORLABS_KEY"

The whole flow in one script

RUN_ID=$(curl -s -X POST "https://api.pandorlabs.com/v1/scrapers/cryptopunks-leaderboard/runs" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{    "input": {      "maxHolders": 100,      "enrichWithOpenSea": false,      "maxEnriched": 100,      "enrichmentDelayMs": 500    }  }' | jq -r '.id')while true; do  RUN=$(curl -s "https://api.pandorlabs.com/v1/runs/$RUN_ID" \    -H "Authorization: Bearer $PANDORLABS_KEY")  STATUS=$(echo "$RUN" | jq -r '.status')  BILLING=$(echo "$RUN" | jq -r '.billing.status')  case "$STATUS" in queued|running) ;; *) [ "$BILLING" != "pending" ] && break ;; esac  sleep 5doneecho "$RUN" | jq '{status, rowCount, billing}'curl -s "https://api.pandorlabs.com/v1/runs/$RUN_ID/rows?page=1&limit=100" \  -H "Authorization: Bearer $PANDORLABS_KEY"

Authorization: Bearer $PANDORLABS_KEY

Keys are scoped: each one only reaches the endpoints its scopes allow. Downloading datasets needs datasets:export (export datasets); starting, following and cancelling runs needs scrapers:run (run scrapers). The scope each endpoint requires is listed in the API reference. Give each script its own key so one can be revoked without breaking the rest.

Downloads

• refreshing

  The dataset was older than maxAgeHours, so its scraper is being re-run. runId points at that run. Nothing has been charged yet.

• ready

  Paid for and downloadable. Fetch the file as often as you like until expiresAt.

• payment_required

  The refresh finished but the balance could not cover it. Top up and request the file again; the charge is retried.

• failedterminal

  The refresh run failed and nothing was charged. Start a new download with a larger maxAgeHours to accept the data that already exists.

• expiredterminal

  Past expiresAt. Start a new download to get the file again.

Runs

• queued

  Accepted and waiting for a worker. Nothing has been fetched yet.

• running

  Collecting. rowCount climbs as rows are parsed; the rows themselves are not readable yet.

• completedterminal

  Finished normally. Billed for the rows it collected.

• failedterminal

  Stopped on an error, reported in error.message. Billed for the rows collected before the failure.

• cancelledterminal

  Stopped by a cancel request. Billed for the rows collected before the cancel.

To stop a run early, send POST /v1/runs/{runId}/cancel. GET /v1/runs lists the runs your account started, and GET /v1/downloads your downloads.

Run billing

Every run carries a billing object next to its status. The rows are readable only when the run has finished and its billing is paid.

• pending

  The run has not been billed yet.

• paid

  Charged billing.credits. The rows are readable. A run that collected no rows is paid at zero credits.

• payment_required

  The balance could not cover the run. Top up, then fetch the run or its rows again; the charge is retried on that request.

• 400Bad request

  An unknown parameter, or a value that fails the scraper’s input schema. The message field lists every problem.

• 401Unauthorized

  The key is missing, malformed, expired or revoked.

• 402Payment required

  Not enough credits: to start a download or a run, or to read a run’s rows or a download’s file while its billing is payment_required. Top up and send the same request again.

• 403Forbidden

  The key is valid but lacks the scope the endpoint needs. The message names the missing scope.

• 404Not found

  No dataset or scraper with that id, or no run or download with that id on your account.

• 409Conflict

  Not ready yet: the rows of a run that is still queued or running, or the file of a download that is still refreshing. Keep polling.

• 410Gone

  The download has expired. Start a new one to get the file again.

• 429Too many requests

  The key went over its per-minute rate limit, or the account already has 5 runs in flight.

{  "statusCode": 400,  "message": [    "maxItems must be a number",    "unknown parameter: colour"  ],  "error": "Bad Request"}

{  "statusCode": 402,  "message": "Not enough credits",  "required": 50,  "balance": 0}

Rate limits

Each key has its own per-minute request limit, shown next to the key in the dashboard. Every response reports where you stand in x-ratelimit-limit and x-ratelimit-remaining. Separately, an account can have at most 5 runs queued or running at once; starting another returns 429 until one finishes or is cancelled.