Media

Wikipedia Articles

Wikipedia article text and attribution by title or search query.

wikipedia-articleswikipediaarticlesresearch

4 rows in the dataset

Download the dataset

Get every row this scraper has collected as one CSV or JSONL file. If the data is older than maxAgeHours, the scraper is re-run with its configured settings first. The key needs the datasets:export scope.

DOWNLOAD_ID=$(curl -s -X POST "https://api.pandorlabs.com/v1/datasets/wikipedia-articles/downloads" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{"maxAgeHours":24,"format":"csv"}' | jq -r '.id')while true; do  STATUS=$(curl -s "https://api.pandorlabs.com/v1/downloads/$DOWNLOAD_ID" \    -H "Authorization: Bearer $PANDORLABS_KEY" | jq -r '.status')  [ "$STATUS" != "refreshing" ] && break  sleep 5donecurl -s "https://api.pandorlabs.com/v1/downloads/$DOWNLOAD_ID/file" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -o dataset.csv

At its current 4 rows, this download costs 1 credit when the data is fresh enough, or 2 credits when it has to be refreshed first.

The rate is 1 credit per 100 rows cached and 2 credits live, charged on the rows in the dataset when the download becomes ready. A failed refresh is not charged. The file stays downloadable for 24 hours. See pricing.

Start a run

To choose the targets yourself, send the parameters below as input. The call returns 202 with the run; the key needs the scrapers:run scope and the account a positive credit balance.

curl -X POST "https://api.pandorlabs.com/v1/scrapers/wikipedia-articles/runs" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{    "input": {      "query": "computer science",      "language": "en",      "maxItems": 10,      "maxPages": 5    }  }'

A run is billed at the live rate once it finishes: 2 credits per 100 rows it collected. A run that collects nothing costs nothing.

Parameters

Every parameter is optional. Anything you leave out uses the default shown here; an unknown name or an invalid value is rejected with a 400 that lists each problem.

titles
string[]
Article titles
Default
none
query
string
Search query
Default
computer science
language
string
Wikipedia language
Default
en
maxItems
number
Maximum articles
Default
10
Range
1 to 100
maxPages
number
Maximum API pages
Default
5
Range
1 to 20

Input as JSON Schema

The same schema the API validates against. Use it as a tool input schema or to generate a typed client.

inputSchema

{  "type": "object",  "additionalProperties": false,  "properties": {    "titles": {      "type": "array",      "items": {        "type": "string"      },      "maxItems": 1000,      "title": "Article titles"    },    "query": {      "type": "string",      "maxLength": 2000,      "title": "Search query",      "default": "computer science"    },    "language": {      "type": "string",      "maxLength": 2000,      "title": "Wikipedia language",      "default": "en"    },    "maxItems": {      "type": "number",      "minimum": 1,      "maximum": 100,      "title": "Maximum articles",      "default": 10    },    "maxPages": {      "type": "number",      "minimum": 1,      "maximum": 20,      "title": "Maximum API pages",      "default": 5    }  }}

Run, poll, read rows

The run is asynchronous. Poll it until it reaches a terminal status and its billing is paid, then page through its rows. Until then the rows endpoint answers 409 (still queued or running) or 402 (payment required).

RUN_ID=$(curl -s -X POST "https://api.pandorlabs.com/v1/scrapers/wikipedia-articles/runs" \  -H "Authorization: Bearer $PANDORLABS_KEY" \  -H "Content-Type: application/json" \  -d '{    "input": {      "query": "computer science",      "language": "en",      "maxItems": 10,      "maxPages": 5    }  }' | jq -r '.id')while true; do  RUN=$(curl -s "https://api.pandorlabs.com/v1/runs/$RUN_ID" \    -H "Authorization: Bearer $PANDORLABS_KEY")  STATUS=$(echo "$RUN" | jq -r '.status')  BILLING=$(echo "$RUN" | jq -r '.billing.status')  case "$STATUS" in queued|running) ;; *) [ "$BILLING" != "pending" ] && break ;; esac  sleep 5doneecho "$RUN" | jq '{status, rowCount, billing}'curl -s "https://api.pandorlabs.com/v1/runs/$RUN_ID/rows?page=1&limit=100" \  -H "Authorization: Bearer $PANDORLABS_KEY"

See the API reference for pagination, cancelling and listing runs.

Output fields

Rows have no fixed schema; these are the fields observed in the rows collected so far. Coverage is the share of rows where the field is present.

Field	Type	Coverage
content_truncated	boolean	100%
images	array	100%
language	string	100%
license_url	string	100%
page_id	number	100%
raw_text	string	100%
references	array	100%
source_dataset_id	string	100%
title	string	100%
created_at	date	100%
updated_at	date	100%
_id	string	100%