Let's get started scraping with the FetchFox API.

Typically, scraping has two steps: crawl and extract. Crawl looks for URLs, and extract converts them into structured data

Get your API key

First, you'll need your API key. You can find it in the FetchFox app at https://fetchfox.ai/settings/api-keys.

Crawl for URLs

To crawl for URLs, you can use FetchFox's /api/crawl endpoint. Let's scrape some Pokemon using the snippet below.

curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"pattern":"https://pokemondb.net/pokedex/*","max_visits":"5","max_depth":"3","proxy":"default","crawl_priority":"random"}'

This will return URLs matching the pattern https://pokemondb.net/pokedex/*, which is all the Pokemon on that page.

The matches will be in the results.hits path on the return JSON. Let's use these URLs for extraction.

Extract data from URLs

Once you have the target URLs, convert them into structured data using /api/extract. Use the snippet below to extract data about Pokemon.

curl -X POST https://api.fetchfox.ai/api/extract \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"urls":["https://pokemondb.net/pokedex/pikachu","https://pokemondb.net/pokedex/ivysaur"],"template":{"name":"Name of the pokemon","number":"Number of the pokemon"},"proxy":"default","extract_mode":"ai"}'

This will get the name and number for each Pokemon. The structured data will be in results.items.

Credits and pricing

Each API request contains a metrics field. This field will indicate the cost for the API call, in dollars. It will also contain a breakdown of the cost. The breakdown shows how much of the cost is charged to AI usage, network and proxy usage, and FetchFox usage.