hot.dev/firecrawl / ::firecrawl

Firecrawl v2 bindings: turn websites into clean, LLM-ready content. Completes the browsing loop with web-search: search finds URLs, Firecrawl reads them as single pages, batches, whole sites, or structured extraction.

Setup: context variable firecrawl.api.key (fc-...), or a per-call api-key in the trailing opts Map of any API-calling function (see ::firecrawl::api).

Quick start

::fc ::firecrawl

page ::fc/scrape("https://example.com", {formats: ["markdown"]})
page.markdown

job ::fc/crawl("https://docs.firecrawl.dev", {limit: 10})
// later (a webhook, a schedule, or the next run):
site ::fc/get-crawl-all(job.id)
if(::fc/is-done(site), map(site.data, (doc) { doc.metadata.sourceURL }), [])

Crawls, batch scrapes, and extractions are async jobs: the start function returns {id}, get-* reads the current status, and is-done says whether it has finished. Learn that a job finished from a webhook (verify it with ::firecrawl::webhooks) or by calling get-* from a later run or a schedule; nothing here blocks waiting for a job. Non-2xx responses are an Err carrying ::firecrawl::api/HttpError.

Values

BASE_URL

BASE_URL: Str "https://api.firecrawl.dev/v2"

Functions

batch-scrape

fn (urls: Vec): Map
fn (urls: Vec, options: Map): Map
fn (urls: Vec, options: Map, opts: Map): Map

POST /batch/scrape - Start scraping many URLs at once. options take the same scrape options as scrape at the top level (formats, onlyMainContent, ...), plus maxConcurrency, ignoreInvalidURLs (default true), and webhook. Returns {id, url, invalidURLs?}; check on it later with get-batch-scrape-all and is-done.

Example

job ::firecrawl/batch-scrape(["https://example.com", "https://firecrawl.dev"], {formats: ["markdown"]})
// in a later run, with job.id saved:
pages ::firecrawl/get-batch-scrape-all(job.id)
if(::firecrawl/is-done(pages), map(pages.data, (doc) { doc.metadata.title }), [])

cancel-batch-scrape

fn (id: Str): Map
fn (id: Str, opts: Map): Map

DELETE /batch/scrape/{id} - Cancel a running batch scrape. Returns {status: "cancelled"}.

cancel-crawl

fn (id: Str): Map
fn (id: Str, opts: Map): Map

DELETE /crawl/{id} - Cancel a running crawl. Returns {status: "cancelled"}.

crawl

fn (url: Str): Map
fn (url: Str, options: Map): Map
fn (url: Str, options: Map, opts: Map): Map

POST /crawl - Start crawling a site. options: limit (page cap, default 10000; always set one), includePaths/excludePaths (regexes), maxDiscoveryDepth, sitemap, crawlEntireDomain, allowSubdomains, scrapeOptions (per-page scrape options), webhook ({url, events?, metadata?}; verify deliveries with ::firecrawl::webhooks/verify-request). Returns {id, url}; check on it later with get-crawl / get-crawl-all and is-done.

credit-usage

fn (): Map
fn (opts: Map): Map

GET /team/credit-usage - Remaining credits: {remainingCredits, planCredits, billingPeriodStart, billingPeriodEnd}. Free, so it doubles as a key check.

extract

fn (urls: Vec, options: Map): Map
fn (urls: Vec, options: Map, opts: Map): Map

POST /extract - Start an LLM-structured extraction across URLs. Provide a prompt and/or a JSON schema; wildcard URLs (https://site.com/*) extract from a whole site. Other options: enableWebSearch, showSources, scrapeOptions. Returns {id, invalidURLs?}: extraction is async, so read the result later with get-extract once is-done.

Example

job ::firecrawl/extract(["https://firecrawl.dev/pricing"], {
  prompt: "List the plan names and their monthly prices"
})
// in a later run, with job.id saved:
out ::firecrawl/get-extract(job.id)
if(::firecrawl/is-done(out), out.data, null)

get-batch-scrape

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /batch/scrape/{id} - One status page, shaped like get-crawl: {status, total, completed, creditsUsed, next?, data}.

get-batch-scrape-all

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /batch/scrape/{id}, following next - The current status with data from every page fetched so far.

get-batch-scrape-errors

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /batch/scrape/{id}/errors - Failed URLs (errors: [{id, timestamp, url, error}]) and robotsBlocked.

get-crawl

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /crawl/{id} - One status page: {status, total, completed, creditsUsed, expiresAt, next?, data: [documents]}. status is "scraping", "completed", "failed", or "cancelled". Results arrive in pages of about 10MB; next is set while more remain. Use get-crawl-all to follow every page.

get-crawl-all

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /crawl/{id}, following next - The current status with data holding the documents from every page fetched so far. While the crawl is still running, next stays set when the cursor has caught up with the crawler.

get-crawl-errors

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /crawl/{id}/errors - Pages that failed (errors: [{id, timestamp, url, error}]) and URLs blocked by robots.txt (robotsBlocked).

get-extract

fn (id: Str): Map
fn (id: Str, opts: Map): Map

GET /extract/{id} - Extraction status: {status, data, expiresAt, tokensUsed?}; status is "processing", "completed", "failed", or "cancelled".

is-done

fn (job: Map): Bool

True when a job status from get-crawl, get-crawl-all, get-batch-scrape, get-batch-scrape-all, or get-extract will not change again: status is "completed", "failed", or "cancelled". Check status itself to tell success from failure.

Example

// On a schedule: check a crawl started earlier
check-crawl fn (id: Str): Any {
  job ::firecrawl/get-crawl-all(id)
  cond {
    not(::firecrawl/is-done(job)) => { null }
    eq(job.status, "completed") => { job.data }
    => { err(`crawl ${id} ended ${job.status}`) }
  }
}

list-active-crawls

fn (): Map
fn (opts: Map): Map

GET /crawl/active - The team's running crawls: {crawls: [{id, teamId, url, options}]}.

map-site

fn (url: Str): Map
fn (url: Str, options: Map): Map
fn (url: Str, options: Map, opts: Map): Map

POST /map - Discover a site's URLs fast (sitemap plus link discovery). options: search (rank links by relevance to a query), limit (default 5000), sitemap ("include"/"skip"/"only"), includeSubdomains, ignoreQueryParameters. Returns {links: [{url, title?, description?}]}; feed the interesting ones to scrape or batch-scrape.

scrape

fn (url: Str): Map
fn (url: Str, options: Map): Map
fn (url: Str, options: Map, opts: Map): Map

POST /scrape - Fetch one URL as clean content. options (camelCase, as Firecrawl documents them) include formats (e.g. ["markdown", "links"], or format objects like {type: "json", prompt: "..."}), onlyMainContent (default true), waitFor (ms), timeout (ms), maxAge (ms of cache to accept), actions, location, proxy. Returns the document (markdown, metadata, ...).

Example

page ::firecrawl/scrape("https://example.com", {formats: ["markdown", "links"]})
page.metadata.title

search

fn (query: Str): Map
fn (query: Str, options: Map): Map
fn (query: Str, options: Map, opts: Map): Map

POST /search - Web search, optionally scraping each hit. options: limit (1-100, default 10), sources (e.g. ["web", "news"]), tbs (time filter, e.g. "qdr:w"), location, country, includeDomains/excludeDomains, scrapeOptions (e.g. {formats: ["markdown"]} to get page content with each hit). Returns the data map: {web: [{title, url, description, markdown?}], news?, images?}.

Example

hits ::firecrawl/search("firecrawl changelog", {limit: 3, scrapeOptions: {formats: ["markdown"]}})
map(hits.web, (hit) { hit.url })