Agent Mag
SUBSCRIBE
Documentation
Web Intelligence

Crawl Endpoint

Plan and inspect multi-page crawl jobs for docs sites, blogs, and product surfaces.

Status: Planned.

POST /v1/crawl is designed for multi-page jobs: docs sites, blogs, marketing pages, knowledge bases, and public product surfaces.

Planned Request

code
{
  "url": "https://docs.example.com",
  "limit": 50,
  "maxDepth": 3,
  "includePaths": ["/docs/**"],
  "excludePaths": ["/blog/**"],
  "formats": ["markdown", "links", "metadata"],
  "agentContext": true
}

Job States

StateMeaning
queuedJob accepted and waiting for a worker.
runningPages are being fetched and normalized.
completedCrawl finished within limits.
partialSome pages failed but usable output exists.
failedNo usable output was produced.

Scrape Endpoint

Fetch a single public URL and return clean content formats for AI agent workflows.

Search And Map

Find, rank, and map public URLs before scraping or crawling them.

On this page

Planned RequestJob States