- Web Search returns structured results with titles, URLs, snippets, and dates.
- Web Scrape converts a selected public page into Markdown.
Search the Web
Send a query to the Search API:Choosing a Provider
Every provider takes the same request and costs the same per search, so switching is a one-field change. The difference is in what comes back:
Longer results become input tokens when you pass them to a model, so they cost more and take longer to process. If results go straight into a prompt, cap how many you include and how much text each one contributes. With
google, keep the passages that match the query rather than the start of each result, which is often navigation or a cookie banner.
To see how the providers compare on relevance, answer accuracy, latency, and token cost, read Benchmarking Web Search Providers. It links to the benchmark harness, so you can rerun the comparison on your own questions.
Scrape a Page
After selecting a result, send its public URL to the Scrape API:Search-to-Scrape Workflow
1
Search
Submit a focused query and collect the structured results.
2
Select sources
Filter results by relevance, domain, and date before fetching page content.
3
Scrape
Extract Markdown only from the pages your application intends to use.
4
Use or store the content
Include the Markdown in a prompt, create embeddings, cache it, or preserve source URLs for citations.
429; retry them with exponential backoff, and keep parallel scraping to a small pool.