Valet fetchers

Valet is a marketplace where AI agents use online services on behalf of a human. Some of those services fetch web pages. Every request from a Valet fetcher is an on-demand fetch triggered by one agent tool call, made for a specific person who asked their agent for that page.

• One page per call. A fetcher retrieves the single URL it was asked for and stops.
• No crawling, no indexing, no recursion. Fetchers never follow links on a page. Besides HTTP redirects to reach the page you served, the only other URLs fetched are by render and screenshot, which load the page in a browser: its own sub-resources and any navigation the page itself starts (a meta refresh or a script redirect).
• Headless browser for render and screenshot. Those two load the page in Chromium so client-side JavaScript runs; the browser loads the page's own sub-resources (scripts, styles, images) exactly as a browser would.
• Nothing is stored for later crawling. The result goes back to the agent that asked; Valet keeps no copy of your page to build an index from.

Every fetcher sends the same honest User-Agent format, with a product token that names the fetcher, its version, and a link to this page:

Valet-<Service>/<version> (+https://askvalet.ai/bot)

For example: Valet-Scrape/0.1.0 (+https://askvalet.ai/bot)

You may also see Valet-Search, Valet-Finance, Valet-Geocode, Valet-PackageLookup, and Valet-FxConvert. Those services only talk to their own data providers' APIs (a search API, a market-data API, a geocoder, package registries, a rates feed) and never fetch arbitrary websites. They use the same User-Agent format so the contact link is always there.

Valet-Scrape, Valet-Render, and Valet-Screenshot read and obey your /robots.txt per RFC 9309 before fetching a page. A disallowed page is not fetched; the agent gets a structured error that quotes the rule, so the person behind it can see why.

What is checked against robots.txt:

• Valet-Scrape: the page, and every HTTP redirect hop on the way to it (each hop against its own site's robots.txt).
• Valet-Render and Valet-Screenshot: every navigation of the browser's main frame — the page, each redirect hop, and any navigation the page itself starts (a meta refresh or a script redirect).
• Not checked: the sub-resources the headless browser loads while rendering (scripts, styles, images, XHR), iframes, and the first navigation of a popup window a page opens.

Each fetcher looks for groups in this order and uses the first that matches:

• its own product token — Valet-Scrape, Valet-Render, or Valet-Screenshot (case-insensitive);
• the family token Valet, which covers every Valet fetcher at once;
• *.

User-agent: Valet
Disallow: /

User-agent: Valet-Screenshot
Disallow: /

User-agent: Valet
Allow: /docs/
Disallow: /

Details that may matter when you write rules:

• Patterns support * as a wildcard and a trailing $ as an end anchor; the longest matching rule wins, and a tie between Allow and Disallow allows.
• Your robots.txt is cached for up to 1 hour per site, so a change takes effect within the hour.
• Only the first 500 KiB of your robots.txt is read, a rule longer than 2,048 characters is ignored, and at most the first 2,000 rules (in file order) of the group a fetcher uses apply to it. A rule past those limits is never seen — keep the rules that must apply inside them.
• If your robots.txt answers with a 4xx status (404, 403, 401, …) it is treated as absent and fetching is allowed — except 429 Too Many Requests, which is treated as unreachable (disallow), not as absent.
• If it answers with a 5xx status or 429, times out, or cannot be reached, fetching is treated as disallowed. That result is remembered for only 60 seconds; after that the fetcher asks for your robots.txt again.
• Valet-Uptime does not read robots.txt: it is a single reachability probe of a URL its user already knows, and it reads nothing from the response body.

A robots.txt refusal (robots_disallowed or robots_unreachable) is a completed, billed call: the agent's request is charged like any other call even though your page was never fetched. A robots_unreachable result carries retry_after_s, an advisory wait in seconds: a sooner retry will usually get the same answer, and is billed too. A robots_disallowed result will not change until your robots.txt does.

• No CAPTCHA solving and no bot-detection evasion of any kind.
• No cookies, no logins, no stored credentials — every request starts from a clean state, so your pages are fetched as an anonymous visitor would see them.
• TLS certificates are validated; a site with a broken certificate is not fetched. The one exception is Valet-Uptime, which checks availability and reports a broken or expiring certificate instead of refusing it.
• The User-Agent is always the honest Valet-… format above. Valet never impersonates a browser or another crawler.
• Private, loopback, link-local, and cloud-metadata addresses are refused outright.

Requests originate from Google Cloud Run in the us-central1 region. There are no fixed source IP addresses, so please allow or block by User-Agent (or with robots.txt) rather than by IP.

Questions, or a fetch you believe should not have happened? Email support@askvalet.ai with the URL, the timestamp, and the User-Agent from your logs, and we will look into it.