STATIC Search crawler protocol

User agent: STATICbot/1.0 (+https://thestaticnetwork.com/help/search/crawler)

The independent crawl pilot indexes public HTTPS HTML on a bounded set of approved origins. It does not sign in, submit forms, carry visitor cookies or bypass access restrictions. JavaScript-only pages are marked as requiring rendering rather than counted as indexed content.

Publisher controls:
- robots.txt User-agent: STATICbot controls crawling. The most specific applicable user-agent group is used.
- Disallow prevents page retrieval. Allow exceptions use the longest matching path, with Allow winning equal-length matches.
- A valid robots.txt crawl-delay is respected, with a minimum host delay of 5 seconds. Origins requesting a delay above 60 seconds are paused in the pilot.
- robots meta, STATICbot meta and X-Robots-Tag noindex directives exclude the page. Duplicate directives are combined conservatively.
- nofollow prevents discovery from page links.
- nosnippet, max-snippet, data-nosnippet and noimageindex constrain the stored/displayed snippet and image metadata.
- Canonicals and redirects are inspected and queued under the same safety, origin and robots rules. They are not automatically followed into other origins.

An existing document is removed from the pilot's live index when a recrawl establishes noindex, a robots denial or a 404/410 response. Private, account and action routes are excluded before retrieval. Explicit operator removals create a tombstone that blocks rediscovery.

The pilot fetches at most 300 network requests per day, processes at most four pages per administrative batch, and caps stored index data. Indexed-page counts represent actual fetched records; first-party metadata feeds and licensed results are separate sources. The current pilot is not a claim of global crawl coverage.

Search indexing permission, use in generated answers and model-training permission are distinct. The crawler does not treat a robots rule as blanket permission for model training. Search does not automatically add private STATIC content to its public index.

Public guide: https://thestaticnetwork.com/help/search