Crawler Policy

Published because an organisation that scores other people on data handling has to be able to answer the same questions about itself. Put this on the public site alongside the Standard.

What the crawler does

TruefieldStandardBot requests publicly available legal and policy pages from vendors on the watchlist, once per week, and stores the text to detect changes over time. It does not attempt to access any page requiring authentication, does not submit forms, does not create accounts, and does not follow links beyond the configured source list.

robots.txt

Checked before every request. A disallowed URL is skipped, logged, and either read from a permitted mirror or reviewed manually by a person. There is no override flag and there should never be one.

Where a vendor blocks general crawling but publishes the same document on an open domain, the open mirror is used and the substitution is recorded in the vendor's config notes.

AI-training directives

Several vendors set directives aimed at AI-training crawlers — GPTBot, CCBot, anthropic-ai, Google-Extended, meta-externalagent — while leaving general crawling open. BallerTV is the current example.

Position: those directives do not name this bot and do not apply to it. Retrieved text is used for one-time analysis against a published checklist and for change detection. It is not used to train, fine-tune, or otherwise improve any model, and it is not redistributed as a corpus.

But the distinction is worth stating out loud rather than relying on. A vendor that blocks training crawlers has expressed a preference about machines consuming its text. Reading that preference narrowly is defensible; pretending not to have noticed it is not. Hence this page.

Commitments that follow:

No retrieved vendor text is used for model training or fine-tuning.
No retrieved vendor text is published as a corpus or dataset.
Quotations in published scores are twelve words or fewer, one per criterion, for the purpose of substantiating a finding.
Any vendor may ask to be excluded from crawling. Their documents will be reviewed manually instead, and the exclusion will be noted in the report — not used against them in scoring.

Rate and identification

One request per source per week. The bot identifies itself by name with a contact URL. It does not rotate user agents, use residential proxies, or take any other step to obscure what it is.

Manual review path

When a document cannot be retrieved — robots disallow, authentication, or a format the crawler cannot parse — the fallback is a person reading it, not a workaround. If a document cannot be obtained lawfully and openly at all, the relevant criterion is scored on what is available and the gap is disclosed in the report's scope column.