Bingbot + GPTBot allowed
This page is designed to be explicitly allowed for both Bingbot and GPTBot, demonstrating a shared-access scenario while keeping other test permutations intact.
Robots intent
- Explicit allow lines for both user agents in robots.txt.
- Useful for testing dual ingestion (search index + LLM crawler) on the same document.
- Content is descriptive to avoid spam signals and clarify test purpose.
Verification steps
- Fetch
/robots.txt and confirm Allow: /tests/bing-gpt-allowed.html under both Bingbot and GPTBot sections.
- Crawl as Bingbot (expect 200 and allowed); crawl as GPTBot (expect 200 and allowed).
- Check sitemaps: this URL is listed in both the root and /tests sitemaps with updated lastmod.
Back to tests index