When new pages remain discovered but not indexed, verify discovery, crawl readiness, page value, template cohorts and time before using manual requests as a diagnostic observation.
Choose schema by the page’s visible job and a documented search feature, then separate rich-result eligibility, machine understanding, validation, and ranking claims.
Reconstruct the availability timeline, crawler-facing status, recovery signals and competing explanations before attributing a search decline to an outage or update.
Diagnose a crawler-facing robots.txt failure across DNS, IPv4 and IPv6, TLS, CDN, WAF, redirects, and origin logs instead of trusting one browser request.
Choose what to do with an expired event, offer, deadline, or annual page using reader intent, replacement equivalence, historical value, accuracy, and recurring demand.
A publisher’s guide to Googlebot, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Applebot, Amazonbot and other documented crawlers—what each one does, which controls it respects, and how to prepare a website for AI search without confusing crawling with citation.
A URL-prefix removal can expose redirect, canonical, sitemap, internal-link, and content-equivalence failures. Diagnose the migration before blaming an update.