Optimizing crawl efficiency for AI agent consumption