Content auditing for compatibility with emerging AI crawler directives