Log Analysis

Google Analytics and Search Console only show human traffic. AI robot visits (GPTBot, ClaudeBot, PerplexityBot) are invisible. Yet if you want to be cited by generative AI, you need to know if and when they crawl your site. Log analysis is the only way to see the invisible and understand how LLMs consume your content.

Each robot identifies itself via its User-Agent in HTTP requests. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended... We configure your infrastructure to automatically parse these User-Agents and categorize them. Thus, you know precisely which pages are read by which AI, and how often.

It depends on your infrastructure and budget. Cloudflare Enterprise offers the most complete native logs. For lower plans, we use custom Cloudflare Workers. sGTM is ideal if you already have a Server-Side infrastructure. For advanced analysis, Botify or OnCrawl remain industry references. We benchmark together the best option.

Cost varies depending on traffic volume and chosen solution. An sGTM + BigQuery configuration can cost a few tens of euros per month for a medium site. Cloudflare Workers is very economical (often free for small volumes). Botify/OnCrawl are more expensive but offer advanced business features. We size the solution to your real needs.

Yes, via the robots.txt file or HTTP headers. But be careful: blocking GPTBot means ChatGPT can no longer cite your content. It's a strategic decision. Log analysis gives you the visibility needed to decide knowingly which bots to allow or block, and on which sections of your site.

A basic setup (sGTM + User-Agent parsing) can be operational in 1-2 weeks. A complete configuration with dashboard, alerts and BigQuery export typically takes 3-4 weeks. Initial audit and benchmark are included to ensure the chosen solution matches your needs.