Crawler Tracking on Qwairy (log analysis)
What is Crawler Tracking?
Crawler Tracking allows you to monitor visits from AI bots such as GPTBot, ClaudeBot, PerplexityBot, and others. You’ll be able to see which pages they visit, how frequently, and with which user agent, giving you transparency into how AI platforms are indexing and referencing your content.How to install Crawler Tracking?
At the moment crawler tracking is only available for websites hosted on Vercel. ✅ Prerequisites- You must be on a Vercel Pro or Enterprise plan.
- You must have access to the Vercel dashboard for the site you want to monitor.
🔧 Step-by-Step Setup
1. Navigate to Log Drains in Vercel
- Go to your project in Vercel.
- Head to:
Settings → Log Drains → Add Log Drain - Choose Custom Log Drain.
2. Configure Source & Environment
- Sources: Select both
StaticandFunction. - Environment: Select
Productiononly. - Delivery Format: Choose
JSON. - Sampling Rate: Keep at
100%.
3. Configure the Log Drain
- Endpoint URL:
https://www.qwairy.co/api/logs/vercel - Custom Header:
Use the token provided in Qwairy (e.g.,x-qwairy-token).
4. Configure Verification Token (Optional)
- Vercel will show a Verification Token.
- You can copy it and save it in Qwairy for later verification.
- Alternatively, click Verify in Vercel if your header is already set up.
5. Complete the Log Drain
- Vercel will ask you to enter a Signature Token.
- Copy that token into Qwairy’s integration screen.
- Click Test Log Drain to validate the connection.
- Once tested, click Add Log Drain to finish the setup.
6. Verify Integration in Qwairy
- Back in Qwairy, toggle Enable Vercel Integration ON.
- Click the Verify button to confirm everything is working properly.
🎉 You’re Done!
You’ll now start seeing:- Real-time logs of AI bots accessing your site
- Breakdown of visits by GPTBot, ClaudeBot, Perplexity, Google-Extended, etc.
- Detailed traffic logs including path, referrer, status, and user agent
- Charts & insights under
AI Traffic Analysis → Crawler Tracking