Botcheck Runbook

How to collect bot traffic from multigigguide.com (hosted on Vercel) and verify which crawlers are real.

Why this exists

I run multigigguide.com on Vercel’s Hobby plan. Hobby does give you runtime logs — but only for an hour. That is useless if you want a durable record of which AI crawlers hit the site, from which IPs, on which paths. Longer retention and Log Drains sit behind paid plans, and I was not going to upgrade just to answer “is that actually GPTBot?”

What I still need is the raw request: IP, user agent, path, referer. Vercel middleware can see those on every request, including bots. So the middleware filters for the bots I care about and POSTs each hit to a small Go collector on a DigitalOcean droplet, where it lands in a plain log file I own.

That log is what botcheck reads. It checks each IP against the ranges Google, OpenAI, Anthropic, and the others publish. User-agent alone is not enough — anyone can spoof GPTBot. The IP check is what makes the hit real or fake.

Vercel’s Observability dashboard can show bot activity at a high level. It does not give me an exportable log I can verify offline. This pipeline does.

How it works

Bot visits multigigguide.com
        │
        ▼
Vercel middleware (middleware.ts)
  └─ user agent matches a tracked bot?
        │ yes: POST the hit (IP, path, UA, referer)
        ▼
https://some-domain.com/_botcollect   (nginx, HTTPS)
        │
        ▼
botcollect (Go service on the droplet)
  └─ appends an nginx-format line to the log
        │
        ▼
/var/lib/botcollect/multigigguide.log
        │  copy to Mac when needed
        ▼
botcheck (Go, on the Mac)
  └─ verifies each hit against vendor-published IP ranges

Where everything lives

Bots tracked: Googlebot, bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User.

1. Check whether bots have visited (droplet)

ssh creinfo@ip-address

How many hits have been collected:

sudo wc -l /var/lib/botcollect/multigigguide.log

The 20 most recent hits:

sudo tail -20 /var/lib/botcollect/multigigguide.log

Watch hits arrive live (press Ctrl+C to stop):

sudo tail -f /var/lib/botcollect/multigigguide.log

Hit counts by user agent:

sudo awk -F'"' '{print $6}' /var/lib/botcollect/multigigguide.log | sort | uniq -c | sort -rn

Most-visited paths:

sudo awk '{print $7}' /var/lib/botcollect/multigigguide.log | sort | uniq -c | sort -rn | head -20

2. Verify which bots are real (droplet, then Mac)

Droplet: make a readable copy. The collector’s folder is locked to its service user, so it can’t be downloaded directly.

sudo cp /var/lib/botcollect/multigigguide.log /tmp/mgg.log && sudo chmod 644 /tmp/mgg.log
exit

Mac: download it and run botcheck.

scp creinfo@ip-address:/tmp/mgg.log ~/Developer/botcheck/multigigguide.log
cd ~/Developer/botcheck
go run . multigigguide.log

go run . with no filename reads sample.log (the host-name.com nginx log).

Reading the report

Notes:

• multigigguide hits always show status 000. The middleware runs before the page is served, so it never sees the real status code. Verification still works fully.

• Anthropic publishes one IP list for ClaudeBot and Claude-User, so the two can’t be told apart by IP.

• The Bing (2024) and Perplexity (2025) lists are older, so a “spoofed” verdict for those two is less certain. Spot-check the top IP:

  host THE-IP

  Real Bingbot resolves to *.search.msn.com. A *.googleusercontent.com or *.amazonaws.com hostname means a rented cloud server, not the real bot.

Log files contain visitor IPs. *.log is in .gitignore, so never force-add them.

3. Test the pipeline end to end

Mac: pretend to be GPTBot.

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://multigigguide.com/

Expect 200. Then, on the droplet:

sudo tail -3 /var/lib/botcollect/multigigguide.log

A new line with your home IP and the GPTBot user agent means everything works. botcheck will correctly mark it as spoofed.

4. Maintenance

Is the collector running? (droplet)

systemctl status botcollect --no-pager
curl http://127.0.0.1:8090/health

Expect active (running) and ok.

Recent collector errors:

sudo journalctl -u botcollect -n 50 --no-pager

Restart it:

sudo systemctl restart botcollect

Clear the log (droplet)

sudo truncate -s 0 /var/lib/botcollect/multigigguide.log

Update the collector after changing its code

Mac:

cd ~/Developer/botcollect
GOOS=linux GOARCH=amd64 go build -o botcollect .
scp botcollect creinfo@ip-address:~

Droplet:

sudo mv ~/botcollect /opt/botcollect/botcollect
sudo chmod 755 /opt/botcollect/botcollect
sudo systemctl restart botcollect
curl http://127.0.0.1:8090/health

Rotate the secret

The secret must match in two places.

1. Mac: generate a new one, and don’t paste it anywhere else.

   openssl rand -hex 24

2. Droplet: update the settings file and restart.

   sudo nano /etc/botcollect.env
   sudo systemctl restart botcollect

3. Vercel: multigigguide → Settings → Environments → Production → Environment Variables → edit BOTCHECK_SECRET. Repeat under Preview.

4. Vercel: redeploy production (Deployments → latest production → ⋯ → Redeploy). The new value only takes effect on a new deployment.

5. Run the end-to-end test in section 3.

Track another bot

Both files need the change.

1. multigigguide, middleware.ts: add the bot’s user-agent token to BOT_RE, spelled exactly as it appears in the user agent (for example bingbot is lowercase). Commit and push.
2. botcheck, main.go: add {"Token", "kind", "https://vendor/ip-list.json"} to the bots list. The IP list must use the {"prefixes": [{"ipv4Prefix": ...}]} format. Commit and push.

5. Troubleshooting

Known limits

• Only the bots listed above are captured. Human visitors and other bots are not.
• Images, CSS, JS and fonts are skipped on purpose. Pages, robots.txt and sitemaps are included.
• No status codes for multigigguide hits (always 000).
• If the droplet is down, hits during that window are lost. The site itself is unaffected.