Trivial anti-crawler with Caddy

Дата публикации: 23-06-2026 11:45:00

With the internet being crawled to death to feed the AI God, it's becoming
seriously annoying to exposed web content on the internet. While
anubis works, it's yet another layer of
complexity. It might be worth deploying and tuning it for high-profile
websites, but for my cgit instance, it's absolutely
overkill.
Instead, I'm taking advantage of Caddy (whose
documentation doesn't have a search feature‽)
matching capabilities to gate access on either the ability to execute
javascript to set a cookie, or having a user-agent string starting with git/
so that repository are still cloneable.
git.dustri.org {
import tls
import noindex
import compress
@unverified {
not header Cookie *not_a_crawler=1*
not header User-Agent git/*
}
handle @unverified {
header Content-Type text/html
respond <<EOF
<script>
document.cookie = 'not_a_crawler=1';
window.location.reload();
</script>
EOF 418
}
reverse_proxy cgit_upstream
}
It's not perfect, trivial to bypass, but strikes the right balance between
simplicity/zero-maintenance and blocking crawlers.
This technique is now used on https://autopkgtest.ubuntu.com.

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	Carrot disclosure: Forgejo	0	9.35	28-04-2026
2	The AI-authored Denial-of-Service attack	0	8.9	07-07-2026
3	swic: a simple web interface for calibre	0	11.32	21-04-2026
4	Bloquer le spam téléphonique chez Free Mobile	0	9.12	16-06-2026
5	spoof: A Simple HTTP Server for Test Environments	0	10	11-06-2026
6	Introducing Workshop: launch sandboxed development environments on Ubuntu with a single command	0	8.9	27-05-2026
7	django-crawl - An in-process site crawler using Django’s test client	0	28.18	28-07-2026
8	Follow-up to Carrot disclosure: Forgejo	0	7.3	30-04-2026
9	Daily Hacker News for 2026-07-20	0	10.73	21-07-2026
10	Daily Hacker News for 2026-07-26	0	11.66	27-07-2026

Классификация: . Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 8.42. Источник: dustri.org.