diff options
| author | Xe Iaso <me@xeiaso.net> | 2025-04-20 20:09:27 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-04-21 00:09:27 +0000 |
| commit | d40b5cfdab11c62dc2ed226bde32b19ea7107f21 (patch) | |
| tree | 2c997fd907c0de22e98b1ab6dc233141e10567e1 /data/botPolicies.json | |
| parent | 022eb59ff337427ae4fe151fd1f24ec411418479 (diff) | |
| download | anubis-d40b5cfdab11c62dc2ed226bde32b19ea7107f21.tar.xz anubis-d40b5cfdab11c62dc2ed226bde32b19ea7107f21.zip | |
lib: move config to yaml (#307)
* lib: move config to yaml
Signed-off-by: Xe Iaso <me@xeiaso.net>
* web: run go generate
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Add Haiku to known instances (#304)
Signed-off-by: Asmodeus <46908100+AsmodeumX@users.noreply.github.com>
* Add headers bot rule (#300)
* Closes #291: add headers support to bot policy rules
* Fix config validator
* update docs for JSON -> YAML
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document http header based actions
Signed-off-by: Xe Iaso <me@xeiaso.net>
* lib: add missing test
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Asmodeus <46908100+AsmodeumX@users.noreply.github.com>
Co-authored-by: Asmodeus <46908100+AsmodeumX@users.noreply.github.com>
Co-authored-by: Neur0toxine <pashok9825@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Diffstat (limited to 'data/botPolicies.json')
| -rw-r--r-- | data/botPolicies.json | 9 |
1 files changed, 8 insertions, 1 deletions
diff --git a/data/botPolicies.json b/data/botPolicies.json index dbc3d35..1993d22 100644 --- a/data/botPolicies.json +++ b/data/botPolicies.json @@ -1,6 +1,13 @@ { "bots": [ { + "name": "cloudflare-workers", + "headers_regex": { + "CF-Worker": ".*" + }, + "action": "DENY" + }, + { "name": "ai-robots-txt", "user_agent_regex": "AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|Brightbot 1.0|Bytespider|CCBot|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo Bot|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|PanguBot|Perplexity-User|PerplexityBot|PetalBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YouBot", "action": "DENY" @@ -680,4 +687,4 @@ } ], "dnsbl": false -} +}
\ No newline at end of file |