diff options
| author | Dennis ten Hoove <36002865+dennis1248@users.noreply.github.com> | 2025-03-17 22:42:42 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-03-17 17:42:42 -0400 |
| commit | f42f5f509a33002cfec811d06d1b858608c17009 (patch) | |
| tree | e2139eba6a8906da92bac92e1c6ddbcffee141c0 /cmd/anubis | |
| parent | 2859f037cdf0cfd7314768152a2a5ad0ec8cd638 (diff) | |
| download | x-f42f5f509a33002cfec811d06d1b858608c17009.tar.xz x-f42f5f509a33002cfec811d06d1b858608c17009.zip | |
cmd/anubis: allow qwantbot scraping (#700)
Like Google and Bing, Qwant maintains its own search index.
Diffstat (limited to 'cmd/anubis')
| -rw-r--r-- | cmd/anubis/botPolicies.json | 7 |
1 files changed, 6 insertions, 1 deletions
diff --git a/cmd/anubis/botPolicies.json b/cmd/anubis/botPolicies.json index b90eba0..6e04a11 100644 --- a/cmd/anubis/botPolicies.json +++ b/cmd/anubis/botPolicies.json @@ -16,6 +16,11 @@ "action": "ALLOW" }, { + "name": "qwantbot", + "user_agent_regex": "\\+https\\:\\/\\/help\\.qwant\\.com/bot/", + "action": "ALLOW" + }, + { "name": "us-artificial-intelligence-scraper", "user_agent_regex": "\\+https\\:\\/\\/github\\.com\\/US-Artificial-Intelligence\\/scraper", "action": "DENY" @@ -62,4 +67,4 @@ } ], "dnsbl": true -}
\ No newline at end of file +} |