feat: Add Open Graph tag support (#195)

* feat: Add Open Graph tag support (og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Fix: Prevent nil pointer dereference in test (og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat!: Implement Open Graph tag caching and passthrough functionality (WIP)

I'm going to sleep. currently tags are passed to renderIndex.

see https://github.com/TecharoHQ/anubis/issues/131

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat: Add configuration for air tool with build and logger settings

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat: Move OG tags to base template (og-tags)

Moves the Open Graph (OG) tags from the index template to
the base template. This allows OG tags to be set on any
page, not just the index.  Also adds a
BaseWithOGTags function to the web package to allow
passing OG tags to the base template.  Removes the
ogTags parameter from the Index function and template.

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Delete CHANGELOG.md

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat: Add language attribute to HTML tag in template

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(tests):  Fix nil pointer ref

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat(og-tags): Add timeout to http client (og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* style: fix line endings & indentation

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* style: add inspection comment for GoBoolExpressions in UnchangingCache

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat(og-tags): Implement Open Graph tag fetching and caching

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(og-tags): Simplify Open Graph tag extraction logic

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(og-tags): Add nil check in isOGMetaTag and enhance test cases

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat(og-tags): Add approved tags and prefixes for Open Graph extraction

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* test(og-tags): Update tests with approved tags and improve clarity

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* chore: Add changelog notes

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix: Improve stability of the target fetcher?

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix: Update template error handling and improve Open Graph tag integration

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* style: format files and remove deubg logs

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat: Credit CELPHASE for mascot design (og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat: Credit CELPHASE for mascot design (og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat: Allow twitter prefixed OG tags by default

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* chore: replace /tmp with /var

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Update docs/docs/CHANGELOG.md

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* Update docs/docs/admin/configuration/open-graph.mdx

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* chore: add fediverse to default prefixes (#og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat(og-tags): Remove og-query-distinct flag

This commit removes the `og-query-distinct` flag and
associated logic.  URLs with different query parameters
will now always be treated as the same cache key for Open
Graph tags.  This simplifies the caching logic and
improves performance.

Additionally, the http client used for fetching OG tags
is now a member of the OGTagCache struct, rather than a
global variable. This improves testability and allows
for more flexible configuration in the future.

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Update docs/docs/admin/configuration/open-graph.mdx

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* docs: remove og tags references

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* refactor: rename url > u to not overlap package name

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Update internal/ogtags/cache.go

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* Update internal/ogtags/cache.go

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* fix(tests): Don't use network when network access is disabled

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Fix: Handle nil URL in GetOGTags (og-tags)

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* chore: sort installation docs alphabetically

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(tests): validate that no duplicate requests are made

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* style(tests): remove unused ok var

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* docs: convert to table fmt

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat(og-tags): Enhance OG tag fetching and caching

Adds additional approved OG tags (`keywords`, `author`), improves

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* chore: update generated templ's after format

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(tests): update integration_test.go to reflect the new behavior of fetchHTMLDocument

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Revert "data/botPolicies: allow iMessage scraper by default (#178)"

This reverts commit 21a9d777

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Fix: Simplify ogTags access in cache test.

Didn't know this was possible! wow!

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Fix: Handle request timeouts when fetching OG tags (#og-tags)

Cache a nil result for half the TTL to avoid repeatedly
requesting a timed-out URL.

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Fix: make OG tags passthrough option function.

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Fix: Handle timeouts and non-200 responses when fetching OG tags (og-tags)

- Cache empty results for timeouts and non-200 status codes
  to avoid spamming the server.
- Use a non-nil empty map to represent empty results in the
  cache, as nil would be a cache miss.

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* feat(og-tags): switch to http.MaxBytesReader

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* chore(og-tags): add noindex, nofollow meta tag and update error line numbers

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

---------

Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>

3 files changed, 60 insertions, 2 deletions

diff --git a/docs/docs/CHANGELOG.md b/docs/docs/CHANGELOG.md
index 0937cc3..d7c6acb 100644
--- a/docs/docs/CHANGELOG.md
+++ b/docs/docs/CHANGELOG.md
@@ -14,7 +14,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added support for native Debian, Red Hat, and tarball packaging strategies including installation and use directions.
 - A prebaked tarball has been added, allowing distros to build Anubis like they could in v1.15.x.
 - The placeholder Anubis mascot has been replaced with a design by [CELPHASE](https://bsky.app/profile/celphase.bsky.social).
-- Allow iMessage's link preview fetcher through Anubis by default.
 - Added a periodic cleanup routine for the decaymap that removes expired entries, ensuring stale data is properly pruned.
 - Added a no-store Cache-Control header to the challenge page
 - Hide the directory listings for Anubis' internal static content
@@ -38,6 +37,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added `zizmor` for GitHub Actions static analysis
 - Fixed most `zizmor` findings
 - Enabled Dependabot
+- Added an air config for autoreload support in development ([#195](https://github.com/TecharoHQ/anubis/pull/195))
+- Added support for [OpenGraph tags](https://ogp.me/) when rendering the challenge page. This allows for social previews to be generated when sharing the challenge page on social media platforms ([#195](https://github.com/TecharoHQ/anubis/pull/195))
 - Added an `--extract-resources` flag to extract static resources to a local folder.
 - Add noindex flag to all Anubis pages ([#227](https://github.com/TecharoHQ/anubis/issues/227)).
 
diff --git a/docs/docs/admin/configuration/open-graph.mdx b/docs/docs/admin/configuration/open-graph.mdx
new file mode 100644
index 0000000..87dd404
--- /dev/null
+++ b/docs/docs/admin/configuration/open-graph.mdx
@@ -0,0 +1,47 @@
+---
+id: open-graph
+title: Open Graph Configuration
+---
+
+# Open Graph Configuration
+
+This page provides detailed information on how to configure [OpenGraph tag](https://ogp.me/) passthrough in Anubis. This enables social previews of resources protected by Anubis without having to exempt each scraper individually.
+
+## Configuration Options
+
+| Name             | Description                                               | Type     | Default | Example                 |
+|------------------|-----------------------------------------------------------|----------|---------|-------------------------|
+| `OG_PASSTHROUGH` | Enables or disables the Open Graph tag passthrough system | Boolean  | `false` | `OG_PASSTHROUGH=true`   |
+| `OG_EXPIRY_TIME` | Configurable cache expiration time for Open Graph tags    | Duration | `24h`   | `OG_EXPIRY_TIME=1h`     |
+
+## Usage
+
+To configure Open Graph tags, you can set the following environment variables, environment file or as flags in your Anubis configuration:
+
+```sh
+export OG_PASSTHROUGH=true
+export OG_EXPIRY_TIME=1h
+```
+
+## Implementation Details
+
+When `OG_PASSTHROUGH` is enabled, Anubis will:
+
+1. Check a local cache for the requested URL's Open Graph tags.
+2. If a cached entry exists and is still valid, return the cached tags.
+3. If the cached entry is stale or not found, fetch the URL, parse the Open Graph tags, update the cache, and return the new tags.
+
+The cache expiration time is controlled by `OG_EXPIRY_TIME`.
+
+## Example
+
+Here is an example of how to configure Open Graph tags in your Anubis setup:
+
+```sh
+export OG_PASSTHROUGH=true
+export OG_EXPIRY_TIME=1h
+```
+
+With these settings, Anubis will cache Open Graph tags for 1 hour and pass them through to the challenge page.
+
+For more information, refer to the [installation guide](../installation).
diff --git a/docs/docs/admin/installation.mdx b/docs/docs/admin/installation.mdx
index adf5cc9..c819e09 100644
--- a/docs/docs/admin/installation.mdx
+++ b/docs/docs/admin/installation.mdx
@@ -53,12 +53,16 @@ Anubis uses these environment variables for configuration:
 | `ED25519_PRIVATE_KEY_HEX_FILE` | unset                   | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set.                                                                                                                                                                         |
 | `METRICS_BIND`                 | `:9090`                 | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information.                                                                                                                                                                                           |
 | `METRICS_BIND_NETWORK`         | `tcp`                   | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information.                                                                                                                                                                                   |
-| `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                |
+| `OG_EXPIRY_TIME`               | `24h`                   | The expiration time for the Open Graph tag cache.                                                                                                                                                                                                                                        |
+| `OG_PASSTHROUGH`               | `false`                 | If set to `true`, Anubis will enable Open Graph tag passthrough.                                                                                                                                                                                                                         |
 | `POLICY_FNAME`                 | unset                   | The file containing [bot policy configuration](./policies.md). See the bot policy documentation for more details. If unset, the default bot policy configuration is used.                                                                                                                |
 | `SERVE_ROBOTS_TXT`             | `false`                 | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file. |
+| `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                |
 | `TARGET`                       | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`.                                                                                                                             |
 | `USE_REMOTE_ADDRESS`           | unset                   | If set to `true`, Anubis will take the client's IP from the network socket. For production deployments, it is expected that a reverse proxy is used in front of Anubis, which pass the IP using headers, instead.                                                                        |
 
+For more detailed information on configuring Open Graph tags, please refer to the [Open Graph Configuration](./configuration/open-graph.mdx) page.
+
 ### Key generation
 
 To generate an ed25519 private key, you can use this command:
@@ -86,6 +90,8 @@ services:
       SERVE_ROBOTS_TXT: "true"
       TARGET: "http://nginx"
       POLICY_FNAME: "/data/cfg/botPolicy.json"
+      OG_PASSTHROUGH: "true"
+      OG_EXPIRY_TIME: "24h"
     ports:
       - 8080:8080
     volumes:
@@ -122,6 +128,10 @@ containers:
         value: "true"
       - name: "TARGET"
         value: "http://localhost:5000"
+      - name: "OG_PASSTHROUGH"
+        value: "true"
+      - name: "OG_EXPIRY_TIME"
+        value: "24h"
     resources:
       limits:
         cpu: 500m