hottub/docs/architecture.md

# Hottub Architecture

## Purpose

Hottub is a Rust server that exposes Hot Tub compatible endpoints for channel discovery, video search, uploader lookups, and site-specific proxying. Most work in this repo is adding or repairing a provider module under `src/providers/`.

## Top-Level Structure

- `src/main.rs`: server bootstrap, env loading, database pool, shared requester/cache, route mounting.
- `src/api.rs`: `/api/status`, `/api/videos`, `/api/uploaders`, `/api/test`, `/api/proxies`.
- `src/providers/mod.rs`: provider trait, provider registry, build-time provider selection, status decoration, runtime validation, panic/error guards.
- `src/providers/*.rs`: one module per channel/provider.
- `src/proxy.rs`: route table for `/proxy/...`.
- `src/proxies/*.rs`: redirect/media/thumb proxy implementations.
- `src/videos.rs`: request/response payloads, `VideoItem`, `VideoFormat`, `ServerOptions`.
- `src/status.rs`: status/channel/group payloads.
- `src/uploaders.rs`: uploader request/profile payloads.
- `src/util/requester.rs`: outbound HTTP with cookies, optional Burp proxying, Jina fallback, and FlareSolverr fallback.
- `build.rs`: compile-time provider registry generation and single-provider build support.

## Startup Flow

1. `main` loads `.env` and ensures `RUST_LOG` is set.
2. It creates the Diesel SQLite pool from `DATABASE_URL`.
3. It creates a shared `Requester`, enables Burp proxying when `PROXY != 0`, and builds the LRU video cache.
4. It starts the public-proxy pool (`util::proxy_pool::start`) unless `PROXY_POOL=0`.
5. It configures provider runtime validation in `providers::configure_runtime_validation`.
6. It spawns a background thread that forces provider initialization via `providers::init_providers_now()`.
7. It starts an `ntex` HTTP server on `0.0.0.0:18080`.

## Runtime Environment

Important environment variables:

- `DATABASE_URL`: required SQLite path.
- `RUST_LOG`: defaults to `warn` if unset.
- `PROXY`: enables Burp proxying when not equal to `0`.
- `BURP_URL`: outbound proxy URL used when `PROXY` is enabled.
- `FLARE_URL`: FlareSolverr endpoint used as the last HTML-fetch fallback.
- `PROXY_POOL`: `0` disables the public-proxy pool (default on).
- `PROXY_POOL_LISTS`: comma-separated proxy list URLs replacing the default TheSpeedX lists.
- `PROXY_POOL_CONCURRENCY`: cap on concurrent proxy checks (default 500).
- `DOMAIN`: used by the `/` redirect target.
- `DISCORD_WEBHOOK`: enables `/api/test` and provider error reporting.

Bundled reference material:

- `docs/hottubapp/📡 Status - Hot Tub Docs.html`
- `docs/hottubapp/🎬 Videos - Hot Tub Docs.html`
- `docs/hottubapp/👤 Uploaders - Hot Tub Docs.html`

Those HTML files are useful when a provider author needs to confirm the expected client payload shape without reading Rust structs first.

## Build-Time Provider Selection

`build.rs` reads `HOT_TUB_PROVIDER` or `HOTTUB_PROVIDER`.

- If unset, every provider in `build.rs` is compiled and registered.
- If set, only that provider is compiled into the binary.
- In a single-provider build, `/api/videos` remaps `"channel": "all"` to the compiled provider.

Generated files in `OUT_DIR` are included by `src/providers/mod.rs`:

- `provider_modules.rs`
- `provider_registry.rs`
- `provider_metadata_fn.rs`
- `provider_selection.rs`

This means adding a new provider always requires updating `build.rs`.

## HTTP Surface

### `/`

Returns a `302` redirect to `hottub://source?url=<DOMAIN-or-request-host>`.

### `/api/status`

Builds the channel list by iterating `ALL_PROVIDERS` and calling `Provider::get_channel`.

Important behavior:

- The `User-Agent` is parsed into `ClientVersion`.
- A provider can hide itself by returning `None`.
- `providers::build_status_response` decorates channels with `groupKey`, top tags, runtime status, and sort order.
- Some heavy status filters are intentionally removed from the client-facing response. The server still accepts them in `/api/videos`.

### `/api/videos`

This is the main provider execution path.

Flow:

1. Parse `VideosRequest`.
2. Normalize `channel`, `sort`, `query`, `page`, and `perPage`.
3. Build `ServerOptions`.
4. If `query` is a full `http://` or `https://` URL, try the `yt-dlp -J` fast path first.
5. Otherwise call `provider.get_videos(...)` through `run_provider_guarded`.
6. For quoted queries like `"teacher"`, apply a literal substring filter after provider fetch.
7. Spawn a background prefetch for the next page.
8. For short videos (`duration <= 120`), populate `preview` from the main URL or first format.

Important behavior:

- Leading `#` is stripped from queries before provider dispatch.
- `"all"` uses `AllProvider` in a normal build, but resolves to the single compiled provider in a single-provider build.
- Older `Hot Tub/38` clients are patched by replacing `video.url` with the last format URL when formats exist.

### `/api/uploaders`

Uploader lookup is optional and provider-specific.

Important behavior:

- At least one of `uploaderId` or `uploaderName` is required.
- If `uploaderId` looks like `channel:id`, the server directly targets that provider.
- Otherwise it scans all providers and returns the best exact-name match.
- Only `hsex`, `omgxxx`, and `vjav` currently implement `get_uploader`.
- In practice, provider-owned uploader IDs should be namespaced, for example `vjav:12345` or `hsex:author_slug`.

### `/api/test`

Sends a Discord error test if `DISCORD_WEBHOOK` is configured.

### `/api/proxies`

Status of the public-proxy pool in `src/util/proxy_pool.rs`: the last list sweep, the fastest exit IPs, and per site (for example `rule34video.com`) the proxies ready for it with their request counters (`ok`/`failed`/`throttled`) and any 429 cooldown. Proxy credentials are never shown.

The pool downloads public proxy lists hourly and keeps only proxies that pass a TLS-verified check against `cdn-cgi/trace` twice with a median under 1.5s, one per exit IP. Providers register a site (`proxy_pool::register_site`) with a cheap check URL; the pool checks its candidates against that URL and rotates requests over the ready ones (`next_proxy` / `report`). rule34video spreads its detail-page enrichment over it to stay under the site's per-IP 429 limit.

## Core Data Structures

### `VideosRequest`

Defined in `src/videos.rs`. Common fields used by providers:

- `channel`
- `sort`
- `query`
- `page`
- `perPage`
- `featured`
- `category`
- `sites`
- `all_provider_sites`
- `filter`
- `language`
- `networks`
- `stars`
- `categories`
- `duration`
- `sexuality`

### `ServerOptions`

The server’s normalized option bag. Providers should read from this instead of reparsing the raw API request.

Important fields:

- `public_url_base`: needed when generating `/proxy/...` URLs.
- `requester`: the shared request client with cookies/debug trace/proxy state.
- `sort`, `sites`, `filter`, `category`, `language`, `network`, `stars`, `categories`, `duration`, `sexuality`.

### `VideoItem`

Minimum useful fields for a provider:

- `id`
- `title`
- `url`
- `channel`
- `thumb`
- `duration`

High-value optional fields:

- `views`
- `rating`
- `uploader`
- `uploaderUrl`
- `uploaderId`
- `tags`
- `uploadedAt`
- `formats`
- `preview`
- `aspectRatio`

Avoid setting `embed` for new providers unless the site truly needs it.

### `VideoFormat`

Use `formats` when:

- the site returns a better direct media URL than the page URL
- HLS or multiple qualities exist
- extra HTTP headers such as `Referer` are required

Use `http_header` or `add_http_header` when the player endpoint needs request headers.

### `Channel` and `ChannelOption`

Each provider’s `get_channel` returns the status metadata exposed by `/api/status`.

Typical option IDs used across the repo:

- `sort`
- `filter`
- `sites`
- `category`
- `language`
- `networks`
- `stars`
- `categories`

Use the same IDs when possible so the server and client behavior stay consistent.

### `UploaderProfile`

If a provider supports `/api/uploaders`, keep the ID routable:

- preferred format: `<channel>:<site-local-id>`
- examples in the repo: `vjav:<user_id>`, `hsex:<author>`, `omgxxx:<kind>:<id>`

This lets `src/api.rs` derive the owning provider immediately.

## Provider Contract

Defined in `src/providers/mod.rs`:

- `async fn get_videos(...) -> Vec<VideoItem>`
- `fn get_channel(clientversion: ClientVersion) -> Option<Channel>`
- `async fn get_uploader(...) -> Result<Option<UploaderProfile>, String>` optional

The server wraps provider execution in:

- `run_provider_guarded` for video paths
- `run_uploader_provider_guarded` for uploader paths

Panics and reported errors trigger runtime validation and optional Discord reporting.

## Runtime Validation and Error Handling

`src/providers/mod.rs` includes a validation subsystem that:

- runs a small sample request against a provider after failures
- checks that enough video items exist
- tries media URLs or format URLs with a `Range` header
- marks repeated failures over time

This means a provider that returns page URLs but no real media/formats may pass visually but still fail operationally.

## Requester Behavior

`src/util/requester.rs` is the standard outbound HTTP layer.

Capabilities:

- shared cookie jar across clones
- optional Burp proxying via `PROXY` and `BURP_URL`
- direct request retries for `429`
- Jina mirror fallback for blocked HTML fetches
- FlareSolverr fallback via `FLARE_URL`
- raw response helpers for media validation and custom headers

Use the shared requester from `ServerOptions` through `requester_or_default`. Do not instantiate a brand-new requester in normal provider fetch paths unless you have a very specific reason.

FlareSolverr note:

- `src/util/flaresolverr.rs` keeps a reusable session pool pattern by rotating a ready session per solve.
- If a provider only works after anti-bot negotiation, the shared requester is the path that benefits from that solved session and cookie state.

## Proxy Subsystem

There are two proxy styles.

### Redirect proxies

These take a provider-specific endpoint and return `302 Location: <resolved-media-url>`.

Examples:

- `/proxy/spankbang/...`
- `/proxy/sxyprn/...`
- `/proxy/pornhd3x/...`
- `/proxy/vjav/...`

### Media or image proxies

These actively fetch media or thumbnails and stream or rewrite the response.

Examples:

- `/proxy/noodlemagazine/...`
- `/proxy/noodlemagazine-thumb/...`
- `/proxy/shooshtime-media/...`
- `/proxy/hanime-cdn/...`

If a site only needs a referer-preserving redirect, use a redirect proxy. If manifests, relative playlist entries, cookies, or binary thumbs need rewriting, use a media/image proxy.

## Best Existing Templates

Use the closest existing provider instead of inventing a new style.

- `src/providers/vjav.rs`: rich API-backed provider with tags, uploader support, and detail enrichment.
- `src/providers/hsex.rs`: HTML scraping with background-loaded filters, uploader support, and direct HLS formats.
- `src/providers/omgxxx.rs`: large filter catalogs and uploader lookup by site/network identity.
- `src/providers/noodlemagazine.rs`: proxied media/thumbs, Jina fallback, and mirrored listing parsing.
- `src/providers/pornhd3x.rs`: complex filter catalogs, detail enrichment, and proxy-generated playback URLs.
- `src/providers/spankbang.rs`: anti-bot handling and a redirect-proxy-based media strategy.

## Important Gotchas

- New providers must export `CHANNEL_METADATA`.
- New providers must be listed in `build.rs` or they will never compile into the registry.
- If a provider returns proxied URLs, it usually also needs `options.public_url_base`.
- Keep filter IDs stable. The `title` is for display; the `id` is what the provider matches on.
- `categories` in `Channel` are not the same as `ChannelOption { id: "categories" }`.
- `/api/status` sanitizes some options away from the client-facing payload. That does not mean the provider option is useless in `/api/videos`.
- If a site needs per-request cookies or a solved user agent, rely on the shared requester.