more providers

This commit is contained in:
Simon
2026-09-07 15:25:03 +00:00
parent 5941a6e607
commit 86c8faca44
10 changed files with 4432 additions and 1 deletions

View File

@@ -2,6 +2,40 @@
This is the implementation checklist for adding a working channel with the least guessing.
## Best Practices (Living)
These rules cut across every step below. They are intentionally grouped at the top so the rest of the playbook can reference them, and they are **living** — add, refine, or reorder as the codebase learns new failure modes. The provider catalog (`docs/provider-catalog.md`) is the ground truth for which patterns have actually held up in production; if a rule here contradicts a catalog note, the catalog wins and this list gets updated.
### Investigation
- **Always use an up-to-date client with a real TLS fingerprint when investigating a site.** A `curl` paste from a terminal looks fine and then 403s the moment the real fetcher goes near it. Use the same `wreq` emulation stack the server ships with (currently `Emulation::Firefox151` in the shared `Requester`, `Emulation::Firefox136` / `Chrome120` in older per-provider paths) and confirm a plain `wreq` GET against the live host returns 200 *before* trusting any URL or header you read off a page. If the investigation client differs from the production client, every "obvious" header you copy is suspect.
- Start from the listing page, not the detail page or the network tab of a single video. The listing tells you the real card shape, pagination, sort, and tag surfaces. Detail pages lie about what is enumerable.
- Confirm each URL shape with at least two pages: home, search page 2, one tag archive, one uploader archive, one detail. A pattern that holds for one page often breaks on the second.
- Record a real browser session only when `wreq` cannot reach the site. If the shared requester plus Jina/FlareSolverr can fetch the page, the production path is the one you are investigating.
### Parser Discipline
- Prefer `serde_json` over regex/HTML scraping whenever the site exposes a JSON shape (Next.js `__NEXT_DATA__`, JSON-LD, hydration blobs, `RSC` payloads). When the site ships an API, scrape the API, not the rendered DOM.
- Treat card metadata as the source of truth for `id`, `title`, `url`, `thumb`, `duration`. Only enrich detail pages for fields the card does not expose, and bound the concurrency (`futures::stream` + `buffer_unordered`).
- Keep `title` text, `id` slug, and uploader identity as separate fields. A mushed title is a debugging nightmare.
### Networking
- Use the shared `Requester` from `ServerOptions`. Local clients drop cookies, lose Burp proxying, and bypass the Jina / FlareSolverr fallbacks. The only legitimate reason to build a fresh `wreq::Client` is a one-off TLS requirement that the shared emulation cannot meet (e.g. a specific Chrome JA3 that the r2.1hanime.com / nhplayer path needs).
- After every fix to a CF/JA3 block, re-validate with `wreq` on the production emulation, not with `curl`. A/B probe with both the old and new fingerprint so you do not confuse a coincidence with a fix.
### Output Shape
- Use `formats` for real media URLs (HLS, multi-quality, signed token), and keep `video.url` on a stable page URL. Never set `video.url` to a tokenized stream that needs a `Referer` to play — the `url` field has no header contract.
- If thumbnails need a `Referer` and the player has no per-thumb header mechanism, proxy them via `/proxy/<id>-thumb/...`. The same applies to manifests with relative URIs.
- Keep filter `id` values stable and machine-targeted. Display `title`s change; ids route.
### Operational
- Add new providers to `build.rs` first, then write the file. A missing entry compiles fine and silently drops the channel.
- After any code change, run `cargo check -q` once for the full build, then `HOT_TUB_PROVIDER=<id> cargo check -q` for the single-provider build. If only the second runs, you will not catch a regression that lives in another provider.
- After modifying the code, refresh the knowledge graph with `graphify update .` so the next investigator can find what you just added without re-deriving it.
## Definition Of Done
A provider is not done when it compiles. It is done when: