Compare commits

...

33 Commits

Author SHA1 Message Date
Simon
cf0ae15f93 hanime: play through a redirect proxy; the signed stream links expire
Items now carry /proxy/hanime/{slug}/{height}.m3u8, which does a fresh
handshake per play and 302s to the current playlist. The handshake moved
to util/hanime_resolve.rs so single-provider builds keep the route.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-11-01 07:48:35 +00:00
Simon
aa00abdd5b rule34video: probe the space-less artist slug (e.g. /models/biggreen/) for multi-word queries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-27 21:45:09 +00:00
Simon
e5a2afcc70 eporner: stream through the server; the signed CDN links are IP-locked
The redirect proxy from 6ba65f1 still played the one-minute "watch on
eporner.com" clip away from home: checked from foreign networks
(check-host.net nodes in BR/DE/HU), a freshly signed link answers 302 ->
static.eporner.com/na.mp4, while the IP that resolved it gets the file. A
redirect therefore only works for a player behind the server's public IP.

/proxy/eporner/{alnum_id}/{height}.mp4 now streams the mp4: resolves the
link (cached 90 min, re-resolved when the CDN answers with a redirect),
forwards Range, and returns a sized body so 206 responses keep
Content-Length. It is no longer a redirect proxy, so app clients keep the
URLs in formats. Verified locally: HEAD, 0-1, mid-file and open-ended ranges
return 206 with Content-Length/Content-Range and valid MP4 data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-22 20:46:14 +00:00
Simon
6ba65f1457 eporner: play through a redirect proxy; the signed CDN links expire after 2h
Expired eporner mp4 links are answered with a 302 to static.eporner.com/
na.mp4, a one-minute "watch it on eporner.com" clip. The links expire
exactly two hours after the /xhr/video call, so anything played from a
favourite, history entry or a list left open past that hit the decoy.

New /proxy/eporner/{alnum_id}/{height} re-resolves the hash/xhr chain at
play time and 302s to the matching (else best) quality. The eporner
provider and melonstube's eporner destinations now emit these proxy URLs;
verified app clients get the best quality as video.url via the existing
redirect-proxy rule. Verified live: proxy -> fresh CDN link -> 206 video/mp4
for app and non-app clients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-22 18:09:23 +00:00
Simon
b131910ea2 rule34video: make queries robust (single-page results, safe search paths, 429-proof enrichment)
- Parser no longer requires a pagination block: single-page results
  (small tags, short searches) used to parse to zero items.
- Search path is slugged like the site's own form: "2b/9s" 404'd, "%20"
  is refused with 403 and "&" truncated the query.
- Probed archives need >= 4 videos to beat keyword search ("elf girl" was a
  one-video tag); conclusive query -> target resolutions are remembered for 6h
  so later pages and the prefetch skip the probes.
- Enriched details are cached per video URL for 2h (tokens last ~12h);
  detail fetches go through get_probe so a lasting 429 stops the batch instead
  of escalating each page to Jina/FlareSolverr. Archive failures serve stale
  cache.
- requester: get_probe waits before retrying 429/5xx and compares redirect
  paths percent-decoded (non-ASCII archives like /tags/pokémon-species/ were
  treated as redirected away); get_with_headers backs off 1/2/4s on 429.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-21 20:17:58 +00:00
Simon
921ad93149 requester: treat DDoS-Guard interstitials as challenges (FlareSolverr fallback); mock-origin test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 20:38:17 +00:00
Simon
ad2abe0057 rule34video: artist, category and tag archives; resolve bare queries by probing; credit artists as uploader
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 17:06:00 +00:00
Simon
0aa8929f29 Docs: the redirect-proxy rule applies to verified Hot Tub app clients only
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:53:08 +00:00
Simon
6c69b46395 Move redirecting-proxy formats to video.url for verified app clients; document the rule
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:48:34 +00:00
Simon
f900987434 archivebate: verified app clients get the proxy URL as item url and no formats
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:39:48 +00:00
Simon
30271b9b7c archivebate: app clients get resolved media URLs with headers, resolved concurrently
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-18 19:30:55 +00:00
Simon
a4364e5be3 archivebate: keep the proxied format when enrichment times out; skip dead search candidates
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 19:54:49 +00:00
Simon
edf6732f8e Query resolution todo, slug helper tests, fix txxx decoder test fixture
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 06:19:18 +00:00
Simon
697836b914 coomer: creator feeds are a bare array; single-word queries that are creator ids go to the creator 2026-10-17 06:16:24 +00:00
Simon
5598216381 sextb: probe actress/studio for names outside the loaded catalogs 2026-10-17 06:07:34 +00:00
Simon
4c29d27b3e redgifs: single-word queries that are creator names go to the creator 2026-10-17 06:05:54 +00:00
Simon
a486bc5502 erome: single-word queries that are uploader names go to the uploader 2026-10-17 06:03:58 +00:00
Simon
270b327f73 fikfap: single-word queries that are profile names go to the creator 2026-10-17 06:01:34 +00:00
Simon
813e2ec2ca clapdat: resolve tag/user names without prefix 2026-10-17 05:58:53 +00:00
Simon
26ef830f6e pornhub: probe pornstar/model/channel for unknown free-form queries 2026-10-17 05:56:31 +00:00
Simon
e34801f6f8 thepornbunny: resolve pornstar/studio/tag names; get_probe treats redirect-away as a miss 2026-10-17 05:53:43 +00:00
Simon
2646ca5659 xgroovy: resolve pornstar/channel names without prefix 2026-10-17 05:51:40 +00:00
Simon
981ec29ad8 xhamster: resolve pornstar/channel names without prefix 2026-10-17 05:50:41 +00:00
Simon
76b9d4e058 xvideos: resolve channel/pornstar names without prefix; channel archives via the JSON video list 2026-10-17 05:48:58 +00:00
Simon
59ea0495a6 porn4fans: resolve model/tag names without prefix 2026-10-17 05:40:28 +00:00
Simon
1f95549167 youporn: resolve pornstar/channel/amateur names without prefix 2026-10-17 05:38:47 +00:00
Simon
1e56fe2da5 tube8: resolve pornstar/channel names without prefix 2026-10-17 05:37:07 +00:00
Simon
bdde02fa8c Add get_probe/probe_first_archive; fullporner and fullpornerorg resolve pornstar names without prefix 2026-10-17 05:34:15 +00:00
Simon
665fb2a0ce hooporn: probe queries as pornstar/studio archives before keyword search
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 05:25:30 +00:00
Simon
99bc80adac Add hooporn provider
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 05:11:17 +00:00
Simon
d72dda5203 pornhub: proxy Referer-gated pix-cdn77 thumbnails
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 21:08:07 +00:00
Simon
6d7b16842d Add txxx provider; cumlouder: route keywords to pornstar /girl/ pages before tags
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 15:47:05 +00:00
Simon
af6674b4ed Route Cloudflare challenges to FlareSolverr on cumlouder and hypnotube raw paths
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 11:59:42 +00:00
41 changed files with 3590 additions and 354 deletions

View File

@@ -361,6 +361,16 @@ const PROVIDERS: &[ProviderDef] = &[
module: "cumlouder",
ty: "CumlouderProvider",
},
ProviderDef {
id: "txxx",
module: "txxx",
ty: "TxxxProvider",
},
ProviderDef {
id: "hooporn",
module: "hooporn",
ty: "HoopornProvider",
},
ProviderDef {
id: "fullporner",
module: "fullporner",

View File

@@ -59,7 +59,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `redtube` | `mainstream-tube` | no | no | Mainstream archive. |
| `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,<link3>,DOOD,<guests>,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). |
| `rule34gen` | `ai` | no | no | AI group example. |
| `rule34video` | `hentai-animation` | no | no | Hentai group example. |
| `rule34video` | `hentai-animation` | no | no | Hentai group example. **Archives:** artists are `/models/{slug}/`, franchises/genres `/categories/{slug}/`, characters and everything else `/tags/{slug}/`; each is paged and sorted through `?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort}&from={page}` (unknown slugs 404). Prefix queries `artist:`/`model:`/`creator:`, `category:`/`franchise:`/`series:`, `tag:`/`character:`; a bare query is probed as artist, then category, then tag, then resolved through `/search_ajax.php?tag=` (characters are tagged `name (franchise)`; an unambiguous exact base-name match wins), else keyword search. Cards are enriched from the detail page, which also yields the credited artist as `uploader` (`rule34video:{slug}`) and artist names as tags. |
| `sextb` | `jav` | no | no | JAV family provider. sextb.net is Cloudflare-gated, so listing/pagination markdown is fetched through a hand-built Jina mirror URL (`mirror_url()`); the target is already an absolute `https://sextb.net/...` URL, so the scheme must be stripped before prepending the `https://r.jina.ai/http://` prefix — concatenating them directly (the pre-fix bug) produces a malformed `.../http://https://...` URL that Jina rejects outright. |
| `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. Listing cards are `div.card` (card→`a.title` for the link/title, `a.image img` for thumb/preview, `div.duration`, `div.list div.box` for views+date, `div.name a.text` for uploader) — the site redesigned from an older `div.item.thumb`/`span.description` layout, so re-verify these selectors against `{BASE_URL}/videos/` (the real default-sort URL, not the bare homepage) if the feed ever goes empty again. |
| `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. |
@@ -81,9 +81,9 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `jable` | `jav` | no | yes | HTML JAV archive scraper; extracts `var hlsUrl` from detail pages; m3u8 format requires Referer + browser User-Agent; proxy route handles HEAD (200 OK) and GET (redirect to watch page) since yt-dlp blocks jable.tv; tag/category/model shortcut queries. |
| `kwiky` | `tiktok` | no | no | Short-form cam clip provider for kwiky.com (ICF/StreamMate platform). Two-step fetch: ID list then metadata batch. Auth: XSRF token extracted from `meta.content = '<TOKEN>'` injected by `/api/state/v1/preload`, cached in an `Arc<RwLock<Option<String>>>` with double-checked locking (same pattern as `xxxtik` bearer token); all API calls also require `x-gateway: 76d59e04-97fa-4ced-aa99-b86ffaf756a2` (platform-identifying fixed UUID observed in browser JS, not stored in cookies or localStorage). Default feed: `GET /api/v1/gateway/v1/quickies/recommended?source=<base64-hotlist>` (cursor-paged, walks up to 5 cursors for higher page numbers). All tag/keyword queries go through `keywordSearch?keyword=...` (offset-paged: `pageToken=(page-1)*perPage`) — the recommended endpoint's `&tag=` param does not reliably filter, so even curated-tag requests use keywordSearch. Media: `media.url` = direct `media.icfcdn.com/*.mp4` (publicly accessible, no Referer/auth), `media.thumbUrl` thumbnail (same CDN, direct), `media.previewUrl` preview clip; both kwiky page URLs (`kwiky.com/quickies/{id}`) and the media CDN URLs pass direct `curl -I` health checks but the page URL is NOT yt-dlp-resolvable (React SPA, generic extractor fails), so `formats` are populated with the direct MP4 URL and `video.url` is the page. Aspect ratio set from `media.width`/`media.height`; `uploadedAt` from RFC3339 `created` field; uploader name/URL/ID from `creator.name` and `creator.id`. 48 curated tags from `/api/v1/gateway/v1/keywords/populartags?gender=f` exposed via `categories` filter option. No proxy needed. |
| `fullporner` | `mainstream-tube` | no | no | HTML scraper for fullporner.com; thumbnail IDs derived from `/thumb/{id}.jpg` URLs and used to build direct `xiaoshenke.net/vid/{id}/720` media redirect URLs (Referer + User-Agent headers required); supports cat:/category:/pornstar:/star: shortcut queries; no proxy needed. |
| `cumlouder` | `mainstream-tube` | no | no | HTML scraper for cumlouder.com (Spanish/amateur tube, same platform family as superporn). Cards are `a.muestra-escena[href^="/porn-video/"]` with lazy thumb in `img[data-src]` (protocol-relative `//th0cdn.cumlouder.com/...`, direct, no Referer/proxy), `h2` title, `.vistas` views, `.minutos` duration (`06:14 m`); relative date only ("2 years ago"), so `uploadedAt` is not set. Default feed `/porn/` (page N≥2 is the bare `/N/` path, not `/porn/N/`); category `/porn-videos/{slug}/{N}/`; tag `/tag/{slug}/{N}/`; series `/series/{slug}/{N}/`. **The site's own `/search/` (POST `form[search]` or `?q=`) is behind a Cloudflare managed challenge — 403 even in a real browser — so keyword queries are routed instead: a query matching a hardcoded category (72, scraped from `/categories/`) goes to the category archive, anything else to `/tag/{slug}/` (unknown tags 404 → empty list).** Supports `cat:`/`category:`, `tag:`, `series:` prefixes and the `categories` option. `video.url` is the detail page, which yt-dlp resolves natively via the generic html5 extractor (the page embeds a signed `m5cdnst.cumlouder.com/*.mp4?secure=…` `<source>`, no Referer needed), so no `formats`/enrichment. The origin intermittently answers a first, uncached request with a Cloudflare 522 (a retry succeeds) — the listing fetch retries up to 3×, and `check.py`/yt-dlp may show a one-off `HTTP=-1`/522 on a cold detail page that is not a provider fault. No proxy needed. |
| `cumlouder` | `mainstream-tube` | no | no | HTML scraper for cumlouder.com (Spanish/amateur tube, same platform family as superporn). Cards are `a.muestra-escena[href^="/porn-video/"]` with lazy thumb in `img[data-src]` (protocol-relative `//th0cdn.cumlouder.com/...`, direct, no Referer/proxy), `h2` title, `.vistas` views, `.minutos` duration (`06:14 m`); relative date only ("2 years ago"), so `uploadedAt` is not set. Default feed `/porn/` (page N≥2 is the bare `/N/` path, not `/porn/N/`); category `/porn-videos/{slug}/{N}/`; tag `/tag/{slug}/{N}/`; series `/series/{slug}/{N}/`. **The site's own `/search/` (POST `form[search]` or `?q=`) is behind a Cloudflare managed challenge — 403 even in a real browser — so keyword queries are routed instead: a query matching a hardcoded category (72, scraped from `/categories/`) goes to the category archive, anything else to `/tag/{slug}/` (unknown tags 404 → empty list).** Pornstar pages live at `/girl/{slug}/{N}/` (same cards/pagination; unknown slug = fast 404). A free-form keyword that is not a category is tried as a pornstar page first, then as a tag archive; `girl:`/`pornstar:`/`star:`/`model:` force the pornstar page. Supports `cat:`/`category:`, `tag:`, `series:` prefixes and the `categories` option. A Cloudflare challenge on the raw listing fetch is handed to the requester's FlareSolverr chain. `video.url` is the detail page, which yt-dlp resolves natively via the generic html5 extractor (the page embeds a signed `m5cdnst.cumlouder.com/*.mp4?secure=…` `<source>`, no Referer needed), so no `formats`/enrichment. The origin intermittently answers a first, uncached request with a Cloudflare 522 (a retry succeeds) — the listing fetch retries up to 3×, and `check.py`/yt-dlp may show a one-off `HTTP=-1`/522 on a cold detail page that is not a provider fault. No proxy needed. |
| `thepornbunny` | `mainstream-tube` | no | yes | KVS-style HTML scraper for thepornbunny.com; 24 items per site page; thumbnails at `https://www.thepornbunny.com/images/thumb/{id}.webp` from `data-original` attribute (no proxy needed); studio exposed as uploader; pornstar names in tags; `/proxy/thepornbunny/{slug}` fetches the video page, extracts `generate_mp4(enc_data, key, rnd, video_id)` args, decrypts `enc_data` via PBKDF2-HMAC-SHA512+AES-256-CBC to get an OK.ru session key, calls `api.ok.ru/fb.do?method=video.get&session_key=KEY&vids=RND` to get signed CDN URLs, and returns 302 to the best-quality okcdn.ru/vkuser.net MP4 URL (no special client headers needed); supports sort: new/popular/rated, 20 hardcoded categories via `categories` option, and tag:/category:/studio:/pornstar: query shortcuts. |
| `eporner` | `mainstream-tube` | no | no | HTML scraper for eporner.com (5M+ videos); card selector `div.mb[data-id]` with inline duration/rating/views/uploader; thumbnails at `static-eu-cdn.eporner.com` (no proxy needed); pagination uses `/{N}/` suffix (page 1 = no suffix, page 2 = `/2/`); supports sort: new/popular/rated/best; 65 hardcoded categories, plus `cat:`/`category:`, `tag:`, `pornstar:`/`star:`, `uploader:`/`profile:` query-prefix routing to the matching archive, a bare-keyword category-name match, and a background-loaded pornstar name→URL map from `/pornstar-list/` (falls back to `/tag/{slug}` keyword search otherwise). `video.url` is the detail page — **not** yt-dlp-resolvable and not directly playable on its own: opening it bare serves a small restricted/preview clip. Real playback needs per-card enrichment (bounded `buffer_unordered(6)`): fetch the detail page for its 32-hex `hash`, derive the site's own hashed token (`calc_hash` — split into four 8-hex chunks, each parsed as u32 and base36-encoded, concatenated; reverse-engineered from the player JS), then call the site's own `GET /xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false` JSON API for direct per-resolution mp4 URLs into `formats[]`. Separately from the hash dance, the resolved CDN mp4 URLs enforce their own Referer check: a request with no Referer or one pointing at eporner.com gets the real file, but any other non-empty Referer gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the exact "this video is only available at eporner.com"-style short clip a client sees if its default (or app-injected) Referer doesn't match, so every format explicitly declares `http_headers: {Referer: "https://www.eporner.com/"}` rather than relying on the client's default. No proxy needed (formats are direct CDN URLs, just header-gated). **2026-09-21:** the `calc_hash`/hash-regex/`alnum_id_from_href`/`resolve_formats` logic moved to `src/util/eporner_resolve.rs` so `melonstube` (a meta-search aggregator that redirects out to eporner.com for ~34% of its own cards) can reuse the exact same resolution without a cross-provider `use`, which would break either provider's `HOT_TUB_PROVIDER=<id>` single-provider build (only the one selected provider module gets compiled; `src/util/*` always does). `eporner.rs` itself now just calls `crate::util::eporner_resolve::resolve_formats(...)`. |
| `eporner` | `mainstream-tube` | no | yes | HTML scraper for eporner.com (5M+ videos); card selector `div.mb[data-id]` with inline duration/rating/views/uploader; thumbnails at `static-eu-cdn.eporner.com` (no proxy needed); pagination uses `/{N}/` suffix (page 1 = no suffix, page 2 = `/2/`); supports sort: new/popular/rated/best; 65 hardcoded categories, plus `cat:`/`category:`, `tag:`, `pornstar:`/`star:`, `uploader:`/`profile:` query-prefix routing to the matching archive, a bare-keyword category-name match, and a background-loaded pornstar name→URL map from `/pornstar-list/` (falls back to `/tag/{slug}` keyword search otherwise). `video.url` is the detail page — **not** yt-dlp-resolvable and not directly playable on its own: opening it bare serves a small restricted/preview clip. Real playback needs per-card enrichment (bounded `buffer_unordered(6)`): fetch the detail page for its 32-hex `hash`, derive the site's own hashed token (`calc_hash` — split into four 8-hex chunks, each parsed as u32 and base36-encoded, concatenated; reverse-engineered from the player JS), then call the site's own `GET /xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false` JSON API for direct per-resolution mp4 URLs into `formats[]`. Separately from the hash dance, the resolved CDN mp4 URLs enforce their own Referer check: a request with no Referer or one pointing at eporner.com gets the real file, but any other non-empty Referer gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the exact "this video is only available at eporner.com"-style short clip a client sees if its default (or app-injected) Referer doesn't match, so every format explicitly declares `http_headers: {Referer: "https://www.eporner.com/"}` rather than relying on the client's default. No proxy needed (formats are direct CDN URLs, just header-gated). **2026-09-21:** the `calc_hash`/hash-regex/`alnum_id_from_href`/`resolve_formats` logic moved to `src/util/eporner_resolve.rs` so `melonstube` (a meta-search aggregator that redirects out to eporner.com for ~34% of its own cards) can reuse the exact same resolution without a cross-provider `use`, which would break either provider's `HOT_TUB_PROVIDER=<id>` single-provider build (only the one selected provider module gets compiled; `src/util/*` always does). `eporner.rs` itself now just calls `crate::util::eporner_resolve::resolve_formats(...)`. **2026-10-22:** the signed CDN URLs expire exactly two hours after the `/xhr/video` call (the leading number in `/{expiry}_{ip}_{n}/`) and an expired link gets the same `na.mp4` decoy, so favourites, history and long-open lists played the one-minute clip. The links are also **IP-locked**: checked from foreign networks (check-host.net nodes) a fresh link answers 302 → `na.mp4`, while the resolving IP gets the file, so a redirect only works for a player behind the server's own public IP (a Jina fetch had misleadingly suggested otherwise). Formats therefore point at the streaming proxy `/proxy/eporner/{alnum_id}/{height}.mp4`: it resolves (cached 90 min per id/height, re-resolved when the CDN answers with a redirect), forwards `Range`, and returns a sized body so `206` responses keep `Content-Length` for AVPlayer. Not a redirect proxy, so app clients keep these in `formats`. All eporner playback bandwidth now flows through the server. melonstube's eporner destinations use the same `resolve_proxied_formats`. |
| `xnxx` | `mainstream-tube` | no | no | HTML scraper for xnxx.com (10M+ videos); unified card parser handles two formats: `div.thumb-block[data-eid]` (search) and `div.thumb-block.video[data-video='{"id":...}']` (hits); eid extracted from `/video-{eid}/{slug}` URL path; thumbnails at `thumb-cdn77.xnxx-cdn.com` and `thumbs-gcore.xnxx-cdn.com` (no proxy, no Referer needed); 0-indexed pagination (page 1 = `/hits`, page N = `/hits/{N-1}`); default feed is `/hits` (most-viewed — xnxx has no chronological listing); search via `/search/{slug}` (works for keywords and tags); supports `tag:`, `cat:`, `category:` query shortcuts; yt-dlp resolves `video.url` natively (XNXX extractor, returns 4-7 HLS formats); no proxy needed. |
| `xgroovy` | `mainstream-tube` | no | no | KVS (Kernel Video Sharing)-style HTML scraper for xgroovy.com; card selector `div.item[data-video-id]` with `data-views`/`data-rating` as clean numeric attributes directly on the card (no text-parsing needed, unlike xhamster); title/url from `a.popito[href] > strong.title`; thumbnails at `i.xgroovy.com` (no proxy, no Referer needed) from `img.thumb[data-jpg|src]`, preview clip from `img[data-preview]`; duration rendered as free text ("N min"/"N sec", not colon-separated) via a bespoke `parse_duration` helper; path-segment pagination (`/new/` = page 1, `/new/2/` = page 2, unlike xhamster's `?page=N` query string); feeds: `/new/` (default), categories via `/categories/{slug}/` (73 hardcoded categories as `categories` option), pornstars via `/pornstars/{slug}/`, channels via `/channels/{slug}/`; search via `/search/{dashed-query}/` (multi-word queries are lowercased and dash-joined); uploader kind inferred from URL path (`/pornstars/` vs `/channels/`) via `div.author-link > a`; supports `cat:`/`category:` query shortcuts plus static category name matching; yt-dlp resolves `video.url` via the generic html5-embed extractor (no dedicated xgroovy extractor, but resolves cleanly to a 1080p format); no proxy needed. |
| `xhamster` | `mainstream-tube` | no | no | HTML scraper for xhamster.com; card selector `div[data-video-type="video"]` with `data-video-id`; thumbnails via `img[data-role="thumb-preview-img"]` at `ic-vt-nss.xhcdn.com` (no proxy, no Referer needed); pagination via `?page=N` query param (browse feeds use infinite-scroll so only search reliably returns different content per page); feeds: `/newest` (default), `/most-viewed`, `/best`; categories via `/categories/{slug}`; channels via `/channels/{slug}`; 43 hardcoded categories as `categories` option; uploader type inferred from URL path (`/channels/` → channel, `/creators/` → creator, `/pornstars/` → pornstar); supports `cat:`/`category:` and `channel:` query shortcuts, plus static category name matching; preview mp4 clips from `data-previewvideo` attribute; yt-dlp resolves `video.url` natively (xHamster extractor, 28 formats); no proxy needed. |
@@ -92,6 +92,8 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `wowxxx` | `studio-network` | no | no | HTML scraper for the wowxxx.to premium aggregator (wow.xxx 301-redirects here now; `BASE_URL` points at the canonical domain directly). The site is Cloudflare-protected — `fetch_html` must NOT pin `Version::HTTP_11` (removed; now `None`), since forcing h1 while the TLS handshake still advertises the browser emulation's h2 ALPN is exactly the mismatch documented for jable.tv's CF gate. Default feed `/latest-updates/`, page 2 `/{N}/` suffix (for example `/latest-updates/2/`), search `/search/{query}/relevance/` with the same page suffix; supports `site:`/`studio:`/`network:`/`model:`/`pornstar:`/`tag:`/`cat:` query shortcuts to direct archive routes; list cards expose preview clips (`cast.wowxxx.to/preview/*.mp4`), thumbnails (`img.wowxxx.to/.../medium@2x/1.jpg`), duration, rating, views, site (as uploader), and model tags; `video.url` is the detail page URL and yt-dlp resolves HTML5 MP4 formats dynamically; no proxy needed. `www.wowxxx.to`/`img.wowxxx.to` are in `check.py`'s CF-protected allowlist since a plain probe still 403s even though the app's own client succeeds. |
| `xlecx` | `hentai-animation` | no | no | DLE (DataLife Engine)-style HTML scraper for xlecx.one, following the `thaiporntv` template. Latest feed `/f/cat={id}/sort={mode}/order=desc/` (page N≥2 appends `page/{N}/`); `sort` options new/popular/commented/rated map to DLE's `date`/`news_read`/`comm_num`/`rating` sort params; `category` option is the 2 hardcoded top-level archives (`videos`, `hentai-video`). Search via `/index.php?do=search&subaction=search&story={query}` (page N adds `search_start={N}`). Tag shortcut (`#slug` query) routes to `/tags/{slug}/` (paginated via `/tags/{slug}/page/{N}/`); the ~1,044-entry tag catalog is background-loaded once at startup from `/tags/` (`.tags-list1 a[href*='/tags/']`, absolute-URL anchors) into an `Arc<RwLock<Vec<FilterOption>>>` and exposed as the `filter`/"Tags" `ChannelOption` — note this option (and any `categories`) is deliberately stripped from the live `/api/status` response by `should_hide_status_option`/`decorate_channel` in `providers/mod.rs` (any `id=="filter"` option titled "Tags" is hidden site-wide), so its absence there is expected, not a bug; it is still honored by `/api/videos`. Cards parsed from `h3.thumb__title` + `img` inside standard DLE thumb blocks; id extracted from the `/{id}-{slug}.html` URL suffix. Thumbnails are direct `xlecx.one/uploads/posts/...` webp (no proxy/Referer needed). `video.url` is the detail page URL, which yt-dlp resolves natively via the generic HTML5 embed extractor; no `formats`/proxy needed. No `/api/uploaders` (no stable uploader identity on the site). |
| `xxxtik` | `tiktok` | yes | no | JSON-API short-form aggregator for xxxtik.com — every post is a moderated repost of a RedGifs clip, so the real media backend is the public **RedGifs v2 API**, not xxxtik itself (new pattern; no other provider currently resolves through a third-party media API). Listing API (`xxxtik-api-iw98m.ondigitalocean.app`, found by grepping the Angular `main-es2015.js` bundle): `GET /post/new`, `/post/top/{week,month,year,all}`, `/post/tag/{name}`, `/post/creator/{username}` — all **cursor**-paginated (`?cursor={lastItemId}&limit=N`; the `cursor` is the numeric `id` of the last item from the previous batch, not an offset/count — reaching page N walks N sequential requests, mirroring `fikfap`'s `fetch_cursor_page`). `GET /search?query=Q` returns tag/profile autocomplete suggestions only (no posts), so free-text search is routed through it to resolve a `Tag`/`Creator` target before a second listing call; `tag:`/`category:`/`cat:` and `user:`/`uploader:`/`creator:` query prefixes skip that lookup. Each post's `source` field is a `redgifs.com/watch/{id}` URL; the provider fetches a short-lived anonymous bearer token from `POST api.redgifs.com/v2/auth/temporary` (cached, refreshed once on a 401) and resolves `GET api.redgifs.com/v2/gifs/{id}` (bounded to 8-way concurrency via `buffer_unordered`) for the real `media.redgifs.com/*.mp4` + poster, both fetchable with zero auth/Referer. `video.url` is the xxxtik page (`https://xxxtik.com/feed/{uuid}`, not yt-dlp-resolvable — Angular SPA, generic extractor fails), with `formats` populated from the resolved redgifs mp4; tags merge xxxtik's own tags with RedGifs' tags. 20 curated tags exposed via `categories` (xxxtik has ~62k tags total, too many to background-load). `/api/uploaders` works via `GET /user/by-username/{name}` + `/post/creator/{name}`, but xxxtik's "creator" accounts are inconsistent: some (e.g. `dlhoodninja`, `besttits`) return real posts matching their profile `_count.posts`; others with a nonzero `_count.posts` (e.g. `bigboobsgw`, count 472) return an empty `/post/creator/` list — likely synthetic curation accounts (`name@default` emails) rather than real uploaders. The provider degrades gracefully (returns the profile with `videos: []`, no error) rather than guessing which accounts are "real". No proxy needed — all media/thumb URLs are publicly fetchable with no Referer or auth. |
| `txxx` | `mainstream-tube` | no | no | JSON-API scraper for txxx.me (mirror of txxx.com; Vue SPA, same KVS-based API family as the hclips/upornia/hdzog network). Listing: `GET /api/json/videos2/86400/str/{sort}/{count}/{section}.{object}.{page}.all...jsond` with sort `latest-updates`/`most-popular`/`top-rated`/`longest`; **the `.json` variant ignores the page number, `jsond` does not**. Category archive = section `categories`, object = category dir slug (numeric ids return `category_not_found`); ~80 top categories hardcoded as the `categories` option (225 exist at `/api/json/categories/86400/str.all.en.jsond`). Search: `GET /api/videos2.php?params=86400/str/relevance/{count}/search..{page}.all..&s={query}` (relevance only; sort is ignored; page works although `params.page` echoes 1). `count` follows `perPage` (max 60). The page URL `/videos/{id}/{dir}/` is **not** yt-dlp-resolvable, so every card is enriched (bounded `buffered(8)`) via `GET /api/videofile.php?video_id={id}&lifetime=8640000`, whose `video_url` is base64 of `/get_file/...` with Cyrillic look-alike letters (А Е М С …) substituted and `~` for `=`; the decoded `https://txxx.me/get_file/...` 302s to a per-client signed `*.ahcdn.com` URL (IP-pinned, so it must NOT be resolved server-side; no Referer needed). Items without formats are dropped. Heights come from the listing `file_formats` (`||_hq.mp4|1278x720|…||_tr.mp4|…`; `_tr.mp4` is the trailer and becomes `preview` via `pv`). Rating 0..5 → ×20; `likes`/`dislikes` ignored; category titles become tags; uploader = content source else member `display_name` (`txxx:{user_id}`; no profile endpoint). Thumbs `tn.txxx.tube` direct. Queries matching a category name route to its archive; `cat:`/`category:` prefix. No models/channels archives (sections `models`/`channels` are rejected by the API). No proxy needed. |
| `hooporn` | `studio-network` | no | no | HTML scraper for hooporn.com (studio/pornstar tube; Cloudflare-fronted but serves plain 200 to the shared requester). Cards `div.video-card` with lazy thumb in `img[data-src]` (`//imgs.hooporn.com/thumb/...jpg?v2`, direct, no Referer), title in `div.video-title a`, `.time` duration, `span.create` unix timestamp → `uploadedAt`; no views/rating. Feed `/home/{N}`, category `/category/{slug}/{N}` (63 hardcoded from `/categories`), pornstar `/pornstar/{slug}/{N}`, studio `/studio/{slug}/{N}` (studio slugs are concatenated, e.g. `familystrokes`), search `/search?q=…&p={N}` (24 per page; `?search=`/`/search/q/N` do not work). Prefix queries `cat:`/`category:`, `pornstar:`/`star:`/`model:`, `studio:`/`site:`; a bare keyword matching a category name routes to the archive; any other query is first probed on the site (`/pornstar/{slug}/1`, then `/studio/{slug-without-dashes}/1`; unknown archives are an instant 404 read through the raw response, Cloudflare challenges are handed to the requester/FlareSolverr chain) and used as that archive if it exists, otherwise it is a keyword search. The probe runs on every page, so page N of a pornstar/studio query stays in that archive. The watch page `/watch/{24hex}` is **not** yt-dlp-resolvable (player is an iframe), and the mp4 is signed (`videos.trendyporn.com/...mp4?md5=…&expires=…`, ~1 day, 403 without the signature, no Referer/IP binding), so every card is enriched (`buffered(8)`): watch page → `iframe[src*="/video/"]` (`//vid.hooporn.com/video/{n}`) → `var video = "…"` into one `formats[]` entry (quality unknown, labelled `auto`); the watch page also yields category tags, pornstar names (as tags) and the studio (uploader `hooporn:{studio-slug}`). Items whose enrichment fails are dropped. No proxy needed. |
## Proxy Routes
@@ -124,6 +126,7 @@ These resolve a provider-specific input into a `302 Location`.
These return binary media or images, sometimes rewriting manifests or forwarding cookies/referers.
- `/proxy/shooshtime-media/{endpoint}*`
- `/proxy/eporner/{alnum_id}/{height}.mp4` (streams the IP-locked signed mp4 with Range passthrough; link cached 90 min)
- `/proxy/noodlemagazine/{endpoint}*`
- `/proxy/noodlemagazine-thumb/{endpoint}*`
- `/proxy/hanime-cdn/{endpoint}*`

View File

@@ -84,3 +84,10 @@ impersonation target; if a minimum client version is needed, gate on `ClientVers
`porndish`, `supjav`, `jable`, `fapello`, `missav`, `wowxxx`, `hentaihaven` return Cloudflare 403 to
yt-dlp/curl from this machine while the server itself returns items. `hsex` returned 429 (rate limit).
## Cloudflare challenge rule
Any request path that bypasses the requester's fallback chain must still hand a Cloudflare challenge (403/503 or
"Just a moment...") to FlareSolverr. `cumlouder` (raw 404-aware listing fetch) now falls back to
`get_with_headers`; `hypnotube` (isolated search session) solves via the shared requester and replays the solved
cookies + user agent in a fresh isolated session.

View File

@@ -26,6 +26,7 @@ These rules cut across every step below. They are intentionally grouped at the t
### Output Shape
- **Verified Hot Tub app clients cannot follow redirects in `formats[].url`.** Redirect proxies (`REDIRECT_PROXIES` in `src/proxy.rs`) are for `video.url` in the app response; `api.rs` moves such formats to `url` for verified app clients as a safety net. Other clients keep the redirect-proxy URLs in `formats`.
- Use `formats` for real media URLs (HLS, multi-quality, signed token), and keep `video.url` on a stable page URL. Never set `video.url` to a tokenized stream that needs a `Referer` to play — the `url` field has no header contract.
- If thumbnails need a `Referer` and the player has no per-thumb header mechanism, proxy them via `/proxy/<id>-thumb/...`. The same applies to manifests with relative URIs.
- Keep filter `id` values stable and machine-targeted. Display `title`s change; ids route.

View File

@@ -0,0 +1,42 @@
# Seamless query resolution todo
Goal: a free-form query ("gal ritchie", "family strokes") should reach the matching
pornstar / studio / channel / tag archive **without** a `pornstar:` / `studio:` / `tag:` prefix,
by asking the site (probe) and falling back to keyword search. Prefixes stay as explicit overrides.
Pattern (see `hooporn`, `cumlouder`): after known-category matching, probe page 1 of each entity
archive for the slug with `Requester::get_probe` (404 = instant `None`, Cloudflare challenge goes to
FlareSolverr, 5xx/429 retried once), use the first archive that yields cards (page N stays in that
archive), else keyword search. A probe is only valid after checking what the site answers for an
unknown slug (404 vs soft-404 that lists unrelated videos).
Status: `[ ]` open, `[x]` done and verified live (feed of the archive + unknown name falls back),
`[-]` not applicable (reason).
| | Provider | Prefix kinds today | Plan / evidence |
|---|---|---|---|
| [x] | `hooporn` | cat, pornstar, studio | reference implementation |
| [x] | `cumlouder` | cat, tag, series, girl | reference implementation (girl page, then tag) |
| [x] | `fullporner` | cat, pornstar | probes `/pornstar/{slug}` (unknown = 404 with a suggestion grid, so status decides); `pornstar:` slug now dashed |
| [x] | `fullpornerorg` | pornstar | probes `/videos/pornstar/{slug}/` (404 for unknown) |
| [x] | `tube8` | tag, cat, channel, pornstar | probes pornstar then channel |
| [x] | `youporn` | tag, channel, pornstar, amateur | probes pornstar, channel, amateur (plus existing background shortcut map) |
| [x] | `porn4fans` | tag, cat, model | probes `/models/{slug}/` then `/tags/{slug}/` (both 404 when unknown) |
| [x] | `xvideos` | tag, cat, uploader/channel/profile | probes `/channels/{slug}/videos/new/0` (JSON; pornstar profiles redirect there). **Also fixed**: the old `channel:`/`uploader:` prefix returned nothing (landing page cards are `/prof-video-click/` links); channels are now read from the JSON video list |
| [x] | `xhamster` | cat, channel | added `Pornstar` target; probes pornstar then channel |
| [x] | `xgroovy` | cat, pornstar, channel | probes pornstar then channel |
| [x] | `thepornbunny` | tag, cat, studio, pornstar | probes pornstar, studio, tag (unknown slugs redirect home: `get_probe` now treats a redirect to another path as a miss) |
| [x] | `pornhub` | channel, pornstar, model, user | background catalogs already matched listed names; unknown names are probed (`pornstar`, `model`, `channels`) and cached in `tag_map` (misses remembered) |
| [x] | `clapdat` | tag, user | probes tag then user |
| [x] | `fikfap` | tag, user | single-word query that is an existing profile (profile endpoint) goes to its posts, empty profile falls back to search |
| [x] | `erome` | uploader, user | single-word query that is an existing `/{name}` with albums goes to the uploader |
| [x] | `redgifs` | tag, user | single-word query that is an existing creator (users endpoint) goes to the creator, empty falls back |
| [x] | `sextb` | (catalog matched) | catalog (actress/studio/genre) match first, then probes `/actress/{slug}` and `/studio/{slug}` for names outside the catalog |
| [x] | `coomer` | u:, @service:user | single-word query that is an existing creator id (`/api/v1/{service}/user/{id}/profile`) goes to the creator. **Also fixed**: creator feeds are a bare JSON array, which the parser rejected (every creator feed was empty) |
| [-] | `ripnsfw`, `hsex`, `omgxxx`, `eporner`, `borntobefuck`, `archivebate` | various | already seamless: bare names match background-loaded creator / studio / tag catalogs (or the whole catalog is filtered client-side) |
| [-] | `perfectgirls`, `okxxx`, `pornhat`, `perverzija`, `porntrex` | sites/models/stars | bare names already resolve through the background-loaded `tag_map`; these clones build their own `wreq` client with no `ServerOptions`, so a live probe needs a separate refactor onto the shared requester |
| [-] | `homoxxx` | models, pornstars, channels, categories, tags | legacy provider (own client, unpaginated archives, no `/models/` index on the site); not worth a probe until it is rewritten |
| [-] | `kwiky`, `animeidhentai`, `hentaitv`, `xnxx`, `blowjobspro`, `txxx` | tag/cat | search already is the entity lookup, or the site has no pornstar/studio archives to probe (txxx: the API rejects `models`/`channels` sections) |
| [-] | `fyptt` | cat | by design: bare category words are common search terms and must not be hijacked |
| [-] | `notfans` | tag, category | tag URLs are hashed ids, so an unknown name cannot be probed; search covers it |
| [-] | `allpornstream`, `jable`, `wowxxx`, `hornyleak`, `hotbunny`, `hentaimama`, `tikporn`, `camsoda` | actor/producer, model, studio, ... | Cloudflare-gated or down from this host (403/522, FlareSolverr wedged), so a probe cannot be verified here; each would add one or two FlareSolverr solves per search. Candidates for the same pattern once checked from the production network |

View File

@@ -714,6 +714,15 @@ async fn videos_post(
.collect();
}
// The Hot Tub app cannot follow a redirect in `formats[].url`: for verified
// app clients, formats that point at a redirecting `/proxy/...` become the
// item's `url` instead.
if clientversion.is_verified_hottub() {
for video in video_items.iter_mut() {
move_redirect_formats_to_url(video);
}
}
if let Some(literal_query) = literal_query.as_deref() {
#[cfg(feature = "debug")]
let before = video_items.len();
@@ -794,6 +803,30 @@ async fn videos_post(
Ok(web::HttpResponse::Ok().json(&videos))
}
/// Moves formats that point at a redirecting proxy to `video.url`.
///
/// If direct (non-redirecting) formats remain they are kept and `url` is left
/// alone; otherwise `url` becomes the first (best-listed) redirecting proxy URL
/// and `formats` is dropped entirely.
pub(crate) fn move_redirect_formats_to_url(video: &mut VideoItem) {
let Some(formats) = video.formats.take() else {
return;
};
let (redirecting, direct): (Vec<_>, Vec<_>) = formats
.into_iter()
.partition(|format| crate::proxy::is_redirect_proxy_url(&format.url));
if redirecting.is_empty() {
video.formats = Some(direct);
return;
}
if direct.is_empty() {
video.url = redirecting[0].url.clone();
video.formats = None;
} else {
video.formats = Some(direct);
}
}
async fn uploaders_post(
uploader_request: web::types::Json<UploadersRequest>,
cache: web::types::State<VideoCache>,
@@ -1082,4 +1115,53 @@ mod tests {
assert_eq!(item.uploader.as_deref(), Some("FreeusePorn"));
assert_eq!(item.formats.as_ref().map(|formats| formats.len()), Some(2));
}
fn item_with_formats(urls: &[&str]) -> VideoItem {
let mut item = VideoItem::new(
"id".to_string(),
"t".to_string(),
"https://site.example/watch/1".to_string(),
"c".to_string(),
"https://thumb".to_string(),
10,
);
item.formats = Some(
urls.iter()
.map(|u| crate::videos::VideoFormat::new(u.to_string(), "q".to_string(), "mp4".to_string()))
.collect(),
);
item
}
#[test]
fn redirect_only_formats_move_to_url() {
let mut item = item_with_formats(&[
"http://h/proxy/hqporner/1080",
"http://h/proxy/hqporner/720",
]);
move_redirect_formats_to_url(&mut item);
assert_eq!(item.url, "http://h/proxy/hqporner/1080");
assert!(item.formats.is_none());
}
#[test]
fn direct_formats_are_kept_and_redirect_ones_dropped() {
let mut item = item_with_formats(&[
"http://h/proxy/hqporner/1080",
"https://cdn.example/v.mp4",
]);
move_redirect_formats_to_url(&mut item);
assert_eq!(item.url, "https://site.example/watch/1");
let formats = item.formats.expect("direct format kept");
assert_eq!(formats.len(), 1);
assert_eq!(formats[0].url, "https://cdn.example/v.mp4");
}
#[test]
fn media_proxy_formats_are_untouched() {
let mut item = item_with_formats(&["http://h/proxy/noodlemagazine/x.mp4"]);
move_redirect_formats_to_url(&mut item);
assert_eq!(item.url, "https://site.example/watch/1");
assert_eq!(item.formats.map(|f| f.len()), Some(1));
}
}

View File

@@ -799,6 +799,39 @@ impl ArchivebateProvider {
.map(|value| value.to_string())
}
/// The proxied format needs no network, so an item always plays through
/// it even when the best-effort header enrichment below times out.
fn with_default_format(item: VideoItem, options: &ServerOptions) -> VideoItem {
let format_url = Self::proxied_video(options, &item.url);
if format_url.is_empty() || item.formats.is_some() {
return item;
}
let mut item = item;
item.formats = Some(vec![VideoFormat::new(
format_url,
"source".to_string(),
"mp4".to_string(),
)]);
item
}
/// For verified Hot Tub app clients: no `formats` at all. The item's `url`
/// is the lazy `/proxy/archivebate` redirect, which the app plays directly
/// (it resolves the media at playback time).
fn proxy_as_item_url(videos: Vec<VideoItem>, options: &ServerOptions) -> Vec<VideoItem> {
videos
.into_iter()
.map(|mut video| {
let proxy_url = Self::proxied_video(options, &video.url);
if !proxy_url.is_empty() {
video.url = proxy_url;
}
video.formats = None;
video
})
.collect()
}
async fn enrich_video(&self, item: VideoItem, options: &ServerOptions) -> VideoItem {
let page_url = item.url.clone();
let format_url = Self::proxied_video(options, &page_url);
@@ -1148,11 +1181,15 @@ impl ArchivebateProvider {
platform: candidate.platform.clone(),
gender: candidate.gender.clone(),
};
let items = self
// A candidate whose profile page is gone (the search index lists
// deleted profiles) must not abort the whole search: try the next.
match self
.fetch_items_for_target(cache.clone(), &target, 1, per_page, &options)
.await?;
if !items.is_empty() {
return Ok(items);
.await
{
Ok(items) if !items.is_empty() => return Ok(items),
Ok(_) => {}
Err(_) => {}
}
}
@@ -1188,6 +1225,14 @@ impl Provider for ArchivebateProvider {
if videos.is_empty() {
return videos;
}
let is_app_client = options
.client_version
.as_ref()
.map(ClientVersion::is_verified_hottub)
.unwrap_or(false);
if is_app_client {
return Self::proxy_as_item_url(videos, &options);
}
stream::iter(videos.into_iter().map(|video| {
let provider = self.clone();
let options = options.clone();
@@ -1199,11 +1244,11 @@ impl Provider for ArchivebateProvider {
.await;
match timeout_result {
Ok(enriched) => enriched,
Err(_) => video,
Err(_) => Self::with_default_format(video, &options),
}
}
}))
.buffer_unordered(4)
.buffered(4)
.collect::<Vec<_>>()
.await
}
@@ -1218,3 +1263,52 @@ impl Provider for ArchivebateProvider {
Some(self.build_channel(clientversion))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn options() -> ServerOptions {
ServerOptions {
client_version: None,
featured: None,
category: None,
sites: None,
filter: None,
language: None,
public_url_base: Some("http://hottub.example".to_string()),
requester: None,
network: None,
stars: None,
categories: None,
duration: None,
sort: None,
sexuality: None,
}
}
#[test]
fn app_clients_get_the_proxy_url_and_no_formats() {
let mut item = VideoItem::new(
"archivebate:1".to_string(),
"t".to_string(),
"https://archivebate.com/watch/1".to_string(),
CHANNEL_ID.to_string(),
"https://cdn.example/t.jpg".to_string(),
10,
);
item.formats = Some(vec![VideoFormat::new(
"x".to_string(),
"source".to_string(),
"mp4".to_string(),
)]);
let out = ArchivebateProvider::proxy_as_item_url(vec![item], &options());
assert_eq!(out.len(), 1);
assert_eq!(
out[0].url,
"http://hottub.example/proxy/archivebate/archivebate.com/watch/1"
);
assert!(out[0].formats.is_none());
}
}

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, report_provider_error, requester_or_default, strip_url_scheme,
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
slug_dashed, strip_url_scheme,
};
use crate::status::*;
use crate::util::time::parse_time_to_seconds;
@@ -149,6 +150,29 @@ impl ClapdatProvider {
}
}
/// A free-form query that names a tag or a user goes to that archive
/// (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Tag { slug: slug.clone() },
Target::User { username: slug },
];
let urls: Vec<String> = candidates
.iter()
.filter_map(|t| self.listing_url(t, 1))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("/video/")
})
.await?;
candidates.into_iter().nth(index)
}
fn listing_url(&self, target: &Target, page: u16) -> Option<String> {
let page = page.max(1);
match target {
@@ -491,7 +515,12 @@ impl Provider for ClapdatProvider {
sort
};
let query_value = query.unwrap_or_default();
let target = self.resolve_target(&query_value, &sort_value);
let mut target = self.resolve_target(&query_value, &sort_value);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
let stubs = match &target {
Target::Search { query } => match self

View File

@@ -8,7 +8,7 @@ use wreq::Version;
use crate::{
DbPool,
api::ClientVersion,
providers::{Provider, requester_or_default},
providers::{Provider, probe_first_archive_with, requester_or_default},
status::{Channel, ChannelOption, FilterOption},
util::cache::VideoCache,
videos::{ServerOptions, VideoFormat, VideoItem},
@@ -147,6 +147,43 @@ impl CoomerProvider {
}
}
/// A single-word query that is an existing creator id (the profile
/// endpoint answers 404 otherwise) goes to that creator's posts, trying the
/// filtered service first, else keyword search.
async fn resolve_keyword(
query: &str,
service_filter: Option<&str>,
options: &ServerOptions,
) -> Option<Target> {
let name = query.trim().trim_start_matches('@');
if name.is_empty()
|| !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')
{
return None;
}
let mut services = vec![Service::Onlyfans, Service::Fansly, Service::Candfans];
if let Some(preferred) = Self::resolve_service(service_filter) {
services.retain(|s| s.as_str() != preferred.as_str());
services.insert(0, preferred);
}
let urls: Vec<String> = services
.iter()
.map(|s| format!("{BASE_URL}/api/v1/{}/user/{name}/profile", s.as_str()))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) =
probe_first_archive_with(&mut requester, &urls, Self::json_headers(), |body| {
body.contains("\"id\"")
})
.await?;
Some(Target::Uploader {
service: services[index].clone(),
username: name.to_string(),
})
}
fn json_headers() -> Vec<(String, String)> {
vec![
("Referer".to_string(), REFERER.to_string()),
@@ -290,13 +327,16 @@ impl CoomerProvider {
)));
}
}
let posts: ApiPostsResponse = serde_json::from_str(&body).map_err(|err| {
let posts: ApiPostsPayload = serde_json::from_str(&body).map_err(|err| {
Error::from(format!(
"json parse failed for {url}: {err}; body[:200]={}",
&body.chars().take(200).collect::<String>()
))
})?;
Ok(posts.posts)
Ok(match posts {
ApiPostsPayload::List(posts) => posts,
ApiPostsPayload::Wrapped(wrapped) => wrapped.posts,
})
}
async fn fetch_items(
@@ -543,6 +583,14 @@ struct ApiError {
error: String,
}
/// The global feed wraps posts in `{posts: [...]}`; a creator's feed is a bare array.
#[derive(Debug, Deserialize)]
#[serde(untagged)]
enum ApiPostsPayload {
List(Vec<ApiPost>),
Wrapped(ApiPostsResponse),
}
#[derive(Debug, Deserialize)]
struct ApiPostsResponse {
#[serde(default)]
@@ -601,8 +649,13 @@ impl Provider for CoomerProvider {
.clamp(1, 64);
let service_filter = options.categories.as_deref().or(options.sites.as_deref());
let target =
let mut target =
Self::resolve_target(&sort_value, query.as_deref(), service_filter);
if let Target::Search { query } = &target {
if let Some(found) = Self::resolve_keyword(query, service_filter, &options).await {
target = found;
}
}
// Bump the ignored `Sort` variant warning off by referencing it.
let _ = Sort::New;
@@ -849,4 +902,18 @@ mod tests {
let post: ApiPost = serde_json::from_str(json).expect("parse");
assert!(CoomerProvider::build_video_item(post).is_none());
}
#[test]
fn parses_bare_array_and_wrapped_post_payloads() {
let bare = r#"[{"id":"1","user":"u","service":"onlyfans","title":"t","file":{},"attachments":[]}]"#;
match serde_json::from_str::<ApiPostsPayload>(bare).expect("bare array") {
ApiPostsPayload::List(posts) => assert_eq!(posts.len(), 1),
other => panic!("expected list, got {other:?}"),
}
let wrapped = r#"{"posts":[{"id":"1"}],"count":1}"#;
match serde_json::from_str::<ApiPostsPayload>(wrapped).expect("wrapped") {
ApiPostsPayload::Wrapped(w) => assert_eq!(w.posts.len(), 1),
other => panic!("expected wrapped, got {other:?}"),
}
}
}

View File

@@ -3,6 +3,7 @@ use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoItem};
use async_trait::async_trait;
@@ -118,6 +119,10 @@ enum Target {
Category(String),
Tag(String),
Series(String),
/// Pornstar page `/girl/{slug}/`.
Girl(String),
/// Free-form keyword: a pornstar page if one exists, else a tag archive.
Keyword(String),
}
impl CumlouderProvider {
@@ -209,12 +214,17 @@ impl CumlouderProvider {
if let Some(rest) = lower.strip_prefix("series:") {
return Target::Series(Self::slugify(rest));
}
for prefix in ["girl:", "pornstar:", "star:", "model:"] {
if let Some(rest) = lower.strip_prefix(prefix) {
return Target::Girl(Self::slugify(rest));
}
}
// The site search is behind a Cloudflare challenge: fall back to a
// category archive, then to the free-form tag archive.
// category archive, then to a pornstar page, then to a tag archive.
if let Some(slug) = Self::category_slug(q) {
return Target::Category(slug.to_string());
}
return Target::Tag(Self::slugify(q));
return Target::Keyword(Self::slugify(q));
}
if let Some(cat) = options.categories.as_deref() {
@@ -243,6 +253,9 @@ impl CumlouderProvider {
Target::Category(slug) => format!("{BASE_URL}/porn-videos/{slug}/{suffix}"),
Target::Tag(slug) => format!("{BASE_URL}/tag/{slug}/{suffix}"),
Target::Series(slug) => format!("{BASE_URL}/series/{slug}/{suffix}"),
Target::Girl(slug) | Target::Keyword(slug) => {
format!("{BASE_URL}/girl/{slug}/{suffix}")
}
}
}
@@ -370,10 +383,30 @@ impl CumlouderProvider {
{
Ok(response) => {
let status = response.status().as_u16();
let body = response.text().await.unwrap_or_default();
let mut body = response.text().await.unwrap_or_default();
if status == 404 {
return Ok(vec![]);
}
// Cloudflare challenge: hand over to the requester's full
// chain (FlareSolverr solve + identity replay).
if status == 403
|| status == 503
|| Requester::looks_like_cloudflare_challenge(&body)
{
match requester.get_with_headers(&url, headers.clone(), None).await {
Ok(solved) => {
body = solved;
if body.contains("muestra-escena") {
html = Some(body);
break;
}
}
Err(e) => {
last_err = e.to_string();
continue;
}
}
}
if body.contains("muestra-escena") {
html = Some(body);
break;
@@ -414,7 +447,22 @@ impl Provider for CumlouderProvider {
let per_page = per_page.parse::<usize>().unwrap_or(30).clamp(1, 60);
let target = Self::pick_target(query.as_deref(), &options);
match Self::fetch_listing(&target, page, per_page, &options).await {
let result = match &target {
Target::Keyword(slug) => {
// A name matches a pornstar page (`/girl/{slug}/`); anything
// else is a tag. An unknown girl is a fast 404 -> empty.
match Self::fetch_listing(&target, page, per_page, &options).await {
Ok(items) if !items.is_empty() => Ok(items),
Ok(_) if page > 1 => Ok(vec![]),
Ok(_) | Err(_) => {
let tag = Target::Tag(slug.clone());
Self::fetch_listing(&tag, page, per_page, &options).await
}
}
}
_ => Self::fetch_listing(&target, page, per_page, &options).await,
};
match result {
Ok(items) => items,
Err(e) => {
report_provider_error(CHANNEL_ID, "get_videos", &e.to_string()).await;
@@ -465,6 +513,10 @@ mod tests {
CumlouderProvider::build_listing_url(&Target::Tag("deepthroat".into()), 2),
"https://www.cumlouder.com/tag/deepthroat/2/"
);
assert_eq!(
CumlouderProvider::build_listing_url(&Target::Girl("karyn".into()), 2),
"https://www.cumlouder.com/girl/karyn/2/"
);
assert_eq!(
CumlouderProvider::build_listing_url(&Target::Category("anal".into()), 1),
"https://www.cumlouder.com/porn-videos/anal/"
@@ -479,8 +531,12 @@ mod tests {
Target::Category("big-tits".into())
);
assert_eq!(
CumlouderProvider::pick_target(Some("deepthroat"), &o),
Target::Tag("deepthroat".into())
CumlouderProvider::pick_target(Some("Karyn"), &o),
Target::Keyword("karyn".into())
);
assert_eq!(
CumlouderProvider::pick_target(Some("girl:Taylor Sands"), &o),
Target::Girl("taylor-sands".into())
);
assert_eq!(
CumlouderProvider::pick_target(Some("series:Nacho Vidal"), &o),

View File

@@ -440,15 +440,26 @@ impl EpornerProvider {
Ok(cards)
}
async fn build_item(requester: &mut Requester, card: EpornerCard) -> VideoItem {
async fn build_item(
requester: &mut Requester,
options: &ServerOptions,
card: EpornerCard,
) -> VideoItem {
// The hash/xhr resolution + Referer handling live in
// `crate::util::eporner_resolve` (shared with other providers that
// redirect out to eporner.com, e.g. `melonstube`) rather than here,
// since single-provider builds can't depend on another provider's
// module — see that module's doc comment.
let formats =
crate::util::eporner_resolve::resolve_formats(requester, &card.alnum_id, &card.page_url)
.await;
// Formats point at the eporner streaming proxy: the CDN links expire
// after two hours and only serve the server's IP; anything else plays
// a one-minute "watch on eporner.com" clip.
let formats = crate::util::eporner_resolve::resolve_proxied_formats(
requester,
options,
&card.alnum_id,
&card.page_url,
)
.await;
let mut item = VideoItem::new(
card.id,
@@ -648,7 +659,8 @@ impl EpornerProvider {
let cards = Self::parse_list_cards(&html)?;
let items = stream::iter(cards.into_iter().take(per_page.max(1)).map(|card| {
let mut req = requester.clone();
async move { Self::build_item(&mut req, card).await }
let options = &options;
async move { Self::build_item(&mut req, options, card).await }
}))
.buffer_unordered(ENRICH_CONCURRENCY)
.collect::<Vec<_>>()

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{Provider, probe_first_archive, report_provider_error, requester_or_default};
use crate::status::*;
use crate::util::parse_abbreviated_number;
use crate::videos::{ServerOptions, VideoItem};
@@ -164,6 +164,26 @@ impl EromeProvider {
(!slug.is_empty()).then_some(slug)
}
/// A single-word query that is an existing uploader name (`/{name}` answers
/// 404 otherwise) goes to that uploader's albums, else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = query.trim().trim_start_matches('@');
if slug.is_empty()
|| !slug
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
{
return None;
}
let url = format!("{}/{slug}", self.url);
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(&mut requester, &[url], &format!("{}/", self.url), |body| {
body.contains("album-link")
})
.await
.map(|_| Target::UploaderPosts { slug: slug.to_string() })
}
fn resolve_target(&self, query: &str, options: &ServerOptions, sort: &str) -> Target {
if let Some(site) = options.sites.as_deref() {
let normalized = site.trim();
@@ -367,7 +387,12 @@ impl Provider for EromeProvider {
};
let query_value = query.unwrap_or_default();
let page_value = page.parse::<u16>().unwrap_or(1);
let target = self.resolve_target(&query_value, &options, &sort_value);
let mut target = self.resolve_target(&query_value, &options, &sort_value);
if let Target::Search { query, .. } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
let url = self.build_url_for_target(&target, page_value);
match self.fetch_page(&options, &url).await {

View File

@@ -639,6 +639,24 @@ impl Provider for FikfapProvider {
.map(str::trim)
.filter(|value| !value.is_empty());
let target = Self::pick_target(normalized_query, &sort, &options);
// A single-word query that is an existing profile name goes to that
// creator's posts (checked against the profile endpoint); a profile
// without posts falls back to the keyword search.
if let Target::Search(query) = &target {
if !query.contains(char::is_whitespace) {
if let Ok(Some(profile)) = self.fetch_user_profile(query, &options).await {
let user = Target::User(profile.username);
if let Ok(items) = self
.fetch_target_items(user, page, per_page, &options)
.await
{
if !items.is_empty() {
return items;
}
}
}
}
}
match self.fetch_target_items(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::time::parse_time_to_seconds;
@@ -134,13 +136,13 @@ impl FullpornerProvider {
}
if let Some(slug) = q.strip_prefix("pornstar:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
page,
};
}
if let Some(slug) = q.strip_prefix("star:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
page,
};
}
@@ -153,6 +155,25 @@ impl FullpornerProvider {
Target::Latest { page }
}
/// A free-form query that names a pornstar goes to her archive (probed on
/// the site; unknown names 404), otherwise it stays a keyword search.
async fn resolve_keyword(query: &str, page: u32, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let probe = Target::Pornstar { slug: slug.clone(), page: 1 };
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::build_listing_url(&probe)],
&format!("{BASE_URL}/"),
|body| body.contains("class=\"video-card\""),
)
.await
.map(|_| Target::Pornstar { slug, page })
}
fn selector(value: &str) -> Result<Selector> {
Selector::parse(value)
.map_err(|e| Error::from(format!("selector `{value}` failed: {e}")))
@@ -394,7 +415,10 @@ impl Provider for FullpornerProvider {
.filter(|q| !q.is_empty())
.map(ToOwned::to_owned);
let target = Self::pick_target(normalized_query.as_deref(), page);
let mut target = Self::pick_target(normalized_query.as_deref(), page);
if let Target::Search { query, .. } = &target {
target = Self::resolve_keyword(query, page, &options).await.unwrap_or(target);
}
match Self::fetch_page(&target, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
use crate::providers::{probe_first_archive, slug_dashed,
Provider, report_provider_error, report_provider_error_background, requester_or_default,
};
use crate::status::*;
@@ -109,12 +109,12 @@ impl FullpornerorgProvider {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
if let Some(slug) = q.strip_prefix("pornstar:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
};
}
if let Some(slug) = q.strip_prefix("star:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
};
}
return Target::Search {
@@ -153,6 +153,25 @@ impl FullpornerorgProvider {
}
}
/// A free-form query that names a pornstar goes to her archive (probed on
/// the site; unknown names 404), otherwise it stays a keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let target = Target::Pornstar { slug };
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::build_listing_url(&target, 1)],
&format!("{BASE_URL}/"),
|body| body.contains("loop-video"),
)
.await
.map(|_| target)
}
fn html_headers(referer: &str) -> Vec<(String, String)> {
vec![
("User-Agent".to_string(), BROWSER_UA.to_string()),
@@ -446,7 +465,10 @@ impl Provider for FullpornerorgProvider {
.filter(|q| !q.is_empty())
.map(ToOwned::to_owned);
let target = Self::pick_target(normalized_query.as_deref());
let mut target = Self::pick_target(normalized_query.as_deref());
if let Target::Search { query } = &target {
target = Self::resolve_keyword(query, &options).await.unwrap_or(target);
}
match Self::fetch_page(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,13 +1,8 @@
use aes_gcm::aead::{Aead, KeyInit, Payload};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use async_trait::async_trait;
use base64::Engine;
use error_chain::error_chain;
use futures::future::join_all;
use serde_json::json;
use sha2::{Digest, Sha256};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::{SystemTime, UNIX_EPOCH};
use std::time::SystemTime;
use std::vec;
use crate::DbPool;
@@ -16,6 +11,7 @@ use crate::db;
use crate::providers::{Provider, report_provider_error, report_provider_error_background};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::hanime_resolve::{self as resolve, HanimeSource, SITE_ORIGIN};
use crate::videos::{self, ServerOptions, VideoItem};
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
@@ -34,14 +30,6 @@ error_chain! {
/// Full catalogue dump. Ignores every query parameter, so filtering, sorting and
/// pagination all happen client side. ~860 KB compressed, CDN cached for an hour.
const INDEX_URL: &str = "https://guest.freeanimehentai.net/api/v11/search_hvs";
const HANDSHAKE_URL: &str = "https://auth.hanime.tv/api/v11/handshake";
const SITE_ORIGIN: &str = "https://hanime.tv";
/// Envelope encryption used for the handshake request/response bodies.
const HANDSHAKE_KEY_SEED: &[u8] = b"htv-insecure-handshake-v1";
const HANDSHAKE_AAD: &[u8] = b"htv-insecure-v1";
/// Pieces of the `x-signature` pre-image, as baked into the site's wasm module.
const SIGNATURE_SECRET: &str = "Xkdi29";
const SIGNATURE_SALT: &str = "mn2";
const INDEX_TTL_SECS: u64 = 3600;
type IndexCache = OnceLock<Mutex<Option<(SystemTime, Arc<Vec<HanimeSearchResult>>)>>>;
@@ -76,99 +64,6 @@ struct HanimeSearchResult {
released_at_unix: u64,
}
#[derive(serde::Deserialize, Debug)]
struct HanimeHandshakePayload {
#[serde(default)]
sources: Vec<HanimeSource>,
}
#[derive(serde::Deserialize, Debug, Clone)]
struct HanimeSource {
#[serde(default)]
src: String,
#[serde(default)]
height: u32,
#[serde(default)]
label: String,
/// `"normal"` for the free streams, `"promotion"` for the premium-only teaser
/// entries which always carry an empty `src`.
#[serde(default)]
kind: String,
}
fn b64() -> base64::engine::general_purpose::GeneralPurpose {
base64::engine::general_purpose::URL_SAFE_NO_PAD
}
fn b64_decode(value: &str) -> Result<Vec<u8>> {
b64()
.decode(value.trim_end_matches('='))
.map_err(|e| Error::from(format!("base64url decode failed: {e}")))
}
fn handshake_cipher() -> Aes256Gcm {
let digest = Sha256::digest(HANDSHAKE_KEY_SEED);
let key = Key::<Aes256Gcm>::from_slice(digest.as_slice());
Aes256Gcm::new(key)
}
/// `{"v":1,"alg":"AES-256-GCM","iv":..,"tag":..,"data":..}`, base64url encoded.
fn seal_envelope(plaintext: &[u8]) -> Result<String> {
let mut iv = [0u8; 12];
rand::fill(&mut iv);
let sealed = handshake_cipher()
.encrypt(
Nonce::from_slice(&iv),
Payload {
msg: plaintext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| Error::from(format!("handshake encrypt failed: {e}")))?;
if sealed.len() < 16 {
return Err(Error::from("handshake ciphertext too short"));
}
let (data, tag) = sealed.split_at(sealed.len() - 16);
let envelope = json!({
"v": 1,
"alg": "AES-256-GCM",
"iv": b64().encode(iv),
"tag": b64().encode(tag),
"data": b64().encode(data),
});
Ok(b64().encode(envelope.to_string().as_bytes()))
}
fn open_envelope(token: &str) -> Result<Vec<u8>> {
let envelope: serde_json::Value = serde_json::from_slice(&b64_decode(token)?)
.map_err(|e| Error::from(format!("handshake envelope is not JSON: {e}")))?;
let field = |name: &str| -> Result<Vec<u8>> {
let raw = envelope
.get(name)
.and_then(|v| v.as_str())
.ok_or_else(|| Error::from(format!("handshake envelope missing {name}")))?;
b64_decode(raw)
};
let iv = field("iv")?;
let mut ciphertext = field("data")?;
ciphertext.extend_from_slice(&field("tag")?);
handshake_cipher()
.decrypt(
Nonce::from_slice(&iv),
Payload {
msg: &ciphertext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| Error::from(format!("handshake decrypt failed: {e}")))
}
fn signature_for(timestamp: u64) -> String {
let pre_image =
format!("{timestamp},{SIGNATURE_SECRET},{SITE_ORIGIN},{SIGNATURE_SALT},{timestamp}");
hex::encode(Sha256::digest(pre_image.as_bytes()))
}
#[derive(Debug, Clone)]
pub struct HanimeProvider;
@@ -242,6 +137,25 @@ impl HanimeProvider {
}
}
/// Fresh free sources for a slug, best quality first.
async fn fetch_sources(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
let mut requester =
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
resolve::fetch_sources(&mut requester, slug)
.await
.map_err(Error::from)
}
/// The stream URLs expire, so players get a stable redirect that does a fresh
/// handshake on every play instead of the signed link itself.
fn proxied_url(options: &ServerOptions, slug: &str, source: &HanimeSource) -> String {
crate::providers::build_proxy_url(
options,
"hanime",
&format!("{slug}/{}.m3u8", source.height),
)
}
fn db_key(slug: &str) -> String {
format!("https://hanime.tv/videos/hentai/{slug}")
}
@@ -319,86 +233,6 @@ impl HanimeProvider {
Ok(items)
}
/// Returns the free `sources` for a slug, best quality first. The auth host
/// occasionally refuses one of a burst of parallel connections, so retry once.
async fn fetch_sources(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
match self.handshake(slug, options).await {
Ok(sources) => Ok(sources),
Err(_) => {
ntex::time::sleep(ntex::time::Millis(500)).await;
self.handshake(slug, options).await
}
}
}
async fn handshake(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
let token = seal_envelope(
json!({
"timestamp_unix": timestamp,
"directive": "htv_player_handshake",
"slug": slug,
})
.to_string()
.as_bytes(),
)?;
let mut requester =
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
let response = requester
.post_json(
HANDSHAKE_URL,
&json!({ "token": token }),
vec![
("Origin".to_string(), SITE_ORIGIN.to_string()),
("Referer".to_string(), format!("{SITE_ORIGIN}/")),
("Accept".to_string(), "application/json".to_string()),
("x-signature-version".to_string(), "web2".to_string()),
("x-signature".to_string(), signature_for(timestamp)),
("x-time".to_string(), timestamp.to_string()),
],
)
.await
.map_err(|e| Error::from(format!("handshake request failed: {e}")))?;
let status = response.status();
if !status.is_success() {
return Err(Error::from(format!("handshake returned HTTP {status}")));
}
// The interesting part of the answer travels in a response header, not the body.
let encrypted = response
.headers()
.get("x-token")
.and_then(|v| v.to_str().ok())
.map(|v| v.to_string())
.ok_or_else(|| Error::from("handshake response is missing x-token"))?;
let payload: HanimeHandshakePayload = serde_json::from_slice(&open_envelope(&encrypted)?)
.map_err(|e| Error::from(format!("Failed to parse handshake payload: {e}")))?;
let mut sources: Vec<HanimeSource> = payload
.sources
.into_iter()
.filter(|s| s.kind == "normal" && !s.src.is_empty())
.collect();
sources.sort_by(|a, b| b.height.cmp(&a.height));
match sources.is_empty() {
true => Err(Error::from("handshake returned no playable sources")),
false => Ok(sources),
}
}
fn source_url(source: &HanimeSource) -> String {
match source.src.starts_with("http") {
true => source.src.clone(),
false => format!("{SITE_ORIGIN}{}", source.src),
}
}
fn quality_label(source: &HanimeSource) -> String {
match source.label.trim_end_matches('p') {
"" => source.height.to_string(),
@@ -452,15 +286,15 @@ impl HanimeProvider {
.iter()
.map(|source| {
videos::VideoFormat::new(
Self::source_url(source),
Self::proxied_url(&options, &hit.slug, source),
Self::quality_label(source),
"m3u8".to_string(),
)
})
.collect();
let stream_url = Self::source_url(&sources[0]);
let stream_url = Self::proxied_url(&options, &hit.slug, &sources[0]);
let duration = self
.fetch_duration(&stream_url, &options)
.fetch_duration(&resolve::source_url(&sources[0]), &options)
.await
.unwrap_or_default();
if let Ok(mut conn) = pool.get() {
@@ -490,11 +324,8 @@ impl HanimeProvider {
.and_then(|mut conn| db::get_video(&mut conn, db_key.clone()).ok().flatten());
match db_result {
Some(video_url) if video_url.contains("m3u8") || video_url.contains("/hls/") => {
let duration = self
.fetch_duration(&video_url, &options)
.await
.unwrap_or_default();
Some(video_url) if video_url.contains("/proxy/hanime/") => {
let duration = 0;
let formats = vec![videos::VideoFormat::new(
video_url.clone(),
"720".to_string(),

606
src/providers/hooporn.rs Normal file
View File

@@ -0,0 +1,606 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
use async_trait::async_trait;
use error_chain::error_chain;
use futures::stream::{self, StreamExt};
use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
use scraper::{Html, Selector};
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
crate::providers::ProviderChannelMetadata {
group_id: "studio-network",
tags: &["studio", "hd", "pornstars", "free"],
};
const BASE_URL: &str = "https://hooporn.com";
const CHANNEL_ID: &str = "hooporn";
const ENRICH_CONCURRENCY: usize = 8;
// Scraped from /categories (`div.channels-card a[href^="/category/"]`).
const CATEGORIES: &[(&str, &str)] = &[
("anal", "anal"),
("ass-licking", "ass licking"),
("atm", "ass to mouth"),
("babe", "babe"),
("bbw", "BBW"),
("big-ass", "big ass"),
("big-tits", "big tits"),
("black", "black"),
("blonde", "blonde"),
("blowjob", "blowjob"),
("bondage", "bondage"),
("brunette", "brunette"),
("busty", "busty"),
("casting", "casting"),
("college", "college"),
("compilation", "compilation"),
("creampie", "creampie"),
("cuckold", "cuckold"),
("czech", "czech"),
("deepthroat", "deepthroat"),
("doggystyle", "doggystyle"),
("dp", "double penetration"),
("ebony", "ebony"),
("fantasy", "fantasy"),
("fetish", "fetish"),
("fingering", "fingering"),
("footjob", "footjob"),
("foursome", "foursome"),
("gangbang", "gangbang"),
("gangbang-creampie", "gangbang creampie"),
("gaping", "gaping"),
("german", "german"),
("gloryhole", "gloryhole"),
("hairy", "hairy"),
("handjob", "handjob"),
("hardcore", "hardcore"),
("homemade", "homemade"),
("interracial", "interracial"),
("latina", "latina"),
("lesbian", "lesbian"),
("lingerie", "lingerie"),
("massage", "massage"),
("masturbation", "masturbation"),
("mature", "mature"),
("milf", "MILF"),
("orgy", "orgy"),
("outdoor", "outdoor"),
("petite", "petite"),
("pov", "POV"),
("public", "public"),
("pussy-licking", "pussy licking"),
("red-head", "red head"),
("riding", "riding"),
("russian", "russian"),
("skinny", "skinny"),
("small-tits", "small tits"),
("solo", "solo"),
("squirt", "squirt"),
("swallow", "swallow"),
("teen", "teen"),
("threesome", "threesome"),
("toys", "toys"),
("wife", "wife"),
];
error_chain! {
foreign_links {
Io(std::io::Error);
}
errors {
Parse(msg: String) {
description("parse error")
display("parse error: {}", msg)
}
}
}
#[derive(Debug, Clone)]
pub struct HoopornProvider;
#[derive(Debug, Clone, PartialEq, Eq)]
enum Target {
Latest,
Category(String),
Pornstar(String),
Studio(String),
Search(String),
/// Free-form query: a pornstar page or a studio page if one exists by that
/// name (probed on the site), else a keyword search.
Keyword(String),
}
/// Detail-page data: the player iframe plus taxonomy links.
#[derive(Debug, Default, PartialEq, Eq)]
struct Detail {
iframe: Option<String>,
tags: Vec<String>,
studio: Option<(String, String)>,
}
impl HoopornProvider {
pub fn new() -> Self {
Self
}
fn build_channel(&self, _clientversion: ClientVersion) -> Channel {
let mut category_options = vec![FilterOption {
id: "all".to_string(),
title: "All".to_string(),
}];
category_options.extend(CATEGORIES.iter().map(|(slug, title)| FilterOption {
id: (*slug).to_string(),
title: Self::title_case(title),
}));
Channel {
id: CHANNEL_ID.to_string(),
name: "HooPorn".to_string(),
description: "Free HD studio videos from HooPorn with category, pornstar and studio archives.".to_string(),
premium: false,
favicon: "https://www.google.com/s2/favicons?sz=64&domain=hooporn.com".to_string(),
status: "active".to_string(),
categories: vec![],
options: vec![
ChannelOption {
id: "sort".to_string(),
title: "Sort".to_string(),
description: "Browse HooPorn by newest videos.".to_string(),
systemImage: "arrow.up.arrow.down".to_string(),
colorName: "blue".to_string(),
options: vec![FilterOption {
id: "new".to_string(),
title: "Latest".to_string(),
}],
multiSelect: false,
},
ChannelOption {
id: "categories".to_string(),
title: "Categories".to_string(),
description: "Browse a HooPorn category archive.".to_string(),
systemImage: "square.grid.2x2".to_string(),
colorName: "orange".to_string(),
options: category_options,
multiSelect: false,
},
],
nsfw: true,
cacheDuration: Some(1800),
ytdlpCommand: None,
}
}
fn title_case(value: &str) -> String {
if value.chars().any(|c| c.is_uppercase()) {
return value.to_string();
}
value
.split(' ')
.map(|word| {
let mut chars = word.chars();
match chars.next() {
Some(first) => first.to_uppercase().collect::<String>() + chars.as_str(),
None => String::new(),
}
})
.collect::<Vec<_>>()
.join(" ")
}
fn slugify(value: &str) -> String {
let mut out = String::new();
let mut last_dash = true;
for c in value.trim().chars() {
if c.is_alphanumeric() {
out.extend(c.to_lowercase());
last_dash = false;
} else if !last_dash {
out.push('-');
last_dash = true;
}
}
out.trim_matches('-').to_string()
}
fn category_slug(value: &str) -> Option<&'static str> {
let slug = Self::slugify(value);
if slug.is_empty() {
return None;
}
CATEGORIES
.iter()
.find_map(|(s, title)| (*s == slug || Self::slugify(title) == slug).then_some(*s))
}
fn pick_target(query: Option<&str>, options: &ServerOptions) -> Target {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
let lower = q.to_ascii_lowercase();
for prefix in ["cat:", "category:"] {
if let Some(rest) = lower.strip_prefix(prefix) {
let slug = Self::category_slug(rest)
.map(str::to_string)
.unwrap_or_else(|| Self::slugify(rest));
return Target::Category(slug);
}
}
for prefix in ["pornstar:", "star:", "model:"] {
if let Some(rest) = lower.strip_prefix(prefix) {
return Target::Pornstar(Self::slugify(rest));
}
}
for prefix in ["studio:", "site:"] {
if let Some(rest) = lower.strip_prefix(prefix) {
return Target::Studio(Self::slugify(rest).replace('-', ""));
}
}
if let Some(slug) = Self::category_slug(q) {
return Target::Category(slug.to_string());
}
return Target::Keyword(q.to_string());
}
if let Some(cat) = options.categories.as_deref() {
let cat = cat.trim();
if !cat.is_empty() && !cat.eq_ignore_ascii_case("all") {
let slug = Self::category_slug(cat)
.map(str::to_string)
.unwrap_or_else(|| Self::slugify(cat));
return Target::Category(slug);
}
}
Target::Latest
}
fn build_listing_url(target: &Target, page: u32) -> String {
match target {
Target::Latest => format!("{BASE_URL}/home/{page}"),
Target::Category(slug) => format!("{BASE_URL}/category/{slug}/{page}"),
Target::Pornstar(slug) => format!("{BASE_URL}/pornstar/{slug}/{page}"),
Target::Studio(slug) => format!("{BASE_URL}/studio/{slug}/{page}"),
Target::Search(query) | Target::Keyword(query) => format!(
"{BASE_URL}/search?q={}&p={page}",
utf8_percent_encode(query, NON_ALPHANUMERIC)
),
}
}
fn selector(value: &str) -> Result<Selector> {
Selector::parse(value).map_err(|e| Error::from(format!("selector `{value}` failed: {e}")))
}
fn absolute(url: &str) -> String {
if let Some(rest) = url.strip_prefix("//") {
format!("https://{rest}")
} else if url.starts_with('/') {
format!("{BASE_URL}{url}")
} else {
url.to_string()
}
}
fn parse_listing(html: &str) -> Result<Vec<VideoItem>> {
let document = Html::parse_document(html);
let card_sel = Self::selector("div.video-card")?;
let link_sel = Self::selector("div.video-title a[href^=\"/watch/\"]")?;
let img_sel = Self::selector("img")?;
let time_sel = Self::selector(".time")?;
let created_sel = Self::selector("span.create")?;
let mut items = Vec::new();
let mut seen = std::collections::HashSet::new();
for card in document.select(&card_sel) {
let Some(link) = card.select(&link_sel).next() else {
continue;
};
let href = link.value().attr("href").unwrap_or_default();
let id = href.trim_start_matches("/watch/").trim_matches('/').to_string();
if id.is_empty() || !seen.insert(id.clone()) {
continue;
}
let title = link.text().collect::<String>().trim().to_string();
if title.is_empty() {
continue;
}
let thumb = card
.select(&img_sel)
.next()
.and_then(|i| i.value().attr("data-src"))
.map(Self::absolute)
.unwrap_or_default();
let duration = card
.select(&time_sel)
.next()
.map(|t| t.text().collect::<String>())
.and_then(|t| parse_time_to_seconds(t.trim()))
.and_then(|s| u32::try_from(s).ok())
.unwrap_or(0);
let mut item = VideoItem::new(
id.clone(),
title,
format!("{BASE_URL}/watch/{id}"),
CHANNEL_ID.to_string(),
thumb,
duration,
);
item.uploadedAt = card
.select(&created_sel)
.next()
.and_then(|c| c.text().collect::<String>().trim().parse::<u64>().ok());
item.aspectRatio = Some(16.0 / 9.0);
items.push(item);
}
Ok(items)
}
fn parse_detail(html: &str) -> Result<Detail> {
let document = Html::parse_document(html);
let iframe_sel = Self::selector("iframe[src*=\"/video/\"]")?;
let tag_sel = Self::selector("p.tag-link a[href^=\"/category/\"]")?;
let star_sel = Self::selector("a[href^=\"/pornstar/\"]")?;
let studio_sel = Self::selector("a[href^=\"/studio/\"]")?;
let mut detail = Detail::default();
detail.iframe = document
.select(&iframe_sel)
.next()
.and_then(|f| f.value().attr("src"))
.map(Self::absolute);
let mut push = |tag: String| {
let tag = tag.trim().trim_start_matches('#').trim().to_string();
if !tag.is_empty() && !detail.tags.iter().any(|t| t.eq_ignore_ascii_case(&tag)) {
detail.tags.push(tag);
}
};
for a in document.select(&tag_sel) {
push(a.text().collect::<String>());
}
for a in document.select(&star_sel) {
push(a.text().collect::<String>());
}
if let Some(a) = document.select(&studio_sel).next() {
let name = a.text().collect::<String>().trim().to_string();
let slug = a
.value()
.attr("href")
.unwrap_or_default()
.trim_start_matches("/studio/")
.trim_matches('/')
.to_string();
if !name.is_empty() && !slug.is_empty() {
detail.studio = Some((name, slug));
}
}
Ok(detail)
}
/// The player frame carries `var video = "https://videos…mp4?md5=…&expires=…"`.
fn parse_player_video(html: &str) -> Option<String> {
let marker = "var video = \"";
let start = html.find(marker)? + marker.len();
let end = html[start..].find('"')? + start;
let url = html[start..end].replace("&amp;", "&");
url.starts_with("http").then_some(url)
}
async fn enrich(items: Vec<VideoItem>, requester: Requester) -> Vec<VideoItem> {
let referer = format!("{BASE_URL}/");
stream::iter(items.into_iter().map(|mut item| {
let mut requester = requester.clone();
let referer = referer.clone();
async move {
let headers = vec![("Referer".to_string(), referer.clone())];
let page = requester
.get_with_headers(&item.url, headers.clone(), None)
.await
.ok()?;
let detail = Self::parse_detail(&page).ok()?;
let frame = detail.iframe?;
let player = requester
.get_with_headers(&frame, headers, None)
.await
.ok()?;
let video_url = Self::parse_player_video(&player)?;
item.formats = Some(vec![
VideoFormat::new(video_url, "auto".to_string(), "mp4".to_string())
.format_id("mp4".to_string()),
]);
if !detail.tags.is_empty() {
item.tags = Some(detail.tags);
}
if let Some((name, slug)) = detail.studio {
item.uploaderUrl = Some(format!("{BASE_URL}/studio/{slug}"));
item.uploaderId = Some(format!("{CHANNEL_ID}:{slug}"));
item.uploader = Some(name);
}
Some(item)
}
}))
.buffered(ENRICH_CONCURRENCY)
.filter_map(|item| async move { item })
.collect::<Vec<_>>()
.await
}
/// Fetches a listing page; `None` for a real 404 (see `Requester::get_probe`).
async fn fetch_html(requester: &mut Requester, url: &str) -> Result<Option<String>> {
let headers = vec![("Referer".to_string(), format!("{BASE_URL}/"))];
requester
.get_probe(url, headers)
.await
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))
}
async fn fetch(target: &Target, page: u32, options: &ServerOptions) -> Result<Vec<VideoItem>> {
let mut requester = requester_or_default(options, CHANNEL_ID, "fetch");
let html = match target {
Target::Keyword(query) => {
// Probe page 1 of the pornstar and studio archive for this
// name; the first that exists wins, otherwise keyword search.
let candidates = [
Target::Pornstar(Self::slugify(query)),
Target::Studio(Self::slugify(query).replace('-', "")),
];
let mut found = None;
for candidate in candidates {
let slug_ok = match &candidate {
Target::Pornstar(s) | Target::Studio(s) => !s.is_empty(),
_ => false,
};
if !slug_ok {
continue;
}
let first = Self::fetch_html(&mut requester, &Self::build_listing_url(&candidate, 1)).await?;
if let Some(body) = first.filter(|b| b.contains("class=\"video-card\"")) {
found = Some(if page <= 1 {
Some(body)
} else {
Self::fetch_html(&mut requester, &Self::build_listing_url(&candidate, page)).await?
});
break;
}
}
match found {
Some(html) => html,
None => Self::fetch_html(&mut requester, &Self::build_listing_url(target, page)).await?,
}
}
_ => Self::fetch_html(&mut requester, &Self::build_listing_url(target, page)).await?,
};
let Some(html) = html else {
return Ok(vec![]);
};
let items = Self::parse_listing(&html)?;
Ok(Self::enrich(items, requester).await)
}
}
#[async_trait]
impl Provider for HoopornProvider {
async fn get_videos(
&self,
_cache: VideoCache,
_pool: DbPool,
_sort: String,
query: Option<String>,
page: String,
_per_page: String,
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u32>().unwrap_or(1).max(1);
let target = Self::pick_target(query.as_deref(), &options);
match Self::fetch(&target, page, &options).await {
Ok(items) => items,
Err(e) => {
report_provider_error(CHANNEL_ID, "get_videos", &e.to_string()).await;
vec![]
}
}
}
fn get_channel(&self, clientversion: ClientVersion) -> Option<Channel> {
Some(self.build_channel(clientversion))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn opts() -> ServerOptions {
ServerOptions {
featured: None,
category: None,
sites: None,
filter: None,
language: None,
public_url_base: None,
requester: None,
network: None,
stars: None,
categories: None,
duration: None,
sort: None,
sexuality: None,
client_version: None,
}
}
#[test]
fn builds_urls() {
assert_eq!(HoopornProvider::build_listing_url(&Target::Latest, 2), "https://hooporn.com/home/2");
assert_eq!(
HoopornProvider::build_listing_url(&Target::Category("milf".into()), 3),
"https://hooporn.com/category/milf/3"
);
assert_eq!(
HoopornProvider::build_listing_url(&Target::Search("big tits".into()), 2),
"https://hooporn.com/search?q=big%20tits&p=2"
);
assert_eq!(
HoopornProvider::build_listing_url(&Target::Studio("mylf".into()), 1),
"https://hooporn.com/studio/mylf/1"
);
}
#[test]
fn routes_queries() {
let o = opts();
assert_eq!(
HoopornProvider::pick_target(Some("Double Penetration"), &o),
Target::Category("dp".into())
);
assert_eq!(
HoopornProvider::pick_target(Some("pornstar:Gal Ritchie"), &o),
Target::Pornstar("gal-ritchie".into())
);
assert_eq!(
HoopornProvider::pick_target(Some("studio:Family Strokes"), &o),
Target::Studio("familystrokes".into())
);
assert_eq!(
HoopornProvider::pick_target(Some("golf"), &o),
Target::Keyword("golf".into())
);
assert_eq!(HoopornProvider::pick_target(None, &o), Target::Latest);
}
#[test]
fn parses_cards_detail_and_player() {
let list = r#"<div class="video-card"><div class="video-card-image">
<a class="popout" href="/watch/abc123"><img src="//cdn.hooporn.com/img/blank.gif" data-src="//imgs.hooporn.com/t.jpg?v2" alt="x"></a>
<div class="time">37:16</div></div><div class="video-card-body"><div class="video-title">
<a class="popout" href="/watch/abc123">A &amp; B - Title</a></div>
<div class="video-view"><span class="create">1792152610</span></div></div></div>"#;
let items = HoopornProvider::parse_listing(list).unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0].title, "A & B - Title");
assert_eq!(items[0].thumb, "https://imgs.hooporn.com/t.jpg?v2");
assert_eq!(items[0].duration, 37 * 60 + 16);
assert_eq!(items[0].uploadedAt, Some(1792152610));
let detail = r##"<iframe src="//vid.hooporn.com/video/182550"></iframe>
<p class="tag-link"><span><a class="popout" href="/category/milf">#milf</a></span></p>
<a href="/pornstar/gal-ritchie">gal ritchie</a><a href="/studio/familystrokes">FAMILY STROKES</a>"##;
let d = HoopornProvider::parse_detail(detail).unwrap();
assert_eq!(d.iframe.as_deref(), Some("https://vid.hooporn.com/video/182550"));
assert_eq!(d.tags, vec!["milf".to_string(), "gal ritchie".to_string()]);
assert_eq!(d.studio, Some(("FAMILY STROKES".to_string(), "familystrokes".to_string())));
let player = r#"var video = "https://videos.trendyporn.com/v/a.mp4?md5=X&amp;expires=1";"#;
assert_eq!(
HoopornProvider::parse_player_video(player).as_deref(),
Some("https://videos.trendyporn.com/v/a.mp4?md5=X&expires=1")
);
}
}

View File

@@ -258,33 +258,74 @@ impl HypnotubeProvider {
/// isolated cookie session instead of the shared requester jar. Page 1 is
/// the redirect target of the POST; page N>1 is fetched directly.
async fn fetch_search_html(&self, query: &str, page: u8) -> Result<String> {
let (status, body) = self.search_once(query, page, None).await?;
if !(status == 403 || status == 503 || Requester::looks_like_cloudflare_challenge(&body)) {
return Ok(body);
}
// Cloudflare challenged the isolated session: solve it through the
// shared requester (FlareSolverr fallback), then replay the solved
// cookies and user agent in a fresh isolated session.
let mut requester = Requester::new();
let home = format!("{}/", self.url);
let _ = requester.get(&home, Some(Version::HTTP_11)).await;
let identity = (
requester.cookie_header_for_url(&home),
Requester::remembered_flaresolverr_user_agent(&home),
);
let (_, body) = self.search_once(query, page, Some(identity)).await?;
Ok(body)
}
async fn search_once(
&self,
query: &str,
page: u8,
identity: Option<(Option<String>, Option<String>)>,
) -> Result<(u16, String)> {
let jar = Arc::new(Jar::default());
let mut user_agent = None;
if let Some((cookies, ua)) = identity {
if let Some(cookies) = cookies {
for pair in cookies.split(';').map(str::trim).filter(|p| !p.is_empty()) {
jar.add(pair, self.url.as_str());
}
}
user_agent = ua;
}
let client = Client::builder()
.tls_cert_verification(false)
.emulation(Emulation::Firefox136)
.cookie_provider(Arc::new(Jar::default()))
.cookie_provider(jar)
.redirect(if page <= 1 { Policy::default() } else { Policy::none() })
.build()?;
let body = format!("q={}&type=videos", query.trim().replace(' ', "+"));
let gate = client
let mut gate = client
.post(format!("{}/searchgate.php", self.url))
.header("Content-Type", "application/x-www-form-urlencoded")
.header("Referer", format!("{}/", self.url))
.body(body)
.send()
.await?;
.body(body);
if let Some(ua) = &user_agent {
gate = gate.header("User-Agent", ua.as_str());
}
let gate = gate.send().await?;
if page <= 1 {
return Ok(gate.text().await?);
let status = gate.status().as_u16();
return Ok((status, gate.text().await?));
}
let slug = query.trim().replace(' ', "%20");
let page_url = format!("{}/search/videos/{}/page{}.html", self.url, slug, page);
let response = client
let mut request = client
.get(page_url)
.header("Referer", format!("{}/search/videos/{}/", self.url, slug))
.send()
.await?;
Ok(response.text().await?)
.header("Referer", format!("{}/search/videos/{}/", self.url, slug));
if let Some(ua) = &user_agent {
request = request.header("User-Agent", ua.as_str());
}
let response = request.send().await?;
let status = response.status().as_u16();
Ok((status, response.text().await?))
}
async fn query(

View File

@@ -494,11 +494,18 @@ impl MelonstubeProvider {
Self::parse_kvs_formats(&html, &origin)
}
/// Through the eporner streaming proxy: its CDN links expire after two
/// hours and only serve the server's IP (else a "watch on eporner.com" clip).
async fn resolve_eporner(
requester: &mut Requester,
options: &ServerOptions,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
crate::util::eporner_resolve::resolve_public_formats(requester, detail_url).await
let alnum_id = crate::util::eporner_resolve::alnum_id_from_href(detail_url)?;
crate::util::eporner_resolve::resolve_proxied_formats(
requester, options, &alnum_id, detail_url,
)
.await
}
fn xhamster_mirror_mp4_regex() -> &'static Regex {
@@ -632,10 +639,11 @@ impl MelonstubeProvider {
async fn resolve_source_formats(
kind: SourceKind,
requester: &mut Requester,
options: &ServerOptions,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
match kind {
SourceKind::Eporner => Self::resolve_eporner(requester, detail_url).await,
SourceKind::Eporner => Self::resolve_eporner(requester, options, detail_url).await,
SourceKind::Kvs => Self::resolve_kvs(requester, detail_url, false).await,
SourceKind::KvsCfGated => Self::resolve_kvs(requester, detail_url, true).await,
SourceKind::XhamsterMirror => Self::resolve_xhamster_mirror(requester, detail_url).await,
@@ -904,6 +912,7 @@ impl MelonstubeProvider {
if !pending.is_empty() {
let resolved = stream::iter(pending.into_iter().map(|(idx, kind, dest_url)| {
let mut req = requester.clone();
let options = &options;
async move {
// Bounded so one slow/hung third-party destination (e.g.
// an unreachable FlareSolverr fallback) can't stall the
@@ -911,7 +920,7 @@ impl MelonstubeProvider {
// its plain destination URL with no formats.
let formats = tokio::time::timeout(
std::time::Duration::from_secs(ENRICH_TIMEOUT_SECS),
Self::resolve_source_formats(kind, &mut req, &dest_url),
Self::resolve_source_formats(kind, &mut req, options, &dest_url),
)
.await
.ok()

View File

@@ -612,6 +612,66 @@ pub fn report_provider_error_background(provider_name: &str, context: &str, msg:
});
}
/// `Some(slug)` joined with `-` ("Gal Ritchie" -> "gal-ritchie") for archive URLs.
pub fn slug_dashed(value: &str) -> String {
let mut out = String::new();
let mut last_dash = true;
for c in value.trim().chars() {
if c.is_alphanumeric() {
out.extend(c.to_lowercase());
last_dash = false;
} else if !last_dash {
out.push('-');
last_dash = true;
}
}
out.trim_matches('-').to_string()
}
/// Same as [`slug_dashed`] without separators ("Family Strokes" -> "familystrokes").
pub fn slug_joined(value: &str) -> String {
slug_dashed(value).replace('-', "")
}
/// Probes candidate archive URLs (pornstar / studio / channel / tag pages) in
/// order and returns the index and body of the first that exists and
/// `has_items` accepts. A miss (404, empty page, transport error) moves on to
/// the next candidate; the caller falls back to keyword search on `None`.
/// Uses `Requester::get_probe`, so an unknown archive answers instantly and a
/// Cloudflare challenge is solved through FlareSolverr.
pub async fn probe_first_archive(
requester: &mut Requester,
urls: &[String],
referer: &str,
has_items: impl Fn(&str) -> bool,
) -> Option<(usize, String)> {
probe_first_archive_with(
requester,
urls,
vec![("Referer".to_string(), referer.to_string())],
has_items,
)
.await
}
/// [`probe_first_archive`] with caller-supplied request headers (JSON APIs
/// that demand a special `Accept`, extra tokens, ...).
pub async fn probe_first_archive_with(
requester: &mut Requester,
urls: &[String],
headers: Vec<(String, String)>,
has_items: impl Fn(&str) -> bool,
) -> Option<(usize, String)> {
for (index, url) in urls.iter().enumerate() {
if let Ok(Some(body)) = requester.get_probe(url, headers.clone()).await {
if has_items(&body) {
return Some((index, body));
}
}
}
None
}
pub fn requester_or_default(
options: &ServerOptions,
provider_name: &str,
@@ -1777,3 +1837,16 @@ mod tests {
);
}
}
#[cfg(test)]
mod slug_tests {
use super::{slug_dashed, slug_joined};
#[test]
fn slugs_names_for_archive_urls() {
assert_eq!(slug_dashed(" Gal Ritchie "), "gal-ritchie");
assert_eq!(slug_dashed("Mia Khalifa!!"), "mia-khalifa");
assert_eq!(slug_joined("Family Strokes"), "familystrokes");
assert_eq!(slug_dashed("???"), "");
}
}

View File

@@ -2,6 +2,7 @@ use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, report_provider_error, report_provider_error_background, requester_or_default,
probe_first_archive,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -260,6 +261,34 @@ impl Porn4fansProvider {
Target::Latest
}
/// A free-form query that names a model or a tag goes to that archive
/// (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = Self::slugify(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Model { slug: slug.clone() },
Target::Tag { slug },
];
let urls: Vec<String> = candidates
.iter()
.map(|t| match t {
Target::Model { slug } => format!("{}/models/{slug}/", self.url),
Target::Tag { slug } => format!("{}/tags/{slug}/", self.url),
_ => String::new(),
})
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) =
probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("custom_list_videos")
})
.await?;
candidates.into_iter().nth(index)
}
/// Build the async `get_block` URL plus the Referer to send with it.
fn build_block_url(&self, target: &Target, sort: &str, page: u32) -> (String, String) {
let sort_by = Self::sort_by(sort);
@@ -788,7 +817,12 @@ impl Provider for Porn4fansProvider {
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u32>().unwrap_or(1).max(1);
let target = self.target_from_request(query.as_deref(), options.categories.as_deref());
let mut target = self.target_from_request(query.as_deref(), options.categories.as_deref());
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
match self
.get_videos_for_target(cache, &target, &sort, page, options)

View File

@@ -1,6 +1,9 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, proxy_aylo_thumb, report_provider_error, requester_or_default,
slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -45,6 +48,8 @@ error_chain! {
pub struct PornhubProvider {
url: String,
tag_map: Arc<RwLock<HashMap<String, TagInfo>>>,
/// Queries already probed on the site that are neither a pornstar, model nor channel.
probe_misses: Arc<RwLock<std::collections::HashSet<String>>>,
}
#[derive(Debug, Clone, Copy)]
@@ -91,6 +96,7 @@ impl PornhubProvider {
let provider = Self {
url: BASE_URL.to_string(),
tag_map: Arc::new(RwLock::new(HashMap::new())),
probe_misses: Arc::new(RwLock::new(std::collections::HashSet::new())),
};
provider.spawn_initial_load();
provider
@@ -588,6 +594,43 @@ impl PornhubProvider {
values.push(normalized.to_string());
}
/// A free-form query that is not in the loaded catalogs may still name a
/// pornstar, model or channel. Probe the site (unknown names 404 or
/// redirect away) and remember the hit in `tag_map` so later pages and
/// queries resolve without another request.
async fn learn_query_target(&self, query: &str, options: &ServerOptions) {
let key = query.trim().to_ascii_lowercase();
let slug = slug_dashed(query);
if slug.is_empty() || self.probe_misses.read().unwrap().contains(&key) {
return;
}
let kinds = [
QueryTargetKind::Pornstar,
QueryTargetKind::Model,
QueryTargetKind::Channel,
];
let urls: Vec<String> = kinds
.iter()
.map(|kind| format!("{}/{}/{slug}/videos", self.url, kind.path_segment()))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "learn_query_target");
let hit = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("pcVideoListItem")
})
.await;
match hit {
Some((index, _)) => {
self.tag_map.write().unwrap().insert(
key,
TagInfo { kind: kinds[index], slug, title: query.trim().to_string() },
);
}
None => {
self.probe_misses.write().unwrap().insert(key);
}
}
}
async fn fetch_listing(
&self,
cache: VideoCache,
@@ -596,6 +639,11 @@ impl PornhubProvider {
query: Option<&str>,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
if self.parse_query_target(q).is_none() {
self.learn_query_target(q, &options).await;
}
}
if query.is_some() && self.tag_map.read().unwrap().is_empty() {
let _ = Self::load_tags(&self.url, Arc::clone(&self.tag_map)).await;
}
@@ -624,7 +672,13 @@ impl PornhubProvider {
};
let items = match self.parse_listing_page(&text, scope) {
Ok(items) => items,
// `pix-cdn77.phncdn.com` thumbs answer 403 without a matching Referer.
Ok(mut items) => {
for item in items.iter_mut() {
item.thumb = proxy_aylo_thumb(&options, &item.thumb);
}
items
}
Err(error) => {
report_provider_error(
CHANNEL_ID,

View File

@@ -560,6 +560,24 @@ impl Provider for RedgifsProvider {
let sort = Sort::from_sort_id(&sort);
let target = Self::pick_target(normalized_query, sort, &options);
// A single-word query that is an existing creator name goes to that
// creator's gifs (checked against the users endpoint); a creator
// without gifs falls back to the keyword search.
if let Target::Query(text, _) = &target {
if !text.contains(char::is_whitespace) {
if let Ok(Some(profile)) = self.fetch_user_profile(text, &options).await {
let creator = Target::Creator(profile.username);
if let Ok(items) = self
.fetch_target_items(creator, page, per_page, &options)
.await
{
if !items.is_empty() {
return items;
}
}
}
}
}
match self.fetch_target_items(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::Provider;
use crate::providers::{Provider, requester_or_default, slug_dashed};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::discord::send_discord_error_report;
@@ -12,10 +12,23 @@ use async_trait::async_trait;
use error_chain::error_chain;
use futures::stream::{self, StreamExt};
use htmlentity::entity::{ICodedDataTrait, decode};
use std::time::{SystemTime, UNIX_EPOCH};
use scraper::{Html, Selector};
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
use std::vec;
const ENRICH_CONCURRENCY: usize = 6;
const ENRICH_CONCURRENCY: usize = 4;
/// The signed `v-acctoken` format URLs last ~12h; keep details well inside that.
const DETAIL_TTL: Duration = Duration::from_secs(2 * 60 * 60);
const DETAIL_CACHE_SIZE: usize = 5000;
const TARGET_TTL: Duration = Duration::from_secs(6 * 60 * 60);
const TARGET_CACHE_SIZE: usize = 2000;
/// A probed archive smaller than this loses to keyword search: "elf girl" is a
/// one-video tag, while the search finds hundreds.
const MIN_PROBED_ARCHIVE_ITEMS: usize = 4;
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
crate::providers::ProviderChannelMetadata {
@@ -36,15 +49,66 @@ error_chain! {
}
}
#[derive(Debug, Clone)]
/// What a request resolves to. Artists live under `/models/`, franchises and
/// genres under `/categories/`, characters and everything else under `/tags/`.
#[derive(Debug, Clone, PartialEq, Eq)]
enum Target {
Latest,
Search(String),
Artist(String),
Category(String),
Tag(String),
}
impl Target {
/// Archive path (`models/{slug}`, ...) for the archive targets.
fn archive_path(&self) -> Option<String> {
match self {
Target::Artist(slug) => Some(format!("models/{slug}")),
Target::Category(slug) => Some(format!("categories/{slug}")),
Target::Tag(slug) => Some(format!("tags/{slug}")),
_ => None,
}
}
}
#[derive(Clone)]
pub struct Rule34videoProvider {
url: String,
/// Enriched items (formats, tags, artist) by video URL. Every listing costs
/// ~23 detail pages, again for the prefetched next page; refetching them
/// on each request is what runs into the site's 429 rate limit.
details: VideoCache,
/// Bare query -> resolved target, so later pages skip the archive probes.
targets: Arc<Mutex<HashMap<String, (Instant, Target)>>>,
}
impl Rule34videoProvider {
pub fn new() -> Self {
let mut details = VideoCache::new();
details.max_size(DETAIL_CACHE_SIZE);
Rule34videoProvider {
url: "https://rule34video.com".to_string(),
details,
targets: Arc::new(Mutex::new(HashMap::new())),
}
}
fn remembered_target(&self, query: &str) -> Option<Target> {
let targets = self.targets.lock().ok()?;
let (time, target) = targets.get(&slug_dashed(query))?;
(time.elapsed() < TARGET_TTL).then(|| target.clone())
}
fn remember_target(&self, query: &str, target: &Target) {
if let Ok(mut targets) = self.targets.lock() {
if targets.len() >= TARGET_CACHE_SIZE {
targets.retain(|_, (time, _)| time.elapsed() < TARGET_TTL);
if targets.len() >= TARGET_CACHE_SIZE {
targets.clear();
}
}
targets.insert(slug_dashed(query), (Instant::now(), target.clone()));
}
}
@@ -93,6 +157,192 @@ impl Rule34videoProvider {
}
}
/// Explicit `artist:` / `category:` / `tag:` style prefixes (the server has
/// already stripped a leading `#`).
fn parse_prefixed(query: &str) -> Option<Target> {
let (kind, value) = query.trim().split_once(':')?;
let slug = slug_dashed(value);
if slug.is_empty() {
return None;
}
match kind.trim().to_ascii_lowercase().as_str() {
"artist" | "artists" | "model" | "models" | "author" | "creator" | "uploader" => {
Some(Target::Artist(slug))
}
"cat" | "category" | "categories" | "franchise" | "series" | "game" => {
Some(Target::Category(slug))
}
"tag" | "tags" | "character" | "char" => Some(Target::Tag(slug)),
_ => None,
}
}
/// Titles of the tags offered by `/search_ajax.php?tag=`.
fn parse_tag_suggestions(html: &str) -> Vec<String> {
let document = Html::parse_fragment(html);
let Ok(selector) = Selector::parse("input[data-title]") else {
return vec![];
};
document
.select(&selector)
.filter_map(|input| input.value().attr("data-title"))
.map(|title| title.trim().to_string())
.filter(|title| !title.is_empty())
.collect()
}
/// The suggested tag whose name (ignoring a trailing "(franchise)") equals
/// the query, e.g. "samus aran" -> "samus aran (metroid)".
fn best_tag_title(titles: &[String], query: &str) -> Option<String> {
let wanted = slug_dashed(query);
if wanted.is_empty() {
return None;
}
let mut exact = titles.iter().filter(|title| {
let base = title.split('(').next().unwrap_or(title);
slug_dashed(base) == wanted || slug_dashed(title) == wanted
});
let first = exact.next()?;
// Several franchises share a character name: ambiguous, leave it to search.
exact.next().is_none().then(|| first.clone())
}
/// A free-form query that names an artist, category or tag goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
/// The second value is false when a probe failed (rate limit, outage) and
/// the answer must not be remembered.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> (Target, bool) {
let slug = slug_dashed(query);
if slug.is_empty() {
return (Target::Search(query.trim().to_string()), true);
}
// Artist slugs are inconsistent on the site: "general-butch" but "biggreen".
let mut candidates = vec![Target::Artist(slug.clone())];
if slug.contains('-') {
candidates.push(Target::Artist(slug.replace('-', "")));
}
candidates.push(Target::Category(slug.clone()));
candidates.push(Target::Tag(slug));
let mut requester = requester_or_default(options, "rule34video", "resolve_keyword");
let referer = format!("{}/", self.url);
let mut conclusive = true;
for candidate in &candidates {
let Some(path) = candidate.archive_path() else {
continue;
};
let url = format!("{}/{path}/", self.url);
let headers = vec![("Referer".to_string(), referer.clone())];
match requester.get_probe(&url, headers).await {
Ok(Some(body)) => {
let items = self.get_video_items_from_html(body).len();
if items >= MIN_PROBED_ARCHIVE_ITEMS {
return (candidate.clone(), true);
}
}
Ok(None) => {}
Err(_) => conclusive = false,
}
}
// Characters are tagged "name (franchise)": ask the site's own tag search.
let suggest_url = format!(
"{}/search_ajax.php?tag={}",
self.url,
utf8_percent_encode(query.trim(), NON_ALPHANUMERIC)
);
let headers = vec![
("Referer".to_string(), referer),
("X-Requested-With".to_string(), "XMLHttpRequest".to_string()),
];
match requester.get_probe(&suggest_url, headers).await {
Ok(Some(body)) => {
let titles = Self::parse_tag_suggestions(&body);
if let Some(title) = Self::best_tag_title(&titles, query) {
return (Target::Tag(slug_dashed(&title)), conclusive);
}
}
Ok(None) => {}
Err(_) => conclusive = false,
}
(Target::Search(query.trim().to_string()), conclusive)
}
async fn resolve_target(&self, query: Option<&str>, options: &ServerOptions) -> Target {
let Some(query) = query.map(str::trim).filter(|q| !q.is_empty()) else {
return Target::Latest;
};
if let Some(target) = Self::parse_prefixed(query) {
return target;
}
if let Some(target) = self.remembered_target(query) {
return target;
}
let (target, conclusive) = self.resolve_keyword(query, options).await;
if conclusive {
self.remember_target(query, &target);
}
target
}
async fn archive(
&self,
cache: VideoCache,
target: &Target,
page: u8,
sort: &str,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
let Some(path) = target.archive_path() else {
return Ok(vec![]);
};
let sort_val = match sort {
"post_date" | "video_viewed" | "rating" | "duration" | "pseudo_random" => sort,
_ => "post_date",
};
let index = format!("rule34video:{path}:{page}:{sort_val}");
if sort_val != "pseudo_random" {
if let Some((time, items)) = cache.get(&index) {
if time.elapsed().unwrap_or_default().as_secs() < 300 {
return Ok(items.clone());
}
}
}
let mut requester = options.requester.clone().ok_or("Requester missing")?;
let url = format!(
"{}/{path}/?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort_val}&from={page}",
self.url
);
let headers = vec![("Referer".to_string(), format!("{}/{path}/", self.url))];
let stale = || cache.get(&index).map(|(_, items)| items).unwrap_or_default();
let text = match requester.get_probe(&url, headers).await {
Ok(Some(text)) => text,
// A 404 is an unknown archive (or a page past the end).
Ok(None) => return Ok(vec![]),
Err(e) => {
eprintln!("rule34video archive fetch failed for {url}: {e}");
return Ok(stale());
}
};
let video_items = self.get_video_items_from_html(text);
let video_items = self.enrich_items(video_items, requester).await;
if video_items.is_empty() {
return Ok(stale());
}
cache.insert(index, video_items.clone());
Ok(video_items)
}
/// `/search/{q}/` takes dashes for spaces. A raw `/` splits the path (404),
/// `%20` is refused (403) and `&` cuts the query short, so keep only words.
fn search_path_segment(query: &str) -> String {
slug_dashed(query)
.split('-')
.map(|word| utf8_percent_encode(word, NON_ALPHANUMERIC).to_string())
.collect::<Vec<_>>()
.join("-")
}
/// Helper to safely extract a string between two delimiters
fn extract_between<'a>(content: &'a str, start_pat: &str, end_pat: &str) -> Option<&'a str> {
let start_idx = content.find(start_pat)? + start_pat.len();
@@ -208,7 +458,7 @@ impl Rule34videoProvider {
let url = format!(
"{}/search/{}/?mode=async&function=get_block&block_id=custom_list_videos_videos_list_search&tag_ids=&sort_by={}&from_videos={}&from_albums={}&_={}",
self.url,
query.replace(" ", "-"),
Self::search_path_segment(query),
sort_val,
page,
page,
@@ -307,30 +557,106 @@ impl Rule34videoProvider {
(formats.into_iter().map(|(_, f)| f).collect(), tags)
}
/// Artists credited on a video page: `(name, slug)` from the "Artist" group.
fn parse_artists(html: &str) -> Vec<(String, String)> {
let document = Html::parse_document(html);
let Ok(group_sel) = Selector::parse("[data-suggest-type=\"model\"] a[href*=\"/models/\"]")
else {
return vec![];
};
let name_sel = Selector::parse(".name").ok();
let mut artists: Vec<(String, String)> = Vec::new();
for link in document.select(&group_sel) {
let href = link.value().attr("href").unwrap_or_default();
let slug = href
.split("/models/")
.nth(1)
.unwrap_or_default()
.trim_matches('/')
.to_string();
let name = name_sel
.as_ref()
.and_then(|sel| link.select(sel).next())
.map(|n| n.text().collect::<String>())
.unwrap_or_else(|| link.text().collect::<String>());
let name = name.split_whitespace().collect::<Vec<_>>().join(" ");
if slug.is_empty() || name.is_empty() || artists.iter().any(|(_, s)| *s == slug) {
continue;
}
artists.push((name, slug));
}
artists
}
fn cached_detail(&self, item: &mut VideoItem) -> bool {
let Some((time, cached)) = self.details.get(&item.url) else {
return false;
};
let Some(cached) = cached.into_iter().next() else {
return false;
};
if time.elapsed().unwrap_or(DETAIL_TTL) >= DETAIL_TTL {
return false;
}
item.formats = cached.formats;
item.tags = cached.tags;
item.uploader = cached.uploader;
item.uploaderUrl = cached.uploaderUrl;
item.uploaderId = cached.uploaderId;
true
}
async fn enrich_items(&self, items: Vec<VideoItem>, requester: Requester) -> Vec<VideoItem> {
let referer = format!("{}/", self.url);
let base = self.url.clone();
// Once the site rate-limits a detail page, the rest of the batch keeps
// the listing data instead of digging the hole deeper.
let throttled = Arc::new(AtomicBool::new(false));
stream::iter(items.into_iter().map(|mut item| {
let mut requester = requester.clone();
let referer = referer.clone();
let base = base.clone();
let throttled = throttled.clone();
async move {
if item.url.is_empty() {
if item.url.is_empty() || self.cached_detail(&mut item) {
return item;
}
let html = requester
.get_with_headers(
&item.url,
vec![("Referer".to_string(), referer)],
None,
)
.await
.unwrap_or_default();
let (formats, tags) = Self::parse_detail(&html);
if throttled.load(Ordering::Relaxed) {
return item;
}
// `get_probe` hands challenges to the full fallback chain but
// reports a lasting 429 as an error rather than escalating every
// detail page to FlareSolverr.
let headers = vec![("Referer".to_string(), referer)];
let html = match requester.get_probe(&item.url, headers).await {
Ok(Some(html)) => html,
Ok(None) => return item,
Err(_) => {
throttled.store(true, Ordering::Relaxed);
return item;
}
};
let (formats, mut tags) = Self::parse_detail(&html);
if !formats.is_empty() {
item.formats = Some(formats);
}
let artists = Self::parse_artists(&html);
if let Some((name, slug)) = artists.first() {
item.uploader = Some(name.clone());
item.uploaderUrl = Some(format!("{base}/models/{slug}/"));
item.uploaderId = Some(format!("rule34video:{slug}"));
}
for (name, _) in &artists {
if !tags.iter().any(|t| t.eq_ignore_ascii_case(name)) {
tags.push(name.clone());
}
}
if !tags.is_empty() {
item.tags = Some(tags);
}
if item.formats.is_some() {
self.details.insert(item.url.clone(), vec![item.clone()]);
}
item
}
}))
@@ -344,15 +670,15 @@ impl Rule34videoProvider {
return vec![];
}
// Safely isolate the video listing section
let video_listing = match Self::extract_between(
&html,
"id=\"custom_list_videos",
"<div class=\"pagination\"",
) {
Some(content) => content,
None => return vec![],
// Isolate the video listing section. A result that fits on one page has
// no pagination block, so the listing then runs to the end.
let Some(start) = html.find("id=\"custom_list_videos") else {
return vec![];
};
let video_listing = &html[start..];
let video_listing = video_listing
.find("<div class=\"pagination\"")
.map_or(video_listing, |end| &video_listing[..end]);
let mut items = Vec::new();
// Skip the first split result as it's the preamble
@@ -449,9 +775,11 @@ impl Provider for Rule34videoProvider {
) -> Vec<VideoItem> {
let page_num = page.parse::<u8>().unwrap_or(1);
let result = match query {
Some(q) => self.query(cache, page_num, &q, &sort, options).await,
None => self.get(cache, page_num, &sort, options).await,
let target = self.resolve_target(query.as_deref(), &options).await;
let result = match &target {
Target::Latest => self.get(cache, page_num, &sort, options).await,
Target::Search(q) => self.query(cache, page_num, q, &sort, options).await,
archive => self.archive(cache, archive, page_num, &sort, options).await,
};
match result {
@@ -517,4 +845,81 @@ mod tests {
let (formats, tags) = Rule34videoProvider::parse_detail("<html></html>");
assert!(formats.is_empty() && tags.is_empty());
}
#[test]
fn parses_prefixed_queries() {
assert_eq!(
Rule34videoProvider::parse_prefixed("artist:General Butch"),
Some(Target::Artist("general-butch".into()))
);
assert_eq!(
Rule34videoProvider::parse_prefixed("category:Super Smash Bros"),
Some(Target::Category("super-smash-bros".into()))
);
assert_eq!(
Rule34videoProvider::parse_prefixed("tag:black widow (marvel)"),
Some(Target::Tag("black-widow-marvel".into()))
);
assert_eq!(Rule34videoProvider::parse_prefixed("samus aran"), None);
}
#[test]
fn picks_the_unambiguous_character_tag() {
let titles = vec![
"samus aran (metroid)".to_string(),
"samus aran's mother".to_string(),
];
assert_eq!(
Rule34videoProvider::best_tag_title(&titles, "samus aran").as_deref(),
Some("samus aran (metroid)")
);
let ambiguous = vec!["peach (mario)".to_string(), "peach (pokemon)".to_string()];
assert_eq!(Rule34videoProvider::best_tag_title(&ambiguous, "peach"), None);
}
#[test]
fn parses_tag_suggestions_and_artists() {
let suggest = r#"<div class="item checkbox-container"><input id="tag_143" type="checkbox" value="143" data-title="samus aran (metroid)"/></div>"#;
assert_eq!(
Rule34videoProvider::parse_tag_suggestions(suggest),
vec!["samus aran (metroid)".to_string()]
);
let page = r#"<div class="col vp-group" data-suggest-type="model"><div class="label">Artist</div>
<span class="chip"><a class="item btn_link" href="https://rule34video.com/models/joigang-va/">
<span class="name">JOIGang (VA)</span></a></span></div>"#;
assert_eq!(
Rule34videoProvider::parse_artists(page),
vec![("JOIGang (VA)".to_string(), "joigang-va".to_string())]
);
}
#[test]
fn parses_a_single_page_listing_without_pagination() {
let html = r#"<div id="custom_list_videos_common_videos_items">
<div class="item thumb video_7"><a class="th js-open-popup" href="https://rule34video.com/video/7/elf/">
<img class="thumb" src="https://rule34video.com/t/7.jpg"><div class="time">0:30</div>
<div class="thumb_title">Elf</div></a></div></div></div>"#;
let items = Rule34videoProvider::new().get_video_items_from_html(html.to_string());
assert_eq!(items.len(), 1);
assert_eq!(items[0].id, "7");
}
#[test]
fn search_segment_uses_dashes_and_drops_path_breaking_punctuation() {
assert_eq!(Rule34videoProvider::search_path_segment("elf girl"), "elf-girl");
assert_eq!(Rule34videoProvider::search_path_segment("2b/9s"), "2b-9s");
assert_eq!(Rule34videoProvider::search_path_segment("Tifa & Aerith"), "tifa-aerith");
assert_eq!(Rule34videoProvider::search_path_segment("Pokémon"), "pok%C3%A9mon");
}
#[test]
fn remembers_resolved_targets_by_slug() {
let provider = Rule34videoProvider::new();
provider.remember_target("Samus Aran", &Target::Tag("samus-aran-metroid".into()));
assert_eq!(
provider.remembered_target("samus aran"),
Some(Target::Tag("samus-aran-metroid".into()))
);
assert_eq!(provider.remembered_target("samus"), None);
}
}

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, report_provider_error, report_provider_error_background, requester_or_default,
Provider, probe_first_archive, report_provider_error, report_provider_error_background,
requester_or_default,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -517,6 +518,28 @@ impl SextbProvider {
Target::Search { slug }
}
/// A free-form query that names an actress or a studio outside the loaded
/// catalogs goes to that archive (probed on the site), else keyword search.
async fn resolve_keyword(&self, slug: &str, options: &ServerOptions) -> Option<Target> {
if slug.is_empty() {
return None;
}
let candidates = [
Target::Actress { slug: slug.to_string() },
Target::Studio { slug: slug.to_string() },
];
let urls: Vec<String> = candidates
.iter()
.map(|t| self.build_url_for_target(t, 1))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("tray-item") && !body.contains("No Video were found")
})
.await?;
candidates.into_iter().nth(index)
}
fn build_url_for_target(&self, target: &Target, page: u16) -> String {
match target {
Target::Section { section, sort } => {
@@ -1092,7 +1115,12 @@ impl SextbProvider {
per_page_limit: usize,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
let target = self.resolve_query_target(query, sort);
let mut target = self.resolve_query_target(query, sort);
if let Target::Search { slug } = &target {
if let Some(found) = self.resolve_keyword(slug, &options).await {
target = found;
}
}
let url = self.build_url_for_target(&target, page);
self.fetch_items_for_url(cache, url, per_page_limit, &options)
.await

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, report_provider_error, requester_or_default,
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -367,6 +368,26 @@ impl ThepornbunnyProvider {
}
}
/// A free-form query that names a pornstar, studio or tag goes to that
/// archive (probed on the site; unknown names redirect home), else keyword
/// search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let urls: Vec<String> = ["pornstar", "studio", "tag"]
.iter()
.map(|kind| format!("{BASE_URL}/{kind}/{slug}/"))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("th-description")
})
.await?;
Some(Target::Archive(urls[index].trim_end_matches('/').to_string()))
}
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat.starts_with(BASE_URL) && cat != "all" {
@@ -476,10 +497,15 @@ impl Provider for ThepornbunnyProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page_limit = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(ref q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
_ => Self::resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page_limit, options)

View File

@@ -1,10 +1,12 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, proxy_aylo_thumb, report_provider_error, requester_or_default,
Provider, build_proxy_url, probe_first_archive, proxy_aylo_thumb, report_provider_error,
requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::parse_abbreviated_number;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
@@ -136,6 +138,25 @@ impl Tube8Provider {
]
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, requester: &mut Requester, query: &str) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Pornstar { slug: slug.clone() },
Target::Channel { slug },
];
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
let (index, _) = probe_first_archive(requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("video-box")
})
.await?;
candidates.into_iter().nth(index)
}
fn target_from_request(query: Option<&str>, sort: &str) -> Target {
if let Some(q) = query.map(str::trim).filter(|v| !v.is_empty()) {
let lower = q.to_ascii_lowercase();
@@ -439,7 +460,13 @@ impl Provider for Tube8Provider {
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let target = Self::target_from_request(query.as_deref(), &sort);
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let mut target = Self::target_from_request(query.as_deref(), &sort);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(&mut requester, query).await {
target = found;
}
}
let url = self.build_url(&target, page);
let old_items = match cache.get(&url) {
@@ -450,7 +477,6 @@ impl Provider for Tube8Provider {
None => vec![],
};
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let text = match requester
.get_with_headers(&url, Self::html_headers(), None)
.await

621
src/providers/txxx.rs Normal file
View File

@@ -0,0 +1,621 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
use async_trait::async_trait;
use base64::Engine;
use error_chain::error_chain;
use futures::stream::{self, StreamExt};
use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
use serde_json::Value;
use std::collections::HashMap;
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
crate::providers::ProviderChannelMetadata {
group_id: "mainstream-tube",
tags: &["tube", "hd", "amateur", "free"],
};
const BASE_URL: &str = "https://txxx.me";
const CHANNEL_ID: &str = "txxx";
const ENRICH_CONCURRENCY: usize = 8;
const MAX_COUNT: usize = 60;
// Top categories by video count from `/api/json/categories/`. The `dir` slug
// is the `object_id` of the `categories` section of the videos API.
const CATEGORIES: &[(&str, &str)] = &[
("hd", "HD"),
("big-tits", "Big Tits"),
("brunette", "Brunette"),
("blowjob", "Blowjob"),
("teens", "Teens (18+)"),
("amateur", "Amateur"),
("blonde", "Blonde"),
("hardcore", "Hardcore"),
("milf", "MILF"),
("anal", "Anal"),
("big-dick", "Big Cock"),
("asian", "Asian"),
("cumshots", "Cumshot"),
("big-butt", "Big Ass"),
("japanese", "Japanese"),
("fetish", "Fetish"),
("dildos-toys", "Toys"),
("small-tits", "Small Tits"),
("pov", "POV"),
("masturbate", "Masturbation"),
("babes", "Babe"),
("lesbian", "Lesbian"),
("deep-throat", "Deepthroat"),
("threesomes", "Threesome"),
("facial", "Facial"),
("interracial", "Interracial"),
("pornstars", "Pornstar"),
("solo-girl", "Solo Female"),
("pantyhose", "Stockings"),
("cunnilingus", "Cunnilingus"),
("mature", "Mature"),
("handjob", "Handjob"),
("tattoos", "Tattoo"),
("doggy-style", "Doggystyle"),
("fingering", "Fingering"),
("hairy", "Hairy"),
("jav-censored", "JAV Censored"),
("creampie", "Creampie"),
("lingerie", "Lingerie"),
("bdsm", "BDSM"),
("ebony", "Ebony"),
("redhead", "Red Head"),
("group-sex", "Group Sex"),
("latina", "Latina"),
("foot-fetish", "Foot Fetish"),
("bbw", "BBW"),
("outdoor", "Outdoor"),
("webcam", "Webcam"),
("european", "European"),
("dp", "Double Penetration"),
("cowgirl", "Cowgirl"),
("skinny", "Skinny"),
("gangbang", "Gangbang"),
("wife", "Wife"),
("german", "German"),
("jav-uncensored", "JAV Uncensored"),
("step-fantasy", "Step Fantasy"),
("couple", "Couple"),
("vintage", "Vintage"),
("female-orgasm", "Female Orgasm"),
("femdom", "Femdom"),
("squirting", "Squirt"),
("casting", "Casting"),
("compilation", "Compilation"),
("russian", "Russian"),
("shaved", "Shaved"),
("public", "Public"),
("massage", "Massage"),
("french", "French"),
("british", "British"),
("high-heels", "High Heels"),
("rimming", "Rimming"),
("swallow", "Swallow Cum"),
("cuckold", "Cuckold"),
("striptease", "Striptease"),
("strapon", "Strapon"),
("close-up", "Close-up"),
("american", "American"),
("grannies", "Granny"),
];
error_chain! {
foreign_links {
Io(std::io::Error);
Json(serde_json::Error);
}
errors {
Parse(msg: String) {
description("parse error")
display("parse error: {}", msg)
}
}
}
#[derive(Debug, Clone)]
pub struct TxxxProvider;
#[derive(Debug, Clone, PartialEq, Eq)]
enum Target {
Latest { sort: &'static str },
Category { dir: String, sort: &'static str },
Search { query: String },
}
/// A listing card plus the pixel heights the site advertises per file suffix.
struct Stub {
item: VideoItem,
heights: HashMap<String, u32>,
}
impl TxxxProvider {
pub fn new() -> Self {
Self
}
fn build_channel(&self, _clientversion: ClientVersion) -> Channel {
let mut category_options = vec![FilterOption {
id: "all".to_string(),
title: "All".to_string(),
}];
category_options.extend(CATEGORIES.iter().map(|(dir, title)| FilterOption {
id: (*dir).to_string(),
title: (*title).to_string(),
}));
Channel {
id: CHANNEL_ID.to_string(),
name: "TXXX".to_string(),
description: "Large free tube with HD videos, categories and search.".to_string(),
premium: false,
favicon: "https://www.google.com/s2/favicons?sz=64&domain=txxx.com".to_string(),
status: "active".to_string(),
categories: vec![],
options: vec![
ChannelOption {
id: "sort".to_string(),
title: "Sort".to_string(),
description: "Sort the TXXX feed or category archive.".to_string(),
systemImage: "arrow.up.arrow.down".to_string(),
colorName: "blue".to_string(),
options: vec![
FilterOption { id: "new".to_string(), title: "Latest".to_string() },
FilterOption { id: "popular".to_string(), title: "Most Viewed".to_string() },
FilterOption { id: "rated".to_string(), title: "Top Rated".to_string() },
FilterOption { id: "longest".to_string(), title: "Longest".to_string() },
],
multiSelect: false,
},
ChannelOption {
id: "categories".to_string(),
title: "Categories".to_string(),
description: "Browse a TXXX category archive.".to_string(),
systemImage: "square.grid.2x2".to_string(),
colorName: "orange".to_string(),
options: category_options,
multiSelect: false,
},
],
nsfw: true,
cacheDuration: Some(1800),
ytdlpCommand: None,
}
}
fn slugify(value: &str) -> String {
let mut out = String::new();
let mut last_dash = true;
for c in value.trim().chars() {
if c.is_alphanumeric() {
out.extend(c.to_lowercase());
last_dash = false;
} else if !last_dash {
out.push('-');
last_dash = true;
}
}
out.trim_matches('-').to_string()
}
fn category_dir(value: &str) -> Option<&'static str> {
let slug = Self::slugify(value);
if slug.is_empty() {
return None;
}
CATEGORIES
.iter()
.find_map(|(dir, title)| (*dir == slug || Self::slugify(title) == slug).then_some(*dir))
}
fn sort_key(sort: Option<&str>) -> &'static str {
match sort.map(|s| s.trim().to_ascii_lowercase()).as_deref() {
Some("popular") | Some("most viewed") | Some("most-popular") | Some("views") => {
"most-popular"
}
Some("rated") | Some("top rated") | Some("top-rated") | Some("rating") => "top-rated",
Some("longest") | Some("duration") => "longest",
_ => "latest-updates",
}
}
fn pick_target(query: Option<&str>, options: &ServerOptions) -> Target {
let sort = Self::sort_key(options.sort.as_deref());
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
let lower = q.to_ascii_lowercase();
for prefix in ["cat:", "category:"] {
if let Some(rest) = lower.strip_prefix(prefix) {
let dir = Self::category_dir(rest)
.map(str::to_string)
.unwrap_or_else(|| Self::slugify(rest));
return Target::Category { dir, sort };
}
}
if let Some(dir) = Self::category_dir(q) {
return Target::Category { dir: dir.to_string(), sort };
}
return Target::Search { query: q.to_string() };
}
if let Some(cat) = options.categories.as_deref() {
let cat = cat.trim();
if !cat.is_empty() && !cat.eq_ignore_ascii_case("all") {
let dir = Self::category_dir(cat)
.map(str::to_string)
.unwrap_or_else(|| Self::slugify(cat));
return Target::Category { dir, sort };
}
}
Target::Latest { sort }
}
fn build_listing_url(target: &Target, page: u32, count: usize) -> String {
match target {
Target::Latest { sort } => format!(
"{BASE_URL}/api/json/videos2/86400/str/{sort}/{count}/..{page}.all...jsond"
),
Target::Category { dir, sort } => format!(
"{BASE_URL}/api/json/videos2/86400/str/{sort}/{count}/categories.{dir}.{page}.all...jsond"
),
Target::Search { query } => format!(
"{BASE_URL}/api/videos2.php?params=86400/str/relevance/{count}/search..{page}.all..&s={}",
utf8_percent_encode(query, NON_ALPHANUMERIC)
),
}
}
fn str_field(value: &Value, key: &str) -> String {
match value.get(key) {
Some(Value::String(s)) => s.trim().to_string(),
Some(Value::Number(n)) => n.to_string(),
_ => String::new(),
}
}
/// `||_hq.mp4|1278x720|3485|...||_tr.mp4|382x216|...` -> {"_hq.mp4": 720, ...}
fn parse_file_heights(file_formats: &str) -> HashMap<String, u32> {
let mut heights = HashMap::new();
for entry in file_formats.split("||").filter(|e| !e.is_empty()) {
let mut parts = entry.split('|');
let (Some(suffix), Some(dims)) = (parts.next(), parts.next()) else {
continue;
};
if let Some(h) = dims.split('x').nth(1).and_then(|h| h.parse::<u32>().ok()) {
heights.insert(suffix.to_string(), h);
}
}
heights
}
fn parse_listing(json: &str) -> Result<Vec<Stub>> {
let root: Value = serde_json::from_str(json)?;
if root.get("error").is_some() {
return Ok(vec![]);
}
let Some(videos) = root.get("videos").and_then(Value::as_array) else {
return Ok(vec![]);
};
let mut out = Vec::new();
for video in videos {
let id = Self::str_field(video, "video_id");
let dir = Self::str_field(video, "dir");
let title = Self::str_field(video, "title");
if id.is_empty() || dir.is_empty() || title.is_empty() {
continue;
}
if Self::str_field(video, "is_private") == "1" {
continue;
}
let url = format!("{BASE_URL}/videos/{id}/{dir}/");
let duration = parse_time_to_seconds(&Self::str_field(video, "duration"))
.and_then(|s| u32::try_from(s).ok())
.unwrap_or(0);
let thumb = Self::str_field(video, "scr");
let mut item = VideoItem::new(
id,
title,
url,
CHANNEL_ID.to_string(),
thumb,
duration,
);
item.views = Self::str_field(video, "video_viewed").parse::<u32>().ok();
if let Ok(rating) = Self::str_field(video, "rating").parse::<f32>() {
if rating > 0.0 {
// The site rates 0..5.
item.rating = Some((rating * 20.0).min(100.0));
}
}
item.aspectRatio = Some(16.0 / 9.0);
let preview = Self::str_field(video, "pv");
if !preview.is_empty() {
item.preview = Some(if preview.starts_with("http") {
preview
} else {
format!("https://{}", preview.trim_start_matches("//"))
});
}
let uploader = {
let channel = Self::str_field(video, "content_source_name");
if channel.is_empty() {
let name = Self::str_field(video, "display_name");
if name.is_empty() { Self::str_field(video, "username") } else { name }
} else {
channel
}
};
if !uploader.is_empty() {
item.uploader = Some(uploader);
let user_id = Self::str_field(video, "user_id");
if !user_id.is_empty() && user_id != "0" {
item.uploaderId = Some(format!("{CHANNEL_ID}:{user_id}"));
}
}
let tags: Vec<String> = Self::str_field(video, "categories")
.split(',')
.map(|t| t.trim().to_string())
.filter(|t| !t.is_empty())
.collect();
if !tags.is_empty() {
item.tags = Some(tags);
}
let posted = Self::str_field(video, "post_date");
if let Ok(dt) = chrono::NaiveDateTime::parse_from_str(&posted, "%Y-%m-%d %H:%M:%S") {
item.uploadedAt = u64::try_from(dt.and_utc().timestamp()).ok();
}
out.push(Stub {
item,
heights: Self::parse_file_heights(&Self::str_field(video, "file_formats")),
});
}
Ok(out)
}
/// `video_url` in `/api/videofile.php` is base64 of the `/get_file/...`
/// path with Cyrillic look-alike letters swapped in and `~` for `=`.
fn decode_video_url(raw: &str) -> Option<String> {
let mapped: String = raw
.chars()
.map(|c| match c {
'А' => 'A', 'В' => 'B', 'С' => 'C', 'Е' => 'E', 'М' => 'M', 'Н' => 'H',
'К' => 'K', 'О' => 'O', 'Р' => 'P', 'Т' => 'T', 'Х' => 'X',
'а' => 'a', 'с' => 'c', 'е' => 'e', 'о' => 'o', 'р' => 'p', 'х' => 'x',
'у' => 'y', 'і' => 'i', 'ј' => 'j',
'~' => '=',
other => other,
})
.collect();
let trimmed = mapped.trim_end_matches('=');
let bytes = base64::engine::general_purpose::STANDARD_NO_PAD
.decode(trimmed)
.or_else(|_| base64::engine::general_purpose::URL_SAFE_NO_PAD.decode(trimmed))
.ok()?;
let path = String::from_utf8(bytes).ok()?;
if path.starts_with("http") {
Some(path)
} else if path.starts_with("/get_file/") {
Some(format!("{BASE_URL}{path}"))
} else {
None
}
}
/// Builds the playable formats (best first) from the videofile response.
fn parse_formats(json: &str, heights: &HashMap<String, u32>) -> Vec<VideoFormat> {
let Ok(Value::Array(entries)) = serde_json::from_str::<Value>(json) else {
return vec![];
};
let mut formats: Vec<(u32, VideoFormat)> = Vec::new();
for entry in entries {
let suffix = Self::str_field(&entry, "format");
if suffix.is_empty() || suffix == "_tr.mp4" {
continue;
}
let Some(url) = Self::decode_video_url(&Self::str_field(&entry, "video_url")) else {
continue;
};
let height = heights.get(&suffix).copied().unwrap_or(0);
let quality = if height > 0 { format!("{height}p") } else { "auto".to_string() };
let mut format = VideoFormat::new(url, quality.clone(), "mp4".to_string())
.format_id(format!("mp4-{}", suffix.trim_end_matches(".mp4").trim_start_matches('_')));
if height > 0 {
format = format.height(height);
}
formats.push((height, format));
}
formats.sort_by(|a, b| b.0.cmp(&a.0));
formats.into_iter().map(|(_, f)| f).collect()
}
async fn enrich(stubs: Vec<Stub>, requester: Requester) -> Vec<VideoItem> {
let referer = format!("{BASE_URL}/");
stream::iter(stubs.into_iter().map(|stub| {
let mut requester = requester.clone();
let referer = referer.clone();
async move {
let Stub { mut item, heights } = stub;
let url = format!(
"{BASE_URL}/api/videofile.php?video_id={}&lifetime=8640000",
item.id
);
let body = requester
.get_with_headers(&url, vec![("Referer".to_string(), referer)], None)
.await
.unwrap_or_default();
let formats = Self::parse_formats(&body, &heights);
if formats.is_empty() {
// The page URL is not yt-dlp-resolvable, so an item without
// formats could never play.
return None;
}
item.formats = Some(formats);
Some(item)
}
}))
.buffered(ENRICH_CONCURRENCY)
.filter_map(|item| async move { item })
.collect::<Vec<_>>()
.await
}
async fn fetch(
target: &Target,
page: u32,
count: usize,
options: &ServerOptions,
) -> Result<Vec<VideoItem>> {
let url = Self::build_listing_url(target, page, count);
let mut requester = requester_or_default(options, CHANNEL_ID, "fetch");
let body = requester
.get_with_headers(
&url,
vec![("Referer".to_string(), format!("{BASE_URL}/"))],
None,
)
.await
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))?;
let stubs = Self::parse_listing(&body)?;
Ok(Self::enrich(stubs, requester).await)
}
}
#[async_trait]
impl Provider for TxxxProvider {
async fn get_videos(
&self,
_cache: VideoCache,
_pool: DbPool,
_sort: String,
query: Option<String>,
page: String,
per_page: String,
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u32>().unwrap_or(1).max(1);
let count = per_page.parse::<usize>().unwrap_or(30).clamp(1, MAX_COUNT);
let target = Self::pick_target(query.as_deref(), &options);
match Self::fetch(&target, page, count, &options).await {
Ok(items) => items,
Err(e) => {
report_provider_error(CHANNEL_ID, "get_videos", &e.to_string()).await;
vec![]
}
}
}
fn get_channel(&self, clientversion: ClientVersion) -> Option<Channel> {
Some(self.build_channel(clientversion))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn opts() -> ServerOptions {
ServerOptions {
featured: None,
category: None,
sites: None,
filter: None,
language: None,
public_url_base: None,
requester: None,
network: None,
stars: None,
categories: None,
duration: None,
sort: None,
sexuality: None,
client_version: None,
}
}
#[test]
fn builds_urls() {
assert_eq!(
TxxxProvider::build_listing_url(&Target::Latest { sort: "latest-updates" }, 2, 30),
"https://txxx.me/api/json/videos2/86400/str/latest-updates/30/..2.all...jsond"
);
assert_eq!(
TxxxProvider::build_listing_url(
&Target::Category { dir: "milf".into(), sort: "most-popular" },
1,
60
),
"https://txxx.me/api/json/videos2/86400/str/most-popular/60/categories.milf.1.all...jsond"
);
assert_eq!(
TxxxProvider::build_listing_url(&Target::Search { query: "big tits".into() }, 3, 60),
"https://txxx.me/api/videos2.php?params=86400/str/relevance/60/search..3.all..&s=big%20tits"
);
}
#[test]
fn routes_queries() {
let o = opts();
assert_eq!(
TxxxProvider::pick_target(Some("Big Tits"), &o),
Target::Category { dir: "big-tits".into(), sort: "latest-updates" }
);
assert_eq!(
TxxxProvider::pick_target(Some("cat:Big Cock"), &o),
Target::Category { dir: "big-dick".into(), sort: "latest-updates" }
);
assert_eq!(
TxxxProvider::pick_target(Some("mia khalifa"), &o),
Target::Search { query: "mia khalifa".into() }
);
assert_eq!(
TxxxProvider::pick_target(None, &o),
Target::Latest { sort: "latest-updates" }
);
}
#[test]
fn decodes_cyrillic_obfuscated_video_url() {
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}i8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}zQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
assert_eq!(
TxxxProvider::decode_video_url(raw).as_deref(),
Some("https://txxx.me/get_file/25/860135ad41383f324fbab2ee8724dbc39cbe2062e2/21808000/21808705/21808705_hq.mp4/?d=3485&br=218&ti=1791646629")
);
}
#[test]
fn parses_listing_and_formats() {
let json = r#"{"videos":[{"video_id":"21808705","title":"A title","dir":"a-title",
"duration":"58:05","video_viewed":"1200","rating":"3.9000","post_date":"2026-10-09 20:34:06",
"file_formats":"||_hq.mp4|1278x720|3485|507453306|117|30|0||_tr.mp4|382x216|7|202168|0|0|0",
"categories":"American,MILF","display_name":"someone","user_id":"6119563",
"scr":"https://tn.txxx.tube/a.jpg","pv":"vp2.txxx.com/c14/a_tr.mp4","is_private":"0"},
{"video_id":"2","title":"Private","dir":"p","duration":"1:00","is_private":"1"}]}"#;
let stubs = TxxxProvider::parse_listing(json).unwrap();
assert_eq!(stubs.len(), 1);
let item = &stubs[0].item;
assert_eq!(item.url, "https://txxx.me/videos/21808705/a-title/");
assert_eq!(item.duration, 58 * 60 + 5);
assert_eq!(item.views, Some(1200));
assert_eq!(item.uploaderId.as_deref(), Some("txxx:6119563"));
assert_eq!(item.preview.as_deref(), Some("https://vp2.txxx.com/c14/a_tr.mp4"));
assert_eq!(item.tags.as_ref().unwrap(), &vec!["American".to_string(), "MILF".to_string()]);
assert_eq!(stubs[0].heights.get("_hq.mp4"), Some(&720));
}
}

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::videos::{ServerOptions, VideoItem};
@@ -426,6 +428,23 @@ impl XgroovyProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("data-video-id")
})
.await?;
candidates.into_iter().nth(index)
}
fn resolve_option_target(&self, options: &ServerOptions) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat != "all" && !cat.is_empty() {
@@ -488,10 +507,15 @@ impl Provider for XgroovyProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
_ => self.resolve_option_target(&options),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -94,6 +96,7 @@ enum Target {
Search(String),
Category(String),
Channel(String),
Pornstar(String),
}
#[derive(Debug, Clone)]
@@ -208,6 +211,7 @@ impl XhamsterProvider {
}
Target::Category(slug) => format!("{BASE_URL}/categories/{slug}"),
Target::Channel(slug) => format!("{BASE_URL}/channels/{slug}"),
Target::Pornstar(slug) => format!("{BASE_URL}/pornstars/{slug}"),
};
if page <= 1 {
base
@@ -385,6 +389,7 @@ impl XhamsterProvider {
match kind.trim().to_ascii_lowercase().as_str() {
"cat" | "category" => return Target::Category(slug),
"channel" | "channels" => return Target::Channel(slug),
"pornstar" | "pornstars" | "star" | "model" => return Target::Pornstar(slug),
_ => {}
}
}
@@ -401,6 +406,23 @@ impl XhamsterProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("video-thumb-info__name")
})
.await?;
candidates.into_iter().nth(index)
}
fn resolve_sort_target(sort: &str) -> Target {
match sort.trim().to_ascii_lowercase().as_str() {
"popular" | "viewed" | "most_viewed" | "mostviewed" => Target::MostViewed,
@@ -472,10 +494,15 @@ impl Provider for XhamsterProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
_ => self.resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -86,6 +88,8 @@ enum Target {
Best,
Search(String),
Archive(String),
/// Channel / pornstar profile, served as JSON by `/channels/{slug}/videos/new/{n}`.
Channel(String),
}
#[derive(Debug, Clone)]
@@ -238,6 +242,9 @@ impl XvideosProvider {
format!("{BASE_URL}/?k={encoded}&p={}", page - 1)
}
}
Target::Channel(slug) => {
format!("{BASE_URL}/channels/{slug}/videos/new/{}", page.saturating_sub(1))
}
Target::Archive(base_url) => {
let base = base_url.trim_end_matches('/');
if page <= 1 {
@@ -528,9 +535,8 @@ impl XvideosProvider {
let slug = value.replace(' ', "_");
return Target::Archive(format!("{BASE_URL}/c/{slug}"));
}
"uploader" | "channel" | "profile" => {
let slug = value.replace(' ', "_").to_ascii_lowercase();
return Target::Archive(format!("{BASE_URL}/{slug}"));
"uploader" | "channel" | "profile" | "pornstar" | "star" | "model" => {
return Target::Channel(slug_dashed(value));
}
_ => {}
}
@@ -545,6 +551,83 @@ impl XvideosProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a channel / pornstar profile goes to its
/// archive (probed on the site; unknown names land on an empty profile),
/// else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let target = Target::Channel(slug);
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::target_url(&target, 1)],
&format!("{BASE_URL}/"),
|body| body.contains("\"videos\":[{"),
)
.await
.map(|_| target)
}
/// Parses the JSON served for a channel / profile video list.
fn parse_channel_json(json: &str, limit: usize) -> Result<Vec<VideoItem>> {
let root: serde_json::Value = serde_json::from_str(json)
.map_err(|e| Error::from(format!("channel json: {e}")))?;
let Some(videos) = root.get("videos").and_then(|v| v.as_array()) else {
return Ok(vec![]);
};
let text = |v: &serde_json::Value, key: &str| -> String {
v.get(key).and_then(|x| x.as_str()).unwrap_or_default().trim().to_string()
};
let mut items = Vec::new();
for video in videos {
let eid = text(video, "eid");
let thumb = text(video, "i");
let title = {
let full = text(video, "tf");
if full.is_empty() { text(video, "t") } else { full }
};
if eid.is_empty() || thumb.is_empty() || title.is_empty() {
continue;
}
let id = video
.get("id")
.map(|x| x.to_string().trim_matches('"').to_string())
.unwrap_or_else(|| eid.clone());
let mut item = VideoItem::new(
id,
Self::decode_html(&title),
format!("{BASE_URL}/video.{eid}/x"),
CHANNEL_ID.to_string(),
thumb,
Self::parse_duration(&text(video, "d")),
);
item.views = Self::parse_views(&text(video, "n"));
item.rating = text(video, "r").trim_end_matches('%').parse::<f32>().ok();
let preview = text(video, "ipu");
if !preview.is_empty() {
item.preview = Some(preview);
}
let name = text(video, "pn");
let slug = text(video, "pu");
if !name.is_empty() {
item.uploader = Some(name);
let slug = slug.trim_matches('/').to_string();
if !slug.is_empty() {
item.uploaderUrl = Some(format!("{BASE_URL}/{slug}"));
item.uploaderId = Some(format!("{CHANNEL_ID}:{slug}"));
}
}
items.push(item);
if items.len() >= limit {
break;
}
}
Ok(items)
}
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat != "all" && !cat.is_empty() {
@@ -602,7 +685,11 @@ impl XvideosProvider {
return Ok(vec![]);
}
let items = Self::parse_listing(&html, per_page)?;
let items = if matches!(target, Target::Channel(_)) {
Self::parse_channel_json(&html, per_page)?
} else {
Self::parse_listing(&html, per_page)?
};
if !items.is_empty() {
cache.insert(cache_key, items.clone());
}
@@ -625,10 +712,15 @@ impl Provider for XvideosProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
_ => Self::resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,8 +1,11 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::parse_abbreviated_number;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoItem};
@@ -248,6 +251,33 @@ impl YoupornProvider {
}
}
/// A free-form query that names a pornstar, channel or amateur goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(
&self,
requester: &mut Requester,
query: &str,
sort: &str,
) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let sort = Self::normalized_sort(sort).to_string();
let candidates = [
Target::Pornstar { slug: slug.clone(), sort: sort.clone() },
Target::Channel { slug: slug.clone(), sort: sort.clone() },
Target::Amateur { slug, sort },
];
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
let referer = format!("{}/", self.url.trim_end_matches('/'));
let (index, _) = probe_first_archive(requester, &urls, &referer, |body| {
body.contains("js_video-box")
})
.await?;
candidates.into_iter().nth(index)
}
fn build_url(&self, target: &Target, page: u8) -> String {
match target {
Target::Latest { sort } => format!(
@@ -543,7 +573,13 @@ impl Provider for YoupornProvider {
) -> Vec<VideoItem> {
let query = query.unwrap_or_default();
let page = page.parse::<u8>().unwrap_or(1);
let target = self.target_from_query(&query, &sort);
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let mut target = self.target_from_query(&query, &sort);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(&mut requester, query, &sort).await {
target = found;
}
}
let video_url = self.build_url(&target, page);
let old_items = match cache.get(&video_url) {
@@ -554,7 +590,6 @@ impl Provider for YoupornProvider {
None => vec![],
};
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let referer = format!("{}/", self.url.trim_end_matches('/'));
let text = match requester
.get_with_headers(&video_url, Self::html_headers(&referer), Some(Version::HTTP_11))

View File

@@ -229,63 +229,90 @@ impl ArchivebateProxy {
embed_url: &str,
requester: &mut Requester,
) -> Option<String> {
let response = requester
.get_raw_with_headers_timeout(
embed_url,
Self::html_headers(detail_url),
Some(StdDuration::from_secs(8)),
None,
)
.await
.ok()?;
if !response.status().is_success() {
return None;
// The embed host is sometimes slow on a first (cold) request and fast on
// the next, so one timed-out attempt is retried.
for _ in 0..2 {
let Ok(response) = requester
.get_raw_with_headers_timeout(
embed_url,
Self::html_headers(detail_url),
Some(StdDuration::from_secs(10)),
None,
)
.await
else {
continue;
};
if !response.status().is_success() {
return None;
}
let html = response.text().await.ok()?;
return Self::extract_mixdrop_media_url(&html);
}
let html = response.text().await.ok()?;
Self::extract_mixdrop_media_url(&html)
None
}
}
/// A resolved Mixdrop media file plus the embed page it came from (the CDN is
/// played with that page as `Referer`).
#[derive(Debug, Clone)]
pub struct ResolvedMedia {
pub media_url: String,
pub embed_url: String,
}
/// Resolves an archivebate watch page (`https://archivebate.com/watch/{id}`)
/// to the real media URL: watch page -> Mixdrop embed -> unpacked
/// `MDCore.wurl`. Shared by the lazy `/proxy/archivebate` redirect and by the
/// provider, which hands the app the media URL directly.
pub async fn resolve_media(requester: &mut Requester, detail_url: &str) -> Option<ResolvedMedia> {
let detail_html = requester
.get_with_headers(
detail_url,
ArchivebateProxy::html_headers("https://archivebate.com/"),
Some(Version::HTTP_11),
)
.await
.unwrap_or_default();
if detail_html.is_empty() {
return None;
}
let mut embeds: Vec<String> = Vec::new();
if let Some(iframe_url) = ArchivebateProxy::first_iframe_source_from_html(&detail_html)
.map(|value| ArchivebateProxy::absolute_url(&value))
{
if ArchivebateProxy::is_mixdrop_host(&iframe_url) {
embeds.push(iframe_url);
}
}
if let Some(embed_url) = ArchivebateProxy::download_fid_from_detail_html(&detail_html)
.map(|value| ArchivebateProxy::absolute_url(&value))
.and_then(|value| ArchivebateProxy::mixdrop_embed_url_from_download_url(&value))
{
embeds.push(embed_url);
}
for embed_url in embeds {
if let Some(media_url) =
ArchivebateProxy::resolve_mixdrop_media_from_embed(detail_url, &embed_url, requester).await
{
return Some(ResolvedMedia { media_url, embed_url });
}
}
None
}
impl crate::proxies::Proxy for ArchivebateProxy {
async fn get_video_url(&self, url: String, requester: web::types::State<Requester>) -> String {
let Some(detail_url) = Self::normalize_detail_request(&url) else {
return String::new();
};
let mut requester = requester.get_ref().clone();
let detail_html = requester
.get_with_headers(
&detail_url,
Self::html_headers("https://archivebate.com/"),
Some(Version::HTTP_11),
)
resolve_media(&mut requester, &detail_url)
.await
.unwrap_or_default();
if detail_html.is_empty() {
return String::new();
}
if let Some(iframe_url) = Self::first_iframe_source_from_html(&detail_html).map(|value| Self::absolute_url(&value)) {
if Self::is_mixdrop_host(&iframe_url) {
if let Some(media_url) =
Self::resolve_mixdrop_media_from_embed(&detail_url, &iframe_url, &mut requester).await
{
return media_url;
}
}
}
if let Some(download_fid) = Self::download_fid_from_detail_html(&detail_html).map(|value| Self::absolute_url(&value)) {
if let Some(embed_url) = Self::mixdrop_embed_url_from_download_url(&download_fid) {
if let Some(media_url) =
Self::resolve_mixdrop_media_from_embed(&detail_url, &embed_url, &mut requester).await
{
return media_url;
}
}
}
String::new()
.map(|resolved| resolved.media_url)
.unwrap_or_default()
}
}

196
src/proxies/eporner.rs Normal file
View File

@@ -0,0 +1,196 @@
use std::collections::HashMap;
use std::rc::Rc;
use std::sync::{Mutex, OnceLock};
use std::time::{Duration, Instant};
use futures::StreamExt;
use http_body_util::BodyExt;
use ntex::http::Method;
use ntex::http::body::{Body, SizedStream};
use ntex::http::header::{ACCEPT_RANGES, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, RANGE};
use ntex::util::Bytes;
use ntex::web::{self, HttpRequest};
use wreq::Client;
use wreq::redirect::Policy;
use wreq_util::Emulation;
use crate::util::eporner_resolve::resolve_ranked;
use crate::util::requester::Requester;
/// The signed CDN links expire two hours after resolving; renew well before.
const LINK_TTL: Duration = Duration::from_secs(90 * 60);
const LINK_CACHE_SIZE: usize = 2000;
/// `{alnum_id}/{height}` -> (resolved at, signed CDN URL).
fn links() -> &'static Mutex<HashMap<String, (Instant, String)>> {
static LINKS: OnceLock<Mutex<HashMap<String, (Instant, String)>>> = OnceLock::new();
LINKS.get_or_init(|| Mutex::new(HashMap::new()))
}
/// Redirects are not followed so the CDN's `na.mp4` decoy shows up as a 302.
fn cdn_client() -> &'static Client {
static CLIENT: OnceLock<Client> = OnceLock::new();
CLIENT.get_or_init(|| {
Client::builder()
.tls_cert_verification(false)
.emulation(Emulation::Firefox151)
.redirect(Policy::none())
.build()
.expect("eporner CDN client")
})
}
/// `{alnum_id}[/{height}][.mp4]`.
fn parse_endpoint(endpoint: &str) -> Option<(String, Option<u32>)> {
let endpoint = endpoint.trim_matches('/').trim_end_matches(".mp4");
let mut parts = endpoint.split('/');
let alnum_id = parts
.next()
.filter(|id| !id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric()))?;
let height = parts.next().and_then(|h| h.parse::<u32>().ok());
Some((alnum_id.to_string(), height))
}
async fn resolve_link(
requester: &web::types::State<Requester>,
alnum_id: &str,
height: Option<u32>,
) -> Option<String> {
let page_url = format!("https://www.eporner.com/video-{alnum_id}/");
let mut requester = requester.get_ref().clone();
let ranked = resolve_ranked(&mut requester, alnum_id, &page_url).await?;
height
.and_then(|h| ranked.iter().find(|(height, _)| *height == h))
.or_else(|| ranked.first())
.map(|(_, format)| format.url.clone())
}
async fn cached_link(
requester: &web::types::State<Requester>,
alnum_id: &str,
height: Option<u32>,
fresh: bool,
) -> Option<String> {
let key = format!("{alnum_id}/{}", height.unwrap_or(0));
if !fresh {
if let Ok(links) = links().lock() {
if let Some((time, url)) = links.get(&key) {
if time.elapsed() < LINK_TTL {
return Some(url.clone());
}
}
}
}
let url = resolve_link(requester, alnum_id, height).await?;
if let Ok(mut links) = links().lock() {
if links.len() >= LINK_CACHE_SIZE {
links.retain(|_, (time, _)| time.elapsed() < LINK_TTL);
if links.len() >= LINK_CACHE_SIZE {
links.clear();
}
}
links.insert(key, (Instant::now(), url.clone()));
}
Some(url)
}
/// Streams the mp4 through this server. The signed CDN links only serve the
/// IP that resolved them (checked from other networks: they answer every
/// other client with a 302 to `static.eporner.com/na.mp4`, a one-minute
/// "watch it on eporner.com" clip), so a redirect to them only ever works for
/// a player that shares the server's public IP.
pub async fn serve_media(
req: HttpRequest,
requester: web::types::State<Requester>,
) -> Result<impl web::Responder, web::Error> {
let endpoint = req.match_info().query("endpoint").to_string();
let Some((alnum_id, height)) = parse_endpoint(&endpoint) else {
return Ok(web::HttpResponse::BadRequest().finish());
};
let range = req
.headers()
.get(RANGE)
.and_then(|value| value.to_str().ok())
.map(str::to_string);
let head = req.method() == Method::HEAD;
let mut upstream = None;
for fresh in [false, true] {
let Some(url) = cached_link(&requester, &alnum_id, height, fresh).await else {
break;
};
let mut request = if head {
cdn_client().head(&url)
} else {
cdn_client().get(&url)
};
request = request.header("Referer", "https://www.eporner.com/");
if let Some(range) = &range {
request = request.header("Range", range);
}
match request.send().await {
// A redirect is the decoy for an expired or foreign link.
Ok(response) if !response.status().is_redirection() => {
upstream = Some(response);
break;
}
_ => continue,
}
}
let Some(upstream) = upstream else {
return Ok(web::HttpResponse::BadGateway().finish());
};
let status = upstream.status();
let headers = upstream.headers().clone();
let mut builder = web::HttpResponse::build(status);
builder.header(
CONTENT_TYPE,
headers
.get(CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.unwrap_or("video/mp4"),
);
builder.header(ACCEPT_RANGES, "bytes");
if let Some(value) = headers.get(CONTENT_RANGE.as_str()).and_then(|v| v.to_str().ok()) {
builder.header(CONTENT_RANGE, value);
}
if head {
// `finish()` would send `Content-Length: 0` over the upstream length.
return Ok(builder.body(Body::None));
}
let length = headers
.get(CONTENT_LENGTH.as_str())
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse::<u64>().ok());
let stream = Box::pin(upstream.into_data_stream().map(|chunk| {
chunk
.map(|bytes| Bytes::copy_from_slice(bytes.as_ref()))
.map_err(|e| std::io::Error::other(e.to_string()))
}));
let Some(length) = length else {
return Ok(builder.streaming(stream));
};
// A sized body keeps `Content-Length` on the (206) responses; a chunked
// one is what AVPlayer refuses to seek in.
let stream = stream.map(|chunk| chunk.map_err(|e| Rc::new(e) as Rc<dyn std::error::Error>));
Ok(builder.body(Body::from_message(SizedStream::new(length, stream))))
}
#[cfg(test)]
mod tests {
use super::parse_endpoint;
#[test]
fn parses_id_and_optional_height() {
assert_eq!(
parse_endpoint("exfeTXHuL1x/720.mp4"),
Some(("exfeTXHuL1x".to_string(), Some(720)))
);
assert_eq!(
parse_endpoint("/exfeTXHuL1x/"),
Some(("exfeTXHuL1x".to_string(), None))
);
assert_eq!(parse_endpoint("../etc"), None);
}
}

71
src/proxies/hanime.rs Normal file
View File

@@ -0,0 +1,71 @@
use ntex::http::Method;
use ntex::http::header::LOCATION;
use ntex::web::{self, HttpRequest};
use crate::util::hanime_resolve::{fetch_sources, pick_source, source_url};
use crate::util::requester::Requester;
/// `{slug}[/{height}][.m3u8]`.
fn parse_endpoint(endpoint: &str) -> Option<(String, Option<u32>)> {
let endpoint = endpoint.trim_matches('/').trim_end_matches(".m3u8");
let mut parts = endpoint.split('/');
let slug = parts
.next()
.filter(|slug| {
!slug.is_empty()
&& slug
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
})?;
let height = parts.next().and_then(|h| h.parse::<u32>().ok());
Some((slug.to_string(), height))
}
/// hanime's stream links expire, so the channel hands out this stable URL and a
/// fresh handshake happens on every play: redirects to the current playlist.
pub async fn redirect_to_media(
req: HttpRequest,
requester: web::types::State<Requester>,
) -> Result<impl web::Responder, web::Error> {
let endpoint = req.match_info().query("endpoint").to_string();
let Some((slug, height)) = parse_endpoint(&endpoint) else {
return Ok(web::HttpResponse::BadRequest().finish());
};
if req.method() == Method::HEAD {
return Ok(web::HttpResponse::Ok().finish());
}
let mut requester = requester.get_ref().clone();
let Ok(sources) = fetch_sources(&mut requester, &slug).await else {
return Ok(web::HttpResponse::BadGateway().finish());
};
let Some(source) = pick_source(&sources, height) else {
return Ok(web::HttpResponse::NotFound().finish());
};
Ok(web::HttpResponse::Found()
.header(LOCATION, source_url(source))
.finish())
}
#[cfg(test)]
mod tests {
use super::parse_endpoint;
#[test]
fn parses_slug_and_height() {
assert_eq!(
parse_endpoint("natsu-zuma-2/720.m3u8"),
Some(("natsu-zuma-2".to_string(), Some(720)))
);
assert_eq!(
parse_endpoint("/natsu-zuma-2"),
Some(("natsu-zuma-2".to_string(), None))
);
}
#[test]
fn rejects_unsafe_slugs() {
assert_eq!(parse_endpoint("../etc/720.m3u8"), None);
assert_eq!(parse_endpoint(""), None);
}
}

View File

@@ -24,7 +24,9 @@ pub mod animeidhentai;
pub mod archivebate;
pub mod clapdat;
pub mod doodstream;
pub mod eporner;
pub mod fikfapthumb;
pub mod hanime;
pub mod hanimecdn;
pub mod hanimethumb;
pub mod heavyfetish;

View File

@@ -20,6 +20,40 @@ use crate::proxies::thaiporntv::ThaipornTvProxy;
use crate::proxies::*;
use crate::util::requester::Requester;
/// Proxies that answer with a `302` to the real media instead of streaming it.
/// The Hot Tub app cannot follow a redirect inside `formats[].url`, so for app
/// clients a format pointing at one of these is moved to `video.url` (see
/// `move_redirect_formats_to_url`). Keep in sync with the `proxy2redirect`
/// routes below; `redirect_proxy_list_matches_routes` enforces it.
pub const REDIRECT_PROXIES: &[&str] = &[
"archivebate",
"clapdat",
"doodstream",
"lulustream",
"sxyprn",
"javtiful",
"spankbang",
"porndish",
"hqporner",
"heavyfetish",
"vjav",
"pornhd3x",
"vidara",
"pimpbunny",
"thaiporntv",
"tube8",
"supjav",
];
/// True for a `/proxy/<redirect proxy>/...` URL (absolute or host-relative).
pub fn is_redirect_proxy_url(url: &str) -> bool {
let Some(rest) = url.split_once("/proxy/").map(|(_, rest)| rest) else {
return false;
};
let name = rest.split('/').next().unwrap_or_default();
REDIRECT_PROXIES.contains(&name)
}
pub fn config(cfg: &mut web::ServiceConfig) {
cfg.service(
web::resource("/archivebate/{endpoint}*")
@@ -123,6 +157,11 @@ pub fn config(cfg: &mut web::ServiceConfig) {
.route(web::post().to(crate::proxies::fikfapthumb::get_image))
.route(web::get().to(crate::proxies::fikfapthumb::get_image)),
)
.service(
web::resource("/hanime/{endpoint}*")
.route(web::get().to(crate::proxies::hanime::redirect_to_media))
.route(web::head().to(crate::proxies::hanime::redirect_to_media)),
)
.service(
web::resource("/hanime-cdn/{endpoint}*")
.route(web::post().to(crate::proxies::hanimecdn::get_image))
@@ -194,6 +233,11 @@ pub fn config(cfg: &mut web::ServiceConfig) {
.route(web::post().to(crate::proxies::melonstube::serve))
.route(web::get().to(crate::proxies::melonstube::serve)),
);
cfg.service(
web::resource("/eporner/{endpoint}*")
.route(web::get().to(crate::proxies::eporner::serve_media))
.route(web::head().to(crate::proxies::eporner::serve_media)),
);
cfg.service(
web::resource("/xiaoshenke/{endpoint}*")
.route(web::post().to(crate::proxies::xiaoshenke::serve_media))
@@ -239,3 +283,39 @@ fn get_proxy(proxy: &str) -> Option<AnyProxy> {
_ => None,
}
}
#[cfg(test)]
mod redirect_proxy_tests {
use super::*;
#[test]
fn recognises_redirect_proxy_urls() {
assert!(is_redirect_proxy_url("http://h/proxy/archivebate/archivebate.com/watch/1"));
assert!(is_redirect_proxy_url("/proxy/hqporner/x"));
assert!(!is_redirect_proxy_url("http://h/proxy/noodlemagazine/x"));
assert!(!is_redirect_proxy_url("https://cdn.example/video.mp4"));
}
#[test]
fn redirect_proxy_list_matches_routes() {
let source = include_str!("proxy.rs");
let config_body = source
.split("pub fn config(")
.nth(1)
.and_then(|rest| rest.split("async fn proxy2redirect(").next())
.expect("config body");
let mut from_routes: Vec<String> = Vec::new();
for block in config_body.split("web::resource(\"/").skip(1) {
let name = block.split("/{endpoint}").next().unwrap_or_default();
let until_next = block.split("web::resource(").next().unwrap_or(block);
if until_next.contains("proxy2redirect") {
from_routes.push(name.to_string());
}
}
let mut listed: Vec<String> = REDIRECT_PROXIES.iter().map(|s| s.to_string()).collect();
from_routes.sort();
from_routes.dedup();
listed.sort();
assert_eq!(listed, from_routes);
}
}

View File

@@ -12,7 +12,7 @@
//! always compiled regardless of provider selection.
use crate::util::requester::Requester;
use crate::videos::VideoFormat;
use crate::videos::{ServerOptions, VideoFormat};
use regex::Regex;
use std::sync::OnceLock;
use wreq::Version;
@@ -101,11 +101,18 @@ fn calc_hash(hash: &str) -> Option<String> {
/// when playback doesn't carry the header the CDN wants. Each format
/// therefore declares an explicit `Referer` so the client's own default (or
/// lack thereof) can't accidentally trip this check.
pub async fn resolve_formats(
///
/// The signed CDN URLs also expire two hours after this call (the leading
/// number in `/{expiry}_{ip}_{n}/` is that deadline) and only serve the IP
/// embedded there; anything else gets the same `na.mp4` decoy.
/// Anything handed to a client should therefore be a
/// [`resolve_proxied_formats`] URL, streamed through this server.
/// Returns `(height, format)`, best first.
pub async fn resolve_ranked(
requester: &mut Requester,
alnum_id: &str,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
) -> Option<Vec<(u32, VideoFormat)>> {
let detail_html = requester
.get_with_headers(page_url, html_headers(page_url), Some(Version::HTTP_11))
.await
@@ -151,25 +158,56 @@ pub async fn resolve_formats(
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
Some(ranked)
}
/// Resolves eporner.com direct mp4 formats for an arbitrary eporner detail
/// page URL, extracting the alnum id from the URL itself. For use by
/// providers that only have a raw destination URL (e.g. a meta-search
/// aggregator's out-link), not a pre-parsed listing card.
pub async fn resolve_public_formats(
/// Resolves the qualities a video offers, but points each one at the
/// `/proxy/eporner/{alnum_id}/{height}.mp4` streaming proxy: the CDN URLs
/// expire after two hours and only serve the IP that resolved them.
pub async fn resolve_proxied_formats(
requester: &mut Requester,
options: &ServerOptions,
alnum_id: &str,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
let alnum_id = alnum_id_from_href(page_url)?;
resolve_formats(requester, &alnum_id, page_url).await
let ranked = resolve_ranked(requester, alnum_id, page_url).await?;
let base = options.public_url_base.as_deref().unwrap_or("");
Some(proxied_formats(base, alnum_id, ranked))
}
/// Builds the URLs like `providers::build_proxy_url`, which this always
/// compiled module cannot rely on in single-provider builds.
fn proxied_formats(
public_url_base: &str,
alnum_id: &str,
ranked: Vec<(u32, VideoFormat)>,
) -> Vec<VideoFormat> {
let base = public_url_base.trim_end_matches('/');
ranked
.into_iter()
.map(|(height, mut format)| {
format.url = format!("{base}/proxy/eporner/{alnum_id}/{height}.mp4");
format
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn proxied_formats_point_at_the_streaming_proxy() {
let ranked = vec![(
720,
VideoFormat::new("https://vid.cdn/x-720p.mp4".into(), "720p".into(), "mp4".into()),
)];
let formats = proxied_formats("https://hottub.example/", "exfeTXHuL1x", ranked);
assert_eq!(formats[0].url, "https://hottub.example/proxy/eporner/exfeTXHuL1x/720.mp4");
// Streamed, not redirected: app clients must keep it in `formats`.
assert!(!crate::proxy::is_redirect_proxy_url(&formats[0].url));
}
#[test]
fn calc_hash_matches_site_algorithm() {
// Reverse-engineered from vjs.js; verified against a live page/xhr pair.

208
src/util/hanime_resolve.rs Normal file
View File

@@ -0,0 +1,208 @@
//! hanime.tv stream resolution, shared by the `hanime` channel and its
//! `/proxy/hanime/` redirect proxy.
//!
//! Lives here (always compiled) rather than in `providers/hanime.rs` because
//! `HOT_TUB_PROVIDER=<id>` single-provider builds only compile the selected
//! provider, while the proxy routes are always registered.
use aes_gcm::aead::{Aead, KeyInit, Payload};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use base64::Engine;
use serde_json::json;
use sha2::{Digest, Sha256};
use std::time::{SystemTime, UNIX_EPOCH};
use crate::util::requester::Requester;
const HANDSHAKE_URL: &str = "https://auth.hanime.tv/api/v11/handshake";
pub const SITE_ORIGIN: &str = "https://hanime.tv";
/// Envelope encryption used for the handshake request/response bodies.
const HANDSHAKE_KEY_SEED: &[u8] = b"htv-insecure-handshake-v1";
const HANDSHAKE_AAD: &[u8] = b"htv-insecure-v1";
/// Pieces of the `x-signature` pre-image, as baked into the site's wasm module.
const SIGNATURE_SECRET: &str = "Xkdi29";
const SIGNATURE_SALT: &str = "mn2";
type Result<T> = std::result::Result<T, String>;
fn err(message: impl Into<String>) -> String {
message.into()
}
#[derive(serde::Deserialize, Debug)]
pub struct HanimeHandshakePayload {
#[serde(default)]
pub sources: Vec<HanimeSource>,
}
#[derive(serde::Deserialize, Debug, Clone)]
pub struct HanimeSource {
#[serde(default)]
pub src: String,
#[serde(default)]
pub height: u32,
#[serde(default)]
pub label: String,
/// `"normal"` for the free streams, `"promotion"` for the premium-only teaser
/// entries which always carry an empty `src`.
#[serde(default)]
pub kind: String,
}
fn b64() -> base64::engine::general_purpose::GeneralPurpose {
base64::engine::general_purpose::URL_SAFE_NO_PAD
}
fn b64_decode(value: &str) -> Result<Vec<u8>> {
b64()
.decode(value.trim_end_matches('='))
.map_err(|e| err(format!("base64url decode failed: {e}")))
}
fn handshake_cipher() -> Aes256Gcm {
let digest = Sha256::digest(HANDSHAKE_KEY_SEED);
let key = Key::<Aes256Gcm>::from_slice(digest.as_slice());
Aes256Gcm::new(key)
}
/// `{"v":1,"alg":"AES-256-GCM","iv":..,"tag":..,"data":..}`, base64url encoded.
fn seal_envelope(plaintext: &[u8]) -> Result<String> {
let mut iv = [0u8; 12];
rand::fill(&mut iv);
let sealed = handshake_cipher()
.encrypt(
Nonce::from_slice(&iv),
Payload {
msg: plaintext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| err(format!("handshake encrypt failed: {e}")))?;
if sealed.len() < 16 {
return Err(err("handshake ciphertext too short"));
}
let (data, tag) = sealed.split_at(sealed.len() - 16);
let envelope = json!({
"v": 1,
"alg": "AES-256-GCM",
"iv": b64().encode(iv),
"tag": b64().encode(tag),
"data": b64().encode(data),
});
Ok(b64().encode(envelope.to_string().as_bytes()))
}
fn open_envelope(token: &str) -> Result<Vec<u8>> {
let envelope: serde_json::Value = serde_json::from_slice(&b64_decode(token)?)
.map_err(|e| err(format!("handshake envelope is not JSON: {e}")))?;
let field = |name: &str| -> Result<Vec<u8>> {
let raw = envelope
.get(name)
.and_then(|v| v.as_str())
.ok_or_else(|| err(format!("handshake envelope missing {name}")))?;
b64_decode(raw)
};
let iv = field("iv")?;
let mut ciphertext = field("data")?;
ciphertext.extend_from_slice(&field("tag")?);
handshake_cipher()
.decrypt(
Nonce::from_slice(&iv),
Payload {
msg: &ciphertext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| err(format!("handshake decrypt failed: {e}")))
}
fn signature_for(timestamp: u64) -> String {
let pre_image =
format!("{timestamp},{SIGNATURE_SECRET},{SITE_ORIGIN},{SIGNATURE_SALT},{timestamp}");
hex::encode(Sha256::digest(pre_image.as_bytes()))
}
/// Returns the free `sources` for a slug, best quality first. The auth host
/// occasionally refuses one of a burst of parallel connections, so retry once.
pub async fn fetch_sources(requester: &mut Requester, slug: &str) -> Result<Vec<HanimeSource>> {
match handshake(requester, slug).await {
Ok(sources) => Ok(sources),
Err(_) => {
ntex::time::sleep(ntex::time::Millis(500)).await;
handshake(requester, slug).await
}
}
}
async fn handshake(requester: &mut Requester, slug: &str) -> Result<Vec<HanimeSource>> {
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
let token = seal_envelope(
json!({
"timestamp_unix": timestamp,
"directive": "htv_player_handshake",
"slug": slug,
})
.to_string()
.as_bytes(),
)?;
let response = requester
.post_json(
HANDSHAKE_URL,
&json!({ "token": token }),
vec![
("Origin".to_string(), SITE_ORIGIN.to_string()),
("Referer".to_string(), format!("{SITE_ORIGIN}/")),
("Accept".to_string(), "application/json".to_string()),
("x-signature-version".to_string(), "web2".to_string()),
("x-signature".to_string(), signature_for(timestamp)),
("x-time".to_string(), timestamp.to_string()),
],
)
.await
.map_err(|e| err(format!("handshake request failed: {e}")))?;
let status = response.status();
if !status.is_success() {
return Err(err(format!("handshake returned HTTP {status}")));
}
// The interesting part of the answer travels in a response header, not the body.
let encrypted = response
.headers()
.get("x-token")
.and_then(|v| v.to_str().ok())
.map(|v| v.to_string())
.ok_or_else(|| err("handshake response is missing x-token"))?;
let payload: HanimeHandshakePayload = serde_json::from_slice(&open_envelope(&encrypted)?)
.map_err(|e| err(format!("Failed to parse handshake payload: {e}")))?;
let mut sources: Vec<HanimeSource> = payload
.sources
.into_iter()
.filter(|s| s.kind == "normal" && !s.src.is_empty())
.collect();
sources.sort_by(|a, b| b.height.cmp(&a.height));
match sources.is_empty() {
true => Err(err("handshake returned no playable sources")),
false => Ok(sources),
}
}
pub fn source_url(source: &HanimeSource) -> String {
match source.src.starts_with("http") {
true => source.src.clone(),
false => format!("{SITE_ORIGIN}{}", source.src),
}
}
/// The source of the requested height, else the best one.
pub fn pick_source(sources: &[HanimeSource], height: Option<u32>) -> Option<&HanimeSource> {
height
.and_then(|h| sources.iter().find(|s| s.height == h))
.or_else(|| sources.first())
}

View File

@@ -6,6 +6,7 @@ pub mod discord;
pub mod eporner_resolve;
pub mod flaresolverr;
pub mod flow_debug;
pub mod hanime_resolve;
pub mod hoster_proxy;
pub mod proxy;
pub mod requester;

View File

@@ -87,7 +87,7 @@ impl Requester {
}
}
fn remembered_flaresolverr_user_agent(url: &str) -> Option<String> {
pub(crate) fn remembered_flaresolverr_user_agent(url: &str) -> Option<String> {
let host = Self::host_key(url)?;
Self::flaresolverr_user_agents()
.read()
@@ -218,10 +218,19 @@ impl Requester {
.unwrap_or_else(|| "none".to_string())
}
fn looks_like_cloudflare_challenge(body: &str) -> bool {
pub(crate) fn looks_like_cloudflare_challenge(body: &str) -> bool {
body.contains("Just a moment...")
|| body.contains("cf-turnstile-response")
|| body.contains("Enable JavaScript and cookies to continue")
|| Self::looks_like_ddos_guard_challenge(body)
}
/// DDoS-Guard interstitials (rule34video) are short JS-check pages, never real content.
fn looks_like_ddos_guard_challenge(body: &str) -> bool {
body.len() < 20_000
&& (body.contains("/.well-known/ddos-guard/")
|| body.contains("check.ddos-guard.net")
|| body.contains("DDoS-Guard</title>"))
}
fn jina_mirror_url(url: &str) -> Option<String> {
@@ -661,6 +670,74 @@ impl Requester {
Ok(response)
}
fn path_of(url: &str) -> Option<String> {
url::Url::parse(url).ok().map(|u| u.path().to_string())
}
/// `Url` percent-encodes a non-ASCII path ("/tags/pok%C3%A9mon/") while the
/// response URI keeps it raw ("/tags/pokémon/"): compare them decoded.
fn same_path(asked: &str, landed: &str) -> bool {
let normalize = |path: &str| {
percent_encoding::percent_decode_str(path)
.decode_utf8_lossy()
.trim_end_matches('/')
.to_lowercase()
};
normalize(asked) == normalize(landed)
}
/// Existence probe for archive pages (pornstar / studio / tag / category).
///
/// Reads the raw response so a real 404/410 answers immediately with
/// `Ok(None)` instead of escalating through the Jina/FlareSolverr chain
/// (minutes per miss). A Cloudflare challenge (403/503/"Just a moment...")
/// is handed to the full chain so FlareSolverr solves it, and 429/5xx are
/// retried once.
pub async fn get_probe(
&mut self,
url: &str,
headers: Vec<(String, String)>,
) -> Result<Option<String>, AnyErr> {
let mut last_err = String::new();
for attempt in 0..2 {
if attempt > 0 {
// An immediate retry of a 429 is all but certain to hit it again.
tokio::time::sleep(std::time::Duration::from_millis(1500)).await;
}
match self.get_raw_with_headers(url, headers.clone()).await {
Ok(response) => {
let status = response.status().as_u16();
// An unknown archive often redirects to the home page or a
// search page instead of answering 404: a redirect that
// lands on a different path is a miss.
let redirected_away = Self::path_of(url)
.zip(Some(response.uri().path().to_string()))
.is_some_and(|(asked, landed)| !Self::same_path(&asked, &landed));
let body = response.text().await.unwrap_or_default();
if status == 404 || status == 410 {
return Ok(None);
}
if status == 403
|| status == 503
|| Self::looks_like_cloudflare_challenge(&body)
{
return self.get_with_headers(url, headers, None).await.map(Some);
}
if redirected_away {
return Ok(None);
}
if status >= 500 || status == 429 {
last_err = format!("HTTP {status}");
continue;
}
return Ok(Some(body));
}
Err(e) => last_err = e.to_string(),
}
}
Err(format!("probe failed for {url}: {last_err}").into())
}
pub async fn get(
&mut self,
url: &str,
@@ -830,7 +907,9 @@ impl Requester {
rate_limit_retries,
MAX_RATE_LIMIT_RETRIES
);
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
// 1s, 2s, 4s: a site-wide limit needs longer than a second to lift.
tokio::time::sleep(std::time::Duration::from_secs(1 << (rate_limit_retries - 1)))
.await;
continue;
} else {
crate::flow_debug!(
@@ -964,6 +1043,14 @@ impl Requester {
mod tests {
use super::Requester;
#[test]
fn probe_paths_compare_percent_decoded() {
let asked = Requester::path_of("https://rule34video.com/tags/pokémon-species/").unwrap();
assert!(Requester::same_path(&asked, "/tags/pokémon-species/"));
assert!(Requester::same_path("/Models/X", "/models/x/"));
assert!(!Requester::same_path(&asked, "/search/"));
}
#[test]
fn new_requesters_share_cookie_jar() {
let a = Requester::new();
@@ -979,4 +1066,88 @@ mod tests {
assert!(cookie_header.contains("shared_cookie=1"));
}
async fn serve(listener: tokio::net::TcpListener, reply: impl Fn(&str) -> String + Send + 'static) {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
loop {
let Ok((mut sock, _)) = listener.accept().await else { return };
let mut buf = vec![0u8; 16384];
let mut len = 0;
// read headers, then the declared body
loop {
let Ok(n) = sock.read(&mut buf[len..]).await else { break };
if n == 0 { break; }
len += n;
let text = String::from_utf8_lossy(&buf[..len]).to_string();
if let Some(head_end) = text.find("\r\n\r\n") {
let want = text
.lines()
.find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0)))
.unwrap_or(0);
if len >= head_end + 4 + want { break; }
}
}
let req = String::from_utf8_lossy(&buf[..len]).to_string();
let _ = sock.write_all(reply(&req).as_bytes()).await;
let _ = sock.shutdown().await;
}
}
fn http(status: &str, ctype: &str, extra: &str, body: &str) -> String {
format!(
"HTTP/1.1 {status}\r\ncontent-type: {ctype}\r\n{extra}content-length: {}\r\nconnection: close\r\n\r\n{body}",
body.len()
)
}
/// DDoS-Guard answers either a bare `403 Access denied.` (blocked fingerprint/IP) or a
/// short JS-check page. Both must end up at FlareSolverr and return the real listing.
#[tokio::test(flavor = "multi_thread")]
async fn ddos_guard_blocks_are_solved_through_flaresolverr() {
// distinct hosts: the cookie jar is process-wide, so a solved cookie would leak across passes
for (interstitial, host) in [(false, "127.0.0.1"), (true, "localhost")] {
let origin = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let flare = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let origin_port = origin.local_addr().unwrap().port();
let flare_port = flare.local_addr().unwrap().port();
tokio::spawn(serve(origin, move |req| {
if req.contains("ddg_ok=1") {
http("200 OK", "text/html", "", "<div class=\"item thumb video_1\">real listing</div>")
} else if interstitial {
http(
"200 OK",
"text/html",
"server: ddos-guard\r\n",
"<html><head><title>DDoS-Guard</title><script src=\"/.well-known/ddos-guard/check?context=free_splash\"></script></head></html>",
)
} else {
http("403 Forbidden", "text/plain", "server: ddos-guard\r\n", "Access denied.")
}
}));
tokio::spawn(serve(flare, move |req| {
let body = if req.contains("sessions.create") {
r#"{"status":"ok","message":"","session":"mock"}"#.to_string()
} else if req.contains("sessions.destroy") {
r#"{"status":"ok","message":""}"#.to_string()
} else {
format!(
r#"{{"status":"ok","message":"","startTimestamp":0,"endTimestamp":1,"version":"mock","solution":{{"url":"http://127.0.0.1:{origin_port}/","status":200,"response":"<div class=\"item thumb video_1\">real listing</div>","headers":{{}},"userAgent":"Mozilla/5.0 mock","cookies":[{{"name":"ddg_ok","value":"1","domain":"","path":"/"}}]}}}}"#
)
};
http("200 OK", "application/json", "", &body)
}));
unsafe { std::env::set_var("FLARE_URL", format!("http://127.0.0.1:{flare_port}/v1")) };
let mut requester = Requester::new();
let body = requester
.get(&format!("http://{host}:{origin_port}/?mode=async"), None)
.await
.expect("requester should fall through to FlareSolverr");
assert!(
body.contains("real listing"),
"interstitial={interstitial}: got {body:?}"
);
}
}
}