diff --git a/docs/provider-catalog.md b/docs/provider-catalog.md index 171e3e0..dda7254 100644 --- a/docs/provider-catalog.md +++ b/docs/provider-catalog.md @@ -33,7 +33,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us | `hsex` | `chinese` | yes | no | Strong template for tags, uploaders, and direct HLS formats. | | `hypnotube` | `fetish-kink` | no | no | Fetish/tube hybrid. | | `javtiful` | `jav` | no | no | JAV channel family. | -| `melonstube` | `mainstream-tube` | no | yes | Meta-search aggregator for melonstube.com — every card is an `/out/?l=&c=&v=3` redirect link whose destination (a third-party host) is embedded, zero-network, in a MessagePack blob under the `l` param (mixed binary/string fields: view/click counts, a JSON date-range blob, an array of related-video ids, and the destination URL itself as one msgpack string). Decoding is local: base64-decode `l`, then regex-extract the `https?://...` destination directly out of the raw (lossy-UTF8-converted) msgpack bytes — **the regex must be a positive allowlist of legal URI characters** (`[A-Za-z0-9\-._~:/?#\[\]@!$&'()*+,;=%]+`), not a denylist of a few excluded characters: the byte immediately following the msgpack string (a length-prefix marker for the next field, e.g. `\xcd`) lossy-converts to U+FFFD (code point 0xFFFD), which a denylist limited to ASCII control chars (`[^\x00-\x1f\\"']`) fails to exclude, silently appending a corrupt trailing character to the URL. Feeds: `/new` (latest), `/popular`, `/rating` (all three are static curated lists, ~120 items, true pagination via `?page=N`); search is NOT the vestigial `/search?q=` GET endpoint (returns generic filler content behind a fake static result counter regardless of query) — the real mechanism mirrors the site's own search form: POST `search_query[query]=` to `/searching/by-form`, which 303-redirects (cookies + redirect auto-followed by the shared `Requester`) to a resolved, genuinely query-relevant destination depending on entity match — `/pornstar/`, `/category/`, `/search/a/` (studio), or `/search/` (generic free text) — all of which support `?page=N` pagination; a non-matching query resolves to a page with zero result cards, which `parse_listing` naturally returns as empty. Destination hosts fall into three buckets: (1) the common case — yt-dlp (`--impersonate chrome-120`) resolves the destination directly, so `video.url` is just the decoded destination URL, no proxy; (2) `JUNK_HOSTS` (currently `fhgte.com`) — observed dead-end paywall/signup funnels with no free playable video, so cards pointing there are dropped entirely rather than surfaced as false-positive results; (3) `HARD_HOSTS` (`manysex.com`, `videomanysex.com`) — yt-dlp cannot resolve these, so `video.url` is routed through `/proxy/melonstube/{host}/{path}`. The proxy (`src/proxies/melonstube.rs`) ports vjav.rs's Cyrillic-homoglyph-obfuscated base64 decode chain to pull the real `get_file` path/query out of `videofile.php`'s `video_url` field, then makes two manual (non-auto-redirect) hops to the final CDN URL: hop 1 needs the manysex.com/videomanysex.com page as `Referer`; hop 2 (to the signed `ahcdn.com` URL) must be sent with **no** Referer at all, because the CDN's signed URL embeds a literal `referer=none,.manysex.com,.gstatic.com` allow-list that rejects the videomanysex.com Referer used on hop 1 — auto-redirect clients that forward the same Referer to every hop get a 403 at hop 2. Before returning the 302, an anti-false-positive check confirms the resolved URL string contains the requested numeric video id and that a ranged GET (`Range: bytes=0-65535`) returns 200/206 with a `video/*`/`octet-stream` content-type, so a paywall/ad/error page can't masquerade as the real stream. Thumbnails (`ttcache.com`) load directly, no proxy. No `/api/uploaders` (aggregator has no stable uploader identity). | +| `melonstube` | `mainstream-tube` | no | yes | Meta-search aggregator for melonstube.com — every card is an `/out/?l=&c=&v=3` redirect link whose destination (a third-party host) is embedded, zero-network, in a MessagePack blob under the `l` param (mixed binary/string fields: view/click counts, a JSON date-range blob, an array of related-video ids, and the destination URL itself as one msgpack string). Decoding is local: base64-decode `l`, then regex-extract the `https?://...` destination directly out of the raw (lossy-UTF8-converted) msgpack bytes — **the regex must be a positive allowlist of legal URI characters** (`[A-Za-z0-9\-._~:/?#\[\]@!$&'()*+,;=%]+`), not a denylist of a few excluded characters: the byte immediately following the msgpack string (a length-prefix marker for the next field, e.g. `\xcd`) lossy-converts to U+FFFD (code point 0xFFFD), which a denylist limited to ASCII control chars (`[^\x00-\x1f\\"']`) fails to exclude, silently appending a corrupt trailing character to the URL. Feeds: `/new` (latest), `/popular`, `/rating` (all three are static curated lists, ~120 items, true pagination via `?page=N`); search is NOT the vestigial `/search?q=` GET endpoint (returns generic filler content behind a fake static result counter regardless of query) — the real mechanism mirrors the site's own search form: POST `search_query[query]=` to `/searching/by-form`, which 303-redirects (cookies + redirect auto-followed by the shared `Requester`) to a resolved, genuinely query-relevant destination depending on entity match — `/pornstar/`, `/category/`, `/search/a/` (studio), or `/search/` (generic free text) — all of which support `?page=N` pagination; a non-matching query resolves to a page with zero result cards, which `parse_listing` naturally returns as empty. Destination hosts fall into three buckets: (1) the common case — yt-dlp (`--impersonate chrome-120`) resolves the destination directly, so `video.url` is just the decoded destination URL, no proxy; (2) `JUNK_HOSTS` (currently `fhgte.com`) — observed dead-end paywall/signup funnels with no free playable video, so cards pointing there are dropped entirely rather than surfaced as false-positive results; (3) `HARD_HOSTS` (`manysex.com`, `videomanysex.com`) — yt-dlp cannot resolve these, so `video.url` is routed through `/proxy/melonstube/{host}/{path}`. The proxy (`src/proxies/melonstube.rs`) ports vjav.rs's Cyrillic-homoglyph-obfuscated base64 decode chain to pull the real `get_file` path/query out of `videofile.php`'s `video_url` field, then makes two manual (non-auto-redirect) hops to the final CDN URL: hop 1 needs the manysex.com/videomanysex.com page as `Referer`; hop 2 (to the signed `ahcdn.com` URL) must be sent with **no** Referer at all, because the CDN's signed URL embeds a literal `referer=none,.manysex.com,.gstatic.com` allow-list that rejects the videomanysex.com Referer used on hop 1 — auto-redirect clients that forward the same Referer to every hop get a 403 at hop 2. Before returning the 302, an anti-false-positive check confirms the resolved URL string contains the requested numeric video id and that a ranged GET (`Range: bytes=0-65535`) returns 200/206 with a `video/*`/`octet-stream` content-type, so a paywall/ad/error page can't masquerade as the real stream. Thumbnails (`ttcache.com`) load directly, no proxy. No `/api/uploaders` (aggregator has no stable uploader identity). **2026-09-21 fix:** melonstube.com started fronting itself with Cloudflare bot management (a real Turnstile "challenge" page, `cf-mitigated: challenge`), which the shared Requester's Firefox151 emulation gets on every path — this made the provider return 0 items (direct fetch, Jina mirror, and FlareSolverr all failed/unreachable). A/B probing many `curl_cffi` fingerprints found Chrome120/124 pass the plain `/new`/`/popular`/`/rating` listing pages but still 403 on `/category/*` and the `/searching/by-form` search POST, while **Safari18** passes all of them; listing and search fetches now try a dedicated `wreq::Client` with `Emulation::Safari18` first and fall back to the shared Requester if Cloudflare's rules change again. Also fixed: `get_videos` previously ignored `perPage` entirely (`let _ = per_page;`), returning the full ~120-card page regardless of the requested size — it now truncates to the requested `perPage` after fetching. | | `missav` | `jav` | no | no | HLS format pattern. | | `noodlemagazine` | `mainstream-tube` | no | yes | Best template for media and thumbnail proxying. | | `okporn` | `mainstream-tube` | no | no | Simple mainstream archive. | diff --git a/src/providers/melonstube.rs b/src/providers/melonstube.rs index 1e869bb..3063341 100644 --- a/src/providers/melonstube.rs +++ b/src/providers/melonstube.rs @@ -12,7 +12,11 @@ use chrono::{Duration as ChronoDuration, Utc}; use error_chain::error_chain; use regex::Regex; use scraper::{Html, Selector}; +use std::sync::Arc; use url::form_urlencoded::Serializer; +use wreq::cookie::Jar; +use wreq::redirect::Policy; +use wreq_util::Emulation; pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata = crate::providers::ProviderChannelMetadata { @@ -218,6 +222,65 @@ impl MelonstubeProvider { Some((destination, id)) } + // melonstube.com started fronting itself with Cloudflare bot management at + // some point after this provider was written: the shared Requester's + // Firefox151 emulation now gets a straight Turnstile "challenge" page on + // every path. Verified live with curl_cffi across many fingerprints — + // Chrome120/124 pass the plain /new /popular /rating listing pages but + // still 403 on /category/* and /searching/by-form; Safari18 is the only + // profile that passes all of them (listing, category/search-result pages, + // and the /searching/by-form POST) — so listing/search fetches try a + // dedicated Safari18 client first and fall back to the shared Requester + // (with its Jina/FlareSolverr fallback chain) if Cloudflare ever changes + // its rules again. + fn build_bypass_client() -> Option { + let jar = Arc::new(Jar::default()); + wreq::Client::builder() + .tls_cert_verification(false) + .emulation(Emulation::Safari18) + .cookie_provider(jar) + .redirect(Policy::default()) + .build() + .ok() + } + + async fn fetch_html_bypass(url: &str) -> Option { + let client = Self::build_bypass_client()?; + let response = client + .get(url) + .header( + "accept", + "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + ) + .header("accept-language", "en-US,en;q=0.5") + .header("referer", BASE_URL) + .send() + .await + .ok()?; + if !response.status().is_success() { + return None; + } + response.text().await.ok() + } + + async fn resolve_search_base_url_bypass(query: &str) -> Option { + let search_url = format!("{BASE_URL}/searching/by-form"); + let body = Self::build_search_request_body(query); + let client = Self::build_bypass_client()?; + let response = client + .post(&search_url) + .header("content-type", "application/x-www-form-urlencoded") + .header("referer", BASE_URL) + .body(body) + .send() + .await + .ok()?; + if !response.status().is_success() { + return None; + } + Some(response.uri().to_string().trim_end_matches('/').to_string()) + } + fn is_junk_host(host: &str) -> bool { JUNK_HOSTS.contains(&host) || host.starts_with("join.") } @@ -388,6 +451,10 @@ impl MelonstubeProvider { } async fn resolve_search_base_url(query: &str, options: &ServerOptions) -> Result { + if let Some(base) = Self::resolve_search_base_url_bypass(query).await { + return Ok(base); + } + let search_url = format!("{BASE_URL}/searching/by-form"); let body = Self::build_search_request_body(query); @@ -440,23 +507,28 @@ impl MelonstubeProvider { } } - let mut requester = requester_or_default(&options, CHANNEL_ID, "fetch_and_parse"); - - let html = requester - .get_with_headers( - &url, - vec![ - ("user-agent".to_string(), BROWSER_UA.to_string()), - ( - "accept".to_string(), - "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8".to_string(), - ), - ("accept-language".to_string(), "en-US,en;q=0.5".to_string()), - ], - Some(wreq::Version::HTTP_11), - ) - .await - .map_err(|e| Error::from(format!("request failed url={url}: {e}")))?; + let html = match Self::fetch_html_bypass(&url).await { + Some(html) if !html.is_empty() => html, + _ => { + let mut requester = requester_or_default(&options, CHANNEL_ID, "fetch_and_parse"); + requester + .get_with_headers( + &url, + vec![ + ("user-agent".to_string(), BROWSER_UA.to_string()), + ( + "accept".to_string(), + "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" + .to_string(), + ), + ("accept-language".to_string(), "en-US,en;q=0.5".to_string()), + ], + Some(wreq::Version::HTTP_11), + ) + .await + .map_err(|e| Error::from(format!("request failed url={url}: {e}")))? + } + }; if html.is_empty() { return Ok(vec![]); @@ -485,13 +557,16 @@ impl Provider for MelonstubeProvider { options: ServerOptions, ) -> Vec { let _ = pool; - let _ = per_page; + let per_page = per_page.parse::().unwrap_or(20).clamp(1, 120); let page = page.parse::().unwrap_or(1); let query_str = query.unwrap_or_default(); let target = Self::resolve_target(&query_str, &sort, page); match self.fetch_and_parse(cache, target, options).await { - Ok(items) => items, + Ok(mut items) => { + items.truncate(per_page); + items + } Err(e) => { report_provider_error(CHANNEL_ID, "get_videos", &e.to_string()).await; vec![]