Fix melonstube provider: Cloudflare bot management now blocks it

melonstube.com started fronting itself with Cloudflare bot management
(a real Turnstile challenge) since this provider was written, so the
shared Requester's Firefox151 emulation got blocked on every path and
the provider returned 0 items.

A/B-probed several browser TLS fingerprints live: Chrome120/124 pass
the plain listing pages but still 403 on /category/* and the
/searching/by-form search POST, while Safari18 passes everywhere.
Listing/search fetches now try a dedicated Safari18-emulated client
first and fall back to the shared Requester if Cloudflare's rules
change again (same pattern as lulustream.rs/doodstream.rs).

Also fixes get_videos ignoring perPage entirely (always returned the
full ~120-card page); it now truncates to the requested size.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QTqf6orbHZ9rFFpVpcgzcR

docs/provider-catalog.md

@@ -33,7 +33,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
 | `hsex` | `chinese` | yes | no | Strong template for tags, uploaders, and direct HLS formats. |
 | `hypnotube` | `fetish-kink` | no | no | Fetish/tube hybrid. |
 | `javtiful` | `jav` | no | no | JAV channel family. |
-| `melonstube` | `mainstream-tube` | no | yes | Meta-search aggregator for melonstube.com — every card is an `/out/?l=<base64>&c=<hash>&v=3` redirect link whose destination (a third-party host) is embedded, zero-network, in a MessagePack blob under the `l` param (mixed binary/string fields: view/click counts, a JSON date-range blob, an array of related-video ids, and the destination URL itself as one msgpack string). Decoding is local: base64-decode `l`, then regex-extract the `https?://...` destination directly out of the raw (lossy-UTF8-converted) msgpack bytes — **the regex must be a positive allowlist of legal URI characters** (`[A-Za-z0-9\-._~:/?#\[\]@!$&'()*+,;=%]+`), not a denylist of a few excluded characters: the byte immediately following the msgpack string (a length-prefix marker for the next field, e.g. `\xcd`) lossy-converts to U+FFFD (code point 0xFFFD), which a denylist limited to ASCII control chars (`[^\x00-\x1f\\"']`) fails to exclude, silently appending a corrupt trailing character to the URL. Feeds: `/new` (latest), `/popular`, `/rating` (all three are static curated lists, ~120 items, true pagination via `?page=N`); search is NOT the vestigial `/search?q=` GET endpoint (returns generic filler content behind a fake static result counter regardless of query) — the real mechanism mirrors the site's own search form: POST `search_query[query]=<term>` to `/searching/by-form`, which 303-redirects (cookies + redirect auto-followed by the shared `Requester`) to a resolved, genuinely query-relevant destination depending on entity match — `/pornstar/<slug>`, `/category/<slug>`, `/search/a/<term>` (studio), or `/search/<term>` (generic free text) — all of which support `?page=N` pagination; a non-matching query resolves to a page with zero result cards, which `parse_listing` naturally returns as empty. Destination hosts fall into three buckets: (1) the common case — yt-dlp (`--impersonate chrome-120`) resolves the destination directly, so `video.url` is just the decoded destination URL, no proxy; (2) `JUNK_HOSTS` (currently `fhgte.com`) — observed dead-end paywall/signup funnels with no free playable video, so cards pointing there are dropped entirely rather than surfaced as false-positive results; (3) `HARD_HOSTS` (`manysex.com`, `videomanysex.com`) — yt-dlp cannot resolve these, so `video.url` is routed through `/proxy/melonstube/{host}/{path}`. The proxy (`src/proxies/melonstube.rs`) ports vjav.rs's Cyrillic-homoglyph-obfuscated base64 decode chain to pull the real `get_file` path/query out of `videofile.php`'s `video_url` field, then makes two manual (non-auto-redirect) hops to the final CDN URL: hop 1 needs the manysex.com/videomanysex.com page as `Referer`; hop 2 (to the signed `ahcdn.com` URL) must be sent with **no** Referer at all, because the CDN's signed URL embeds a literal `referer=none,.manysex.com,.gstatic.com` allow-list that rejects the videomanysex.com Referer used on hop 1 — auto-redirect clients that forward the same Referer to every hop get a 403 at hop 2. Before returning the 302, an anti-false-positive check confirms the resolved URL string contains the requested numeric video id and that a ranged GET (`Range: bytes=0-65535`) returns 200/206 with a `video/*`/`octet-stream` content-type, so a paywall/ad/error page can't masquerade as the real stream. Thumbnails (`ttcache.com`) load directly, no proxy. No `/api/uploaders` (aggregator has no stable uploader identity). |
+| `melonstube` | `mainstream-tube` | no | yes | Meta-search aggregator for melonstube.com — every card is an `/out/?l=<base64>&c=<hash>&v=3` redirect link whose destination (a third-party host) is embedded, zero-network, in a MessagePack blob under the `l` param (mixed binary/string fields: view/click counts, a JSON date-range blob, an array of related-video ids, and the destination URL itself as one msgpack string). Decoding is local: base64-decode `l`, then regex-extract the `https?://...` destination directly out of the raw (lossy-UTF8-converted) msgpack bytes — **the regex must be a positive allowlist of legal URI characters** (`[A-Za-z0-9\-._~:/?#\[\]@!$&'()*+,;=%]+`), not a denylist of a few excluded characters: the byte immediately following the msgpack string (a length-prefix marker for the next field, e.g. `\xcd`) lossy-converts to U+FFFD (code point 0xFFFD), which a denylist limited to ASCII control chars (`[^\x00-\x1f\\"']`) fails to exclude, silently appending a corrupt trailing character to the URL. Feeds: `/new` (latest), `/popular`, `/rating` (all three are static curated lists, ~120 items, true pagination via `?page=N`); search is NOT the vestigial `/search?q=` GET endpoint (returns generic filler content behind a fake static result counter regardless of query) — the real mechanism mirrors the site's own search form: POST `search_query[query]=<term>` to `/searching/by-form`, which 303-redirects (cookies + redirect auto-followed by the shared `Requester`) to a resolved, genuinely query-relevant destination depending on entity match — `/pornstar/<slug>`, `/category/<slug>`, `/search/a/<term>` (studio), or `/search/<term>` (generic free text) — all of which support `?page=N` pagination; a non-matching query resolves to a page with zero result cards, which `parse_listing` naturally returns as empty. Destination hosts fall into three buckets: (1) the common case — yt-dlp (`--impersonate chrome-120`) resolves the destination directly, so `video.url` is just the decoded destination URL, no proxy; (2) `JUNK_HOSTS` (currently `fhgte.com`) — observed dead-end paywall/signup funnels with no free playable video, so cards pointing there are dropped entirely rather than surfaced as false-positive results; (3) `HARD_HOSTS` (`manysex.com`, `videomanysex.com`) — yt-dlp cannot resolve these, so `video.url` is routed through `/proxy/melonstube/{host}/{path}`. The proxy (`src/proxies/melonstube.rs`) ports vjav.rs's Cyrillic-homoglyph-obfuscated base64 decode chain to pull the real `get_file` path/query out of `videofile.php`'s `video_url` field, then makes two manual (non-auto-redirect) hops to the final CDN URL: hop 1 needs the manysex.com/videomanysex.com page as `Referer`; hop 2 (to the signed `ahcdn.com` URL) must be sent with **no** Referer at all, because the CDN's signed URL embeds a literal `referer=none,.manysex.com,.gstatic.com` allow-list that rejects the videomanysex.com Referer used on hop 1 — auto-redirect clients that forward the same Referer to every hop get a 403 at hop 2. Before returning the 302, an anti-false-positive check confirms the resolved URL string contains the requested numeric video id and that a ranged GET (`Range: bytes=0-65535`) returns 200/206 with a `video/*`/`octet-stream` content-type, so a paywall/ad/error page can't masquerade as the real stream. Thumbnails (`ttcache.com`) load directly, no proxy. No `/api/uploaders` (aggregator has no stable uploader identity). **2026-09-21 fix:** melonstube.com started fronting itself with Cloudflare bot management (a real Turnstile "challenge" page, `cf-mitigated: challenge`), which the shared Requester's Firefox151 emulation gets on every path — this made the provider return 0 items (direct fetch, Jina mirror, and FlareSolverr all failed/unreachable). A/B probing many `curl_cffi` fingerprints found Chrome120/124 pass the plain `/new`/`/popular`/`/rating` listing pages but still 403 on `/category/*` and the `/searching/by-form` search POST, while **Safari18** passes all of them; listing and search fetches now try a dedicated `wreq::Client` with `Emulation::Safari18` first and fall back to the shared Requester if Cloudflare's rules change again. Also fixed: `get_videos` previously ignored `perPage` entirely (`let _ = per_page;`), returning the full ~120-card page regardless of the requested size — it now truncates to the requested `perPage` after fetching. |
 | `missav` | `jav` | no | no | HLS format pattern. |
 | `noodlemagazine` | `mainstream-tube` | no | yes | Best template for media and thumbnail proxying. |
 | `okporn` | `mainstream-tube` | no | no | Simple mainstream archive. |

src/providers/melonstube.rs

@@ -12,7 +12,11 @@ use chrono::{Duration as ChronoDuration, Utc};
 use error_chain::error_chain;
 use regex::Regex;
 use scraper::{Html, Selector};
+use std::sync::Arc;
 use url::form_urlencoded::Serializer;
+use wreq::cookie::Jar;
+use wreq::redirect::Policy;
+use wreq_util::Emulation;
 
 pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
     crate::providers::ProviderChannelMetadata {
@@ -218,6 +222,65 @@ impl MelonstubeProvider {
         Some((destination, id))
     }
 
+    // melonstube.com started fronting itself with Cloudflare bot management at
+    // some point after this provider was written: the shared Requester's
+    // Firefox151 emulation now gets a straight Turnstile "challenge" page on
+    // every path. Verified live with curl_cffi across many fingerprints —
+    // Chrome120/124 pass the plain /new /popular /rating listing pages but
+    // still 403 on /category/* and /searching/by-form; Safari18 is the only
+    // profile that passes all of them (listing, category/search-result pages,
+    // and the /searching/by-form POST) — so listing/search fetches try a
+    // dedicated Safari18 client first and fall back to the shared Requester
+    // (with its Jina/FlareSolverr fallback chain) if Cloudflare ever changes
+    // its rules again.
+    fn build_bypass_client() -> Option<wreq::Client> {
+        let jar = Arc::new(Jar::default());
+        wreq::Client::builder()
+            .tls_cert_verification(false)
+            .emulation(Emulation::Safari18)
+            .cookie_provider(jar)
+            .redirect(Policy::default())
+            .build()
+            .ok()
+    }
+
+    async fn fetch_html_bypass(url: &str) -> Option<String> {
+        let client = Self::build_bypass_client()?;
+        let response = client
+            .get(url)
+            .header(
+                "accept",
+                "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+            )
+            .header("accept-language", "en-US,en;q=0.5")
+            .header("referer", BASE_URL)
+            .send()
+            .await
+            .ok()?;
+        if !response.status().is_success() {
+            return None;
+        }
+        response.text().await.ok()
+    }
+
+    async fn resolve_search_base_url_bypass(query: &str) -> Option<String> {
+        let search_url = format!("{BASE_URL}/searching/by-form");
+        let body = Self::build_search_request_body(query);
+        let client = Self::build_bypass_client()?;
+        let response = client
+            .post(&search_url)
+            .header("content-type", "application/x-www-form-urlencoded")
+            .header("referer", BASE_URL)
+            .body(body)
+            .send()
+            .await
+            .ok()?;
+        if !response.status().is_success() {
+            return None;
+        }
+        Some(response.uri().to_string().trim_end_matches('/').to_string())
+    }
+
     fn is_junk_host(host: &str) -> bool {
         JUNK_HOSTS.contains(&host) || host.starts_with("join.")
     }
@@ -388,6 +451,10 @@ impl MelonstubeProvider {
     }
 
     async fn resolve_search_base_url(query: &str, options: &ServerOptions) -> Result<String> {
+        if let Some(base) = Self::resolve_search_base_url_bypass(query).await {
+            return Ok(base);
+        }
+
         let search_url = format!("{BASE_URL}/searching/by-form");
         let body = Self::build_search_request_body(query);
 
@@ -440,23 +507,28 @@ impl MelonstubeProvider {
             }
         }
 
-        let mut requester = requester_or_default(&options, CHANNEL_ID, "fetch_and_parse");
+        let html = match Self::fetch_html_bypass(&url).await {
-
+            Some(html) if !html.is_empty() => html,
-        let html = requester
+            _ => {
-            .get_with_headers(
+                let mut requester = requester_or_default(&options, CHANNEL_ID, "fetch_and_parse");
-                &url,
+                requester
-                vec![
+                    .get_with_headers(
-                    ("user-agent".to_string(), BROWSER_UA.to_string()),
+                        &url,
-                    (
+                        vec![
-                        "accept".to_string(),
+                            ("user-agent".to_string(), BROWSER_UA.to_string()),
-                        "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8".to_string(),
+                            (
-                    ),
+                                "accept".to_string(),
-                    ("accept-language".to_string(), "en-US,en;q=0.5".to_string()),
+                                "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
-                ],
+                                    .to_string(),
-                Some(wreq::Version::HTTP_11),
+                            ),
-            )
+                            ("accept-language".to_string(), "en-US,en;q=0.5".to_string()),
-            .await
+                        ],
-            .map_err(|e| Error::from(format!("request failed url={url}: {e}")))?;
+                        Some(wreq::Version::HTTP_11),
+                    )
+                    .await
+                    .map_err(|e| Error::from(format!("request failed url={url}: {e}")))?
+            }
+        };
 
         if html.is_empty() {
             return Ok(vec![]);
@@ -485,13 +557,16 @@ impl Provider for MelonstubeProvider {
         options: ServerOptions,
     ) -> Vec<VideoItem> {
         let _ = pool;
-        let _ = per_page;
+        let per_page = per_page.parse::<usize>().unwrap_or(20).clamp(1, 120);
         let page = page.parse::<u32>().unwrap_or(1);
         let query_str = query.unwrap_or_default();
         let target = Self::resolve_target(&query_str, &sort, page);
 
         match self.fetch_and_parse(cache, target, options).await {
-            Ok(items) => items,
+            Ok(mut items) => {
+                items.truncate(per_page);
+                items
+            }
             Err(e) => {
                 report_provider_error(CHANNEL_ID, "get_videos", &e.to_string()).await;
                 vec![]