simon/hottub
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resolve melonstube's top-20 destination hosts into direct formats

Browse Source 
melonstube is a meta-search aggregator that redirects out to ~120
third-party hosts. A live crawl of /new, /popular, and /rating
(~3,350 out-links) found the previous "assume yt-dlp can resolve it"
fallback was hiding real failures on the highest-traffic hosts:

- eporner.com is ~34% of all melonstube traffic by itself, and like
  the eporner channel (fixed earlier), serves a restricted preview
  clip without its hash/xhr dance.
- fapcat.com/momvids.com 403 a plain fetch (Cloudflare).
- Several others just needed real extraction instead of a guess.

Per-card enrichment now dispatches by destination host to:
- Eporner: reuses eporner's own hash/xhr + Referer resolution, moved
  to src/util/eporner_resolve.rs so both providers' single-provider
  builds still compile (HOT_TUB_PROVIDER=<id> only compiles the one
  selected provider module; src/util/* always compiles).
- Kvs (pornfreenow, analdin, babepump, w1mp, fetishshrine, pervclips,
  xozilla, ok.xxx, ebonypeek, xgroovy): a shared get_file extractor,
  the same KVS backend several other Hot Tub providers already scrape.
- KvsCfGated (fapcat, momvids): same extractor via the Safari18
  bypass client melonstube.com itself already needs.
- XhamsterMirror (xh.partners): a genuine xHamster mirror instance,
  Referer built from the actual resolved host.
- SomePorn: HLS playlist URL built from the page's video id directly.
- Pornone: direct <source> tag extraction.

flirt4free.com (a live-cam page, not a VOD) moved into the dropped-
host bucket alongside the existing junk/join.* hosts.

Enrichment is bounded (concurrency, a 10s per-card timeout, and
capped to the requested perPage rather than the full ~240-card page)
so a slow or rate-limited destination can't stall the response.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Simon
2026-09-22 08:28:05 +00:00
parent a9da585f45
commit c244461d0b
5 changed files with 746 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/provider-catalog.md
View File

File diff suppressed because one or more lines are too long

170
src/providers/eporner.rs
View File

@@ -8,14 +8,13 @@ use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number; use crate::util::parse_abbreviated_number;
use crate::util::requester::Requester; use crate::util::requester::Requester;
use crate::util::time::parse_time_to_seconds; use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoFormat, VideoItem}; use crate::videos::{ServerOptions, VideoItem};
use async_trait::async_trait; use async_trait::async_trait;
use error_chain::error_chain; use error_chain::error_chain;
use futures::stream::{self, StreamExt}; use futures::stream::{self, StreamExt};
use htmlentity::entity::{ICodedDataTrait, decode}; use htmlentity::entity::{ICodedDataTrait, decode};
use regex::Regex;
use scraper::{ElementRef, Html, Selector}; use scraper::{ElementRef, Html, Selector};
use std::sync::{Arc, OnceLock, RwLock}; use std::sync::{Arc, RwLock};
use std::{collections::HashMap, thread, vec}; use std::{collections::HashMap, thread, vec};
use wreq::Version; use wreq::Version;
@@ -351,18 +350,6 @@ impl EpornerProvider {
digits.parse::<f32>().ok().map(|v| v / 100.0) digits.parse::<f32>().ok().map(|v| v / 100.0)
} }
fn alnum_id_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"/video-([A-Za-z0-9]+)").expect("valid regex"))
}
fn alnum_id_from_href(href: &str) -> Option<String> {
Self::alnum_id_regex()
.captures(href)
.and_then(|c| c.get(1))
.map(|m| m.as_str().to_string())
}
fn parse_list_cards(html: &str) -> Result<Vec<EpornerCard>> { fn parse_list_cards(html: &str) -> Result<Vec<EpornerCard>> {
let document = Html::parse_document(html); let document = Html::parse_document(html);
let card_sel = Self::selector("div.mb[data-id]")?; let card_sel = Self::selector("div.mb[data-id]")?;
@@ -390,7 +377,7 @@ impl EpornerProvider {
if page_url.is_empty() { if page_url.is_empty() {
continue; continue;
} }
let Some(alnum_id) = Self::alnum_id_from_href(href) else { let Some(alnum_id) = crate::util::eporner_resolve::alnum_id_from_href(href) else {
continue; continue;
}; };
@@ -453,115 +440,15 @@ impl EpornerProvider {
Ok(cards) Ok(cards)
} }
fn hash_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r#"hash\s*[:=]\s*['"]([0-9a-f]{32})"#).expect("valid regex"))
}
fn height_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"^(\d+)").expect("valid regex"))
}
fn base36(mut n: u32) -> String {
const DIGITS: &[u8] = b"0123456789abcdefghijklmnopqrstuvwxyz";
if n == 0 {
return "0".to_string();
}
let mut buf = Vec::new();
while n > 0 {
buf.push(DIGITS[(n % 36) as usize]);
n /= 36;
}
buf.reverse();
String::from_utf8(buf).unwrap_or_default()
}
/// Reverse-engineered from the site's player JS: the page's 32-hex-char
/// `hash` is split into four 8-hex-char chunks, each parsed as a u32 and
/// base36-encoded, then concatenated. The `/xhr/video/{id}` format API
/// requires this derived value or it serves a restricted/preview
/// response instead of the real sources.
fn calc_hash(hash: &str) -> Option<String> {
if hash.len() != 32 || !hash.is_ascii() {
return None;
}
let mut out = String::new();
for chunk_start in (0..32).step_by(8) {
let chunk = &hash[chunk_start..chunk_start + 8];
let n = u32::from_str_radix(chunk, 16).ok()?;
out.push_str(&Self::base36(n));
}
Some(out)
}
/// Fetches the detail page for its `hash`, then calls the site's own
/// `/xhr/video/{alnum_id}` JSON API (the same one the site's own player
/// uses) to get real direct mp4 URLs across all available resolutions.
/// Without this, the client is left to open the bare page URL, which
/// eporner serves as a small restricted/preview clip rather than the
/// full video.
///
/// The resolved CDN mp4 URLs also enforce a Referer check independent of
/// the hash/xhr dance above: a request whose Referer is absent or points
/// at eporner.com is served the real file, but any other non-empty
/// Referer gets a 302 to a static `.../na.mp4` ("not available") decoy
/// clip — the exact "this video is only available at eporner.com"-style
/// clip users see when playback doesn't carry the header the CDN wants.
/// Each format therefore declares an explicit `Referer` so the client's
/// own default (or lack thereof) can't accidentally trip this check.
async fn resolve_formats(
requester: &mut Requester,
alnum_id: &str,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
let detail_html = Self::fetch_html(requester, page_url).await.ok()?;
let hash = &Self::hash_regex().captures(&detail_html)?[1];
let calc = Self::calc_hash(hash)?;
let api_url = format!(
"{BASE_URL}/xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false"
);
let json_text = requester
.get_with_headers(&api_url, Self::html_headers(page_url), Some(Version::HTTP_11))
.await
.ok()?;
let value: serde_json::Value = serde_json::from_str(&json_text).ok()?;
if value.get("available").and_then(|v| v.as_bool()) == Some(false) {
return None;
}
let mp4_sources = value.get("sources")?.get("mp4")?.as_object()?;
let mut ranked: Vec<(u32, VideoFormat)> = Vec::new();
for (key, entry) in mp4_sources {
let Some(src) = entry.get("src").and_then(|v| v.as_str()) else {
continue;
};
let label = entry
.get("labelShort")
.and_then(|v| v.as_str())
.unwrap_or(key.as_str());
let height = Self::height_regex()
.captures(label)
.and_then(|c| c[1].parse::<u32>().ok())
.unwrap_or(0);
let format = VideoFormat::new(src.to_string(), label.to_string(), "mp4".to_string())
.format_id(key.clone())
.format_note(key.clone())
.height(height)
.http_header("Referer".to_string(), format!("{BASE_URL}/"));
ranked.push((height, format));
}
if ranked.is_empty() {
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
}
async fn build_item(requester: &mut Requester, card: EpornerCard) -> VideoItem { async fn build_item(requester: &mut Requester, card: EpornerCard) -> VideoItem {
let formats = Self::resolve_formats(requester, &card.alnum_id, &card.page_url).await; // The hash/xhr resolution + Referer handling live in
// `crate::util::eporner_resolve` (shared with other providers that
// redirect out to eporner.com, e.g. `melonstube`) rather than here,
// since single-provider builds can't depend on another provider's
// module — see that module's doc comment.
let formats =
crate::util::eporner_resolve::resolve_formats(requester, &card.alnum_id, &card.page_url)
.await;
let mut item = VideoItem::new( let mut item = VideoItem::new(
card.id, card.id,
@@ -813,36 +700,5 @@ impl Provider for EpornerProvider {
} }
} }
#[cfg(test)] // The calc_hash / alnum_id_from_href unit tests moved to
mod tests { // `crate::util::eporner_resolve` along with the functions themselves.
use super::*;
#[test]
fn calc_hash_matches_site_algorithm() {
// Reverse-engineered from vjs.js; verified against a live page/xhr pair.
let hash = "06aadb04e6f6e853e25f5a583f173617";
assert_eq!(hash.len(), 32);
assert_eq!(
EpornerProvider::calc_hash(hash).unwrap(),
"1ulk3o1s31fmb1qt66aghi70uv"
);
}
#[test]
fn calc_hash_rejects_wrong_length() {
assert!(EpornerProvider::calc_hash("abc").is_none());
}
#[test]
fn alnum_id_from_href_extracts_path_segment() {
assert_eq!(
EpornerProvider::alnum_id_from_href("/video-jQDTzcHvtjv/vj-blacked/"),
Some("jQDTzcHvtjv".to_string())
);
assert_eq!(
EpornerProvider::alnum_id_from_href("https://www.eporner.com/video-5Ssyrtrs9FG/-/"),
Some("5Ssyrtrs9FG".to_string())
);
assert_eq!(EpornerProvider::alnum_id_from_href("/hd-porn/95008/foo/"), None);
}
}

576
src/providers/melonstube.rs
View File

@@ -5,14 +5,17 @@ use crate::providers::{
}; };
use crate::status::*; use crate::status::*;
use crate::util::cache::VideoCache; use crate::util::cache::VideoCache;
use crate::videos::{ServerOptions, VideoItem}; use crate::util::requester::Requester;
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
use async_trait::async_trait; use async_trait::async_trait;
use base64::{Engine as _, engine::general_purpose::STANDARD}; use base64::{Engine as _, engine::general_purpose::STANDARD};
use chrono::{Duration as ChronoDuration, Utc}; use chrono::{Duration as ChronoDuration, Utc};
use error_chain::error_chain; use error_chain::error_chain;
use futures::stream::{self, StreamExt};
use regex::Regex; use regex::Regex;
use scraper::{Html, Selector}; use scraper::{Html, Selector};
use std::sync::Arc; use std::collections::HashSet;
use std::sync::{Arc, OnceLock};
use url::form_urlencoded::Serializer; use url::form_urlencoded::Serializer;
use wreq::cookie::Jar; use wreq::cookie::Jar;
use wreq::redirect::Policy; use wreq::redirect::Policy;
@@ -39,6 +42,75 @@ const HARD_HOSTS: &[&str] = &["manysex.com", "videomanysex.com"];
// false-positive "ad" results. // false-positive "ad" results.
const JUNK_HOSTS: &[&str] = &["fhgte.com"]; const JUNK_HOSTS: &[&str] = &["fhgte.com"];
// Live-cam profile pages, not a completed VOD — dropped the same way as
// junk hosts rather than surfaced as a broken "video".
const LIVE_CAM_HOSTS: &[&str] = &["flirt4free.com"];
// Destination hosts that share the classic KVS (Kernel Video Sharing)
// `get_file/{N}/{hash}/{bucket}/{id}/{id}[_variant].mp4[?token]` pattern
// already used by several other Hot Tub KVS-style providers (porn4fans.rs,
// xgroovy.rs, thepornbunny.rs), reachable directly through the shared
// Requester with no anti-bot bypass needed. Verified live against a real
// card from each host.
const KVS_HOSTS: &[&str] = &[
"pornfreenow.com",
"analdin.xxx",
"babepump.com",
"w1mp.com",
"fetishshrine.com",
"pervclips.com",
"xozilla.xxx",
"ok.xxx",
"ebonypeek.com",
"xgroovy.com",
];
// Same KVS backend as `KVS_HOSTS`, but Cloudflare-fronted: a plain fetch
// gets a 403 challenge (verified live), so these go through the same
// Safari18 bypass client `melonstube.com` itself needs before the KVS
// extraction applies.
const KVS_CF_GATED_HOSTS: &[&str] = &["fapcat.com", "momvids.com"];
// Redirect gateways into an xHamster mirror instance — verified live to be
// genuine xHamster infrastructure (same xhcdn.com CDN, same embedded
// `"sources":{"mp4":{...}}` JSON xhamster.com itself uses), just reached
// through an affiliate short-link rather than xhamster.com directly.
const XHAMSTER_MIRROR_HOSTS: &[&str] = &["xh.partners"];
// `some.porn`'s HLS master playlist can be built directly from the numeric
// id in the page URL — no page fetch required.
const SOME_PORN_HOSTS: &[&str] = &["some.porn"];
// pornone.com embeds direct, unauthenticated `<source src="...">` mp4 tags
// at multiple resolutions right on the detail page.
const PORNONE_HOSTS: &[&str] = &["pornone.com"];
// Bounded concurrency for the per-card destination-page enrichment pass.
// Deliberately higher than most other providers' enrichment concurrency:
// this fans out to many *different* third-party hosts (not repeated calls
// to one site), so there's no single-origin rate limit to respect, and a
// low value compounds badly with `ENRICH_TIMEOUT_SECS` when several cards
// in one page happen to hit a slow/unreachable host back to back.
const ENRICH_CONCURRENCY: usize = 8;
// Per-card cap so one slow or unreachable destination host can't stall the
// whole listing response.
const ENRICH_TIMEOUT_SECS: u64 = 10;
// Upper bound on how many cards get enriched per listing fetch, regardless
// of how many cards the page actually has (~240 for /new) — matches the
// max `perPage` `get_videos` accepts, so any valid request is fully
// enriched while still bounding worst-case fetch fan-out.
const MAX_ENRICH_PER_FETCH: usize = 120;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum SourceKind {
Eporner,
Kvs,
KvsCfGated,
XhamsterMirror,
SomePorn,
Pornone,
}
error_chain! { error_chain! {
foreign_links { foreign_links {
Io(std::io::Error); Io(std::io::Error);
@@ -244,7 +316,7 @@ impl MelonstubeProvider {
.ok() .ok()
} }
async fn fetch_html_bypass(url: &str) -> Option<String> { async fn fetch_html_bypass(url: &str, referer: &str) -> Option<String> {
let client = Self::build_bypass_client()?; let client = Self::build_bypass_client()?;
let response = client let response = client
.get(url) .get(url)
@@ -253,7 +325,7 @@ impl MelonstubeProvider {
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
) )
.header("accept-language", "en-US,en;q=0.5") .header("accept-language", "en-US,en;q=0.5")
.header("referer", BASE_URL) .header("referer", referer)
.send() .send()
.await .await
.ok()?; .ok()?;
@@ -281,48 +353,336 @@ impl MelonstubeProvider {
Some(response.uri().to_string().trim_end_matches('/').to_string()) Some(response.uri().to_string().trim_end_matches('/').to_string())
} }
fn bare_host(host: &str) -> &str {
host.strip_prefix("www.").unwrap_or(host)
}
fn is_junk_host(host: &str) -> bool { fn is_junk_host(host: &str) -> bool {
JUNK_HOSTS.contains(&host) || host.starts_with("join.") let bare = Self::bare_host(host);
JUNK_HOSTS.contains(&bare) || LIVE_CAM_HOSTS.contains(&bare) || host.starts_with("join.")
} }
fn is_hard_host(host: &str) -> bool { fn is_hard_host(host: &str) -> bool {
HARD_HOSTS.contains(&host) HARD_HOSTS.contains(&Self::bare_host(host))
} }
fn parse_listing(&self, html: &str, options: &ServerOptions) -> Vec<VideoItem> { fn classify_source(host: &str) -> Option<SourceKind> {
let bare = Self::bare_host(host);
if bare == "eporner.com" {
Some(SourceKind::Eporner)
} else if KVS_HOSTS.contains(&bare) {
Some(SourceKind::Kvs)
} else if KVS_CF_GATED_HOSTS.contains(&bare) {
Some(SourceKind::KvsCfGated)
} else if XHAMSTER_MIRROR_HOSTS.contains(&bare) {
Some(SourceKind::XhamsterMirror)
} else if SOME_PORN_HOSTS.contains(&bare) {
Some(SourceKind::SomePorn)
} else if PORNONE_HOSTS.contains(&bare) {
Some(SourceKind::Pornone)
} else {
None
}
}
fn kvs_mp4_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| {
Regex::new(
r#"(?:https?://[^"'\s\\<>]+?)?get_file/\d+/[0-9a-f]+/\d+/\d+/[^"'\s\\<>]+\.mp4[^"'\s\\<>]*"#,
)
.expect("valid regex")
})
}
fn kvs_height_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"_(\d{3,4})p\.mp4").expect("valid regex"))
}
/// Ranks a KVS `get_file` URL for sorting only — not every site tags its
/// higher-quality variant with a numeric resolution; several (analdin,
/// xozilla) instead suffix it with a bare "hd" (`{id}hd.mp4` vs the
/// default `{id}.mp4`), which carries no exact pixel height.
fn kvs_quality_score(url: &str) -> u32 {
if let Some(height) = Self::kvs_height_regex()
.captures(url)
.and_then(|c| c[1].parse::<u32>().ok())
{
return height;
}
if url.contains("hd.mp4") { 1 } else { 0 }
}
/// Extracts every distinct `get_file` mp4 URL from a KVS-CMS detail page
/// (the same backend porn4fans.rs/xgroovy.rs/thepornbunny.rs already
/// scrape), regardless of which JS variable name the site wraps them in
/// — some expose named `video_url`/`video_alt_url` flashvars, others just
/// inline the URL directly, so this scans the raw HTML for the URL shape
/// itself. Short teaser/thumbnail-preview variants (`_s.mp4`,
/// `_preview.mp4`) are excluded so a card never resolves to a decoy clip.
fn parse_kvs_formats(html: &str, origin: &str) -> Option<Vec<VideoFormat>> {
let mut seen = HashSet::new();
let mut ranked: Vec<(u32, VideoFormat)> = Vec::new();
for m in Self::kvs_mp4_regex().find_iter(html) {
let raw = m.as_str();
if raw.contains("_s.mp4") || raw.contains("_preview.mp4") || raw.contains("_thumb") {
continue;
}
let url = if raw.starts_with("http") {
raw.to_string()
} else {
format!("{origin}/{raw}")
};
if !seen.insert(url.clone()) {
continue;
}
let score = Self::kvs_quality_score(&url);
let height = Self::kvs_height_regex()
.captures(&url)
.and_then(|c| c[1].parse::<u32>().ok())
.unwrap_or(0);
let quality = if height > 0 {
format!("{height}p")
} else if url.contains("hd.mp4") {
"hd".to_string()
} else {
"source".to_string()
};
let format = VideoFormat::new(url, quality.clone(), "mp4".to_string())
.format_note(quality)
.height(height)
.http_header("Referer".to_string(), format!("{origin}/"));
ranked.push((score, format));
}
if ranked.is_empty() {
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
}
async fn resolve_kvs(
requester: &mut Requester,
detail_url: &str,
cf_gated: bool,
) -> Option<Vec<VideoFormat>> {
let origin = url::Url::parse(detail_url).ok()?.origin().ascii_serialization();
let html = if cf_gated {
Self::fetch_html_bypass(detail_url, &format!("{origin}/")).await?
} else {
requester
.get_with_headers(
detail_url,
vec![
("user-agent".to_string(), BROWSER_UA.to_string()),
(
"accept".to_string(),
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
.to_string(),
),
],
None,
)
.await
.ok()?
};
Self::parse_kvs_formats(&html, &origin)
}
async fn resolve_eporner(
requester: &mut Requester,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
crate::util::eporner_resolve::resolve_public_formats(requester, detail_url).await
}
fn xhamster_mirror_mp4_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| {
Regex::new(r#""(\d{3,4})p":"(https:\\/\\/[^"]+?\.mp4[^"]*)""#).expect("valid regex")
})
}
/// `xh.partners`-style gateways redirect into a live xHamster mirror
/// instance whose page embeds a `"sources":{"mp4":{"144p":"...",...}}`
/// JSON block identical in shape to xhamster.com's own. The signed CDN
/// URLs require a Referer from the exact mirror host that served them
/// (verified live), so the Referer is built from the actual final URL
/// after following the gateway's redirect rather than assumed.
async fn resolve_xhamster_mirror(
requester: &mut Requester,
gateway_url: &str,
) -> Option<Vec<VideoFormat>> {
let response = requester
.get_raw_with_headers(
gateway_url,
vec![("User-Agent".to_string(), BROWSER_UA.to_string())],
)
.await
.ok()?;
if !response.status().is_success() {
return None;
}
let final_host = response.uri().host()?.to_string();
let html = response.text().await.ok()?;
let anchor = html.find(r#""sources":{"mp4":{"#)?;
let window_end = (anchor + 6000).min(html.len());
let window = &html[anchor..window_end];
let referer = format!("https://{final_host}/");
let mut ranked: Vec<(u32, VideoFormat)> = Vec::new();
for cap in Self::xhamster_mirror_mp4_regex().captures_iter(window) {
let Ok(height) = cap[1].parse::<u32>() else {
continue;
};
let url = cap[2].replace(r"\/", "/");
let format = VideoFormat::new(url, format!("{height}p"), "mp4".to_string())
.format_note(format!("{height}p"))
.height(height)
.http_header("Referer".to_string(), referer.clone());
ranked.push((height, format));
}
if ranked.is_empty() {
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
}
fn some_porn_id_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"/video/(\d+)").expect("valid regex"))
}
/// `some.porn`'s HLS master playlist is addressable directly from the
/// numeric video id in the page URL — no page fetch needed. Verified
/// live: `s.cdn3x.com/get-playlist` returns a real segmented `.m3u8`
/// with no session/cookie requirement.
fn resolve_some_porn(detail_url: &str) -> Option<Vec<VideoFormat>> {
let id = Self::some_porn_id_regex().captures(detail_url)?.get(1)?.as_str();
let url = format!("https://s.cdn3x.com/get-playlist?id={id}&q=720p");
Some(vec![
VideoFormat::m3u8(url, "720p".to_string(), "hls".to_string())
.format_note("720p".to_string()),
])
}
fn pornone_source_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| {
Regex::new(r#"<source src="(https://[^"]+?\.mp4[^"]*)""#).expect("valid regex")
})
}
fn pornone_resolution_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"_(\d+)x(\d+)_").expect("valid regex"))
}
/// pornone.com embeds direct, unauthenticated `<source src="...">` mp4
/// tags at multiple resolutions right on the detail page — no Referer
/// required (verified live).
async fn resolve_pornone(
requester: &mut Requester,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
let html = requester
.get_with_headers(
detail_url,
vec![("user-agent".to_string(), BROWSER_UA.to_string())],
None,
)
.await
.ok()?;
let mut seen = HashSet::new();
let mut ranked: Vec<(u32, VideoFormat)> = Vec::new();
for cap in Self::pornone_source_regex().captures_iter(&html) {
let url = cap[1].to_string();
if !seen.insert(url.clone()) {
continue;
}
let height = Self::pornone_resolution_regex()
.captures(&url)
.and_then(|c| c[2].parse::<u32>().ok())
.unwrap_or(0);
let quality = if height > 0 {
format!("{height}p")
} else {
"source".to_string()
};
let format = VideoFormat::new(url, quality.clone(), "mp4".to_string())
.format_note(quality)
.height(height);
ranked.push((height, format));
}
if ranked.is_empty() {
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
}
async fn resolve_source_formats(
kind: SourceKind,
requester: &mut Requester,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
match kind {
SourceKind::Eporner => Self::resolve_eporner(requester, detail_url).await,
SourceKind::Kvs => Self::resolve_kvs(requester, detail_url, false).await,
SourceKind::KvsCfGated => Self::resolve_kvs(requester, detail_url, true).await,
SourceKind::XhamsterMirror => Self::resolve_xhamster_mirror(requester, detail_url).await,
SourceKind::SomePorn => Self::resolve_some_porn(detail_url),
SourceKind::Pornone => Self::resolve_pornone(requester, detail_url).await,
}
}
fn parse_listing(
&self,
html: &str,
options: &ServerOptions,
) -> (Vec<VideoItem>, Vec<(usize, SourceKind, String)>) {
let document = Html::parse_document(html); let document = Html::parse_document(html);
let card_sel = match Selector::parse("div.card[data-public-id]") { let card_sel = match Selector::parse("div.card[data-public-id]") {
Ok(s) => s, Ok(s) => s,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let link_sel = match Selector::parse("a.item-link[href]") { let link_sel = match Selector::parse("a.item-link[href]") {
Ok(s) => s, Ok(s) => s,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let img_sel = match Selector::parse("img.item-image") { let img_sel = match Selector::parse("img.item-image") {
Ok(s) => s, Ok(s) => s,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let model_sel = match Selector::parse("a.pornstar-label") { let model_sel = match Selector::parse("a.pornstar-label") {
Ok(s) => s, Ok(s) => s,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let source_sel = match Selector::parse(r#"a.item-source[href^="/source/"]"#) { let source_sel = match Selector::parse(r#"a.item-source[href^="/source/"]"#) {
Ok(s) => s, Ok(s) => s,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let dur_re = match Regex::new(r#"class="badge[^"]*"[^>]*>\s*(\d{1,2}:\d{2}(?::\d{2})?)\s*</span>"#) { let dur_re = match Regex::new(r#"class="badge[^"]*"[^>]*>\s*(\d{1,2}:\d{2}(?::\d{2})?)\s*</span>"#) {
Ok(r) => r, Ok(r) => r,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let ago_re = match Regex::new(r#"</a><span class="item-source[^"]*">([^<]+)</span>"#) { let ago_re = match Regex::new(r#"</a><span class="item-source[^"]*">([^<]+)</span>"#) {
Ok(r) => r, Ok(r) => r,
Err(_) => return vec![], Err(_) => return (vec![], vec![]),
}; };
let mut items = Vec::new(); let mut items = Vec::new();
let mut pending: Vec<(usize, SourceKind, String)> = Vec::new();
for card in document.select(&card_sel) { for card in document.select(&card_sel) {
let public_id = match card.value().attr("data-public-id") { let public_id = match card.value().attr("data-public-id") {
@@ -381,14 +741,13 @@ impl MelonstubeProvider {
let proxy_target = strip_url_scheme(&destination_url); let proxy_target = strip_url_scheme(&destination_url);
let proxy_url = build_proxy_url(options, CHANNEL_ID, &proxy_target); let proxy_url = build_proxy_url(options, CHANNEL_ID, &proxy_target);
if !proxy_url.is_empty() { if !proxy_url.is_empty() {
let mut format = crate::videos::VideoFormat::new( let mut format =
proxy_url, VideoFormat::new(proxy_url, "auto".to_string(), "video/mp4".to_string());
"auto".to_string(),
"video/mp4".to_string(),
);
format.add_http_header("Referer".to_string(), destination_url.clone()); format.add_http_header("Referer".to_string(), destination_url.clone());
item = item.formats(vec![format]); item = item.formats(vec![format]);
} }
} else if let Some(kind) = Self::classify_source(host) {
pending.push((items.len(), kind, destination_url.clone()));
} }
if let Some(ts) = ago_re if let Some(ts) = ago_re
@@ -427,7 +786,7 @@ impl MelonstubeProvider {
items.push(item); items.push(item);
} }
items (items, pending)
} }
fn resolve_target(query: &str, sort: &str, page: u32) -> Target { fn resolve_target(query: &str, sort: &str, page: u32) -> Target {
@@ -480,6 +839,7 @@ impl MelonstubeProvider {
&self, &self,
cache: VideoCache, cache: VideoCache,
target: Target, target: Target,
per_page: usize,
options: ServerOptions, options: ServerOptions,
) -> Result<Vec<VideoItem>> { ) -> Result<Vec<VideoItem>> {
let url = match &target { let url = match &target {
@@ -507,34 +867,70 @@ impl MelonstubeProvider {
} }
} }
let html = match Self::fetch_html_bypass(&url).await { let mut requester = requester_or_default(&options, CHANNEL_ID, "fetch_and_parse");
let html = match Self::fetch_html_bypass(&url, BASE_URL).await {
Some(html) if !html.is_empty() => html, Some(html) if !html.is_empty() => html,
_ => { _ => requester
let mut requester = requester_or_default(&options, CHANNEL_ID, "fetch_and_parse"); .get_with_headers(
requester &url,
.get_with_headers( vec![
&url, ("user-agent".to_string(), BROWSER_UA.to_string()),
vec![ (
("user-agent".to_string(), BROWSER_UA.to_string()), "accept".to_string(),
( "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
"accept".to_string(), .to_string(),
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" ),
.to_string(), ("accept-language".to_string(), "en-US,en;q=0.5".to_string()),
), ],
("accept-language".to_string(), "en-US,en;q=0.5".to_string()), Some(wreq::Version::HTTP_11),
], )
Some(wreq::Version::HTTP_11), .await
) .map_err(|e| Error::from(format!("request failed url={url}: {e}")))?,
.await
.map_err(|e| Error::from(format!("request failed url={url}: {e}")))?
}
}; };
if html.is_empty() { if html.is_empty() {
return Ok(vec![]); return Ok(vec![]);
} }
let items = self.parse_listing(&html, &options); let (mut items, mut pending) = self.parse_listing(&html, &options);
// Only enrich as many cards as this request can actually use — the
// page itself may have ~240 cards, but per_page is what the caller
// asked for. Cached results still hold the full parsed list for a
// later, larger per_page request; only the cards beyond this bound
// stay on the plain-destination-URL fallback for *this* response.
pending.truncate(per_page.min(MAX_ENRICH_PER_FETCH));
if !pending.is_empty() {
let resolved = stream::iter(pending.into_iter().map(|(idx, kind, dest_url)| {
let mut req = requester.clone();
async move {
// Bounded so one slow/hung third-party destination (e.g.
// an unreachable FlareSolverr fallback) can't stall the
// whole listing response — the card just falls back to
// its plain destination URL with no formats.
let formats = tokio::time::timeout(
std::time::Duration::from_secs(ENRICH_TIMEOUT_SECS),
Self::resolve_source_formats(kind, &mut req, &dest_url),
)
.await
.ok()
.flatten();
(idx, formats)
}
}))
.buffer_unordered(ENRICH_CONCURRENCY)
.collect::<Vec<_>>()
.await;
for (idx, formats) in resolved {
if let Some(formats) = formats {
if let Some(item) = items.get_mut(idx) {
item.formats = Some(formats);
}
}
}
}
if !items.is_empty() { if !items.is_empty() {
cache.insert(url, items.clone()); cache.insert(url, items.clone());
@@ -562,7 +958,7 @@ impl Provider for MelonstubeProvider {
let query_str = query.unwrap_or_default(); let query_str = query.unwrap_or_default();
let target = Self::resolve_target(&query_str, &sort, page); let target = Self::resolve_target(&query_str, &sort, page);
match self.fetch_and_parse(cache, target, options).await { match self.fetch_and_parse(cache, target, per_page, options).await {
Ok(mut items) => { Ok(mut items) => {
items.truncate(per_page); items.truncate(per_page);
items items
@@ -662,9 +1058,99 @@ mod tests {
assert!(!MelonstubeProvider::is_hard_host("www.429men.com")); assert!(!MelonstubeProvider::is_hard_host("www.429men.com"));
assert!(MelonstubeProvider::is_junk_host("fhgte.com")); assert!(MelonstubeProvider::is_junk_host("fhgte.com"));
assert!(MelonstubeProvider::is_junk_host("join.nastyclub.com")); assert!(MelonstubeProvider::is_junk_host("join.nastyclub.com"));
assert!(MelonstubeProvider::is_junk_host("www.flirt4free.com"));
assert!(!MelonstubeProvider::is_junk_host("www.429men.com")); assert!(!MelonstubeProvider::is_junk_host("www.429men.com"));
} }
#[test]
fn classifies_source_hosts() {
use super::SourceKind;
assert_eq!(
MelonstubeProvider::classify_source("www.eporner.com"),
Some(SourceKind::Eporner)
);
assert_eq!(
MelonstubeProvider::classify_source("eporner.com"),
Some(SourceKind::Eporner)
);
assert_eq!(
MelonstubeProvider::classify_source("xgroovy.com"),
Some(SourceKind::Kvs)
);
assert_eq!(
MelonstubeProvider::classify_source("www.analdin.xxx"),
Some(SourceKind::Kvs)
);
assert_eq!(
MelonstubeProvider::classify_source("www.fapcat.com"),
Some(SourceKind::KvsCfGated)
);
assert_eq!(
MelonstubeProvider::classify_source("xh.partners"),
Some(SourceKind::XhamsterMirror)
);
assert_eq!(
MelonstubeProvider::classify_source("some.porn"),
Some(SourceKind::SomePorn)
);
assert_eq!(
MelonstubeProvider::classify_source("pornone.com"),
Some(SourceKind::Pornone)
);
assert_eq!(MelonstubeProvider::classify_source("www.429men.com"), None);
// Hosts still resolved via manysex.com's dedicated proxy must not
// also be picked up here — the two paths are mutually exclusive in
// `parse_listing`.
assert_eq!(MelonstubeProvider::classify_source("manysex.com"), None);
}
#[test]
fn parses_kvs_formats_from_html() {
let html = r#"
var flashvars = {
video_url: 'https://www.analdin.xxx/get_file/20/6d9d5daae562d9c7a41c88b6ac9210e5/812000/812240/812240.mp4/',
video_alt_url: 'https://www.analdin.xxx/get_file/20/28c406693124ceee1097b4d31cd0fdbb/812000/812240/812240hd.mp4/',
video_url_text: '480p',
video_alt_url_text: '720p',
};
var preview = 'https://www.analdin.xxx/get_file/20/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/812000/812240/812240_preview.mp4/';
"#;
let formats = MelonstubeProvider::parse_kvs_formats(html, "https://www.analdin.xxx")
.expect("finds formats");
assert_eq!(formats.len(), 2);
// Higher-labelled resolution ranked first.
assert!(formats[0].url.contains("812240hd.mp4"));
assert!(formats[1].url.contains("812240.mp4"));
for format in &formats {
assert!(!format.url.contains("_preview.mp4"));
assert_eq!(
format.http_headers_pairs(),
vec![("Referer".to_string(), "https://www.analdin.xxx/".to_string())]
);
}
}
#[test]
fn parses_kvs_formats_excludes_short_teaser_variant() {
let html = r#"video_url: 'https://w1mp.com/get_file/6/4a69e7a72396b25724d177cb557a2b68/561000/561838/561838_s.mp4/'"#;
assert!(MelonstubeProvider::parse_kvs_formats(html, "https://w1mp.com").is_none());
}
#[test]
fn resolves_some_porn_playlist_url_from_page_id() {
let formats = MelonstubeProvider::resolve_some_porn(
"https://some.porn/video/278305/the-worlds-hottest-italian?utm_campaign=933ab2ff",
)
.expect("builds a format");
assert_eq!(formats.len(), 1);
assert_eq!(
formats[0].url,
"https://s.cdn3x.com/get-playlist?id=278305&q=720p"
);
}
#[test] #[test]
fn parses_cards_from_html() { fn parses_cards_from_html() {
let html = r##" let html = r##"
@@ -684,7 +1170,8 @@ mod tests {
let provider = MelonstubeProvider::new(); let provider = MelonstubeProvider::new();
let options = make_options(); let options = make_options();
let items = provider.parse_listing(html, &options); let (items, pending) = provider.parse_listing(html, &options);
assert!(pending.is_empty());
assert_eq!(items.len(), 1); assert_eq!(items.len(), 1);
let item = &items[0]; let item = &items[0];
assert_eq!(item.id, "rrUbb7lXkQG"); assert_eq!(item.id, "rrUbb7lXkQG");
@@ -712,7 +1199,8 @@ mod tests {
let provider = MelonstubeProvider::new(); let provider = MelonstubeProvider::new();
let options = make_options(); let options = make_options();
let items = provider.parse_listing(html, &options); let (items, pending) = provider.parse_listing(html, &options);
assert!(pending.is_empty());
assert_eq!(items.len(), 1); assert_eq!(items.len(), 1);
let item = &items[0]; let item = &items[0];
assert!(item.url.starts_with("https://videomanysex.com/video/3519093/")); assert!(item.url.starts_with("https://videomanysex.com/video/3519093/"));

198
src/util/eporner_resolve.rs Normal file
View File

@@ -0,0 +1,198 @@
//! eporner.com direct-mp4 resolution, shared across providers.
//!
//! `eporner.rs` is the primary user of this (the `eporner` channel itself),
//! but other providers that redirect out to eporner.com (e.g. `melonstube`,
//! a meta-search aggregator) need the exact same handling to avoid leaving
//! the client with a bare page URL — see [`resolve_formats`] for why.
//!
//! Lives under `src/util/` rather than being imported directly from
//! `eporner.rs` because `HOT_TUB_PROVIDER=<id>` single-provider builds only
//! compile the one selected provider module; a cross-provider `use` would
//! break every other provider's single-provider build. `src/util/*` is
//! always compiled regardless of provider selection.
use crate::util::requester::Requester;
use crate::videos::VideoFormat;
use regex::Regex;
use std::sync::OnceLock;
use wreq::Version;
const BASE_URL: &str = "https://www.eporner.com";
const FIREFOX_UA: &str = "Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0";
const HTML_ACCEPT: &str =
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8";
fn html_headers(referer: &str) -> Vec<(String, String)> {
vec![
("User-Agent".to_string(), FIREFOX_UA.to_string()),
("Accept".to_string(), HTML_ACCEPT.to_string()),
("Referer".to_string(), referer.to_string()),
]
}
fn alnum_id_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"/video-([A-Za-z0-9]+)").expect("valid regex"))
}
/// Extracts the alphanumeric video id eporner uses in its page URL
/// (`/video-{alnum_id}/...`) and its `/xhr/video/{alnum_id}` format API.
pub fn alnum_id_from_href(href: &str) -> Option<String> {
alnum_id_regex()
.captures(href)
.and_then(|c| c.get(1))
.map(|m| m.as_str().to_string())
}
fn hash_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r#"hash\s*[:=]\s*['"]([0-9a-f]{32})"#).expect("valid regex"))
}
fn height_regex() -> &'static Regex {
static RE: OnceLock<Regex> = OnceLock::new();
RE.get_or_init(|| Regex::new(r"^(\d+)").expect("valid regex"))
}
fn base36(mut n: u32) -> String {
const DIGITS: &[u8] = b"0123456789abcdefghijklmnopqrstuvwxyz";
if n == 0 {
return "0".to_string();
}
let mut buf = Vec::new();
while n > 0 {
buf.push(DIGITS[(n % 36) as usize]);
n /= 36;
}
buf.reverse();
String::from_utf8(buf).unwrap_or_default()
}
/// Reverse-engineered from the site's player JS: the page's 32-hex-char
/// `hash` is split into four 8-hex-char chunks, each parsed as a u32 and
/// base36-encoded, then concatenated. The `/xhr/video/{id}` format API
/// requires this derived value or it serves a restricted/preview response
/// instead of the real sources.
fn calc_hash(hash: &str) -> Option<String> {
if hash.len() != 32 || !hash.is_ascii() {
return None;
}
let mut out = String::new();
for chunk_start in (0..32).step_by(8) {
let chunk = &hash[chunk_start..chunk_start + 8];
let n = u32::from_str_radix(chunk, 16).ok()?;
out.push_str(&base36(n));
}
Some(out)
}
/// Fetches the detail page for its `hash`, then calls the site's own
/// `/xhr/video/{alnum_id}` JSON API (the same one the site's own player
/// uses) to get real direct mp4 URLs across all available resolutions.
/// Without this, the client is left to open the bare page URL, which
/// eporner serves as a small restricted/preview clip rather than the full
/// video.
///
/// The resolved CDN mp4 URLs also enforce a Referer check independent of
/// the hash/xhr dance above: a request whose Referer is absent or points at
/// eporner.com is served the real file, but any other non-empty Referer
/// gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the
/// exact "this video is only available at eporner.com"-style clip users see
/// when playback doesn't carry the header the CDN wants. Each format
/// therefore declares an explicit `Referer` so the client's own default (or
/// lack thereof) can't accidentally trip this check.
pub async fn resolve_formats(
requester: &mut Requester,
alnum_id: &str,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
let detail_html = requester
.get_with_headers(page_url, html_headers(page_url), Some(Version::HTTP_11))
.await
.ok()?;
let hash = &hash_regex().captures(&detail_html)?[1];
let calc = calc_hash(hash)?;
let api_url = format!(
"{BASE_URL}/xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false"
);
let json_text = requester
.get_with_headers(&api_url, html_headers(page_url), Some(Version::HTTP_11))
.await
.ok()?;
let value: serde_json::Value = serde_json::from_str(&json_text).ok()?;
if value.get("available").and_then(|v| v.as_bool()) == Some(false) {
return None;
}
let mp4_sources = value.get("sources")?.get("mp4")?.as_object()?;
let mut ranked: Vec<(u32, VideoFormat)> = Vec::new();
for (key, entry) in mp4_sources {
let Some(src) = entry.get("src").and_then(|v| v.as_str()) else {
continue;
};
let label = entry
.get("labelShort")
.and_then(|v| v.as_str())
.unwrap_or(key.as_str());
let height = height_regex()
.captures(label)
.and_then(|c| c[1].parse::<u32>().ok())
.unwrap_or(0);
let format = VideoFormat::new(src.to_string(), label.to_string(), "mp4".to_string())
.format_id(key.clone())
.format_note(key.clone())
.height(height)
.http_header("Referer".to_string(), format!("{BASE_URL}/"));
ranked.push((height, format));
}
if ranked.is_empty() {
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
}
/// Resolves eporner.com direct mp4 formats for an arbitrary eporner detail
/// page URL, extracting the alnum id from the URL itself. For use by
/// providers that only have a raw destination URL (e.g. a meta-search
/// aggregator's out-link), not a pre-parsed listing card.
pub async fn resolve_public_formats(
requester: &mut Requester,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
let alnum_id = alnum_id_from_href(page_url)?;
resolve_formats(requester, &alnum_id, page_url).await
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn calc_hash_matches_site_algorithm() {
// Reverse-engineered from vjs.js; verified against a live page/xhr pair.
let hash = "06aadb04e6f6e853e25f5a583f173617";
assert_eq!(hash.len(), 32);
assert_eq!(calc_hash(hash).unwrap(), "1ulk3o1s31fmb1qt66aghi70uv");
}
#[test]
fn calc_hash_rejects_wrong_length() {
assert!(calc_hash("abc").is_none());
}
#[test]
fn alnum_id_from_href_extracts_path_segment() {
assert_eq!(
alnum_id_from_href("/video-jQDTzcHvtjv/vj-blacked/"),
Some("jQDTzcHvtjv".to_string())
);
assert_eq!(
alnum_id_from_href("https://www.eporner.com/video-5Ssyrtrs9FG/-/"),
Some("5Ssyrtrs9FG".to_string())
);
assert_eq!(alnum_id_from_href("/hd-porn/95008/foo/"), None);
}
}

1
src/util/mod.rs
View File

@@ -3,6 +3,7 @@ use std::process::Command;
pub mod cache; pub mod cache;
pub mod discord; pub mod discord;
pub mod eporner_resolve;
pub mod flaresolverr; pub mod flaresolverr;
pub mod flow_debug; pub mod flow_debug;
pub mod hoster_proxy; pub mod hoster_proxy;