diff --git a/docs/provider-catalog.md b/docs/provider-catalog.md index 02ace07..b5080a4 100644 --- a/docs/provider-catalog.md +++ b/docs/provider-catalog.md @@ -59,7 +59,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us | `redtube` | `mainstream-tube` | no | no | Mainstream archive. | | `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,,DOOD,,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). | | `rule34gen` | `ai` | no | no | AI group example. | -| `rule34video` | `hentai-animation` | no | no | Hentai group example. | +| `rule34video` | `hentai-animation` | no | no | Hentai group example. **Archives:** artists are `/models/{slug}/`, franchises/genres `/categories/{slug}/`, characters and everything else `/tags/{slug}/`; each is paged and sorted through `?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort}&from={page}` (unknown slugs 404). Prefix queries `artist:`/`model:`/`creator:`, `category:`/`franchise:`/`series:`, `tag:`/`character:`; a bare query is probed as artist, then category, then tag, then resolved through `/search_ajax.php?tag=` (characters are tagged `name (franchise)`; an unambiguous exact base-name match wins), else keyword search. Cards are enriched from the detail page, which also yields the credited artist as `uploader` (`rule34video:{slug}`) and artist names as tags. | | `sextb` | `jav` | no | no | JAV family provider. sextb.net is Cloudflare-gated, so listing/pagination markdown is fetched through a hand-built Jina mirror URL (`mirror_url()`); the target is already an absolute `https://sextb.net/...` URL, so the scheme must be stripped before prepending the `https://r.jina.ai/http://` prefix — concatenating them directly (the pre-fix bug) produces a malformed `.../http://https://...` URL that Jina rejects outright. | | `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. Listing cards are `div.card` (card→`a.title` for the link/title, `a.image img` for thumb/preview, `div.duration`, `div.list div.box` for views+date, `div.name a.text` for uploader) — the site redesigned from an older `div.item.thumb`/`span.description` layout, so re-verify these selectors against `{BASE_URL}/videos/` (the real default-sort URL, not the bare homepage) if the feed ever goes empty again. | | `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. | diff --git a/src/providers/rule34video.rs b/src/providers/rule34video.rs index 8c81bb1..c690e22 100644 --- a/src/providers/rule34video.rs +++ b/src/providers/rule34video.rs @@ -1,6 +1,6 @@ use crate::DbPool; use crate::api::ClientVersion; -use crate::providers::Provider; +use crate::providers::{Provider, probe_first_archive, requester_or_default, slug_dashed}; use crate::status::*; use crate::util::cache::VideoCache; use crate::util::discord::send_discord_error_report; @@ -12,7 +12,9 @@ use async_trait::async_trait; use error_chain::error_chain; use futures::stream::{self, StreamExt}; use htmlentity::entity::{ICodedDataTrait, decode}; +use scraper::{Html, Selector}; use std::time::{SystemTime, UNIX_EPOCH}; +use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode}; use std::vec; const ENRICH_CONCURRENCY: usize = 6; @@ -36,6 +38,29 @@ error_chain! { } } +/// What a request resolves to. Artists live under `/models/`, franchises and +/// genres under `/categories/`, characters and everything else under `/tags/`. +#[derive(Debug, Clone, PartialEq, Eq)] +enum Target { + Latest, + Search(String), + Artist(String), + Category(String), + Tag(String), +} + +impl Target { + /// Archive path (`models/{slug}`, ...) for the archive targets. + fn archive_path(&self) -> Option { + match self { + Target::Artist(slug) => Some(format!("models/{slug}")), + Target::Category(slug) => Some(format!("categories/{slug}")), + Target::Tag(slug) => Some(format!("tags/{slug}")), + _ => None, + } + } +} + #[derive(Debug, Clone)] pub struct Rule34videoProvider { url: String, @@ -93,6 +118,156 @@ impl Rule34videoProvider { } } + /// Explicit `artist:` / `category:` / `tag:` style prefixes (the server has + /// already stripped a leading `#`). + fn parse_prefixed(query: &str) -> Option { + let (kind, value) = query.trim().split_once(':')?; + let slug = slug_dashed(value); + if slug.is_empty() { + return None; + } + match kind.trim().to_ascii_lowercase().as_str() { + "artist" | "artists" | "model" | "models" | "author" | "creator" | "uploader" => { + Some(Target::Artist(slug)) + } + "cat" | "category" | "categories" | "franchise" | "series" | "game" => { + Some(Target::Category(slug)) + } + "tag" | "tags" | "character" | "char" => Some(Target::Tag(slug)), + _ => None, + } + } + + /// Titles of the tags offered by `/search_ajax.php?tag=`. + fn parse_tag_suggestions(html: &str) -> Vec { + let document = Html::parse_fragment(html); + let Ok(selector) = Selector::parse("input[data-title]") else { + return vec![]; + }; + document + .select(&selector) + .filter_map(|input| input.value().attr("data-title")) + .map(|title| title.trim().to_string()) + .filter(|title| !title.is_empty()) + .collect() + } + + /// The suggested tag whose name (ignoring a trailing "(franchise)") equals + /// the query, e.g. "samus aran" -> "samus aran (metroid)". + fn best_tag_title(titles: &[String], query: &str) -> Option { + let wanted = slug_dashed(query); + if wanted.is_empty() { + return None; + } + let mut exact = titles.iter().filter(|title| { + let base = title.split('(').next().unwrap_or(title); + slug_dashed(base) == wanted || slug_dashed(title) == wanted + }); + let first = exact.next()?; + // Several franchises share a character name: ambiguous, leave it to search. + exact.next().is_none().then(|| first.clone()) + } + + /// A free-form query that names an artist, category or tag goes to that + /// archive (probed on the site; unknown names 404), else keyword search. + async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Target { + let slug = slug_dashed(query); + if slug.is_empty() { + return Target::Search(query.trim().to_string()); + } + let candidates = [ + Target::Artist(slug.clone()), + Target::Category(slug.clone()), + Target::Tag(slug), + ]; + let urls: Vec = candidates + .iter() + .filter_map(|t| t.archive_path()) + .map(|path| format!("{}/{path}/", self.url)) + .collect(); + let mut requester = requester_or_default(options, "rule34video", "resolve_keyword"); + let referer = format!("{}/", self.url); + if let Some((index, _)) = probe_first_archive(&mut requester, &urls, &referer, |body| { + body.contains("item thumb video_") + }) + .await + { + return candidates[index].clone(); + } + + // Characters are tagged "name (franchise)": ask the site's own tag search. + let suggest_url = format!( + "{}/search_ajax.php?tag={}", + self.url, + utf8_percent_encode(query.trim(), NON_ALPHANUMERIC) + ); + let headers = vec![ + ("Referer".to_string(), referer), + ("X-Requested-With".to_string(), "XMLHttpRequest".to_string()), + ]; + if let Ok(Some(body)) = requester.get_probe(&suggest_url, headers).await { + let titles = Self::parse_tag_suggestions(&body); + if let Some(title) = Self::best_tag_title(&titles, query) { + return Target::Tag(slug_dashed(&title)); + } + } + Target::Search(query.trim().to_string()) + } + + async fn resolve_target(&self, query: Option<&str>, options: &ServerOptions) -> Target { + let Some(query) = query.map(str::trim).filter(|q| !q.is_empty()) else { + return Target::Latest; + }; + if let Some(target) = Self::parse_prefixed(query) { + return target; + } + self.resolve_keyword(query, options).await + } + + async fn archive( + &self, + cache: VideoCache, + target: &Target, + page: u8, + sort: &str, + options: ServerOptions, + ) -> Result> { + let Some(path) = target.archive_path() else { + return Ok(vec![]); + }; + let sort_val = match sort { + "post_date" | "video_viewed" | "rating" | "duration" | "pseudo_random" => sort, + _ => "post_date", + }; + let index = format!("rule34video:{path}:{page}:{sort_val}"); + if sort_val != "pseudo_random" { + if let Some((time, items)) = cache.get(&index) { + if time.elapsed().unwrap_or_default().as_secs() < 300 { + return Ok(items.clone()); + } + } + } + + let mut requester = options.requester.clone().ok_or("Requester missing")?; + let url = format!( + "{}/{path}/?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort_val}&from={page}", + self.url + ); + let headers = vec![("Referer".to_string(), format!("{}/{path}/", self.url))]; + let text = match requester.get_probe(&url, headers).await { + Ok(Some(text)) => text, + // A 404 is an unknown archive (or a page past the end). + Ok(None) => return Ok(vec![]), + Err(e) => return Err(format!("archive fetch failed for {url}: {e}").into()), + }; + let video_items = self.get_video_items_from_html(text); + let video_items = self.enrich_items(video_items, requester).await; + if !video_items.is_empty() { + cache.insert(index, video_items.clone()); + } + Ok(video_items) + } + /// Helper to safely extract a string between two delimiters fn extract_between<'a>(content: &'a str, start_pat: &str, end_pat: &str) -> Option<&'a str> { let start_idx = content.find(start_pat)? + start_pat.len(); @@ -307,11 +482,44 @@ impl Rule34videoProvider { (formats.into_iter().map(|(_, f)| f).collect(), tags) } + /// Artists credited on a video page: `(name, slug)` from the "Artist" group. + fn parse_artists(html: &str) -> Vec<(String, String)> { + let document = Html::parse_document(html); + let Ok(group_sel) = Selector::parse("[data-suggest-type=\"model\"] a[href*=\"/models/\"]") + else { + return vec![]; + }; + let name_sel = Selector::parse(".name").ok(); + let mut artists: Vec<(String, String)> = Vec::new(); + for link in document.select(&group_sel) { + let href = link.value().attr("href").unwrap_or_default(); + let slug = href + .split("/models/") + .nth(1) + .unwrap_or_default() + .trim_matches('/') + .to_string(); + let name = name_sel + .as_ref() + .and_then(|sel| link.select(sel).next()) + .map(|n| n.text().collect::()) + .unwrap_or_else(|| link.text().collect::()); + let name = name.split_whitespace().collect::>().join(" "); + if slug.is_empty() || name.is_empty() || artists.iter().any(|(_, s)| *s == slug) { + continue; + } + artists.push((name, slug)); + } + artists + } + async fn enrich_items(&self, items: Vec, requester: Requester) -> Vec { let referer = format!("{}/", self.url); + let base = self.url.clone(); stream::iter(items.into_iter().map(|mut item| { let mut requester = requester.clone(); let referer = referer.clone(); + let base = base.clone(); async move { if item.url.is_empty() { return item; @@ -324,10 +532,21 @@ impl Rule34videoProvider { ) .await .unwrap_or_default(); - let (formats, tags) = Self::parse_detail(&html); + let (formats, mut tags) = Self::parse_detail(&html); if !formats.is_empty() { item.formats = Some(formats); } + let artists = Self::parse_artists(&html); + if let Some((name, slug)) = artists.first() { + item.uploader = Some(name.clone()); + item.uploaderUrl = Some(format!("{base}/models/{slug}/")); + item.uploaderId = Some(format!("rule34video:{slug}")); + } + for (name, _) in &artists { + if !tags.iter().any(|t| t.eq_ignore_ascii_case(name)) { + tags.push(name.clone()); + } + } if !tags.is_empty() { item.tags = Some(tags); } @@ -449,9 +668,11 @@ impl Provider for Rule34videoProvider { ) -> Vec { let page_num = page.parse::().unwrap_or(1); - let result = match query { - Some(q) => self.query(cache, page_num, &q, &sort, options).await, - None => self.get(cache, page_num, &sort, options).await, + let target = self.resolve_target(query.as_deref(), &options).await; + let result = match &target { + Target::Latest => self.get(cache, page_num, &sort, options).await, + Target::Search(q) => self.query(cache, page_num, q, &sort, options).await, + archive => self.archive(cache, archive, page_num, &sort, options).await, }; match result { @@ -517,4 +738,51 @@ mod tests { let (formats, tags) = Rule34videoProvider::parse_detail(""); assert!(formats.is_empty() && tags.is_empty()); } + + #[test] + fn parses_prefixed_queries() { + assert_eq!( + Rule34videoProvider::parse_prefixed("artist:General Butch"), + Some(Target::Artist("general-butch".into())) + ); + assert_eq!( + Rule34videoProvider::parse_prefixed("category:Super Smash Bros"), + Some(Target::Category("super-smash-bros".into())) + ); + assert_eq!( + Rule34videoProvider::parse_prefixed("tag:black widow (marvel)"), + Some(Target::Tag("black-widow-marvel".into())) + ); + assert_eq!(Rule34videoProvider::parse_prefixed("samus aran"), None); + } + + #[test] + fn picks_the_unambiguous_character_tag() { + let titles = vec![ + "samus aran (metroid)".to_string(), + "samus aran's mother".to_string(), + ]; + assert_eq!( + Rule34videoProvider::best_tag_title(&titles, "samus aran").as_deref(), + Some("samus aran (metroid)") + ); + let ambiguous = vec!["peach (mario)".to_string(), "peach (pokemon)".to_string()]; + assert_eq!(Rule34videoProvider::best_tag_title(&ambiguous, "peach"), None); + } + + #[test] + fn parses_tag_suggestions_and_artists() { + let suggest = r#"
"#; + assert_eq!( + Rule34videoProvider::parse_tag_suggestions(suggest), + vec!["samus aran (metroid)".to_string()] + ); + let page = r#""#; + assert_eq!( + Rule34videoProvider::parse_artists(page), + vec![("JOIGang (VA)".to_string(), "joigang-va".to_string())] + ); + } }