Fix javtiful: parse redesigned card markup, drop per-item detail fetches
The site moved to article.video-card; the old front-video-card parser found nothing and the unbounded detail fan-out timed out. The page URL now resolves natively in yt-dlp, so it is returned directly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
This commit is contained in:
70
docs/provider-fix-todo.md
Normal file
70
docs/provider-fix-todo.md
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
# Provider fix todo
|
||||||
|
|
||||||
|
Source: `check.py` + search probe audit, 2026-10-10. Serial re-run of every flagged channel,
|
||||||
|
so timeouts below are not load artifacts. Update the status column as each item is fixed and
|
||||||
|
verified (`HOT_TUB_PROVIDER=<id>` build, `/api/videos` feed + search + page 2, `yt-dlp` on
|
||||||
|
`video.url` or `formats[0].url`, `python3 check.py <id>`).
|
||||||
|
|
||||||
|
Status: `[ ]` open, `[~]` in progress, `[x]` fixed and verified, `[-]` not reproducible / not a provider bug (reason noted).
|
||||||
|
|
||||||
|
## 1. Channels returning no items
|
||||||
|
|
||||||
|
| | Channel | Symptom | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [x] | `javtiful` | feed times out (150s) then 0 items; search 0 | site redesign (`article.video-card`); rewrote parser, page URL as `video.url` (yt-dlp-native), no per-item detail fetches. 23 items, p2/search OK |
|
||||||
|
| [ ] | `sextb` | 0 items on feed, search and page 2 | |
|
||||||
|
| [ ] | `camsoda` | 0 items on every request | live cams, CF + Jina-only fetch (see memory) |
|
||||||
|
| [ ] | `hentaimama` | 0 items or 85-150s timeouts | |
|
||||||
|
| [ ] | `hotbunny` | 0 items; first request 78s | |
|
||||||
|
|
||||||
|
## 2. Playback: page URL not resolvable by yt-dlp / formats not playable
|
||||||
|
|
||||||
|
| | Channel | Symptom | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [ ] | `thaiporntv` | page URL "Unsupported URL", no `formats` | populate formats from detail page |
|
||||||
|
| [ ] | `pornmz` | page URL "Unsupported URL", no `formats` | populate formats from detail page |
|
||||||
|
| [ ] | `coomer` | page URL "Unsupported URL"; media host CF-gated here | check whether `formats` are populated for videos |
|
||||||
|
| [ ] | `tube8` | page URL 404; `/proxy/tube8/{id}` 502 | |
|
||||||
|
| [ ] | `hypnotube` | page URLs 404 on all sampled items | URL construction |
|
||||||
|
| [ ] | `fullpornerorg` | `lulustream` proxy 403 / 502 | |
|
||||||
|
| [ ] | `ripnsfw` | `doodstream` format 502 | |
|
||||||
|
| [ ] | `freepornvideosxxx` | yt-dlp 502 on page URL | may be transient, verify |
|
||||||
|
| [ ] | `hdporn92` | m3u8 formats time out (curl + yt-dlp) | morencius.com streams |
|
||||||
|
|
||||||
|
## 3. Thumbnails
|
||||||
|
|
||||||
|
| | Channel | Symptom | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [ ] | `xxdbx` | malformed thumbs `https:/7004….jpg` | built as `"https:" + src` |
|
||||||
|
| [ ] | `ph-shorties` | thumbs return 403 | |
|
||||||
|
| [ ] | `noodlemagazine` | thumbs unreachable (-1 / 400) in check.py | thumb proxy |
|
||||||
|
|
||||||
|
## 4. Search
|
||||||
|
|
||||||
|
| | Channel | Symptom | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [ ] | `fapello` | "girl" and "milf" both 0 | |
|
||||||
|
| [ ] | `chaturbate` | "girl" and "milf" both 0 | |
|
||||||
|
| [ ] | `cumlouder` | unknown tag takes 146s before returning 0 | own provider, retry path on 404 |
|
||||||
|
| [ ] | `vjav` | "girl" 0, "milf" 10 | may be term-specific |
|
||||||
|
| [ ] | `xxxtik` | "girl" 0 (not re-run serially) | |
|
||||||
|
| [ ] | `hypnotube` | "milf" 0, "girl" 20 | may be term-specific |
|
||||||
|
|
||||||
|
## 5. Slow channels
|
||||||
|
|
||||||
|
| | Channel | Symptom | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [ ] | `aps` | 30-98s per request; check.py 30s timeout | |
|
||||||
|
| [ ] | `hdporn92` | feed about 66s per request | shares item with section 2 |
|
||||||
|
| [ ] | `porndish` | 14-23s feed/search | |
|
||||||
|
|
||||||
|
## 6. Tooling
|
||||||
|
|
||||||
|
| | Item | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| [ ] | `check.py` "url points to hottub server" | contradicts the hottub skill (proxy-needing videos use the proxy as `item.url`). Affects spankbang, hqporner, pornhd3x, pimpbunny, sxyprn, noodlemagazine. Decide: exempt proxy URLs that resolve, rather than hard-fail. |
|
||||||
|
|
||||||
|
## Environmental / needs the production IP (not worked here)
|
||||||
|
|
||||||
|
`porndish`, `supjav`, `jable`, `fapello`, `missav`, `wowxxx`, `hentaihaven` return Cloudflare 403 to
|
||||||
|
yt-dlp/curl from this machine while the server itself returns items. `hsex` returned 429 (rate limit).
|
||||||
@@ -1,20 +1,16 @@
|
|||||||
use crate::DbPool;
|
use crate::DbPool;
|
||||||
use crate::api::ClientVersion;
|
use crate::api::ClientVersion;
|
||||||
use crate::providers::{Provider, build_proxy_url, strip_url_scheme};
|
use crate::providers::Provider;
|
||||||
use crate::status::*;
|
use crate::status::*;
|
||||||
use crate::util::cache::VideoCache;
|
use crate::util::cache::VideoCache;
|
||||||
use crate::util::discord::{format_error_chain, send_discord_error_report};
|
use crate::util::parse_abbreviated_number;
|
||||||
use crate::util::requester::Requester;
|
|
||||||
use crate::util::time::parse_time_to_seconds;
|
use crate::util::time::parse_time_to_seconds;
|
||||||
use crate::videos::{ServerOptions, VideoItem};
|
use crate::videos::{ServerOptions, VideoItem};
|
||||||
|
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
use error_chain::error_chain;
|
use error_chain::error_chain;
|
||||||
use futures::future::join_all;
|
|
||||||
use htmlentity::entity::{ICodedDataTrait, decode};
|
use htmlentity::entity::{ICodedDataTrait, decode};
|
||||||
use std::sync::{Arc, RwLock};
|
|
||||||
use std::vec;
|
use std::vec;
|
||||||
use titlecase::Titlecase;
|
|
||||||
use wreq::Version;
|
use wreq::Version;
|
||||||
|
|
||||||
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
|
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
|
||||||
@@ -40,16 +36,13 @@ error_chain! {
|
|||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct JavtifulProvider {
|
pub struct JavtifulProvider {
|
||||||
url: String,
|
url: String,
|
||||||
categories: Arc<RwLock<Vec<FilterOption>>>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl JavtifulProvider {
|
impl JavtifulProvider {
|
||||||
pub fn new() -> Self {
|
pub fn new() -> Self {
|
||||||
let provider = Self {
|
Self {
|
||||||
url: "https://javtiful.com".to_string(),
|
url: "https://javtiful.com".to_string(),
|
||||||
categories: Arc::new(RwLock::new(vec![])),
|
}
|
||||||
};
|
|
||||||
provider
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn build_channel(&self, clientversion: ClientVersion) -> Channel {
|
fn build_channel(&self, clientversion: ClientVersion) -> Channel {
|
||||||
@@ -61,18 +54,7 @@ impl JavtifulProvider {
|
|||||||
premium: false,
|
premium: false,
|
||||||
favicon: "https://www.google.com/s2/favicons?sz=64&domain=javtiful.com".to_string(),
|
favicon: "https://www.google.com/s2/favicons?sz=64&domain=javtiful.com".to_string(),
|
||||||
status: "active".to_string(),
|
status: "active".to_string(),
|
||||||
categories: self
|
categories: vec![],
|
||||||
.categories
|
|
||||||
.read()
|
|
||||||
.map(|categories| categories.iter().map(|c| c.title.clone()).collect())
|
|
||||||
.unwrap_or_else(|e| {
|
|
||||||
crate::providers::report_provider_error_background(
|
|
||||||
"javtiful",
|
|
||||||
"build_channel.categories_read",
|
|
||||||
&e.to_string(),
|
|
||||||
);
|
|
||||||
vec![]
|
|
||||||
}),
|
|
||||||
options: vec![ChannelOption {
|
options: vec![ChannelOption {
|
||||||
id: "sort".to_string(),
|
id: "sort".to_string(),
|
||||||
title: "Sort".to_string(),
|
title: "Sort".to_string(),
|
||||||
@@ -80,35 +62,85 @@ impl JavtifulProvider {
|
|||||||
systemImage: "list.number".to_string(),
|
systemImage: "list.number".to_string(),
|
||||||
colorName: "blue".to_string(),
|
colorName: "blue".to_string(),
|
||||||
options: vec![
|
options: vec![
|
||||||
FilterOption {
|
|
||||||
id: "relevance".into(),
|
|
||||||
title: "Relevance".into(),
|
|
||||||
},
|
|
||||||
FilterOption {
|
FilterOption {
|
||||||
id: "latest".into(),
|
id: "latest".into(),
|
||||||
title: "Latest".into(),
|
title: "Latest".into(),
|
||||||
},
|
},
|
||||||
FilterOption {
|
FilterOption {
|
||||||
id: "popular".into(),
|
id: "popular".into(),
|
||||||
title: "Popular".into(),
|
title: "Popular This Week".into(),
|
||||||
|
},
|
||||||
|
FilterOption {
|
||||||
|
id: "most_viewed".into(),
|
||||||
|
title: "Most Viewed".into(),
|
||||||
|
},
|
||||||
|
FilterOption {
|
||||||
|
id: "most_liked".into(),
|
||||||
|
title: "Most Liked".into(),
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
multiSelect: false,
|
multiSelect: false,
|
||||||
}],
|
}],
|
||||||
nsfw: true,
|
nsfw: true,
|
||||||
cacheDuration: Some(1800),
|
cacheDuration: Some(1800),
|
||||||
ytdlpCommand: Some("yt-dlp --impersonate chrome-120".to_string())
|
ytdlpCommand: Some("yt-dlp --impersonate chrome-120".to_string()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn push_unique(target: &Arc<RwLock<Vec<FilterOption>>>, item: FilterOption) {
|
/// Maps the app's sort ids to the site's `?sort=` values. The default
|
||||||
if let Ok(mut vec) = target.write() {
|
/// listing (no parameter) is newest first.
|
||||||
if !vec.iter().any(|x| x.id == item.id) {
|
fn sort_param(sort: &str) -> &'static str {
|
||||||
vec.push(item);
|
match sort {
|
||||||
}
|
"popular" => "sort=popular_week&",
|
||||||
|
"most_viewed" => "sort=most_viewed&",
|
||||||
|
"most_liked" => "sort=most_liked&",
|
||||||
|
_ => "",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn fetch_listing(
|
||||||
|
&self,
|
||||||
|
cache: VideoCache,
|
||||||
|
video_url: String,
|
||||||
|
page: u8,
|
||||||
|
options: &ServerOptions,
|
||||||
|
) -> Result<Vec<VideoItem>> {
|
||||||
|
let old_items = match cache.get(&video_url) {
|
||||||
|
Some((time, items)) if time.elapsed().unwrap_or_default().as_secs() < 60 * 5 => {
|
||||||
|
return Ok(items.clone());
|
||||||
|
}
|
||||||
|
Some((_, items)) => items.clone(),
|
||||||
|
None => vec![],
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut requester =
|
||||||
|
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
|
||||||
|
let text = match requester.get(&video_url, Some(Version::HTTP_2)).await {
|
||||||
|
Ok(text) => text,
|
||||||
|
Err(e) => {
|
||||||
|
crate::providers::report_provider_error(
|
||||||
|
"javtiful",
|
||||||
|
"fetch_listing.request",
|
||||||
|
&format!("url={video_url}; error={e}"),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
return Ok(old_items);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
// Out-of-range pages are served as page 1 again; only trust the grid
|
||||||
|
// when the pager marks the requested page as current.
|
||||||
|
if page > 1 && !text.contains(&format!("aria-current=\"page\">{}<", page)) {
|
||||||
|
return Ok(vec![]);
|
||||||
|
}
|
||||||
|
let video_items = self.get_video_items_from_html(&text);
|
||||||
|
if video_items.is_empty() {
|
||||||
|
return Ok(old_items);
|
||||||
|
}
|
||||||
|
cache.remove(&video_url);
|
||||||
|
cache.insert(video_url, video_items.clone());
|
||||||
|
Ok(video_items)
|
||||||
|
}
|
||||||
|
|
||||||
async fn get(
|
async fn get(
|
||||||
&self,
|
&self,
|
||||||
cache: VideoCache,
|
cache: VideoCache,
|
||||||
@@ -116,58 +148,13 @@ impl JavtifulProvider {
|
|||||||
sort: &str,
|
sort: &str,
|
||||||
options: ServerOptions,
|
options: ServerOptions,
|
||||||
) -> Result<Vec<VideoItem>> {
|
) -> Result<Vec<VideoItem>> {
|
||||||
let sort_string = match sort {
|
let video_url = format!(
|
||||||
"latest" => "sort=latest&",
|
"{}/videos?{}page={}",
|
||||||
"popular" => "sort=popular&",
|
self.url,
|
||||||
_ => "",
|
Self::sort_param(sort),
|
||||||
};
|
page
|
||||||
let video_url = format!("{}/videos?{}page={}", self.url, sort_string, page);
|
);
|
||||||
println!("Fetching URL: {}", video_url);
|
self.fetch_listing(cache, video_url, page, &options).await
|
||||||
let old_items = match cache.get(&video_url) {
|
|
||||||
Some((time, items)) => {
|
|
||||||
if time.elapsed().unwrap_or_default().as_secs() < 60 * 5 {
|
|
||||||
return Ok(items.clone());
|
|
||||||
} else {
|
|
||||||
items.clone()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None => {
|
|
||||||
vec![]
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let mut requester =
|
|
||||||
crate::providers::requester_or_default(&options, module_path!(), "missing_requester");
|
|
||||||
let text = match requester.get(&video_url, Some(Version::HTTP_2)).await {
|
|
||||||
Ok(text) => text,
|
|
||||||
Err(e) => {
|
|
||||||
crate::providers::report_provider_error(
|
|
||||||
"javtiful",
|
|
||||||
"get.request",
|
|
||||||
&format!("url={video_url}; error={e}"),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
return Ok(old_items);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
if page > 1
|
|
||||||
&& !text.contains(&format!(
|
|
||||||
"aria-current=\"page\">{}</a>",
|
|
||||||
page
|
|
||||||
))
|
|
||||||
{
|
|
||||||
return Ok(vec![]);
|
|
||||||
}
|
|
||||||
let video_items: Vec<VideoItem> = self
|
|
||||||
.get_video_items_from_html(text.clone(), &mut requester, &options)
|
|
||||||
.await;
|
|
||||||
if !video_items.is_empty() {
|
|
||||||
cache.remove(&video_url);
|
|
||||||
cache.insert(video_url.clone(), video_items.clone());
|
|
||||||
} else {
|
|
||||||
return Ok(old_items);
|
|
||||||
}
|
|
||||||
Ok(video_items)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn query(
|
async fn query(
|
||||||
@@ -177,271 +164,83 @@ impl JavtifulProvider {
|
|||||||
query: &str,
|
query: &str,
|
||||||
options: ServerOptions,
|
options: ServerOptions,
|
||||||
) -> Result<Vec<VideoItem>> {
|
) -> Result<Vec<VideoItem>> {
|
||||||
let sort_string = match options.sort.as_deref().unwrap_or("") {
|
let sort = options.sort.clone().unwrap_or_default();
|
||||||
"latest" => "sort=latest&",
|
|
||||||
"popular" => "sort=popular&",
|
|
||||||
_ => "",
|
|
||||||
};
|
|
||||||
let video_url = format!(
|
let video_url = format!(
|
||||||
"{}/search?{}q={}&page={}",
|
"{}/search?{}q={}&page={}",
|
||||||
self.url,
|
self.url,
|
||||||
sort_string,
|
Self::sort_param(&sort),
|
||||||
query.replace(" ", "+"),
|
query.trim().replace(' ', "+"),
|
||||||
page
|
page
|
||||||
);
|
);
|
||||||
// Check our Video Cache. If the result is younger than 1 hour, we return it.
|
self.fetch_listing(cache, video_url, page, &options).await
|
||||||
let old_items = match cache.get(&video_url) {
|
|
||||||
Some((time, items)) => {
|
|
||||||
if time.elapsed().unwrap_or_default().as_secs() < 60 * 5 {
|
|
||||||
return Ok(items.clone());
|
|
||||||
} else {
|
|
||||||
let _ = cache.check().await;
|
|
||||||
return Ok(items.clone());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None => {
|
|
||||||
vec![]
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let mut requester =
|
|
||||||
crate::providers::requester_or_default(&options, module_path!(), "missing_requester");
|
|
||||||
let text = match requester.get(&video_url, Some(Version::HTTP_2)).await {
|
|
||||||
Ok(text) => text,
|
|
||||||
Err(e) => {
|
|
||||||
crate::providers::report_provider_error(
|
|
||||||
"javtiful",
|
|
||||||
"query.request",
|
|
||||||
&format!("url={video_url}; error={e}"),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
return Ok(old_items);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
if page > 1
|
|
||||||
&& !text.contains(&format!(
|
|
||||||
"aria-current=\"page\">{}</a>",
|
|
||||||
page
|
|
||||||
))
|
|
||||||
{
|
|
||||||
return Ok(vec![]);
|
|
||||||
}
|
|
||||||
let video_items: Vec<VideoItem> = self
|
|
||||||
.get_video_items_from_html(text.clone(), &mut requester, &options)
|
|
||||||
.await;
|
|
||||||
if !video_items.is_empty() {
|
|
||||||
cache.remove(&video_url);
|
|
||||||
cache.insert(video_url.clone(), video_items.clone());
|
|
||||||
} else {
|
|
||||||
return Ok(old_items);
|
|
||||||
}
|
|
||||||
Ok(video_items)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn get_video_items_from_html(
|
fn between<'a>(text: &'a str, start: &str, end: &str) -> Option<&'a str> {
|
||||||
&self,
|
let from = text.find(start)? + start.len();
|
||||||
html: String,
|
let rest = &text[from..];
|
||||||
requester: &mut Requester,
|
Some(&rest[..rest.find(end)?])
|
||||||
options: &ServerOptions,
|
}
|
||||||
) -> Vec<VideoItem> {
|
|
||||||
if html.is_empty() || html.contains("404 Not Found") {
|
|
||||||
return vec![];
|
|
||||||
}
|
|
||||||
|
|
||||||
let block = match html
|
fn get_video_items_from_html(&self, html: &str) -> Vec<VideoItem> {
|
||||||
.split("front-pagination")
|
// Cards are `<article class="video-card">`; sponsored slots add a
|
||||||
.next()
|
// `video-card--partner` modifier and link off-site, so skip them.
|
||||||
.and_then(|s| s.split("front-video-grid").nth(1))
|
html.split("<article class=\"video-card")
|
||||||
{
|
|
||||||
Some(b) => b,
|
|
||||||
None => {
|
|
||||||
eprint!("Javtiful Provider: Failed to get block from html");
|
|
||||||
let e = Error::from(ErrorKind::Parse("html".into()));
|
|
||||||
send_discord_error_report(
|
|
||||||
e.to_string(),
|
|
||||||
Some(format_error_chain(&e)),
|
|
||||||
Some("Javtiful Provider"),
|
|
||||||
Some(&format!("Failed to get block from html:\n```{html}\n```")),
|
|
||||||
file!(),
|
|
||||||
line!(),
|
|
||||||
module_path!(),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
return vec![];
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let futures = block
|
|
||||||
.split("\"front-video-card\"")
|
|
||||||
.skip(1)
|
.skip(1)
|
||||||
.filter(|seg| !seg.contains("front-ad-card"))
|
.filter(|seg| !seg.starts_with("video-card--partner") && !seg.starts_with(" video-card--partner"))
|
||||||
.map(|el| self.get_video_item(el.to_string(), requester.clone(), options));
|
.filter_map(|seg| self.get_video_item(seg))
|
||||||
|
|
||||||
join_all(futures)
|
|
||||||
.await
|
|
||||||
.into_iter()
|
|
||||||
.inspect(|r| {
|
|
||||||
if let Err(e) = r {
|
|
||||||
eprint!("Javtiful Provider: Failed to get video item:{}\n", e);
|
|
||||||
// Prepare data to move into the background task
|
|
||||||
let msg = e.to_string();
|
|
||||||
let chain = format_error_chain(&e);
|
|
||||||
|
|
||||||
// Spawn the report into the background - NO .await here
|
|
||||||
tokio::spawn(async move {
|
|
||||||
let _ = send_discord_error_report(
|
|
||||||
msg,
|
|
||||||
Some(chain),
|
|
||||||
Some("Javtiful Provider"),
|
|
||||||
Some("Failed to get video item"),
|
|
||||||
file!(), // Note: these might report the utility line
|
|
||||||
line!(), // better to hardcode or pass from outside
|
|
||||||
module_path!(),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.filter_map(Result::ok)
|
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn get_video_item(
|
fn get_video_item(&self, seg: &str) -> Option<VideoItem> {
|
||||||
&self,
|
let href = Self::between(seg, "href=\"", "\"")?;
|
||||||
seg: String,
|
if !href.starts_with("/video/") {
|
||||||
mut requester: Requester,
|
return None;
|
||||||
options: &ServerOptions,
|
}
|
||||||
) -> Result<VideoItem> {
|
let video_url = format!("{}{}", self.url, href);
|
||||||
let video_url = format!(
|
let id = href.trim_start_matches("/video/").split('/').next()?.to_string();
|
||||||
"{}{}",
|
if id.is_empty() {
|
||||||
self.url,
|
return None;
|
||||||
seg.split(" href=\"")
|
}
|
||||||
.nth(1)
|
|
||||||
.and_then(|s| s.split('"').next())
|
|
||||||
.ok_or_else(|| ErrorKind::Parse(format!("video url\n\n{seg}")))?
|
|
||||||
.to_string()
|
|
||||||
);
|
|
||||||
|
|
||||||
let mut title = match seg.contains("front-video-title") {
|
let title_raw = Self::between(seg, "class=\"video-card__title\" title=\"", "\"")
|
||||||
true => seg
|
.or_else(|| Self::between(seg, "alt=\"Thumbnail for ", "\""))?
|
||||||
.split("front-video-title")
|
.trim();
|
||||||
.nth(1)
|
let title = decode(title_raw.as_bytes())
|
||||||
.and_then(|s| s.split('>').nth(1))
|
|
||||||
.and_then(|s| s.split('<').next())
|
|
||||||
.ok_or_else(|| ErrorKind::Parse(format!("video title\n\n{seg}")))?
|
|
||||||
.trim()
|
|
||||||
.to_string(),
|
|
||||||
false => seg
|
|
||||||
.split("alt=\"")
|
|
||||||
.nth(1)
|
|
||||||
.and_then(|s| s.split('\"').next())
|
|
||||||
.ok_or_else(|| ErrorKind::Parse(format!("video title\n\n{seg}")))?
|
|
||||||
.trim()
|
|
||||||
.to_string(),
|
|
||||||
};
|
|
||||||
|
|
||||||
title = decode(title.as_bytes())
|
|
||||||
.to_string()
|
.to_string()
|
||||||
.unwrap_or(title)
|
.unwrap_or_else(|_| title_raw.to_string());
|
||||||
.titlecase();
|
|
||||||
let id = video_url
|
|
||||||
.split("/video/")
|
|
||||||
.filter(|s| !s.is_empty())
|
|
||||||
.nth(1)
|
|
||||||
.ok_or_else(|| ErrorKind::Parse(format!("video id\n\n{seg}")))?
|
|
||||||
.to_string();
|
|
||||||
let thumb_block = seg.split("<img ").nth(1);
|
|
||||||
|
|
||||||
let thumb = match thumb_block {
|
let thumb = Self::between(seg, "data-front-lazy-src=\"", "\"")
|
||||||
Some(block) => format!("{}{}", self.url,block
|
.map(|t| {
|
||||||
.split("data-front-lazy-src=\"")
|
if t.starts_with("http") {
|
||||||
.nth(1)
|
t.to_string()
|
||||||
.and_then(|s| s.split('"').next())
|
} else {
|
||||||
.unwrap_or("")
|
format!("{}{}", self.url, t)
|
||||||
.to_string()),
|
}
|
||||||
None => "".to_string(),
|
|
||||||
};
|
|
||||||
let mut preview = seg
|
|
||||||
.split("data-front-video-preview-src=\"")
|
|
||||||
.nth(1)
|
|
||||||
.and_then(|s| s.split('"').next())
|
|
||||||
.unwrap_or("")
|
|
||||||
.to_string();
|
|
||||||
let raw_duration = seg
|
|
||||||
.split("class=\"front-duration-tag\">")
|
|
||||||
.nth(1)
|
|
||||||
.and_then(|s| s.split('<').next())
|
|
||||||
.unwrap_or("")
|
|
||||||
.to_string();
|
|
||||||
let duration = parse_time_to_seconds(&raw_duration).unwrap_or(0) as u32;
|
|
||||||
let (tags, views) = self.extract_media(&video_url, &mut requester).await?;
|
|
||||||
|
|
||||||
if preview.len() == 0 {
|
|
||||||
preview = format!("https://trailers.jav.si/preview/{id}.mp4");
|
|
||||||
}
|
|
||||||
let proxy_url = build_proxy_url(
|
|
||||||
options,
|
|
||||||
"javtiful",
|
|
||||||
&strip_url_scheme(video_url.clone().as_str()),
|
|
||||||
);
|
|
||||||
let video_item = VideoItem::new(id, title, proxy_url, "javtiful".into(), thumb, duration)
|
|
||||||
.tags(tags)
|
|
||||||
.preview(preview)
|
|
||||||
.views(views);
|
|
||||||
Ok(video_item)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn extract_media(
|
|
||||||
&self,
|
|
||||||
url: &str,
|
|
||||||
requester: &mut Requester,
|
|
||||||
) -> Result<(Vec<String>, u32)> {
|
|
||||||
let text = requester
|
|
||||||
.get(url, Some(Version::HTTP_2))
|
|
||||||
.await
|
|
||||||
.map_err(|e| Error::from(format!("{}", e)))?;
|
|
||||||
let tags = text
|
|
||||||
.split("related-actress")
|
|
||||||
.next()
|
|
||||||
.and_then(|s| s.split("video-comments").next())
|
|
||||||
.and_then(|s| s.split(">Tags").nth(1))
|
|
||||||
.and_then(|s| s.split("</div>").next())
|
|
||||||
.map(|tag_block| {
|
|
||||||
tag_block
|
|
||||||
.split("<a ")
|
|
||||||
.skip(1)
|
|
||||||
.filter_map(|tag_el| {
|
|
||||||
tag_el
|
|
||||||
.split('>')
|
|
||||||
.nth(1)
|
|
||||||
.and_then(|s| s.split('<').next())
|
|
||||||
.map(|s| {
|
|
||||||
decode(s.as_bytes())
|
|
||||||
.to_string()
|
|
||||||
.unwrap_or(s.to_string())
|
|
||||||
.titlecase()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
.collect()
|
|
||||||
})
|
})
|
||||||
.unwrap_or_else(|| vec![]);
|
.unwrap_or_default();
|
||||||
for tag in &tags {
|
let preview = Self::between(seg, "data-front-video-preview-src=\"", "\"")
|
||||||
Self::push_unique(
|
.unwrap_or("")
|
||||||
&self.categories,
|
.to_string();
|
||||||
FilterOption {
|
|
||||||
id: tag.to_ascii_lowercase().replace(" ", "+"),
|
// `<span class="video-card__details"><strong>FHD</strong>02:49:46</span>`
|
||||||
title: tag.to_string(),
|
let duration = Self::between(seg, "class=\"video-card__details\">", "</span>")
|
||||||
},
|
.and_then(|d| d.rsplit("</strong>").next())
|
||||||
);
|
.and_then(|d| parse_time_to_seconds(d.trim()))
|
||||||
}
|
.unwrap_or(0) as u32;
|
||||||
let views = text
|
|
||||||
.split(" views</span>")
|
// `<div class="video-card__meta">12.9K views <span ..>·</span> 6 hours ago</div>`
|
||||||
.next()
|
let views = Self::between(seg, "class=\"video-card__meta\">", "views")
|
||||||
.and_then(|s| s.split('>').last())
|
.and_then(|v| parse_abbreviated_number(&v.trim().replace(',', "")))
|
||||||
.and_then(|s| s.trim().replace(".", "").replace(",", "").parse::<u32>().ok())
|
|
||||||
.unwrap_or(0);
|
.unwrap_or(0);
|
||||||
|
|
||||||
Ok((tags, views))
|
// The page URL resolves natively in yt-dlp (html5 `<source>`), so no
|
||||||
|
// proxy and no formats are needed.
|
||||||
|
let mut item = VideoItem::new(id, title, video_url, "javtiful".into(), thumb, duration)
|
||||||
|
.views(views);
|
||||||
|
if !preview.is_empty() {
|
||||||
|
item = item.preview(preview);
|
||||||
|
}
|
||||||
|
Some(item)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -474,3 +273,32 @@ impl Provider for JavtifulProvider {
|
|||||||
Some(self.build_channel(v))
|
Some(self.build_channel(v))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::JavtifulProvider;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parses_cards_and_skips_partner_slots() {
|
||||||
|
let html = r#"<div class="video-grid">
|
||||||
|
<article class="video-card"><a href="/video/114451/royd-358" class="video-card__thumbnail" data-front-video-preview-src="https://x.jav.si/p.mp4"><img src="/ph.svg" data-front-lazy-src="/uploads/t.jpg" alt="Thumbnail for ROYD-358 T"><span class="video-card__details"><strong>FHD</strong>02:49:46</span></a>
|
||||||
|
<div class="video-card__copy"><a href="/video/114451/royd-358" class="video-card__title" title="ROYD-358 I’ll & You">ROYD-358</a><div class="video-card__meta">12.9K views <span aria-hidden="true">·</span> 6 hours ago</div></div></article>
|
||||||
|
<article class="video-card video-card--partner"><a href="https://ads.example/c" class="video-card__thumbnail"><img data-front-lazy-src="/a.jpg" alt="Partner thumbnail for Ad"></a></article>
|
||||||
|
</div>"#;
|
||||||
|
let items = JavtifulProvider::new().get_video_items_from_html(html);
|
||||||
|
assert_eq!(items.len(), 1);
|
||||||
|
assert_eq!(items[0].id, "114451");
|
||||||
|
assert_eq!(items[0].url, "https://javtiful.com/video/114451/royd-358");
|
||||||
|
assert_eq!(items[0].thumb, "https://javtiful.com/uploads/t.jpg");
|
||||||
|
assert_eq!(items[0].duration, 2 * 3600 + 49 * 60 + 46);
|
||||||
|
assert_eq!(items[0].views, Some(12900));
|
||||||
|
assert!(items[0].formats.is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn maps_sort_values() {
|
||||||
|
assert_eq!(JavtifulProvider::sort_param("latest"), "");
|
||||||
|
assert_eq!(JavtifulProvider::sort_param("popular"), "sort=popular_week&");
|
||||||
|
assert_eq!(JavtifulProvider::sort_param("most_viewed"), "sort=most_viewed&");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user