Commit Graph

678 Commits

Author SHA1 Message Date
Simon
63c34179ff rule34video: stream playback through the server; signed links are IP-bound
The v-acctoken signature covers the IP that loaded the video page. Since
enrichment goes through pool proxies, every served link 403'd for clients.
Served formats now point at /proxy/rule34video/{id}/{slug}/{height}.mp4,
which reloads the page from the server's IP (1h link cache) and streams
the mp4 with range support. The pool still supplies the metadata.

The streaming response and link cache moved from the eporner proxy into
util/media_stream.rs, shared by both. USE_MEDIA_PROXY and the redirect
handler are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEp724zDpCWNaSeCrMSvjx
2026-11-06 10:43:02 +00:00
Simon
29d526b5fc proxy pool: spread rule34video enrichment over checked public proxies
Rework the unused ALL_PROXIES list into util/proxy_pool.rs:
- hourly sweep of the public lists, TLS-verified cloudflare trace checks,
  >=2 successes with a median under 1.5s, one proxy per exit IP; proxies
  that show our own IP are dropped, two failures in a row evict
- per-site sets checked against a site URL (rule34video: robots.txt),
  round-robin, 10-min per-site cooldown on 429/challenge pages, live
  request outcomes fed back
- started from main (PROXY_POOL=0 disables; PROXY_POOL_LISTS,
  PROXY_POOL_CONCURRENCY), /api/proxies shows its status without credentials
- wreq "socks" feature enabled: SOCKS proxies silently failed before

rule34video enrichment is back, through the pool: full detail pages
(1080p..360p formats, tags, artist), 8 concurrent, one 4s deadline per
listing; late pages keep running and land in the 2h detail cache, which
cached listings are re-merged with on every serve. No ready proxies means
no enrichment (no direct fallback). Requester::get_via sends a pooled
request with certificate checks, sharing the session's cookie jar.

Also adds an ignored live probe showing /embed/ shares the 429 limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEp724zDpCWNaSeCrMSvjx
2026-11-05 21:59:28 +00:00
Simon
4f83dae3fa rule34video: drop per-video enrichment; add on-demand page resolver proxy (off)
Listings no longer fetch every video page for formats/tags/artist, which
ran into the site's 429 limit. /proxy/rule34video/{id}/{slug} resolves the
page at play time and 302s to the signed mp4; the resolver lives in
util/rule34video_resolve.rs. Handing out proxy URLs is gated behind
USE_MEDIA_PROXY, currently false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEp724zDpCWNaSeCrMSvjx
2026-11-05 20:26:50 +00:00
Simon
cf0ae15f93 hanime: play through a redirect proxy; the signed stream links expire
Items now carry /proxy/hanime/{slug}/{height}.m3u8, which does a fresh
handshake per play and 302s to the current playlist. The handshake moved
to util/hanime_resolve.rs so single-provider builds keep the route.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-11-01 07:48:35 +00:00
Simon
aa00abdd5b rule34video: probe the space-less artist slug (e.g. /models/biggreen/) for multi-word queries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-27 21:45:09 +00:00
Simon
e5a2afcc70 eporner: stream through the server; the signed CDN links are IP-locked
The redirect proxy from 6ba65f1 still played the one-minute "watch on
eporner.com" clip away from home: checked from foreign networks
(check-host.net nodes in BR/DE/HU), a freshly signed link answers 302 ->
static.eporner.com/na.mp4, while the IP that resolved it gets the file. A
redirect therefore only works for a player behind the server's public IP.

/proxy/eporner/{alnum_id}/{height}.mp4 now streams the mp4: resolves the
link (cached 90 min, re-resolved when the CDN answers with a redirect),
forwards Range, and returns a sized body so 206 responses keep
Content-Length. It is no longer a redirect proxy, so app clients keep the
URLs in formats. Verified locally: HEAD, 0-1, mid-file and open-ended ranges
return 206 with Content-Length/Content-Range and valid MP4 data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-22 20:46:14 +00:00
Simon
6ba65f1457 eporner: play through a redirect proxy; the signed CDN links expire after 2h
Expired eporner mp4 links are answered with a 302 to static.eporner.com/
na.mp4, a one-minute "watch it on eporner.com" clip. The links expire
exactly two hours after the /xhr/video call, so anything played from a
favourite, history entry or a list left open past that hit the decoy.

New /proxy/eporner/{alnum_id}/{height} re-resolves the hash/xhr chain at
play time and 302s to the matching (else best) quality. The eporner
provider and melonstube's eporner destinations now emit these proxy URLs;
verified app clients get the best quality as video.url via the existing
redirect-proxy rule. Verified live: proxy -> fresh CDN link -> 206 video/mp4
for app and non-app clients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-22 18:09:23 +00:00
Simon
b131910ea2 rule34video: make queries robust (single-page results, safe search paths, 429-proof enrichment)
- Parser no longer requires a pagination block: single-page results
  (small tags, short searches) used to parse to zero items.
- Search path is slugged like the site's own form: "2b/9s" 404'd, "%20"
  is refused with 403 and "&" truncated the query.
- Probed archives need >= 4 videos to beat keyword search ("elf girl" was a
  one-video tag); conclusive query -> target resolutions are remembered for 6h
  so later pages and the prefetch skip the probes.
- Enriched details are cached per video URL for 2h (tokens last ~12h);
  detail fetches go through get_probe so a lasting 429 stops the batch instead
  of escalating each page to Jina/FlareSolverr. Archive failures serve stale
  cache.
- requester: get_probe waits before retrying 429/5xx and compares redirect
  paths percent-decoded (non-ASCII archives like /tags/pokémon-species/ were
  treated as redirected away); get_with_headers backs off 1/2/4s on 429.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-21 20:17:58 +00:00
Simon
921ad93149 requester: treat DDoS-Guard interstitials as challenges (FlareSolverr fallback); mock-origin test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 20:38:17 +00:00
Simon
ad2abe0057 rule34video: artist, category and tag archives; resolve bare queries by probing; credit artists as uploader
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 17:06:00 +00:00
Simon
0aa8929f29 Docs: the redirect-proxy rule applies to verified Hot Tub app clients only
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:53:08 +00:00
Simon
6c69b46395 Move redirecting-proxy formats to video.url for verified app clients; document the rule
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:48:34 +00:00
Simon
f900987434 archivebate: verified app clients get the proxy URL as item url and no formats
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:39:48 +00:00
Simon
30271b9b7c archivebate: app clients get resolved media URLs with headers, resolved concurrently
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-18 19:30:55 +00:00
Simon
a4364e5be3 archivebate: keep the proxied format when enrichment times out; skip dead search candidates
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 19:54:49 +00:00
Simon
edf6732f8e Query resolution todo, slug helper tests, fix txxx decoder test fixture
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 06:19:18 +00:00
Simon
697836b914 coomer: creator feeds are a bare array; single-word queries that are creator ids go to the creator 2026-10-17 06:16:24 +00:00
Simon
5598216381 sextb: probe actress/studio for names outside the loaded catalogs 2026-10-17 06:07:34 +00:00
Simon
4c29d27b3e redgifs: single-word queries that are creator names go to the creator 2026-10-17 06:05:54 +00:00
Simon
a486bc5502 erome: single-word queries that are uploader names go to the uploader 2026-10-17 06:03:58 +00:00
Simon
270b327f73 fikfap: single-word queries that are profile names go to the creator 2026-10-17 06:01:34 +00:00
Simon
813e2ec2ca clapdat: resolve tag/user names without prefix 2026-10-17 05:58:53 +00:00
Simon
26ef830f6e pornhub: probe pornstar/model/channel for unknown free-form queries 2026-10-17 05:56:31 +00:00
Simon
e34801f6f8 thepornbunny: resolve pornstar/studio/tag names; get_probe treats redirect-away as a miss 2026-10-17 05:53:43 +00:00
Simon
2646ca5659 xgroovy: resolve pornstar/channel names without prefix 2026-10-17 05:51:40 +00:00
Simon
981ec29ad8 xhamster: resolve pornstar/channel names without prefix 2026-10-17 05:50:41 +00:00
Simon
76b9d4e058 xvideos: resolve channel/pornstar names without prefix; channel archives via the JSON video list 2026-10-17 05:48:58 +00:00
Simon
59ea0495a6 porn4fans: resolve model/tag names without prefix 2026-10-17 05:40:28 +00:00
Simon
1f95549167 youporn: resolve pornstar/channel/amateur names without prefix 2026-10-17 05:38:47 +00:00
Simon
1e56fe2da5 tube8: resolve pornstar/channel names without prefix 2026-10-17 05:37:07 +00:00
Simon
bdde02fa8c Add get_probe/probe_first_archive; fullporner and fullpornerorg resolve pornstar names without prefix 2026-10-17 05:34:15 +00:00
Simon
665fb2a0ce hooporn: probe queries as pornstar/studio archives before keyword search
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 05:25:30 +00:00
Simon
99bc80adac Add hooporn provider
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 05:11:17 +00:00
Simon
d72dda5203 pornhub: proxy Referer-gated pix-cdn77 thumbnails
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 21:08:07 +00:00
Simon
6d7b16842d Add txxx provider; cumlouder: route keywords to pornstar /girl/ pages before tags
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 15:47:05 +00:00
Simon
af6674b4ed Route Cloudflare challenges to FlareSolverr on cumlouder and hypnotube raw paths
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 11:59:42 +00:00
Simon
2addcc2e7c Deliver ytdlpCommand for an allowlist of verified channels
decorate_channel ignored every provider-declared ytdlpCommand except
pimpbunny's. Send the declared command for freepornvideosxxx and missav,
where plain yt-dlp is verified to fail; other channels stay unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 10:28:57 +00:00
Simon
40eab41b2b Fix hanime-cdn relative path prefixing and stale vidara test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 09:36:13 +00:00
Simon
5a0fb649ee todo: aps and porndish are Cloudflare/FlareSolverr bound 2026-10-10 09:34:05 +00:00
Simon
2d35f29f44 Fix xxxtik empty results; replay solved identity on raw requests
Raw requester calls did not adopt the remembered FlareSolverr user agent,
so a RedGifs token minted under it was rejected on every gif lookup.
Tag/creator pages that mix non-RedGifs sources are refilled from later
cursor batches instead of coming back empty.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 09:33:36 +00:00
Simon
5176178607 vjav: fall back to keyword search when an implicit uploader/tag match is empty
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 09:24:56 +00:00
Simon
0824941c23 Fix chaturbate search URL (swapped format args) and stale cache branch
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 09:21:02 +00:00
Simon
2fccea0fc2 Fix noodlemagazine thumb proxy; check.py verifies proxied urls
The thumb proxy dropped the preview query string, rejected extensionless
getVideoPreview paths and warmed up through the slow fallback chain on
every image. check.py now verifies that a proxied video.url resolves
instead of failing it outright.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 09:17:30 +00:00
Simon
e4c5cb2f0a todo: mark xxdbx and cumlouder fixed 2026-10-10 09:10:05 +00:00
Simon
ae516aba91 Fix xxdbx site-relative thumbs; cumlouder answers 404 immediately
xxdbx assets are now /id.jpg, not //host/..; prefixing https: produced
https:/id.jpg. cumlouder's unknown-tag 404 went through the requester's
mirror/FlareSolverr fallbacks (146s); read the raw response and return
an empty list at once while still retrying 5xx.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 09:09:58 +00:00
Simon
95700c67a5 check.py: apply the channel's status ytdlpCommand to yt-dlp probes; log open decision
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 08:36:25 +00:00
Simon
765fcb5579 Fix fullpornerorg: drop deleted hoster files, keep myvidplay host
Deleted lulustream files stayed listed and 502'd. myvidplay embeds lost
their host and were resolved against lulustream; route them through the
doodstream proxy (now allowing myvidplay/playmogo) with the host kept.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 08:33:18 +00:00
Simon
a1e6021766 Fix hypnotube search sessions; check.py confirms HEAD failures with GET
The site serves one results page per searchgate POST, so later searches
and page 2 returned nothing on the shared cookie session. Use an isolated
session per (query, page). check.py now retries a refused HEAD (403/404/
405/501) with a ranged GET before reporting a URL as unreachable.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 08:24:15 +00:00
Simon
78acac2150 Fix tube8 ids and Referer-gated Aylo thumbnails
Sponsored channel cards carry a stale data-video-id that 404s; use the
card link target. The signed pix-cdn77 thumbnail hosts 403 without a
Referer, so route them (tube8, ph-shorties) through a new host-restricted
aylo-thumb proxy. Also correct the pornhub-thumb route that sat under a
doubled /proxy prefix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 08:16:03 +00:00
Simon
a20a4ab642 Fix coomer: offset pagination, per-query cache key, video-only posts
The API ignores page= (every page 2+ repeated page 1), searches shared a
single cache entry, image-only posts were listed as unplayable videos, and
gzip/br bodies were not decoded because wreq had no decompression features.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-10 08:04:00 +00:00