Add ripnsfw provider (Doodstream/Lulustream leak aggregator)
ripnsfw.com serves its entire catalogue as a published-Google-Sheet CSV with no native pagination/search API, so the provider fetches and parses that CSV once (cached 180s) and does feed/search/pagination/sort in memory. Each row's Doodstream/Lulustream embed links resolve to formats[] via the existing (previously unused) doodstream/lulustream redirect proxies. Also fixes check.py's follow_proxy_redirect, which used HEAD even though these redirect-proxy routes only accept GET/POST, so it never actually resolved the redirect; extends the CF-protected host list (suffix matching + ripnsfw.com's client-only-SPA 404 page, dood.video, tnmr.org) so known sandbox/CDN-IP-reputation failures are reported as warnings instead of errors. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QTqf6orbHZ9rFFpVpcgzcR
This commit is contained in:
@@ -57,6 +57,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
|
||||
| `porntrex` | `mainstream-tube` | no | no | KVS-style HTML archive with direct MP4 formats and tag-aware search shortcuts. |
|
||||
| `redgifs` | `amateur-homemade` | yes | no | Direct integration against the public RedGifs v2 API (same backend `xxxtik` proxies through, but consumed natively here). Auth: anonymous bearer token from `GET api.redgifs.com/v2/auth/temporary`, cached in an `Arc<RwLock<Option<String>>>` with double-checked locking, refreshed once on a 401 via a centralized `authed_get` helper. Default feed: `GET /v2/feeds/trending/{popular,established}?page=&count=` (honors `count` exactly). Free-text query and `tag:`/`category:`/`cat:` shortcuts hit `GET /v2/gifs/search?query=|tags=&order=&page=&count=`; `user:`/`uploader:`/`creator:` shortcuts hit `GET /v2/users/{username}/search?page=&count=`. Uploader profile (`GET /v1/users/{username}`) backs `/api/uploaders`, with `profileContent:true` pulling the creator's own gif listing for `videoCount`/`totalViews`. Listing JSON already carries duration/poster/tags/uploader per item, so no per-item enrichment call is needed (`build_video_item` is a plain sync function, unlike `xxxtik`'s `buffer_unordered` resolve step). `video.url` is the yt-dlp-native `redgifs.com/watch/{id}` page; `formats` intentionally left unset. Thumbnails (`urls.poster`/`urls.thumbnail`) are directly hotlinkable, no proxy needed. 20 curated tags exposed via `categories`. **Known upstream quirk:** `/v2/gifs/search` (query/tag targets only — trending and creator listings are unaffected) returns noticeably fewer items than the requested `count` on some pages (e.g. `count=20` → ~14); not compensated for by over-fetching, since inflating `count` while keeping the same `page` would desync the API's own offset-based pagination cursor (`page × count`) and risk skipped/duplicated items across pages. |
|
||||
| `redtube` | `mainstream-tube` | no | no | Mainstream archive. |
|
||||
| `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,<link3>,DOOD,<guests>,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). |
|
||||
| `rule34gen` | `ai` | no | no | AI group example. |
|
||||
| `rule34video` | `hentai-animation` | no | no | Hentai group example. |
|
||||
| `sextb` | `jav` | no | no | JAV family provider. |
|
||||
@@ -98,6 +99,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
|
||||
These resolve a provider-specific input into a `302 Location`.
|
||||
|
||||
- `/proxy/doodstream/{endpoint}*`
|
||||
- `/proxy/lulustream/{endpoint}*`
|
||||
- `/proxy/sxyprn/{endpoint}*`
|
||||
- `/proxy/javtiful/{endpoint}*`
|
||||
- `/proxy/spankbang/{endpoint}*`
|
||||
|
||||
Reference in New Issue
Block a user