WHY JAPANESE PEOPLE
or: everything that went wrong writing a FANZA / R18.dev crawler for XBVR
R18.com is dead, FANZA is geofenced, so R18.dev is what we've got. It mirrors FANZA's data faithfully. That's the problem. Below is every gotcha that came up building a crawler that scrapes JAV VR "properly" (per studio, toggles, rescrapes) instead of pasting IDs into a box one by one.
Numbers come from a survey of an entire VR category on R18.dev: 25,071 content IDs
But that doesn't cover all of the VR scenes either. VR Exclusive? High-Quality VR? Who knows?!
1. There is no such thing as "a studio"
FANZA has three things that all sort of mean "who made this":
- Maker (the company)
- Label (a brand line under a maker... usually)
- DVD prefix (the
VRKMinVRKM-139)
None of them line up with each other. In either direction.
One label, many prefixes
The KMPVR label (25637) carries VRKM, KMVR, EXVR, BMVR, KBVR, QRVR and HOTVR.
One prefix, many labels
84exvr00010 (EXVR-010) is label KMPVR. 84exvr00149 (EXVR-149) is label "Outer Space Variety VR". Same prefix, same maker (K M Produce), different label.
P-BOX VR is a maker AND a label
Label P-BOX VR (46146) shows up under maker P-BOX VR (40534). It also shows up under maker K M Produce. Same label. Two makers. Why? Fuck you, that's why.
KMPVR-Color- and KMPVR-bibi- are also both a maker and a label
And they don't agree with themselves:
| Label | Maker | |
|---|---|---|
| KMPVR-Color- (SAVR) | 25997: 1,273 scenes | 46628: 1,288 scenes |
| KMPVR-bibi- (BIBIVR) | 26288: 621 scenes | 40404: 637 scenes |
The maker versions drag in extra stuff: Dokuji Pictures, AVOPVR ("VR-1 Grand Prix"), MTBVR, KMTV, Tenjiku VR (TJVR). The only way to know what's actually in there is to diff 13 pages of each.
KBVR: the Neo Armstrong Cyclone Jet Armstrong Cannon of DVD prefixes
One prefix, bolted together out of however many labels it takes:
- REAL VR NEO
- KMPVR
- KMPVR-Color-
- more? Probably. Nobody has found the end of it yet.
Plus two different content ID maker codes, 84kbvr and h_1241kbvr, and all of it is KMP anyway. So "scrape the KBVR label" isn't a thing. There is no KBVR label. There's a KBVR-shaped thing that fires out of three or more labels at once.
Group labels that span makers
- 極グループ (2066129): Koala VR, VR Paradise (VRPRD) and Cute Meat Paradise VR (NKKVR) all under one label. So you can't use the label, you have to use the three makers.
- HHH-VR (HHH Group): Hunter, Oyashoku, Royal, Jackpot System.
- WAAP Group VR: Waap and Dream Ticket.
- Marrion Group VR: makers "Unfinished" (45867) and "Unfinished-VR" (6707). Yes, two makers.
- アウトビジョンVR (OVVR): shows up under S1, MOODYZ, Idea Pocket, PREMIUM, MUTEKI, Staring Crew and AQUA. Any time you scrape one of those by maker you have to subtract it by hand. Because they're weird physical releases of VR?
Makers that aren't VR
You can't just scrape "K M Produce" or "SOD Create" by maker, you get their whole 2D catalog.
MAX-A's maker is 7,946 scenes and zero VR on the sampled pages; the VR lives on three separate labels.
FALENO VR shares its label with DAHLIA, and FALENO's own big label is 4,037 scenes with zero VR.
FS.Knights' "Furasupi" label: 145 scenes, 25 of them VR. So you HAVE to fetch every scene's detail page and check for category 6793 (VR Exclusive), and remember the rejects, or you refetch 120 non-VR pages every single run.
2. R18.dev itself as an endpoint, and FANZA's source data
- List pages: 100 per page, newest first. Page 1 of the VR category is ~6 weeks in the future.
- The category list doesn't tell you the label. The label list doesn't tell you the category. Pick your poison, then fetch details anyway.
release_datecan benull. Those sort to the very end of the list. Don't treat null as "future", treat it as "old".label_idcan be null entirely (MILU VR,h_1145mivr00081). That scene is in no label list at all.label_name_encan be null with only the Japanese name set (DOC ERO VR).label_name_encan contain a newline."Koala VR\nThe Ultimate". Real data.- Category 307935 (8KVR) has
category_name_en: nullon the list endpoint and"8KVR"on the detail endpoint. Match on IDs, never on names. - Rate limit: a flat 1 request per second gets you HTTP 429 after about 50 fetches. Looks like ~20 requests per 30 seconds, not documented anywhere.
3. Content IDs are a crime scene
The content ID (84vrkm00139) is FANZA's real key. It looks parseable. It isn't.
- Maker number prefix, sometimes:
84exvr00010andexvr00439,84vrkm00139andvrkm00352. Same label, some have the84, some don't. - Content IDs lose their maker prefix over time:
h_1155crvrcrvr,h_1248kiwvrkiwvr. - SOD:
13dsvris maker code1+ series3DSVR. Strip "leading digits" like you do for everyone else and you getDSVR, which is a different studio (Das!). - VR総研:
wvr90012is WVR9-012.wvr100001is WVR1-001.wvr9c00001is WVR9C-001.wvr00006d086is WVR6D-086. The section number is hiding in the zeros. Only for maker codesh_1337/h_1390, because other makers use plainWVR-001(5539wvr,h_1239wvr). - Too many digits:
rvr000001,84kmvr000165. And84kmvr00165ALSO exists and is the same scene. So is13dsvr02001vs13dsvr002001(3DSVR-2001). - Reissues:
hnvr00144re01,84kmvr00799re01. Identical metadata. The original can be gone (84kmvr00799is a 404, onlyre01exists). A carelessre\d+$regex also eats5725ore00001andkore00004, which are not reissues, they're just prefixes that end in "re". - Letter suffixes:
oycvr00025athroughdare parts with no base release.urvrsp00494ais a FANZA special edition (88 min vs 76 forurvrsp00494, same date, same everything). Then there's plain junk likeajvrbx00004mtn,5648burlesquevrs00001d,h_1171r00489vr. - Content ID says one thing, DVD ID says another:
h_1241dpvr00050is DPKM-001.h_1068dftvr00001is FTVR-001 (the other ten areh_1068ftvr).
4. "DVD ID" is a fan convention, not a fact
- 11,314 of the 25,071 VR scenes have no dvd_id at all. Post-R18.com digital-only releases never had a "DVD"
- The ones that do have one are a mess:
- lowercase:
ftvr-007,tami0003,3dsvr-0140 - no dash:
CAFR009 - letters after the dash:
WVR6-D071,YP-Y003,Booyah-VR0002 - 5 digits:
MIVR-00077. 2 digits:ZVR-01.
- lowercase:
- 4 digits: SOD prints
3DSVR-0137on the actual cover, so keep it. Every other 4 digit prefix (AVVR, SODVR, VRM, KPVR...) is just padding and never went past 999.
Same DVD ID, different scene
| DVD ID | One | The other |
|---|---|---|
| FTVR-0NN | FuuTube, 2017-2019 | Fantasy Star, 2021-2023 |
| VVVR-00N | KMP ViVidVR, 2017-2018 | some other maker, 2020 |
| YPP-00N | VR総研 (dvd says YP-P00N), 2020 |
Yellow Pinkman, 2021 |
| WVR-00N | 5539wvr, 2017 |
h_1239wvr, 2017-2018 |
| DPVR-001, 002 | KMP Dokuji, 2017 | Global Media, 2017 |
| OPG-001 | h_1157opg, 2016 |
h_1402opg, 2019 |
| PRVR-001 | Peters MAX, 2018 | PREMIUM, 2019 |
And DPVR is still live: KMP DPVR-033 is from 2018, Global Media's "dpvr00033" is from 2025.
Global Media's "3D Mature" line stopped getting dvd_ids after 2017, so it just reuses the prefix.
5. Compilations, best-ofs and imports
- Not every compilation has the Compilation category (6003). There's also 6608 (Actress Best Compilation), 6609 (More Than 16 Hours Of Footage), 7267 (Set Items), and KMP's "VR BEST" series where
vrkm01903is 1,136 minutes long and has none of those categories. - SODSVR is always 3DSVR best-ofs. Tags not guaranteed.
- Compilations get brand new numbers. Some of them are actually 4K60 remasters of older releases, so you can't even just ignore them wholesale.
- Foreign imports (category 4022): SLR Originals, Imagine VR, HAYABUSA etc. are on FANZA as censored copies of Western VR scenes you can get uncensored from the studio directly.
6. Covers
- "NOW PRINTING" placeholder covers come back as HTTP 404 with an actual JPEG body: 2,732 bytes,
last-modified: Tue, 18 Dec 2007. Your browser shows a picture. Your code sees a 404. - Some preorders already have the real cover weeks early. Some old scenes have a permanent 404 because they got delisted. So "404 = skip" is wrong too; only gate recent / upcoming scenes.
7. Actress names
- Actresses who never got an official English name come back as one word, family name first, machine-romanized: 藍瀬ミナ
Aisemina(Mina Aise). 宇佐美すいUsamisui(Sui Usami). - Titles have a "this is machine translated" flag. Actress names don't.
- The fix: the actress image filename is
aise_mina.jpg, so you can split it. Except 一条みお's image isitizyou_mio.jpg(Kunrei romanization) while her name isIchijou(Hepburn). Long vowels. Placeholder images. No images. - Sometimes FANZA doesn't credit the actress at all. Empty array.
- Titles are self-censored:
D***k Girl.
8. One scene, four files
Many releases ship as 1.mp4 through 4.mp4 or more, and FANZA calls that one product. Meanwhile StashDB wants each part as its own
scene. So now you also need a way to split one scraped scene into several.
TL;DR
Nothing is a key. Labels aren't studios, makers aren't studios, prefixes aren't studios, content IDs change, DVD IDs collide or don't exist, names are in the wrong order, null shows up wherever it feels like it, and the placeholder image is a 404.