ERP/RP and erotica raw data collection

RP/ERP forum scrapes

Listed below are archives containing RP (roleplaying) and ERP (erotic roleplaying) messages scraped from openly accessible forums, in CSV format. For non-story forums, average message length can usually give a good indication of their general quality (i.e. effort put into writing them); on good quality RP forums it is often well over 1 kB/message. Besides miscellaneous information about data quality, the notes provide message count, mean length, standard deviation and message length percentiles.

Keep in mind that most messages in the CSV files are in raw HTML and are not ready for use for LLM training or fine-tuning. Further processing will be needed for that.

Usage notes

All message archive files are in .csv format. They can be easily processed in python using the pandas data processing library as follows:

1
2

import pandas
data = pandas.read_csv(filename)

After doing so, most of the time the data will have 5 or more fields as in the following example:

1
2
3
4
5
6
7

b.loc[211]
Out[211]: 
thread_title                              new idea, but need some help
thread_href          https://forum.literotica.com/threads/new-idea-...
message_timestamp                               Nov 3, 2000 at 3:00 PM
message_username                                   husband's nightmare
message              I want to do a role play based o the movie Dre...

Some details

• Messages have been parsed by chronological order using threads first, then messages. In other words, messages from the same thread will be consecutive in the .csv files.
• message contains the message body in html format. To become fully usable for fine-tuning LLMs it will likely have to be rendered as text. The Python library BeautifulSoup could be used for this.
• message_timestamp has not been normalized and thus may have different formats depending on the forum.
• Sometimes, thread labels or tags have been added too as separate fields.

Suggested strategy for parsing and filtering the files

• Forums can have a lot of off-topic or non-RP threads. The latter most of the time include "OOC" in the thread title. Such threads could be filtered away.
  • For Elliquiy, it is suggested to filter away threads titles containing the strings: OOC, O.O.C. (out of character, i.e. non-RP/ERP talk); Character Sheet, Character Profile, Character Thread, Character List, Character Roster (these all contain character descriptions). However, note that IC or in-character means that the thread is in-topic, i.e. it contains proper role-play.
  • Note that sometimes Character and OOC can be in the same thread.
• For privacy reasons it might be best to anonymize usernames and scramble URLs originating from the forum where the messages have been posted.
• Typically OOC (out of character) messages within in-topic threads will be enclosed with double parentheses (( )), single parentheses with OOC marking (OOC: ); more rarely square parentheses are used as well [ ] [OOC: ]. However, often just single parentheses ( ) without other indications are used for OOC.
• Threads tend to have a more or less consistent posting style. It is suggested to filter complete threads away if the posts contain too many one-liners.
• Oftentimes there will only be 1 opening post, perhaps due to a failure to attract attention. In principle, these threads could be filtered away, but depending on the forum they are often also high-effort posts.

Stories website scrapes

These archives contain short stories written by a single author instead of roleplay between two or more persons.