Silly Tavern on Single Board Computer (SBC)

Why Use Silly Tavern on an SBC?

Running Silly Tavern on a Single Board Computer (SBC) offers several advantages:

Intranet Access: Easily access your models and cards from any web browser within your local network.
Centralized Management: Consolidate all cards and API endpoints in one location for access from multiple devices.
Learning Opportunity: Gain hands-on experience with lower-level networking concepts (focused here on Ubuntu).
Compact and Efficient: A headless setup (no monitor or keyboard) keeps the system compact—about the size of a deck of cards—and uses only ~15W of power.

An SBC with enough memory to run Ubuntu.
- This guide uses an Orange Pi 3B with a 64GB card (overkill but effective)
- RPi alternatives like Orange Pi are affordable via AliExpress or similar vendors; as long as there's a modern Linux system available you should be fine
- Raspberry Pi is compatible but may be pricier; recent price and availability issues are a consideration.
- Memory required to run is unclear, but my attempt to get it working on an RPi Zero W with 512K ran out of memory (aside from requiring unofficial repo for npm to get that working.) Any SBC in a RPi 3/3+ format will probably be fine.
Access to intranet via ethernet or Wi-Fi (this guide assumes you're using WiFi)
Monitor, keyboard, and mouse for initial setup (these can be set aside after going headless).

Other Essentials:

Basic Linux knowledge and a willingness to learn more.
ChatGPT to troubleshoot and answer questions along the way.

How to Set It Up

This is a rough guide outlining the process I followed, including key steps and challenges encountered. It's designed with minimal guidance but highlights areas where I got stuck. If additional help is needed, querying ChatGPT for specific issues is effective—especially as it gains context about your setup. Or perhaps below will be enough.

Install Ubuntu on your SBC. For Orange Pi this comes preloaded as OrangePi OS, which is Ubuntu, running XFCE as a Xwin GUI manager. Follow instruction for your SBC if it's not.
Get machine set up with monitor, keyboard, mouse. We'll use the windowed display to do initial setup, then turn it off later.
Install Git on your machine: sudo apt-get install git
Install SillyTavern on your machine git clone https://github.com/SillyTavern/SillyTavern -b release
Ensure SillyTavern works and can be seen on the local browser
- I ran into whole issue here around Node.js not being up to date. As of writing, ST require v18 or better. ST errors on this aren't obv (at least not to me)
  - Check Node.js version node -v
  - List all installed Node.js versions nvm list
  - Install Node.js v18 nvm install 18, nvm use 18
  - Uninstall Node.js v16 (for example) when they refuse to let go nvm uninstall 16
- Might be better off uninstalling the whole thing and starting over from scratch
- Getting NodeJS, NVM, NPM working is critical and bounds in what hardware you can use. There are alt repos for non-standard hardware, but TLDR if it runs a modern version of Ubuntu you should be OK.
Get SBC hooked up to the WiFI network. Use the windowed tools...
Follow the ST guide to get ST to start talking on your network (https://docs.sillytavern.app/usage/remoteconnections/)
Edit the ~/SillyTavern/default/config.yaml and set autorun: false to stop ST from opening up a browser on startup
Test access to the SBC using port 8000 on one of your computers
Get SSH set up on your daily driver (Windows: PuTTY, for example)
Hook into your SBC via SSH with your daily driver
- This turned into another whole thing. The TLDR is if its not working, uninstall SSH and reinstall it
  - sudo apt update
  - sudo apt-get remove --purge openssh-server
  - sudo apt-get install openssh-server
- Default SSH is via port 22
So now ST will run in an open Xterm window. We want it to run in the background, and restart every time the machine powers up
- We will us pm2 to do this
  - sudo npm install -g pm2
  - From ST directory: pm2 start start.sh
  - pm2 startup, pm2 save to keep it running after reboot
At this point: ST is running in background, we can access it via WiFi, and we can SSH into machine. So, we can (and should) shut off the Xwin GUI. Below is for XFCE, steps will be similar for other OS
- sudo systemctl set-default multi-user.target Start in command line
- sudo reboot Restart, you should get a simple CLI now
- To get Xwin GUI back: sudo systemctl set-default graphical.target (permanent) sudo systemctl start lightdm (one time)
- You can also uninstall the XFCE package to save space (I didn't bother with this step.)
You can monitor the ST in SSH via pm2 logs, which will allow you to see the entire context being sent via API
Unplug the keyboard, mouse, monitor from SBC, and put SBC in its new home
Everything should be running now; last bit are some security issues
- Change the default uname pwd: sudo passwd orangepi for example
- Set up basic firewall, and open just the ports you'll need, using ufw. Do below in order...
  - sudo apt install ufw
  - sudo ufw allow ssh
  - sudo ufw enable Ensure SSH is allowed first obv
  - sudo ufw status Check all's working
  - sudo ufw allow 80 These 3 open up basic HTTP and port 8000 which ST uses
  - sudo ufw allow 443
  - sudo ufw allow 8000

Where to Place the SBC with ST

Once the setup is complete, you can position the SBC anywhere with power and access to Wi-Fi or your intranet. Remote management is possible via SSH.

Remote Maintenance Items

To monitor the context (useful if you're writing cards and can't figure out why the LLM is responding differently than expected)
- SSH into SBC
- pm2 log will show context ala the terminal window
I found SBC would fall off the WiFi network if the network reset. To get the SBC to monitor and re-start network, see bottom of this document for a chunk of code to fix that.
I found on network reset, the IP address would change... You can lock it down from your router:
- Log into your router (usually via web at 192.168.1.1 or some such)
- Look for the DHCP Client List to see connected devices and their MAC addresses. Identify your headless system and note the MAC address
- Look for Address Reservation or IP-MAC Binding options on your router, Click Add or Reserve, and save it

Optional Configurations (Not Implemented)

Internet Access: Set up ST for access from anywhere over the internet. While the developers advise against this, they do provide instructions.
Peer-to-Peer Setup: Configure ST for peer-to-peer operation. This may be useful when traveling, though running it locally might suffice.

Other Resources

Code

Simple network monitoring and reconnect script that periodically checks the connection and attempts to reconnect if disconnected.

1. Use a Shell Script to Monitor the Connection

Create a script that pings a reliable external server (like Google's DNS server 8.8.8.8) and reconnects if there's no response.

Script:

⎗
✓
#!/bin/bash

# Target for connectivity check
TARGET="8.8.8.8"
# Network interface (adjust as needed, e.g., wlan0 for Wi-Fi)
INTERFACE="wlan0"

# Check connection and attempt reconnect if disconnected
while true; do
    if ! ping -c 1 -W 1 $TARGET > /dev/null 2>&1; then
        echo "$(date): Connection lost. Attempting to reconnect..."
        sudo ifconfig $INTERFACE down
        sleep 5
        sudo ifconfig $INTERFACE up
    else
        echo "$(date): Connection OK."
    fi
    sleep 30 # Check every 30 seconds
done

2. Save and Run the Script

Save the script as network-monitor.sh.
Make it executable:
⎗

✓
1
chmod +x network-monitor.sh
Run it in the background:
⎗

✓
1
nohup ./network-monitor.sh &

3. Automate at Startup
To ensure the script runs on boot:

Edit the crontab:
⎗

✓
1
crontab -e
Add the following line:
⎗

✓
1
@reboot /path/to/network-monitor.sh