Linux Safe Web Service

What is this

Guide to setting up a web service like Gradio without allowing it to talk to the internet. This will keep you safe from any hidden bullshit baked into the program, but WILL ALSO STOP IT FROM BEING ABLE TO INSTALL PLUGINS OR DO UPDATES FROM WITHIN THE WEB INTERFACE OR ACCESS THE INTERNET IN ANY WAY. You'll need to do those things from the command line.

This guide has modern Debian Linux in mind, but should be adaptable to any other distro
We're going to assume you're setting up an already installed and working oobabooga AI server in this example

Create the user

You'll first need a local user service account to run the code
Its best if this user has as few permissions as possible on your system
sudo adduser --disabled-login --disabled-password --shell /bin/false ai

and make sure to change ownership on the files it will need access to
sudo chown -R ai:ai /opt/text-generation-webui/start_linux.sh

This adds a user that can't log in interactively and basically can't do anything except eventually run your AI service

Create a shell script to start the service

If you create a script to start the service, you can pass custom arguments and have some indirection for any other changes you want to make later.

sudo nano /usr/local/bin/start_ai.sh

#!/bin/sh
/opt/text-generation-webui/start_linux.sh --model miqu-70b-q5/miqu-1-70b.q5_K_M.gguf --tensorcores --threads=8

and make it executable
sudo chmod +x /usr/local/bin/start_ai.sh

Create a system service to start/stop/get status on your service

sudo nano /etc/systemd/system/ai.service

[Unit]
Description=Text Generation Web Backend
After=network-online.target

[Service]
ExecStart=/usr/local/bin/start_ai.sh
User=ai
Group=ai
IPAddressDeny=any
IPAddressAllow=localhost

[Install]
WantedBy=multi-user.target

enable the service and start it
systemctl enable ai.service
systemctl start ai

It will now ONLY be able to talk to 127.0.0.1. You can now talk to http://localhost:7860 on that computer
Now you can either use ssh port forwarding
ssh yourusername@IP_ADDRESS_OF_THE_AI_COMPUTER -L 7860:IP_ADDRESS_OF_THE_AI_COMPUTER:7860
and go to http://localhost:7860, or set up an apache or nginx proxy with https

nginx config

This is a basic config that works.
You'll need to use some other guide to install nginx, install and enable the right modules, enable this site within nginx and do any ssl cert config yourself
sudo nano /etc/nginx/sites-available/ai.conf

server {
listen 443 ssl;
listen [::]:443 ssl;
server_name chat.yourdomain.local;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_ecdh_curve secp384r1;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver_timeout 5s;
location / {
proxy_pass http://127.0.0.1:7860/; # Change this if you're running on a different port
proxy_buffering off;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}