Marehood

LiteLLM Proxy Setup

A simple, automated setup for running your own LiteLLM Proxy server with Gemini and OpenRouter models.

Prerequisites

Before you begin, ensure you have:

Windows Operating System – These scripts are designed for Windows
Python Installed – Python 3.8 or higher is recommended
API Keys:
- Gemini API Key
- OpenRouter API Key

Note: You need at least ONE API key (either Gemini or OpenRouter)

Setup Guide

Some browsers may freak out because of the .bat files in the zip.
Here's the code if you want to create your own .bat files:

Setup
Run

Step 1: Initial Setup

Download all the provided batch files into a new, empty folder
Launch setup_litellm.bat
The script will:
- Check if Git is installed and install it if needed
- Clone the LiteLLM repository
- Modify the requirements to ensure compatibility
- Install DSPy and other dependencies
- Create configuration files with predefined models

Step 2: Add Your API Keys

After the initial setup completes:

Open the created text files:
- GeminiKeys.txt – For your Gemini API keys
- OpenRouterKeys.txt – For your OpenRouter API keys
Add your API keys to these files:
- One key per line
- No quotes or special formatting needed
- Save the files

Step 3: Launch the Proxy Server

Run run_litellm.bat
The script will:
- Check that you've added at least one API key
- Generate the configuration file with your keys
- Launch the LiteLLM proxy server
Once running, you can access the LiteLLM UI at:
http://localhost:4000
To stop the server, press Ctrl+C in the terminal window

Step 4: SillyTavern

Alt Tag

Custom Endpoint: http://0.0.0.0:4000
Custom API Key: sk-1234

Streaming

Streaming works with this proxy, but there is a known issue with fallback behavior.

Let's say one of your Gemini keys is out of quota and is randomly selected by the proxy:

Streaming OFF: The proxy will detect the issue and automatically switch to another key.
Streaming ON: The proxy will not detect the issue. SillyTavern will display an error, and you'll need to swipe again and hope the proxy doesn't select the same key (it's random).

If you add more keys, the chance of this happening decreases.

If you enjoy using streaming, it's generally best to keep it ON, and only disable it temporarily if you start seeing error messages related to quota limits. You can re-enable streaming once the quotas reset.

As mentioned: the more keys you add, the lower the chance you'll run into this issue and need to disable streaming at all.

Available Models

The proxy is pre-configured with:

Gemini Models

Gemini 2.5 Pro Experimental 2025-03-25
Gemini 2.0 Pro Experimental
Gemini 2.0 Flash
Gemini 2.0 Flash Thinking Experimental 2025-01-21
Gemini 2.0 Flash Thinking Experimental 2024-12-19
Gemini Experimental 2024-12-06
Gemini Experimental 2024-11-21
Gemini Experimental 2024-11-14
Gemini 1.5 Pro

OpenRouter Models

DeepSeek R1
DeepSeek V3 0324
DeepSeek R1 Zero
DeepSeek V3 Base
Quasar Alpha
Llama 4 Maverik

Customizing Your Setup

Adding More API Keys

Simply add additional API keys to the corresponding text files:

One key per line
The proxy will use all valid keys when generating configurations

Adding Custom Models

To add your own models:

Edit the model text files:
- GeminiModels.txt – For Gemini models
- OpenRouterModels.txt – For OpenRouter models

Follow the format:

⎗

1	Display Name;model_identifier

For Gemini example:

⎗

1	Gemini 1.0 Pro;gemini/gemini-1.0-pro

For OpenRouter example:

⎗

1	Claude Opus;anthropic/claude-3-opus:free (Lol, funny)

Run run_litellm_proxy.bat again to generate an updated configuration

Advanced Configuration

For advanced configuration options:

Directly edit litellm/proxy_config.yaml after it's generated
Refer to the LiteLLM documentation for configuration options
Restart the proxy server after making changes

Troubleshooting

Common Issues

"Git installation failed"
- Try installing Git manually from git-scm.com
"No API keys found"
- Ensure you've added at least one API key to one of the key files
"Failed to install dependencies"
- Check your Python installation
- Try running pip install --upgrade pip before retrying
Proxy server crashes on startup
- Check if another service is using port 4000
- Verify you have the correct API keys

Getting Help

For more information:

LiteLLM Documentation: docs.litellm.ai
LiteLLM GitHub Repository: github.com/BerriAI/litellm

Updating

To update the LiteLLM version:

Delete the litellm folder
Run setup_litellm.bat again to clone the latest version
Add your API keys again if needed
Run run_litellm_proxy.bat to configure and start the proxy