Koboldcpp Colab Guide

This is the same guide seen in the Koboldcpp Colab, but mirrored to a rentry instead. To follow it you will need to navigate to this URL:

Step 1. Keeping Google Colab Running

Google Colab has a tendency to timeout after a period of inactivity. If you want to ensure your session doesn't timeout abruptly, you can use the following widget.

Starting the Widget for Audio Player:

Starting the Widget for Audio Player

How the Widget Looks When Playing:

How the Widget Looks When Playing

Follow the visual cues in the images to start the widget and ensure that the notebook remains active.

Step 2. Decide your Model

Pick a model and the quantization from the dropdowns, then run the cell like how you did earlier.

Select your Model and Quantization:

Select your Model and Quantization

Alternatively, you can specify a model manually.

Manual Model Option:

Manual Model Option

5_K_M 13b models should work with 4k (maybe 3k?) context on Colab, since the T4 GPU has ~16GB of VRAM. You can now start the cell, and after 1-3 minutes, it should end with your API link that you can connect to in SillyTavern:

SillyTavern Link

Pub: 10 Sep 2023 14:37 UTC
Edit: 10 Sep 2023 14:45 UTC
Views: 3599