Koboldcpp Colab Guide
This is the same guide seen in the Koboldcpp Colab, but mirrored to a rentry instead. To follow it you will need to navigate to this URL:
https://colab.research.google.com/github/kalomaze/koboldcpp/blob/alternate_colab/Koboldcpp_Colab_(Improved_Edition).ipynb
Step 1. Keeping Google Colab Running
Google Colab has a tendency to timeout after a period of inactivity. If you want to ensure your session doesn't timeout abruptly, you can use the following widget.
Starting the Widget for Audio Player:
How the Widget Looks When Playing:
Follow the visual cues in the images to start the widget and ensure that the notebook remains active.
Step 2. Decide your Model
Pick a model and the quantization from the dropdowns, then run the cell like how you did earlier.
Select your Model and Quantization:
Alternatively, you can specify a model manually.
Manual Model Option:
5_K_M 13b models should work with 4k (maybe 3k?) context on Colab, since the T4 GPU has ~16GB of VRAM. You can now start the cell, and after 1-3 minutes, it should end with your API link that you can connect to in SillyTavern:
