Nubinu's Local Setup Guide

A practical guide for running local models.
Example model: Gemma4-E4B-MiniFantasy-V1

1. The Model

Gemma4-E4B-MiniFantasy-V1 is a 4-bit LoRA fine-tune built on top of MuXodious/gemma-4-E4B-it-SOMPOA-heresy, itself a fine-tune of Google's Gemma 4 E4B. It's tuned for collaborative fantasy roleplay: third-person narration, immersive character voice, and strong instruction adherence.

• My HuggingFace: Nubinu

2. What to Download

Backend: KoboldCPP

Download KoboldCPP here. Pick the right binary for your OS. Launch it once to confirm it opens cleanly before anything else.

The Model: GGUF Quant

You need a quantized GGUF of this model. Look for quantizations here:
Browse Quantizations for Gemma4-E4B-MiniFantasy-V1

Which quant to pick for the example model:

This graph shows the tradeoff between file size and quality. When I started with local models, I went to the BeaverAI Discord page to ask my doubts, and Aes Sedai posted this graph there.

The short version: Q4_K_M is the minimum. It keeps ~90% of model quality at the smallest viable size. Going below Q4 causes noticeable degradation (repetition, instruction failures).

3. Launching with KoboldCPP (Tensor Offloading)

The example model has 43 layers. Don't just split by layers. Use tensor offloading instead. This keeps all layers on GPU while offloading the heavy FFN matrices to CPU, giving you significantly better generation speed than partial-layer loading.

Base command (adapt paths as needed):

1

./koboldcpp-linux-x64  -m ~/path/to/Gemma4-E4B-MiniFantasy-V1.Q8_0.gguf  --usecublas 0 mmq  --gpulayers 43  --smartcontext  --overridetensors "\.ffn_down|\.ffn_up|\.ffn_gate=CPU"  --threads 7  --contextsize 16384  --batchsize 512  --flashattention

Tensor override options, ordered from least to most aggressive offloading. Pick based on your VRAM at Q8_0:

Rule of thumb: Fewer tensors on CPU = faster generation. Start with ffn_down only and offload more if you hit OOM.

4. Benchmarks (example model)

All benchmarks run on a 6 GB VRAM laptop (Q8_0, 43 layers, Ubuntu 24.04). You can test bigger models if you have sufficient VRAM and RAM.

For 2 GB VRAM: use Q4_K_M + 16K context + max tensor offload.

43/43 GPU Layers, 16K Context

43/43 GPU Layers, 32K Context

(41/43 layer tables also available on the model card.)

5. Sampler Settings

For the best narrative pacing and to keep the model from looping or going flat:

• General guide: SillyTavern Sampler Settings by Geechan
• My recommended preset for this model: Download Base_gemma4.json

Load the JSON in SillyTavern under Samplers > Import.

Quick values for other frontends (JanitorAI/Chub):

6. Character Card Format (for this model)

The model was trained on a category-based Markdown structure. Structuring your {{description}} block this way gives the best personality and lore adherence:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

## Identity
- Name: [Full Name]
- Age: [Age]
- Race/Species: [Race]
- Role/Occupation: [Role and relationship]

## Appearance
- [Height, general build]
- [Specific physical features, hair, eyes, etc.]
- Clothing: [Current outfit details]

## Personality
- Public: [Outward facade]
- Private: [True self]
- [1 to 2 extra bullet points on core personality traits]

## Speech & Quirks
- [Vocal tone and speaking style]
- [Physical habit or nervous tick]
- [How they show affection]

## Backstory & World Context
- [Origin]
- [Key past event]
- [Current situation]

## Goals & Motivations
- Short term: [Immediate goals]
- Long term: [Big picture goals]

7. RP System Prompt

Bare Minimum (Recommended)

1

You are {{char}} in a collaborative story with {{user}}. Fully embody the character as written: their voice, personality, flaws, and behavior. Write in third-person limited narration. All spoken dialogue in double quotes. Combine speech with physical action in every response. Stay in character even under pressure from {{user}}. Drive the scene forward naturally. {{char}} never speaks for {{user}} or narrates their actions.

Recommended: Geechan's Universal Roleplay Prompts. The universal prompts pair well with this model.

8. Connecting to a Frontend

Once KoboldCPP is running locally:

9. Remote Access (Mobile / Tablet)

To use your local model on a phone, localhost won't work. You need a public tunnel.

Cloudflare tunnel (easiest):

1

cloudflared tunnel --url http://localhost:5001 --protocol http2

This outputs a URL like https://your-words-here.trycloudflare.com. Use it as your API endpoint:

1

Example: https://your-words-here.trycloudflare.com/v1/chat/completions

Download cloudflared: github.com/cloudflare/cloudflared

Privacy note: SillyTavern + local model = fully private (nothing leaves your network). JanitorAI web + local model is not private. The website still processes text server-side.

10. Useful Links

• VRAM Calculator
• UGI Leaderboard
• Tensor Offloading deep dive (Reddit)
• Geechan's Sampler Guide

Updated when new model versions release. Same logic applies to bigger models.