Running KoboldCPP ROCM on Linux with Docker

Try this if your distro packages ROCM terribly like mine. You might need to figure out how to tell Docker to not store everything in your root directory to avoid running out of space.

Get koboldcpp rocm

Make a folder (we'll call it docker_folder_name from now on) and clone https://github.com/YellowRoseCx/koboldcpp-rocm into it. Make a separate folder in your home directory for storing models, we'll call it model_folder.

Dockerfile

In the docker folder, make a Dockerfile with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# syntax=docker/dockerfile:1

FROM archlinux:latest
WORKDIR /app
ENV PIP_BREAK_SYSTEM_PACKAGES 1
COPY . .
RUN pacman -Syyu --noconfirm
RUN pacman-db-upgrade
RUN pacman -S --noconfirm rocm-hip-sdk make gcc perl python python-pip python-setuptools which cblas openblas clblast tk
RUN pip install customtkinter
RUN cd koboldcpp-rocm && make clean && make LLAMA_HIPBLAS=1 -j4

This just pulls an arch image, installs koboldcpp's dependencies, and builds koboldcpp_rocm. You can build the image with: docker build -t <docker_folder_name> . Wait for the build to finish.

Running the image

Once the image is built. Run it with docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --env="DISPLAY" --volume="$HOME/.Xauthority:/root/.Xauthority:rw" -v $HOME/<model_folder>:/model_folder<docker_folder_name>
This command gives docker pretty full access to your system. You'll start out in /app, which should contain koboldcpp-rocm. You should be able to run koboldcpp.py from there. Your model folder will be accessible from the root directory.

Running the GUI

If you want to run the koboldcpp gui, you need to run xhost + in another terminal to loosen Xorg's access restrictions. This should work on just about any DE/WM, regardless of if you're using wayland.