Miqumaxx box

Richfag cpumaxxing rig

The big problem with running really high-parameter large-language models, is that inference requires massive amounts of memory bandwidth.

There are three ways:

• Stack GPUs for sweet, sweet GDDR bandwidth eg. v100maxx
• UMA richfag (Appleshit)
• CPuMAXx NUMA insanity
  Dual-socket EPYC Genoa with 24-channels of DDR5-4800 at 920GB/s is what I went with

If you stack GPUs you will eventually:

• Run out of power on a PSU you can run on a standard breaker, so either put in a 20A or bigger breaker and run a 1600W+ PSU, or gang PSUs together. In any case, GPUs are power-hungry
• Need a shit-ton of fans and cooling. Also: noisy, hot hellbox
• Run out of PCIe lanes, slots and literal room in your case. You can use PCIe risers, potentially split larger slots into 1x, use mining frames and boards, etc, but all these things mean your Frankenrig is all over the place and kind of crazy
• Spend as much or more than other approaches. Also, trying to save slots and space by getting more VRAM per card means you're spending orders of magnitude more money. A single 80GB card is $15k+ even if you get a bargain somewhere
• Need to do model training to justify your investment in that many GPUs
• Laugh at anyone trying to do layer offload for any model you can fit in your VRAM. If you can do that, you'll be the fastest game in town
• Realize that even if you can squeeze the model in, you can't fit the model, the context, the tts model and the image gen model all into your VRAM. OOMing due to context can be heartbreaking.

If you go for Appleshit:

• You get a less supported arch. Metal works, but seems to have less performance than should be theoretically available
• Are stuck with exactly what you have for memory. Can't upgrade on-die RAM!
• Have basically no PCIe lanes for GPUs. Might be able to do something with Thunderbolt, but then you're back into heavy jank territory
• Can't actually use all that memory for inference. There are hacks to use more, but you'll never use anywhere close to 100%
• macos
• Spending as much as, or more than the other solutions for a less expandable solution.
• It is stylish, quiet and compact, though

If you CPuMAXx:

• You're spending a pile of cash up front ($6k USD as of the writing of this)
• Need to fill up all RAM slots to get max bandwidth, so an upgrade means swapping all sticks. BUT its easy to get 384GB/768GB/1.5TB or more of fast RAM. Try doing that with GPUs!
• Have to deal with NUMA bullshit, restricting max performance unless things are coded and balanced just-so
• Have tons of PCIe lanes for expandability
• Have tons of general purpose compute for other things. I run lots of selfhost VMs and labs. make -j 128 is satisfying
• Leave any GPU you put in completely free for things it does well: Processing context, imagegen, TTS/STT, etc
• If A100 become cheap on eBay, you can still just start stacking them
• If some other tech that drops into a PCIe slot becomes available (Groq but cheap), then you can still stack those
• Can run the whole works on a 1000W PSU easily
• Run relatively cool, but CPU coolers can be hard to find
• My rig is quiet because I run big fans at low speeds. ymmv
• Can upgrade the CPU later when higher-end EPYC inevitably become cheap on eBay when companies start mass-upgrading
• Need a giant case to house the monstrosity. The CPU dies are HUGE, the memory slots take up massive real-estate, and it just doesn't fit nicely in a reasonably sized case
• You likely aren't going to be doing any training, bruh. CPUs aint GPUs. You can quant and merge pretty well though.
• You can play with frankenmerges and new models at FP16 release time. No llama.cpp integration PR lag!
• You can run Miqu 70b Q5 at 6T/s+ without doing anything special. More speedups likely (theoretically 20T/s+)
• Maybe someday I'll be laughing because there'll be some crazy huge model that you need more than 192GB to even run. I'm not holding my breath.

BOM (parts sourced from QVL) :

• eBay auction for an "Engineering Sample" gear. I got a Gigabyte MZ73-LM1 with two AMD EPYC GENOA 9334 QS 64c/128t Processors
• 24 sticks of M321R4GA3BB6-CQK Samsung 1x 32GB DDR5-4800 RDIMM PC5-38400R Dual Rank x8 Module from memory.net (cheaper than eBay)
• Good quality PSU (I used a FSP 1000W HydroPTM X Pro because I got a deal. No issues)
• Whatever storage you need. Warning that the m2 slot is NVME only. It has a 4x SATA breakout included with the MB. You can add up to 16 more drives with SlimSAS 8i breakout cables
• Giant case or whatever ghetto solution you want. I used an Antec 900 I had lying around, but needed to drill and tap new standoff holes. Make sure airflow is good and you can mount some big, slow moving fans to keep air cycling through without a lot of noise
• A couple of good SP5 CPU coolers. I got "CoolServer" brand 4U height coolers and they'll keep things room temperature at idle and only about 30C increase at full load on all cores
• Still worth it to throw a 24GB GPU in the thing. You can use the onboard video for a console so all VRAM is available for other things. I'm running a 24GB A5000, and even though all my inference is CPU, it still gets a shitton of use. eg. You can get fast text-to-speech and have SD renders of the scene as it progresses

Linux Details

• I'm running Linux Debian (Trixie) headless so all possible horsepower is going to inference.
• Newer (6.6+) Linux kernels have big speedups on EPYC!
• I found turning off Transparent Hugepages to be key for remaining stable through high memory pressure situations
• nginx proxy on all services with the actual backend interfaces firewalled off from external interfaces. Who needs shit leaking to the internet? You're running local for a reason I assume
• Newer distros have gcc13, so you have to dick around with CUDA devkits to get things to compile (mainly forcing gcc 12 to be your compiler and editing apt sources to depend on libtinfo6 instead of libtinfo5 in nvidia packages)

BIOS settings:

• Don't even try to run in BIOS mode. I found it slow and unreliable. UEFI all the way.
• Make sure the xGMI link is running at full speed (4x)
• Turn off extraneous devices to free up PCIe lanes
• Tweak the NUMA layout to work for your workload (NPS0 is single image and is probably best if you don't intend to do any tweaking or multiuser stuff)

Stupid NUMA tricks

You can set NPS to higher than zero and use the new --numa isolate and numactl flags in llama.cpp to run multiple LLMs, each isolated to its own NUMA node. Each one will run relatively fast (limited to memory bandwidth to those sockets/CCXs) and not impact the others. Its useful for multi-user or multi-agent setups.
eg.
numactl -N3 -m3 ./main -m miqu.gguf --no-mmap --numa numactl -t 16 -s 3955 -p "Hello"
will isolate to (zero-based) numa node 3.

You can see your numa map with numactl -H
Use gnu parallel to test simultaneous execution scenarios to get the ideal NPS for your situation.