themachine

The Quest for themachine ⚔️

I've been fascinated by the potential of large language models (LLMs) to revolutionize the way we interact with technology. However, running these models requires serious computational horsepower.

That's why I set out to build a custom machine learning server, dubbed themachine, capable of running multiple LLMs concurrently and handling large context sizes. In this post, I'll share the story of how I built themachine, its specs, and some of the exciting projects and experiments I've run on it.

From Humble Beginnings

Building themachine wasn't easy. I started with a modest setup featuring a pair of NVIDIA GeForce RTX 3090 GPUs, a desktop motherboard, and 96GB of RAM. But... those GPUs wouldn't fit quite right, so I had to get creative. I designed and 3d-printed a custom PCIe riser bracket to orient them vertically.

Even with the addition of a 4060Ti, I quickly realized that this configuration was holding me back. It was time for an upgrade. I swapped out my hardware for some serious server-grade gear, including an Epyc CPU and the ASRock ROMED8-2T motherboard, which boasts seven PCIe slots. This allowed me to add even more GPUs to the mix.

A Custom Case for a Custom Machine

With all this new hardware, I needed a case that could keep up. Unfortunately, there weren't many options available that could fit everything, so I designed and built a custom case using 2020 extrusion. The end result was a modular design that could grow over time.

When taking apart the original machine to migrate parts over, I noticed the GPUs were sagging... The PCIE mount (3d-printed PLA) had deformed from the heat!

The Never-Ending Upgrade

For a while, everything was good and the universe was at peace.

But it didn't last. A month or so later, it becase clear that four GPUs just weren't enough... we needed... more.

The release of Llama3.1 405b was on the horizon, and I wanted to be able to run it at a reasonable speed. This also meant I could run Llama3.1 70b with full context as well as multiple large LLMs concurrently. Little did I know, this would require some serious power - four 1600w power supplies, to be exact. We had definitely exceeded the capabilities of a standard US household electrical circuit!

Fortunately, I had designed the case with modularity in mind, allowing me to expand it one tier at a time.

The modular design worked well and provided a compact housing for the whole system.

Overcoming the Final Hurdles

Getting to this point took some elbow grease, but the real test was yet to come. The cheap PCIe risers I had been using were causing PCIe bus errors, and the system wouldn't stay stable. I sourced higher-quality risers from c-payne.com, which have worked like a charm. To keep the risers and cards aligned and mounted, I designed and built custom mounting plates.

Finally, after all the blood, sweat, and tears, we had all 12 GPUs working in harmony! Well, almost. The top tier of cards still can't reliably run at PCIe 4.0 speeds and had to be stepped down to 3.0. I suspect I need to add PCIe redrivers to get them running flawlessly, but for now, the impact is negligible since my workloads are mostly inference-oriented.

Capabilities and Applications

themachine is capable of running Llama3.1 405b at a 4.5bpw quant as well as multiple LLMs concurrently like in the 70-120B param size like Llama3.1-70b @6.0bpw with Mistral-Large-Instruct-2407 @8.0bpw and at least than 32k contexts for each. This provides decent headroom for various tasks.

Some potential uses for themachine include:

• Running multiple LLMs for various tasks, such as language translation, text summarization, and sentiment analysis
• Developing and training custom LLMs and other machine learning models for specific applications
• Experimenting with applications of LLMs for various tasks

Projects and Experiments

I've run several exciting projects and experiments on themachine, including:

• A CLI assistant, kalle, which is powered by models running on themachine (it can also use third party LLM APIs)
• Generating synthetic conversations for an experiment with intent classification
• A cli color palette tool that generates color palettes from a source image using a number of methods including: kmeans, medcut, and dbscan
• An experiment in LLM driven software maintenance where the logs and code are analyzed, the code updated, and the LLM updated code is deployed without a human making any code edits

What's Next?

themachine is a powerful machine learning server that is capable of running multiple LLMs concurrently and handling large context sizes. Its custom design and technical specifications make it an ideal setup for natural language processing, text generation, and other applications.

I'm excited to continue exploring the possibilities of themachine and sharing my experiences with the community. If you're interested in building your own machine learning server or learning more about themachine, I'd be happy to share more details and answer any questions you may have.