We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

[u/saltexx]

I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from now on everything happens in public.

It's an inference engine in Rust and C++ with our own CUDA kernels. One binary with OpenAI and Anthropic style APIs, loads GGUF and safetensors. We run about 300B tokens a year through it at work.

Some numbers: Qwen3.8-27B FP8 on one RTX PRO 6000, spec decoding off on every engine:

• vs vLLM faster in 13 of 13 cells, 1.02x to 1.19x (so not huge)
• vs SGLang faster in 10 of 13, behind in 2, level in 1
• vs llama.cpp Q8_0 faster in 13 of 13, 1.5x to 37x
• 32 clients at 1024 in / 1024 out: 1062 tok/s, vLLM 958, SGLang 844

Full board with the losses: https://truespar.com/paddock/benchmarks/qwen38-27b

What it does not do yet: CUDA only, Windows and Linux. Validated on Blackwell (5090, RTX PRO 4500/5000/6000, B200) and Ampere (an A6000 was the bring-up card, 30-series works). Ada kernels ship but nobody has run a board on them so the engine refuses to start unless you set PADDOCK_UNVALIDATED_ARCH=1. Hopper and A100 kernels are in the tree without a board. No Mac, no ROCm, no Vulkan. One model per GPU, no tensor parallel.

https://github.com/truespar/paddock

Thankful for any help and input!

[u/Human-Gas-1288]

does this works on Windows？

[u/saltexx → u/Human-Gas-1288]

Yes, Windows and Linux x64. You need an NVIDIA driver 580 or newer.

[u/takoulseum]

One model per gpu means the model must fit in a single gpu as no tp so pipeline parallel not supported too?

[u/saltexx → u/takoulseum]

Yeah no TP or pipeline split yet, model has to fit on one card. Kind of on purpose tbh, we want the single GPU path fast first. Will come later!

[u/N34257]

Two questions: does it support INT8 (I'm thinking entirely in terms of my 170HX here)? Also, I didn't see any mention of MTP or prefill performance...

[u/AI_spell]

One thing I’d watch in a custom engine is KV-cache allocation and attention behavior when the context crosses a block boundary. A benchmark can look great at 4k then hit a latency or OOM cliff at 8k or 16k, especially with variable batch sizes, so reporting max context along with batch and concurrency would be useful.

[u/BodyPhysical]

would suggest to please build release binaries which could be zipped and downloaded through the github repo

[u/saltexx → u/BodyPhysical]

Thanks, agreed. Coming up

[u/JollyJoker3]

Asked Claude if it's worth trying, it essentially said no;

The llama.cpp column is broken, not slow. 16 s to first token at one client on a PRO 6000 for a 1k prompt is not a plausible number for llama.cpp. They ran -c 262144 -np 32, which splits into 8k slots but forces an enormous KV allocation up front. The 37.5x claim rests on that column, and I wouldn't trust it.

[u/AppealSame4367 → u/JollyJoker3]

Sad, so the tests including the presentation are AI slopped :-/

[u/saltexx → u/JollyJoker3]

Flags and the GGUF are on the board page, -c 262144 -np 32 is 32 slots of 8k like the vLLM run, KV alloc is at startup. Post the flags you'd use and I'll rerun it.

[u/JollyJoker3 → u/saltexx]

No clue what flags to use and apparently what I said was a Claude hallucination. Something is broken though. Both "Short chat" and "Long generation" take 128 tokens in. Time to first token is 1.4s for one, 17s for the other.