Scheduling LLM inference across consumer GPUs
The bottleneck isn't compute. It's what you choose to run, and when — a scheduler that picks the right backend per operation beats brute-force parallelism on hardware that was never meant to share a workload.
Consumer GPUs weren't built to share inference workloads the way a datacenter cluster is. The obvious move — throw more parallelism at the problem — runs into memory bandwidth and thermal limits fast.
The more useful lever turned out to be scheduling: detecting which backend (CUDA, ROCm, Metal, OpenCL, CPU fallback) actually wins for a given operation on the machine in front of you, and caching that choice instead of re-deciding every call.
Where this got interesting: the fastest backend for a matmul isn't always the fastest backend for a conv2d on the same card, and that gap widens on older or thermally-throttled hardware. A one-size-fits-all dispatch loses real throughput.
Open question: how much of this generalizes past matrix ops — attention kernels, quantized inference — before the per-operation cache stops paying for itself.