Mini PCs Run Larger AI Models but Suffer from Slow Memory Speeds

A compact Intel-based mini computer can now load artificial intelligence models that exceed the memory limits of a dedicated graphics card found in a typical gaming tower. This shift challenges the assumption that high-end discrete GPUs are the only viable option for running large local AI workloads.

However, the convenience of fitting these large files into unified system memory comes with a significant performance penalty. While the mini PC can hold the data, its processing speed is often cut in half due to hardware limitations that affect real-time usability for daily tasks.

Mixture of Experts models change memory requirements

Traditional AI models require all their numerical weights to be loaded into memory simultaneously, creating a hard cap on file size. Newer architectures known as Mixture of Experts, or MoE, alter this dynamic by activating only a small subset of parameters for each input token.

This efficiency allows models with billions of total parameters to run on devices with standard consumer-grade memory. For example, a model with 35 billion parameters can operate with only three billion active at a time, reducing the immediate memory footprint significantly.

Single-channel memory halves token generation speed

Testing a MINISFORUM MS-03 unit revealed that while it successfully loaded large quantized models, its output speed lagged behind manufacturer claims. The device achieved only 9 to 13 tokens per second, far below the 22 tokens per second quoted in official specifications.

The primary cause of this slowdown is the use of a single stick of 32GB memory, which forces the system to run at single-channel speeds. This architectural limitation effectively halves the data transfer rate, making the machine suitable for occasional batch processing but less ideal for interactive, real-time applications.

Older gaming cards remain faster for active layers

Despite the mini PC’s ability to host larger files, an older RTX 3080 graphics card can still outperform it in specific scenarios. By offloading the static expert weights to system RAM and keeping only the active attention layers on the GPU, the dedicated card leverages its faster memory bandwidth.

This hybrid approach allows the gaming system to maintain higher token rates for the same model sizes, provided the model fits within the hybrid memory scheme. The trade-off is that the mini PC wins on capacity, while the dedicated GPU wins on raw speed for the active components.

According to reporting by XDA Developers, the choice between these two setups depends on whether the user prioritizes the ability to load massive files or the speed of generating responses. Neither option is universally superior, as each excels in different aspects of local AI inference.