Enterprise Security Requirements Push Russian Developer to Deploy 125B MoE LLM Locally on Consumer Hardware

A cybersecurity journalist and internal platform developer has published a detailed account of migrating production AI agents from cloud APIs to a fully local 125B-parameter MoE model after repeated demands from enterprise clients regarding data residency and information security.

The author works with Russian state corporations, banks, and industrial holdings where commercial secrets, FSTEC requirements, and physical network segmentation are standard. When AI agents began writing code, investigating incidents from traces, and handling customer task statements, security teams immediately asked where the prompts, code fragments, and contextual data were being sent.

Cloud APIs were ruled out because they transmit task formulations containing project names, counterparties, amounts, responsible persons, internal URLs, server names, database schemas, access keys, and sometimes regulated personal data. Masking was implemented as a hygiene layer but proved insufficient: indirect reconstruction remained possible, masked context degraded model quality, code and configuration files could not be masked, and metadata about query patterns still left the perimeter.

After rejecting smaller 30-35B models for insufficient tool-calling and long-context reliability, the author selected the Qwen3.8-Flash-Next model (qwen4exp architecture). This 125B MoE model activates only about 6B parameters per token and keeps the majority of experts in system RAM, allowing it to run on a single RTX 5060 Ti 16 GB GPU.

The workstation consists of an RTX 5060 Ti 16 GB (Blackwell), Xeon E5-2640 v3 (8 cores / 16 threads), 128 GiB DDR4 ECC, and fast SSD storage. The author reports that DDR4 bandwidth became the primary bottleneck, leading to extensive tuning of expert placement between CPU and GPU.

Initial experiments with LM Studio and llama.cpp yielded only 3-9 tokens per second. Switching to the specialized Strata inference engine (MIT license), built specifically for the qwen4exp architecture, produced a dramatic improvement after calibration: prefill reached 437 tokens per second and generation reached 28.0 / 24.8 tokens per second (English / Russian) on a ~10k-token prompt.

Key Strata features leveraged include adaptive expert caching in VRAM, parallel CPU/GPU expert computation, built-in speculative decoding via the model’s MTP layer, chunked prefill, and KV streaming that keeps most of the cache in pinned RAM while only the active window resides in VRAM. Context lengths up to 512k tokens were tested with minimal decode penalty.

The final production configuration uses IQ3_S quantization, a single 512k context slot, GPU vision encoding, and calibrated parameters for expert offloading and speculation threshold. The resulting system now runs real agentic workloads at 22-25 tokens per second while keeping every byte inside the workstation perimeter.

The author notes that local inference also serves as a resilient backup channel during subscription loss or internet outages and provides a valuable testbed for validating agent skills before deploying them against frontier cloud models.