Building Secure On-Prem AI Assistants: How to Keep Corporate Data Inside Closed Contours

Many companies abandon AI assistant projects the moment security teams declare that sensitive data cannot leave the organization. Yet it is entirely possible to run large language models inside a fully closed contour without any external data transmission.

The author outlines four possible locations for hosting the model. The most open option is a public cloud API suitable only for prototypes with anonymized data. The next level places the model inside the company’s own cloud account. More restrictive environments run inference on company-owned servers in an internal data center, a configuration common in finance, healthcare, and government. The strictest setup launches the model directly on an employee’s laptop with no internet connection at all.

Access rights must be separated into three distinct categories rather than treated as a single switch. Read access lets the system retrieve documents to draft answers. Write access allows updates to databases or records. Execute access triggers external actions such as sending emails or initiating payments. Most practical use cases require only read permissions, which significantly reduces objections from security teams.

Because models can still make mistakes, the article recommends explicit human-in-the-loop checkpoints for any irreversible operations. Technical safeguards include spending and volume limits, idempotency keys, comprehensive audit logging, and an emergency “stop” button that revokes execute rights without shutting down the entire service.

Local performance concerns are addressed by noting that modern open-weight models such as Llama, Qwen, and Mistral, along with Russian alternatives like GigaChat and YandexGPT, deliver sufficient quality for corporate tasks when paired with retrieval-augmented generation. Quantization to INT4 or INT8 further reduces hardware requirements, enabling deployment on modest GPU setups or even high-end laptops.

Additional operational lessons include honest calculation of total cost of ownership for on-premise GPUs and the recognition that cleaning and maintaining the knowledge base frequently represents the majority of the project effort.