Building Secure MLOps Platforms in Air-Gapped Environments for DevOps Engineers

The article presents a practical guide for DevOps engineers tasked with building an MLOps platform inside a fully closed, air-gapped environment. Up to 80 percent of machine-learning projects reportedly never reach production because of missing reproducible processes for data, models, and deployment.

MLOps extends classic DevOps practices to data and models. Data becomes a first-class artifact that must be versioned with DVC and stored in MinIO, while experiments are tracked in MLflow backed by PostgreSQL. A Model Registry with champion and challenger aliases allows safe model rollouts without changing application code.

The platform runs on two physical segments: a Kubernetes cluster handling serving, storage, and monitoring, and a separate GPU server used exclusively for training via Docker containers with NVIDIA runtime. This separation prevents expensive GPU resources from being locked to Kubernetes nodes.

Key components include ArgoCD with Apps-of-Apps pattern, GitLab CI/CD, FastAPI inference service, Celery and Redis for GPU task queuing, JupyterLab for experiments, OpenBao instead of Vault, External Secrets Operator, Trivy and Bandit scanning, and Prometheus plus Grafana monitoring. All Helm charts are vendored inside the Git repository to eliminate external dependencies.

Security measures specific to closed contours cover private Harbor registry, SOPS encryption with age keys delivered via ArgoCD CMP sidecar, wildcard TLS certificates distributed by ClusterExternalSecret, and a Docker Socket Proxy that restricts container operations to the minimum required privileges.

The author deliberately avoids Kubeflow because of its cloud-oriented design and heavy CRDs, and replaces Airflow with the lighter Celery queue. The resulting stack provides a reproducible, auditable foundation that can later be scaled when project volume increases.