Multi-Tenancy Isolation in VictoriaMetrics, VictoriaLogs and Grafana: Building Client Separation in a Shared Stack

Voltir provides monitoring for servers, 1C systems and databases as both a cloud service and on-premise installation. In the cloud all clients share one stack containing vmstorage, vmselect, vminsert, VictoriaLogs, Grafana, vmalert, Alertmanager and Postgres. Separate stacks per customer would require ten full copies of caches and ten update cycles per release, so the company keeps a single stack and implements isolation manually at each layer.

Every tenant receives a numeric account_id. VictoriaMetrics routes writes and reads through paths such as /insert/<account>/… and /select/<account>/…, while VictoriaLogs and VictoriaTraces read the account from the AccountID header. Nginx sits in front and calls the platform API via auth_request to determine the tenant from the Bearer token before forwarding requests. Storage endpoints are never exposed externally.

Each tenant has its own Grafana organization whose data sources are created by the platform with the account ID embedded in the URL or secureJsonData header. Users receive at most the Editor role; Org Admin is deliberately withheld because an administrator could change the data-source URL or header and read another tenant’s metrics or logs. The platform explicitly checks role values and rejects anything higher than Editor for tenant users.

Write authorization compares the account supplied in the URL with the account belonging to the token; a mismatch returns 403. For logs and traces the platform returns the correct account in the auth_request response and nginx overwrites any client-supplied header. VictoriaLogs only understands the AccountID header and silently ignores X-Tenant-ID, writing everything to account 0 if the header is missing.

A single vmalert instance reads all rule files. Each tenant’s rules file contains groups narrowed by an extra_filters clause containing the tenant’s account ID. VictoriaMetrics appends this filter to every selector, preventing a rule from seeing data outside its tenant. Global rules without filters were removed after one self-test rule began scanning thousands of deleted tenants and consuming excessive CPU.

During testing a rule containing the Russian string “пять минут” instead of a valid duration value caused vmalert to reject the entire rules directory on reload. Subsequent changes to other tenants stopped being applied, and a container restart would have left alerting completely disabled. The platform now validates durations, severity levels and Go templates before writing rule files, performs atomic replacement under lock, and reverts on reload failure.

Deleted tenants left behind rule files, Grafana organizations, users and time series. Over several months the team accumulated 196 orphaned rule files, 705 phantom Grafana users and more than three thousand stale accounts visible in /admin/tenants. A single cleanup module now removes every external artifact when a tenant is deleted, and an hourly reconciliation job compares files, organizations and storage accounts against the database, failing CI if any residue remains.

Resource isolation remains incomplete. All tenants share memory, CPU and disk. Heavy parallel queries from one dashboard or high-cardinality series from a single Kubernetes cluster can still trigger OOM events that affect every tenant. Per-container memory limits, reduced cache sizes in VictoriaLogs and request queuing mitigate but do not eliminate the shared-resource risk.