Habr•October 26, 2026•🇷🇺Translated from Russian

Multi-Tenancy Isolation in VictoriaMetrics, VictoriaLogs and Grafana: Building Client Separation in a Shared Stack

Voltir provides monitoring for servers, 1C systems and databases as both a cloud service and on-premise installation. In the cloud all clients share one stack containing vmstorage, vmselect, vminsert, VictoriaLogs, Grafana, vmalert, Alertmanager and Postgres. Separate stacks per customer would require ten full copies of caches and ten update cycles per release, so the company keeps a single stack and implements isolation manually at each layer.

Every tenant receives a numeric account_id. VictoriaMetrics routes writes and reads through paths such as /insert/<account>/… and /select/<account>/…, while VictoriaLogs and VictoriaTraces read the account from the AccountID header. Nginx sits in front and calls the platform API via auth_request to determine the tenant from the Bearer token before forwarding requests. Storage endpoints are never exposed externally.

Each tenant has its own Grafana organization whose data sources are created by the platform with the account ID embedded in the URL or secureJsonData header. Users receive at most the Editor role; Org Admin is deliberately withheld because an administrator could change the data-source URL or header and read another tenant’s metrics or logs. The platform explicitly checks role values and rejects anything higher than Editor for tenant users.

Write authorization compares the account supplied in the URL with the account belonging to the token; a mismatch returns 403. For logs and traces the platform returns the correct account in the auth_request response and nginx overwrites any client-supplied header. VictoriaLogs only understands the AccountID header and silently ignores X-Tenant-ID, writing everything to account 0 if the header is missing.

A single vmalert instance reads all rule files. Each tenant’s rules file contains groups narrowed by an extra_filters clause containing the tenant’s account ID. VictoriaMetrics appends this filter to every selector, preventing a rule from seeing data outside its tenant. Global rules without filters were removed after one self-test rule began scanning thousands of deleted tenants and consuming excessive CPU.

During testing a rule containing the Russian string “пять минут” instead of a valid duration value caused vmalert to reject the entire rules directory on reload. Subsequent changes to other tenants stopped being applied, and a container restart would have left alerting completely disabled. The platform now validates durations, severity levels and Go templates before writing rule files, performs atomic replacement under lock, and reverts on reload failure.

Deleted tenants left behind rule files, Grafana organizations, users and time series. Over several months the team accumulated 196 orphaned rule files, 705 phantom Grafana users and more than three thousand stale accounts visible in /admin/tenants. A single cleanup module now removes every external artifact when a tenant is deleted, and an hourly reconciliation job compares files, organizations and storage accounts against the database, failing CI if any residue remains.

Resource isolation remains incomplete. All tenants share memory, CPU and disk. Heavy parallel queries from one dashboard or high-cardinality series from a single Kubernetes cluster can still trigger OOM events that affect every tenant. Per-container memory limits, reduced cache sizes in VictoriaLogs and request queuing mitigate but do not eliminate the shared-resource risk.

Related articles

AntiMalware•Other

Yandex Tests Compass Device for Taxi Navigation During GPS Outages

Yandex is conducting trials of a compact Compass device in 10,000 taxis connected to its service to maintain accurate vehicle positioning when GPS signals are lost or jammed. The device connects to the OBD diagnostic port and pulls wheel speed data via the CAN bus while using its built-in gyroscope and accelerometer to determine direction of travel. These readings are sent over Bluetooth to the Yandex Pro driver app, which combines them with the last known GPS coordinates and frequent Wi-Fi access point scans to update the map in real time. The initiative addresses ongoing navigation problems that have affected Russian cities since spring 2022 and have been compounded by regular mobile internet outages since spring 2025. Earlier, Yandex tested a similar OBD-connected gadget called Yandex Pathfinder in 2023. The company has not disclosed timelines for completing the current trial or rolling the solution out more widely, though it plans to make Compass available to ordinary drivers later. Yandex navigation apps have also offered an offline step-by-step routing mode since 2025.

Habr•Other

Apple Attributes Widespread Service Outage to Faulty Internal IT Update

Apple has confirmed that the large-scale outage affecting its systems and user services on November 17, 2026, was caused by an incorrect internal update within the company's IT infrastructure. The incident impacted multiple consumer-facing services including the App Store, iCloud, Apple Pay, and Apple ID authentication. Users reported inability to download applications, sign in to accounts, or activate devices during the disruption. Internal corporate tools such as AppleWeb, AppleConnect, employee VPN access, corporate calendars, and the internal incident-tracking site were also taken offline. Retail operations in Apple Stores worldwide experienced interruptions to sales, repairs, and customer service processes. Company engineers restored most systems within four hours and stated they were working to return remaining services to normal operation as quickly as possible. Apple explicitly ruled out any external cyberattack or third-party interference as the cause of the outage.

Habr•Other

Global Outage Disrupts Apple Services Including App Store, iCloud and Apple Pay

On November 17, 2026, widespread disruptions affected multiple Apple services worldwide, including the App Store, iCloud, Apple Pay, and Apple ID authentication systems. The incident impacted users attempting to download applications on iPhone devices, sign into accounts, or activate new hardware. Apple confirmed problems with Apple Music and App Store functionality, along with performance degradation or complete unavailability of iCloud for all users. The outage was explicitly stated as unrelated to any country-level internet traffic filtering or restrictions. Service interruptions prevented normal operation of cloud storage, music streaming, and payment processing features tied to Apple ID. No official timeline for full restoration was provided in the initial reports from the company.

Habr•Other

Forbidden Fruit: Recovering Delisted iOS Apps Wirelessly via Reverse-Engineered Apple Protocols

A detailed technical investigation demonstrates how to restore delisted applications to iOS and iPadOS devices directly on the target hardware without cables, computers, or Apple-signed developer certificates. Researchers performed extensive reverse engineering of Apple binaries including Configurator, CoreADI, AuthKit, AppleMediaServices, and legacy FairPlay components to replicate cryptographic device provisioning and authentication flows in pure Go. The work reconstructs ADI provisioning for both IdMS and AMD states, SRP-based GrandSlam authentication, SAP envelope signing, and white-box AES transformations previously executed only inside Apple binaries. All steps were validated on iOS 17.3.1 through 18.2.1 and iPadOS 18.7.1, enabling a single-device workflow that obtains purchase history, downloads .ipa files, and installs them via a custom server recognized by the system installer. The resulting implementation eliminates dependencies on macOS drivers, iTunes, Unicorn emulation, and closed-source libraries while remaining cross-platform.