Guardrails Filter Tackles Complex LLM Streaming and Tool Call Challenges to Protect Sensitive Data

Developers at Cloud.ru have shared the technical challenges encountered while building Guardrails Filter, a dedicated protection layer that prevents personal data from reaching large language models. The filter scans incoming JSON requests, locates phone numbers, email addresses, passport numbers, SNILS and names using regular expressions, and replaces them with stable placeholders such as <PHONE_1> or <PERSON_1>.

A simple substitution approach quickly proved inadequate. When two different phone numbers appear in the same conversation, replacing both with the generic token <PHONE> makes it impossible to restore the correct values after the model replies. The team therefore maintains an internal mapping table that guarantees the same real value always receives the same placeholder throughout the dialogue history.

Because LLMs are stateless between requests, Guardrails Filter must re-process the entire conversation history on every turn, rebuild the mapping table, mask the data, and then restore original values once the response arrives. With ordinary non-streaming JSON responses the task is manageable, yet streaming via SSE introduces far greater complexity.

In streaming mode, responses arrive in small chunks that may split a placeholder across multiple events. The filter therefore buffers several events until enough characters accumulate to determine whether a complete placeholder is present. Once a placeholder is identified, the original value is substituted and a new chunk is emitted to the client. This buffering adds a small latency but prevents broken placeholders from reaching users.

Tool calling adds another layer of difficulty. Model-generated function calls may contain masked values inside JSON arguments that themselves are serialized inside another JSON object. The filter must fully reconstruct each tool-call argument, validate that all braces are closed, and only then perform demasking. Failure to do so can produce invalid JSON that breaks downstream agent pipelines.

Support for the Anthropic Messages API required a completely separate implementation because its event format differs significantly from OpenAI’s Chat Completions streaming. The Messages API uses explicit lifecycle events for tool calls, while Chat Completions delivers deltas inside larger JSON objects. Separate code paths and approximately 1,000 additional lines of tests were written to handle both providers consistently.

In total, the streaming logic alone grew to roughly 1,500 lines of code, backed by more than 3,000 lines of tests. The engineering effort focused on correctly identifying the end of reasoning blocks, the completion of tool-call arguments, and the safe delivery of any remaining buffered data once the model signals the end of a response.