LangGraph Architecture Combines Hybrid RAG with YARA and Sigma Engines for Streaming Log Analysis

A research team has developed an architectural pattern titled LangGraph, hybrid RAG + Signature Engine that creates a universal graph for processing streaming data, demonstrated on cybersecurity logs but designed to be modular for other text analysis tasks.

Architecture Overview

Logs are ingested by Vector with a Lua chunker producing 250-line segments and 20-line overlap, then published to a Kafka topic. An aiokafka consumer with retry logic and dead-letter queue forwards batches to an 8-node LangGraph DAG whose results are persisted in PostgreSQL.

The graph contains two parallel branches. The AI branch routes filtered logs through Agent 1 for event grouping and anomaly detection, Description Agent for natural-language summaries, and Agent 2 for hybrid RAG retrieval against MITRE ATT&CK. The deterministic branch parses logs and runs them through custom YARA and Sigma engines. All outputs converge at Agent 3, which produces the final report, validates hypotheses, and flags events requiring manual review.

Hybrid RAG Component

Agent 2 improves queries with an LLM, executes hybrid search combining vector similarity (weight 0.6) and BM25 (weight 0.4) over a Russian-translated MITRE ATT&CK collection in ChromaDB, then applies LLM re-ranking on the top-3 candidates. This approach reduces token usage while maintaining classification accuracy for techniques such as T1110 and T1190.

Deterministic Engines and Rule Generation

The YARA engine compiles .yar files and matches concatenated log fields using the native yara-python library. The Sigma engine parses YAML rules into selection conditions evaluated with contains, startswith, endswith, and regex operators. When RAG identifies an uncovered technique, Agent 3 triggers an LLM-based YARA rule generator that iterates up to three times with validation before storing the rule in a pending queue for analyst approval.

Performance and Metrics

On a 43-minute synthetic dataset of approximately 850 lines containing 38 ground-truth MITRE ATT&CK techniques, the pipeline recorded 85.7% precision, 78.9% recall, and an F1 score of 0.82 while processing 3.5 lines per second. The system was developed as part of a project practicum at UrFU and tested with Gemini 2.5 Flash.