Building a Cybersecurity News Aggregator: Story Clustering, Seven Importance Signals and Strict Filtering Thresholds
The idea for an information security news aggregator emerged years ago with the goal of delivering only the most relevant stories while eliminating noise. Early versions relied on the rut5_base_sum_gazeta summarization model and TextRank for importance scoring, but performance remained unsatisfactory. The current system processes roughly one thousand items daily from more than 200 sources and publishes only about 0.5 percent of them. A hybrid approach now combines a transparent seven-feature formula that scores story importance with an LLM that removes irrelevant content and generates concise summaries. The formula incorporates signals such as confirmation count, CVSS severity, exploitation status from KEV, source tier, proximity to Russian readers, depth of analysis, and publication speed. Penalties are applied for vendor self-promotion, webinars, and job postings. Stories are clustered using CVE identifiers, shared trigrams, and named entities to avoid duplicate posts while preserving original reporting.
HabrSep 2, 2026Translated from Russian