Agents don’t query like your detections do

A security team of four has run a major SIEM since February. Nobody on the team writes SPL. Nobody opens the console or looks at the dashboards.

Every question goes through an AI agent, which writes the query, runs it, reads the result, and answers in plain language. The team is now asking why they pay for a SIEM whose interface, UEBA and SOAR they never touch.

They are early adopters, not outliers. The chat box is the least interesting part. Watch the query load instead.

The queries multiply, and stop being predictable

Your SIEM already answers mostly to machines: correlation searches fire on cron, dashboard panels refresh on a timer. On one production tenant ingesting more than 15 TB a day, we counted roughly 154,000 queries in one day, most of them scheduled detections returning nothing.

Agents multiply that load and change its shape. A detection that ran one fixed query now fires follow-ups whenever something looks off. Triage that was one analyst reading one screen fans out queries per alert. Tens of queries per investigation, times the alerts you triage daily, times every detection cycle: a mid-sized SOC reaches tens of thousands of machine-generated queries a day before a human opens a case.

The old machine load was tunable: a fixed query with a fixed lookback on a fixed cadence, and platforms spent years optimizing for it with summary indexes, accelerated data models and cached results. An agent brings none of it: every query is new text, the lookback follows the hypothesis, and each query depends on the last answer. Your platform is tuned for the known set, with a human handling whatever falls outside it. Agents make the outside the main event.

What that does to the architecture

What changes	What it demands of the data layer
One question fans out into dozens of queries, some parallel, some chained on the last answer	Concurrency that scales out, not a queue
Lookbacks follow the hypothesis, not a configured window	Long time ranges that don’t cost a fortune
Pivots land on fields nobody planned for	No schema, sort order or catalog decided up front
Following one user, session or process needs full-fidelity events, years back	Entity pivots that work across the whole retention window, not just the hot tier
An agent that times out concludes instead of waiting	Predictable latency, and aggregation at the source
Sweeps and investigations run hot for minutes, then go quiet for hours	Compute you stop paying for between questions

Older data is where the pivot dies. Full-fidelity events age out of the hot tier, and what remains splits two ways. The fast path is summarized: rollups and summary indexes keep long-window dashboards responsive, keep the trend, and lose the entity. The complete path is unreachable: raw events sit in frozen archives or S3 with every field intact, where searching them means a restore ticket, a different query language and meter, or a different system entirely running Athena- or Spark-shaped scans. Splunk’s own docs say it plainly: frozen data is not searchable until you thaw it.

An analyst knows the archive exists and files the ticket. An agent queries what is queryable, gets a partial answer, and concludes.

Latency stops being a comfort metric. A slow search annoys an analyst, who waits. An agent that times out concludes from whatever came back. Answer size fails the same way: with a finite context window, a query returning 400 MB of raw events is as useless as one that never returns. And little of this gets checked. Where validation exists, it confirms the query ran, not that the conclusion was right.

The new work arrives in bursts. The minute-cadence detection loop is continuous and predictable; always-warm compute suits it. What agents add is different: a nightly sweep across ninety days, a chain of follow-ups when a detection fires, an investigation that runs hot for twenty minutes and stops. Weld compute to storage and the nodes that answered a 03:00 sweep are still billing at 03:15, at 04:00, and every minute between. And those queries are written in the hope that they return nothing, which most of the time they do.

Always-on compute to run queries that correctly find nothing is the most expensive possible way to buy nothing.

In Splunk, that burst has two paths, and each one meters it. SmartStore keeps compute static: before an indexer can search older data, the cache manager must fetch bucket copies from remote storage into local cache; the search then runs on the same always-on indexer fleet. Customers tell us that is too slow for real investigation. Federated Search for S3 meters the questions instead: queries draw down a Data Scan Unit entitlement, one DSU per 10 TB scanned, sized against what you project to search over the year ahead. They are also written in SPL2, not the SPL your saved searches use. One path makes the agent wait. The other makes every question billable, so the sensible response is to throttle your agents. Which is the opposite of why you bought them.

Why Lumi handles this workload

The engine was built for machines, not consoles

Lumi is built on Apache Druid, an engine made for applications and APIs issuing high volumes of concurrent queries against high-cardinality data. Its documentation names the target: “highly-concurrent APIs that need fast aggregations.” A decade-old design goal, not an AI-era rewrite.

The published record:

Deployment	Concurrency	Scale
Pinterest	1,000+ QPS at p99 under 250 ms, 2,000+ nodes	over 1 PB
Confluent	250+ QPS, tiering cut p95 by 75%	3M+ events/sec ingest
Target	200 QPS peak, 4M queries/day	~3 trillion rows
eBay	100+ concurrent queries, no impact on ingest or latency	100k+ events/sec ingest
Netflix	—	10 trillion rows, 15M events/sec

What matters is how Druid gets those numbers. It never needs the query in advance: no summary index to build first, no data model to accelerate, no sort order that decides winners. Every query, including one nobody has run before, goes through the same machinery, with segments scanned in parallel across the cluster and a 500 ms target per segment. Caching helps the repeats; the floor under a brand-new query is the scan itself. That floor is what an agent needs, and it is the part most platforms bolt on afterwards.

 That buys you four things. A sprawling investigation can’t starve your detection cycle, because bursty work runs on its own compute pools. The dataset everyone wants at once answers from several nodes at once. When query load grows, you add compute; nothing gets redesigned. And agents get answers instead of exports: aggregation runs where the data lives, so kilobytes of result cross the wire instead of megabytes of logs.

Nothing has to be planned in advance

Lumi discovers your schema on write and persists it: no schema to declare, no partition key, no sort order, no Glue catalog. A source you connect this morning is queryable this morning, including the field nobody thought to model.

High cardinality is a design goal rather than a budget problem: Druid uses dictionary-encoded columns with compressed per-value bitmap indexes, a very different cost curve from indexing raw log text.

You stop paying for the silence

The detection loop stays on warm compute, where it belongs. Everything bursty runs on pools that start when a query arrives, pull what they need from object storage, answer, and shut down after an idle timeout. Billing follows the minute, and a nightly ninety-day sweep becomes a scheduling decision rather than a procurement cycle.

Cold start is the usual objection, and scheduled work never feels it: pools take tens of seconds to start, and a sweep firing at 03:00 has nobody waiting. Interactive work pays it once. Cache is shared across the pool, so when fifty agents fire on the same cron, the first pays the cold load and the rest arrive warm. Segments load at up to 1.3 GB/s per worker against a 1.8 GB/s hardware ceiling, decompressing and querying as they go.

Query cost tracks compute time, not questions asked or bytes scanned. Idle time costs nothing. In most deployments we see, ingest dominates the bill and querying is a small fraction.

Longer retention, lower cost

On that same tenant, 92% of queries looked back less than ten days, 95% less than thirty, and the remainder was almost entirely all-time. That is not a fact about security investigation. It is a habit, learned from a platform that kept thirty days. The thin tail is the part that grows when the caller never learned to flinch at a long time range.

Lumi makes the tail affordable. Managed data compresses roughly 6:1 against raw for sizing, and the same indexed segments sit on local disk or in object storage without changing the query. For archives you never ingested, point Lumi at the bucket: it builds a compressed, indexed artifact holding the raw data plus a full index in less space than the raw data gzipped, originals recoverable byte for byte. Standard SPL, no restore ticket, no second system. Data you dropped from the SIEM years ago becomes something your agent can ask about.

Your search head keeps working

Not every SOC is the four-person team in the opening. The more common shape is two hundred analysts fluent in SPL, with years of saved searches and correlation rules behind them, and a data layer has to serve both ends of that transition.

Lumi speaks SPL natively and federates transparently. Your console, saved searches and dashboards keep working, and nobody gets retrained. It also speaks SQL over JDBC and Spark SQL, and its MCP server gives agents a SQL surface directly. SQL is the language models generate best, so agents get their strongest language and analysts keep theirs. A pitch that starts with “move everything to SQL” is asking you to rebuild every detection and retrain every analyst around the newest, most replaceable part of your architecture.

BTG Pactual cut security data costs by more than 70% without replacing Splunk. Their analysts kept their search head, their queries and their dashboards. What changed was the layer underneath.

The bottom line

A data platform is not a SOC. Something still has to run your detections, raise the alert and drive the response: your SIEM, your SOAR, or code you maintain. The teams furthest down this road have replaced the interface, not the detection engine.

The number to manage is time to answer across your full retention window: how long a real question takes when it reaches back as far as your obligations do, not as far as your hot tier does. For most SIEMs that reads as infinity past the cache, because the honest answer is “we would have to rehydrate that.”

Agents will query far more of your security data than your analysts ever did, on a schedule rather than a shift pattern. Your data layer should answer the questions nobody planned for. It shouldn’t bill you for the quiet in between.

Attending Splunk .conf this year? Join us at the Imply Lumi Lounge on September 15 to hear how BTG Pactual cut security data costs by more than 70% without replacing Splunk. You can also explore live Lumi demos, talk with Imply’s technical experts, and enjoy complimentary food and drinks, all directly across from the Colorado Convention Center.

RSVP for the Imply Lumi Lounge