Lumi Virtual Tier is generally available today on Lumi Cloud and Lumi Enterprise. Customers are already using it to keep historical data searchable without always-on infrastructure. Compute spins up when a query arrives and shuts down when the work is done, so historical searches never compete with the compute powering dashboards and detections.
We previewed Virtual Tier earlier this year and outlined the broader approach in May: align infrastructure to each workload, with always-on compute for detections and dashboards and on-demand compute for investigations.
This post is the practical follow-up: what you can use now and what’s coming next.
Why we built it
Every SIEM platform lets you tier data. The problem is where tiering ends up. Data ages out of the hot tier, moves through progressively slower storage, and eventually lands in an archive that has to be restored or rehydrated before anyone can search it. Additionally, historical scans can create resource contention: investigative queries — the breach sweep, the audit pull, and the “what changed last quarter” questions — compete with dashboards and detections when they run on the same compute.
The result is a familiar trade-off. Pay hot-tier prices for data you rarely query, accept the delay of restoring it from an archive, or cut retention and hope you never need to query what you cut.
Virtual Tier is how Lumi gets you to “store everything, query what matters.” Keep your most-queried data on Hot Tier for interactive performance. Retain everything else in object storage, fully queryable, on demand, at commodity-storage prices. You can retain historical security data for investigations without keeping compute running around the clock or taking resources away from dashboards and detections.
How it works
Virtual Tier has two parts.
Virtual Storage. Tiering rules define what stays in Hot Storage. Everything else remains in object storage and is loaded into a transient cache only when a query touches it, then is evicted when it’s no longer needed. There is no second copy of your data and no rehydration job.
Virtual Compute. Isolated, on-demand compute pools serve queries against Virtual Storage. You pick a t-shirt size and a maximum idle time. The pool starts when a query arrives, pulls what it needs from object storage, returns results, and shuts down after the idle window. You pay only while it’s running.
Routing is automatic. If a query’s entire time range can be served from Hot Storage, Lumi runs it on Hot Compute. Otherwise, the entire query runs on a Virtual Compute Pool, even when it spans both recent and historical data. Queries are never split across the two, so a historical deep-dive won’t compete for the resources your real-time monitoring depends on.
Nothing about your workflow changes. You use the same query languages and tools across both tiers, whether you’re running SPL from Splunk, LogQL from Grafana, or an MCP-connected AI agent.
With both Virtual Tier and Loglake now generally available on the same on-demand compute, Lumi has one storage engine supporting three different workflows: persistent compute for detections and dashboards, on-demand compute for investigation over ingested data, and in-place query for logs that never left your data lake.
What’s available now
Here is what you can use today:
- Tiering rules. Set how long data stays in Hot Tier. Data past the window stays queryable through Virtual Storage.
- Configurable Virtual Compute Pools. Choose a t-shirt size based on workload requirements. Set an idle time anywhere from 15 minutes to 3 days. We recommend starting with Large and tuning based on your workload.
- Multiple pools per account. Each pool processes queries in isolation and can be sized independently. For example, you can give the SOC’s incident-investigation traffic its own pool, separate from compliance reporting.
- Pool routing by IAM key. Map Federated Search keys to specific pools, so each team’s searches land on the compute you sized for them.
- Guardrails against runaway scans. Restrict Virtual Tier access per IAM key to prevent accidental historical searches from spinning up Virtual Compute. Configure query timeouts per key to accommodate long-running investigation workloads, and use admin permissions to control who can manage tiering rules and Virtual Compute.
- Usage and billing visibility. Storage and search metrics show how much data sits in each tier and how well each tier is utilized. The billing page breaks out search and retention consumption by Hot Tier and Virtual Tier.
- Custom sizes on Lumi Enterprise. Define your own t-shirt sizes in Terraform, mapping each size to the instance types and node counts that fit your environment.
- Multiple clusters per pool. A pool scales out to additional clusters as query load grows and scales back in when it drops. Max concurrency stops being a fixed ceiling per pool, and you get finer price-performance control without over-provisioning a larger size.
What’s coming next
We’re continuing to add control over the cost and performance of Virtual Compute. Coming soon is support for multiple instance families. Today all default t-shirt sizes use the same ratio of compute to storage. We’re adding families that balance the two differently, so you can pick a storage-heavy family for large, infrequent scans at the lowest cost, or a compute-heavy one when concurrency and latency matter more.
Getting started
Virtual Tier is generally available and on by default for new Lumi Cloud and Lumi Enterprise customers. If you’re an existing customer, contact your account team or Imply support representative to enable it. The docs walk through creating your first pool and tiering rule.
If you’re new to Lumi, request a demo.