Imply Lumi Major Release Preview: Continuing the Journey Towards Decoupled Observability/SIEM

We are getting ready to introduce the next major expansion of Imply Lumi and the observability warehouse.

When we introduced the industry’s first observability warehouse, the goal was clear: decouple the observability stack by separating storage, compute, and access. Since then, we have worked closely with numerous enterprise customers (some examples here, here, and here) to bring that vision to life, making more observability data accessible while preserving existing workflows.

Splunk has been the first ecosystem we have deeply integrated with, enabling organizations to optimize and extend existing environments without disrupting the workflows they already rely on. This next phase of Lumi expands that vision even further.

Query Unstructured Logs Where They Live

Imply Lumi Loglake is the next major expansion of the observability warehouse.

It enables teams to query unstructured logs directly where they live in object storage.

As observability data volumes continue to grow, many organizations are rethinking the economics of traditional indexing architectures. At modern observability/SIEM scale, fully indexing everything has become economically impractical.

In response, teams are increasingly moving logs into object storage for cost and flexibility. But once the data is there, it is still difficult to use for observability/SIEM workloads. Complex data pipelines, predefined schemas, and rehydration workflows often get in the way of actually working with the data.

Loglake removes those barriers.

Instead of forcing teams to prepare and move data before they can work with it, Lumi can query unstructured logs directly in object storage without requiring a predefined schema. No pipelines to build. No data to rehydrate. No additional infrastructure to manage.

This is about making data in object storage immediately usable for real world observability workloads as simply and as cheaply as possible.

Expanding the Observability Ecosystem

Loglake is only one part of this next phase of Lumi. We are also expanding how the observability warehouse integrates across the broader observability ecosystem.

Most organizations are not looking to replace the tools they already rely on. They want a simpler way to make more of their observability data accessible across those environments.

That is why Lumi continues to expand support for the platforms teams already use for storage, visualization, and ingestion.

With Databricks, teams can query observability data directly where it lives in the lakehouse without requiring additional pipelines, transformations, or data movement.

With Grafana, teams can access the same observability data in Lumi directly from the dashboards they already use, without duplication or additional pipelines. You can learn more in our blog, Query Lumi from Grafana Now in Private Preview.

With expanded SPL support including search head SPL, teams can continue using the same Splunk queries, dashboards, and operational workflows while accessing far more data through Lumi.

The goal is not to force teams into a new ecosystem. It is to make observability data easier to access and work with across the environments teams already have.

Smarter Infrastructure Management for Modern Observability

Alongside new query capabilities and ecosystem integrations, this next phase of Lumi also introduces new approaches to managing observability infrastructure at scale.

Not all observability workloads behave the same way. Some observability workloads require fast, always on access for monitoring and detection. Others are bursty and investigative in nature, only queried during incidents, compliance requests, or historical analysis.

Lumi is designed to align infrastructure to those different access patterns.

With Virtual Tier and Elastic Compute, teams can keep data in lower cost storage while dynamically routing investigative workloads to elastic compute resources only when needed.

Real time observability workloads remain isolated from large scale historical investigations, allowing teams to scale each independently without forcing always on infrastructure across all data.

This architecture is already being used by customers like BTG Pactual to extend their Splunk environments with longer retention, broader visibility, and more scalable investigations.

To learn more, check out our BTG Pactual ebook:
BTG Pactual ebook

The result is a more decoupled observability architecture where investigative workloads can scale independently without forcing teams to pay for always on infrastructure across all data.

Continuing the Shift Toward Decoupled Observability/SIEM

Observability/SIEM architectures are continuing to shift toward more decoupled models, where storage, compute, and access can scale independently while preserving the workflows teams already rely on.

Over the next few weeks, we will share deeper looks at the capabilities coming to Lumi, including Loglake, ecosystem integrations, and new approaches to elastic observability infrastructure.

We will formally showcase these new capabilities at the Databricks Data + AI Summit in San Francisco (6/15), where we will be hosting demos, technical sessions, and live discussions about the future of decoupled observability architectures.

If you are attending the summit, stop by the Imply booth (#571) to see Lumi in action!