Imply Lumi Loglake Demo
In this demo, you'll see exactly how teams are using Lumi to do more with Splunk and modern observability stacks — without replatforming. We'll also give you a first look at Lumi Loglake — our newest feature...
Watch nowHey, folks. Thanks for joining today's webinar. My name is Matt Morrissey, and for the most part, I will be driving today's webinar. Later, I'll be joined by Peter Marshall and Timmy Fries, who will each walk through demos later in the session. So before we get going, quick housekeeping note. Please feel free to drop questions into the chat anytime. We will address them live or during the q and a at the end. So for today, we're going to focus on three things. One is revisit why Imply Lumi exists. Two, highlight what's materially stronger since our last update. And then three, show where we are continuing to invest. Now everything we do with Lumi is really centered around one core challenge. How do you retain more telemetry without your observability costs growing at the same rate? So let's get into it. Okay. Quick context on Imply. We were founded by the original creators of Apache Druid, which is a real time distributed database built for high concurrency analytics at scale. Across our customers, we consistently see sub one hundred millisecond performance at terabyte to petabyte scale. And because of that, we've had a we've had a front row seat to help companies manage streaming event data. And most importantly, where their observability architectures start to strain as volumes grow. So what we've seen is this. Most observability stacks follow a pretty simple model, ingest data, index it upfront, and keep it searchable at all times. Now that model is is powerful. It makes data immediately available. It creates fast interactive search, and it gives teams confidence in in what they're seeing. But that design choice does have consequences. When you index everything upfront, what you're essentially doing is committing infrastructure the moment data arrives. And as a result, cost, well, it scales directly with telemetry volume. And telemetry, it it doesn't just grow with the business. It's it's growing with machines. When you think about every microservice, every container, every every cloud integration, they're all generating more data by default. And because everything is indexed upfront, more data means more infrastructure, more indexers, more operational overhead. And eventually, that that math stops working for you. And when teams hit that point, they they don't immediately rearchitect. What they do is they they try to cope. And there are ways to reduce cost. You can prune data before it's indexed, or you can move high volume sources over to object storage. Now both approaches absolutely lower your steady state spend, but they they introduced a trade off. You are you're ultimately reducing your visibility because cold data isn't immediately searchable and that pruned data isn't there when you need it. Now these these aren't bad decisions. They're they're rational responses to economic pressure, but they don't address the underlying architectural constraints. So what's driving that constraint? What we've learned is this, observability stacks are running two fundamentally different workloads. You've got detection and you've got investigation. Detection, that's running twenty four by seven. It's very steady. It's predictable. It's low latency access to very recent data because you're monitoring what's happening in the moment right now. Investigations are are very different. They are they're triggered when a detection finds something. And now you need to explore. Why did this happen? Has it happened before? How far back does it go? These are bursty by nature. They often are are scanning months of historical telemetry. You've got multiple analysts jumping in to help, so your concurrency spikes. So bottom line is that these are completely different workload profiles, but most stacks most observability stacks force both through the same always on indexing model. And that's that's creating a lot of friction. So once you once you recognize that friction, the the question becomes how how do you remove it? The answer isn't pruning more data. It's it's separating workload behavior from the infrastructure design. And when detection and investigation are forced through the same the same execution model, your infrastructure is always going to be misaligned with one of them. You're either over provisions most of the time or you're under provisions when it matters when it matters most. Now decoupling changes that. Retention no longer dictates permanent infrastructure growth, and historical search no longer requires always on peak capacity. When you decouple, your detections get to remain stable and your investigations get to behave very differently. That's how you remove the cost performance trade off, not by choosing between cost and visibility, but by aligning infrastructure with workload behavior. And that's why we built implied Lumi. Lumi is a high performance, high efficiency observability warehouse for logs. Architecturally, Lumi decouples storage from compute so you can retain a lot more data without permanently expanding infrastructure. And it's designed to sit underneath the tools teams are already using. So, for example, from Splunk's point of view, Loomi simply looks like another federated search target. Splunk remains your home base, so same SPL, your same Splunk dashboards, your same workflows. Nothing changes for the end user, but what does change is how much data you can actually keep searchable. So I think the best way to explain this is to hand it over to Peter who is gonna walk through a short demo so you can so you can see Lumi in action. So, Peter, over to you. Thanks for that team. Great to meet you all. Welcome to the webinar. I'm Peter Marshall. I'm director of developer relations here at Inply. I'm gonna talk to you a little bit more about Lumi's approach to search and query handling, that thing that helps us search that data at very high speeds, and a little bit about the compression, that thing that's reducing the storage footprint because those two things together is what makes Loomi really cost effective. So my task here today is to translate all the good stuff you've been hearing about and about from my colleagues and to show you what that looks like in the real world. So let's get going. I'm gonna open a browser here, and let's log into a leading observability platform. Now note there are no add ons. There's no custom applications. We're just looking at standard features that come with this tool off the shelf. So in the background, my team has been preloading a demo dataset here. We've got some web data from a make believe ecommerce website. So let me answer the first question you might have, which is how easy is it to look at this data if I'm looking in Loomi? Well, it's very easy. We are dual loading this exact same dataset into Loomi. And to get to it, I'm gonna take exactly the same search command that I used to show you the local demo data. I'm just gonna make one small change. When I hit go, these two technologies are now working together. They're searching for the matching records and computing the final results from a copy of the data that I have in Lumi. And in the results, as you see here, I'm getting exactly the same raw data, the same fields, the same results because the commands I use, the language I use stays the same even though I'm now using Loomi. So let's now turn to that exciting advantage of Loomi, speed. To see that more clearly, let's get a bit more complicated. I'm gonna run this more complicated command on the local data. And meanwhile, let's open the same search. I'm gonna put this now so it goes and searches Lumi. This is exactly the kind of command that any of your users who use these tools day in, day out will be very familiar with. And look how much faster results came back when imply Loomi is involved. Meanwhile, that local search, let's have a look. Oh, yeah. That's still working all the way. So my team undertook comprehensive benchmark testing. We looked at a range of different searches and queries and observability tools consistently get at least four x performance boost when observability is stored in, and those answers are being computed in partnership with imply Lumi. That kind of speed really counts when you care about MTTR, when you're looking at speedy access to the latest data, when you're asking challenging questions, when you're trying to really dig into what you know about your environment. So how do we set this all up? It's really quite easy. Lumi presents itself as part of the infrastructure that these tools aren't used to. It's about maximizing fit, about minimizing friction when it comes to integration. That means in this main tool, all the searches, the dashboards, the alerts, everything benefits. But Lumi doesn't just make searching and querying faster. It's also about how stores the data. And as you can see here, this sample dataset was compressed by over ninety percent. Now that's really not unusual. Lumi's approach to intelligently compressing all sorts of observability data is what opens up options for rethinking about how and where you store your observability data. You might reconsider your retention policies and avoid that pain of rehydration and whether it's now possible to collect and search things like Kubernetes or VPC flow logs. Maybe today, you've been thinking that's gonna be way too expensive to store, let alone in terms of compute to try and search this stuff. Again, LUMI isn't about replacing what you have. It's about doing more with what you have. Implies that to fit really neatly into existing architecture, both for collecting data and for generating insights. You know, this gets to the art of how we are gonna deliver on what the market is calling decoupled observability. So with that in mind, let's look at some of these integrations. So this is the integrations page here inside Lumine. To load data into this environment, we actually use OpenTelemetry. If you're Splunk users, well, you can see there's HEC and s two s connectors That's really easy to configure, therefore, on your forwarders to push the data through into Lumi. And we also have a pull mechanism in here in the demo environment that picks up data automatically from s three buckets. So how about getting your data out? Well, starting here with Splunk, Lumi appears as a remote search head. So looking up to search the data that we have stored in Lumi as with all of our integrations, it's just copy and paste. We're building Lumi to fit neatly into your architecture. We also see here we've got Grafana. And in Grafana, Lumi appears as Loki. We've got a JDBC connected to BI tools. We have an MCP server. So I have a bit of fun with this. I've connected this demo environment to desktop. I have a play around with natural language search. I have a colleague who's using language chain modules to build a chatbot for anomaly detection. That's really So under the hood, in summary then, we've got Lumi's speed. We've got its compression. That's what's making querying and searching all this data really cost effective. Well, that decoupled approach to integration, that's what's opening up the possibility of fewer silos, a better collaboration between teams, more efficient workflows. You can preserve your workflows and use the tools that you already have. Just making small configuration changes, and you can do more thanks to that high compression, thanks to that really great performance. When you're ready to activate this decoupled observability architecture, you can then take advantage of those native integrations and open up your Lumi data to other tools as well. So I hope that was all helpful, useful, and interesting for you. With that, I'm gonna hand back over to the team. I'm gonna stay in the background here and answer any questions you might have. With that, back over to you. Okay. Thanks, Peter. Awesome as as usual. Now that we have finished, a refresher on Lumi, let's switch gears and talk about how how we're making it better, how we're making it stronger. I'm gonna talk about three areas. One is more control, two is faster search, and three is deeper integration with Splunk. Okay. First area we've strengthened is operational control. You now have more more precision over life cycle and ingestion integrity. So filter based deletion and time based retention policies. What these allow you to do is you decide what stays and what goes without having to do blunt pruning. We've also strengthened ingestion visibility, with new pipeline simulation previews and unparsable event surfacing. Bottom line here is that you're able to see issues before they impact production. So, again, this is about operational confidence, more control, more visibility. Okay. The second area we've strengthened is accelerating large historical investigations without impacting real time detection. Now as retention grows, teams need to backfill and load large datasets. We've optimized historical ingestion with partitioning and dedicated processing queues. So historical loads no longer competes with real time events. Ultimately, detections remain steady and historical search remains fast. Again, bottom line here is that this is this is about aligning infrastructure with your workload behavior. Okay. The third area we've strengthened is how seamlessly Lumi integrates with existing Splunk workflows. So as Peter talked about, the the goal isn't to introduce a new tool, a parallel tool. This is about allowing you to do more with your Splunk environment without changing how you work. So as shown on the slide, we've expanded SPL coverage and and deepened support for core Splunk capabilities. So that means the searches your teams already rely on can run directly against LumiData. You can run your existing Splunk dashboards against LumiData without rewriting them. And you can also use transparent federated search to analyze data across Splunk and Lumi together. And I think I think the best way to understand this is to to show it in action. Again, now I'm gonna turn it over to Timmy so you can see some of these features in action. Over to you, Timmy. Cool. Cool. Yeah. Thanks. So this is the sort of basic Splunk UI. Splunk users don't love. And up until this point, if you want to use Splunk to query data against movie, what you've done is created a federated provider and then a federated index top of that federated provider. And when issuing a search, you include this federated polling keyword. That basically tells Splunk, hey. Go look for the data in Lumi and, you know, return the results. So, you know, we'll we'll just enter a search here and return return with the results. This is all using Splunk's, what they call their standard mode, federated provider. And the one of the sort of hiccups that that comes along with this is you have to add this federated full in keyword before every query. So what we are rolling out is support for what's called Splunk's transparent node federated search. And what that does is it moves this need to specify the federated pull in. It's pretty similar in terms of setting things up where a user creates a federated provider in Splunk. So I'll show you I've already sort of done that. Basically, you go to Lumi. You get a username and password. You come back to Splunk, Enter that information and create a federated provider. This is the one I'm gonna be using. But you'll see the name of the provider's docs, data model dev, and in particular, it's called transparent rather than up until this point, you'd be using what's called. And so what this lets me do is issue a search, but I no longer have to use that that federated colon keyword. So this is, you know, the about the most basic search you can do. Index equals bane, and it's returning results. And you know that it's returning results from Lumi because this Splunk fender writer field is actually the exact same federated writer I created before. Docs, data model, dev. Splunk no longer looks for that federated full in keyword to know, hey. Let me reset to Lumi. Splunk just by default assume assumes every query needs to be sent to Lumi. We've added this added this thing called allowed indexes. So this is an example of in LUMI, I'm creating a key, which gives me, like, the username and password to create a federated provider. And in particular, you'll see this thing called allowed indexes where I can specify, okay, allow any any a query that has any index, can be processed by Lumi. Can do none, which essentially blocks all of them, or I can do, like, specific indexes. So in the case that I, in the example search I did earlier, I would have entered an allowed index of main. And so a query that says index equals to main is we're saying allowed to be processed, by Illumina, so we'll actually return results. Yeah. So that's sort of the way that we are taking advantage of transparent federated search to remove this, federated polling keyword. The other benefit of transparent federated search, and really this is actually sort of the core reason behind why we implemented it, is called knowledge bundle replication. So, basically, when you enable a transparent federated provider, Splunk does this thing called knowledge bundle replication. Splunk communicates with Lumi. Some objects that are included in the knowledge bundle that gets replicated are data models and lookups that were otherwise sort of unavailable to our users. Transparent mode federated search is sort of the means, the the mechanism by which we support data models, lookups, and sort of federated colon free searches. Yeah. I hope that all makes sense. There'll be there'll be docs on all this coming, but that's just sort of, like, a high level view of how this all connects. Awesome. Okay. Thank you, Timmy. Appreciate it. Okay. So everything we've discussed so far applies to what we call LumiCloud, our SaaS offering. But for many teams, many enterprise teams, SaaS isn't a technical decision. It's it's governance. So because of that, we're excited to introduce Imply Lumi Enterprise, our bring your own cloud deployment model. Lumi Enterprise runs entirely inside your AWS account. So your data stays in your s three buckets, encrypted with your KMS keys, and governed by your AIM policies. And most importantly, imply has zero I'm permissions inside your account. So nothing is pushed into your environment. Your infrastructure pulls only the approved updates. Your security team controls egress, and your compliance team can audit every interaction. So we're we're excited about this. Okay. Now that we have covered, what Lumi is and how we're making it stronger, let's transition a bit and talk about where we are continuing to invest. Okay. When we when we talk about continuing to remove, this cost performance trade off, this is what we mean. Every query enters the system the same way, but not every workload behaves the same way. Workload rules determine where that query should run. So if it's a real time detection workload, so think monitoring, dashboards, alerts, it runs on always on compute. So it's predictable. It's low latency. It's isolated from burst traffic. What this means ultimately is that the performance for your detections remain very much protected. Now if it's an investigation workload where you're doing, historical look backs, where you're running an audits, or you're doing a forensic review, it routes that very differently. Now your compute scales on demand, your ephemeral resources spin up, and all of that happens transparently with the same queries, same dashboards, and same alerts. The user experience doesn't change, but the infrastructure does. Now our continued investment is about strengthening this model, smarter routing, better isolation, and greater elasticity. So, ultimately, you can retain more data and align your infrastructure with your workload behavior without changing how your teams work. Okay. Now let's look at what this looks like in the real world. I'm gonna talk about a global investment bank and how they were working through how to put a cost structure in place that would keep their Splunk environment very much sustainable for the long haul. Okay. Like most large security organizations, they were running two fundamentally different workloads on a single platform. Detection was steady, continuous monitoring, predictable usage patterns, strict low latency requirements. But their investigations were were very different. They were bursty. They required high concurrency with a lot of different analysts. You're doing multi month look backs. You're you're doing audit and forensic reviews. Their architecture again treated them the same, but the workloads behaved very differently. They looked for ways to resolve this, alleviate this, but everywhere they turn, they've you know, they forced or they faced some sort of trade off. If they size the infrastructure to handle investigative peaks, they overpaid most of the time. If they size for a steady state detection, then their investigative performance took a hit during spikes. If they pruned that data, they lost visibility. They moved it to cold storage, their searches lost their interactivity. Now instead of replacing Splunk, the bank adjusted the architecture. Splunk remained in place, so their dashboard stayed exactly as they were, but they introduced Lumi as a decoupled data layer. That allowed them to retain an additional five terabytes per day of security telemetry that now this is data that previously was not stored in Splunk at all. Now that telemetry, this five terabytes is now fully queryable through their existing Splunk workflows. So, ultimately, the team gains visibility into more data without expanding their Splunk infrastructure. Investigative workloads no longer dictated permanent infrastructure sizing, and select detection workloads could run on Lumi where where it made sense. So overall, they saw really, really strong results by deploying implied Lumi over a seventy percent reduction in Splunk related observability costs. And, again, they were able to add an additional five terabytes per day and keep that data fully queryable within their Splunk environment. Ultimately, because of this, they were able to expand visibility inside their existing Splunk dashboards while putting the right cost structure in place to to keep Splunk sustainable moving forward for them. Okay. Before we close, I I wanted to pause and open it up for questions. If anything came up during the demo or earlier in the session, again, drop your questions into the chat. While those come in, I'll start with one that I that I hear frequently. I get asked if, if we're already using SmartStore to offload data to s three, how is Loomi, different from that? And the answer is that SmartStore improves storage economics by moving data to object storage. Absolutely. But Loomi goes further. It it changes how search executes. So what I mean by that is for investigative workloads, Loomi does two things. First, it queries the data directly in place on object storage. And second, it can scale compute independently so large investigations don't require permanently expanding infrastructure. Ultimately, Lumi allows you to to to have detection and investigation run separate and don't compete for the same resources. So it's not just cheaper storage. With Loomi, it's a different execution model for search. Okay. So, if there aren't any other questions, I think we'll leave it there. Again, I wanna say thanks on behalf of myself as well as Peter and Timmy. Thank you everyone for joining today. If this conversation, sparked some interest, if it resonated with you in terms of the pressures you're seeing around retention, cost, or performance, we'd love to continue continue to talk. So please feel free to reach out directly or or connect with us, however you may say choose. Again, really appreciate your time, and, have a great rest of your day. Thank you again.
Observability teams must retain more data, investigate faster, and control costs without disrupting existing tools. This live Imply Lumi update shows new ingestion, retention, search, Splunk interoperability, and a demo of investigations on large historical datasets.
Imply Lumi Loglake Demo
In this demo, you'll see exactly how teams are using Lumi to do more with Splunk and modern observability stacks — without replatforming. We'll also give you a first look at Lumi Loglake — our newest feature...
Watch nowImply Lumi Observability Warehouse Demo
In this 30-minute session, you'll see a demo of Imply Lumi — the observability data layer built to help you store more, search faster, and reduce cost without changing your existing tools.
Watch nowLunch & Learn: Imply Lumi Observability Warehouse Demo
In this 30-minute session, you'll see a demo of Imply Lumi — the observability data layer built to help you store more, search faster, and reduce cost without changing your existing tools.
Watch now