to a city near you
Denver was just the beginning. Be the first to know when Lumi Lounge opens near you — plus get access to every resource and session on-demand.
Victória Mai · Cyber Security Engineer, BTG Pactual
Watch now
I would talk about what happens to a SIEM in an AgenTex sock. It's an AI sock talk. I'm going to try to make this one a little bit different. We're not selling agents. We are not selling a platform to run agents. We're selling a data layer that makes agents work really, really well. So how do you build an AI SOC? What does it look like? I think what it looks like, humans and agents working side by side. This is not an idea we came up with. There are some people that say they're gonna automate everything. I don't buy it, I don't know, maybe a few years from now. I think humans and agents working side by side, just like we do it in software engineering, is the way it's gonna work. So in software engineering, humans write code, agents write code, humans review code, agents review code, humans push things, agents push things. Humans and agents essentially do the same work. I think it actually makes sense to think of it in the same way. Humans and agents build detections, humans and agents hunt for threats. Probably the first investigation is, always done by an agent just because they're so fast at doing the first investigation, but then, agents will close out some investigations on their own, humans will close out others. There's gonna be a lot of sort of side by side work, work, work. But what data layer do agents need to work well? They don't query like detections do. Detections are tending to look at recent data with a known look back. They're a known quantity. They run on hot compute. You can optimize in advance. They're very, very regular. That is a workload that is easy to optimize for. Data infrastructure guy like me, that's a boring workload. The Asian workload is an exciting workload to optimize for. It's exciting for me and maybe terrifying for, you know, if you're not thinking about it all day. There's a stream of new queries, dynamic look back. Each query depends on the answer to the last one. Each query brings up new questions. All the evidence. You can't optimize this in advance. Reason this happens is because agents thrive on data and context. They're always trying to build more context. There's a detection that fires that says maybe there's a malicious login. And, you know, the detection system is very basic. It's doing very basic queries. It expects an investigator to figure out if this is really happening or not. And what does an agentic investigator do? It looks at the user, it looks at the IP, it looks at the device, it looks at the history, it looks at things over weeks, months, maybe even years, you know, it'll take as much data as you can feed it. They follow the evidence. But humans make do without it. Humans have experience, humans have intuition that guide them when data is not available, when content is not available. Agents, at least today's agents, don't have a thing. If they don't have data, they don't make good decisions. So how do we size a data layer for workloads that may come where a simple detection, like maybe a malicious happened, leads to queries, you know, filtering on the same device, the same music, same IP going back years. It's a very spiky workload and we need a data layer that matches to that very spiky workload. Splunk has some products that play in this area that attempt to make workloads on long range archival data work a little better. One is called Splunk SmartStore. We store index data and object storage. There's a problem with Splunk Smart Storage, which is that the compute is not particularly elastic. In Splunk Smart store, we are, taking older data and we are storing it on S3 and as you query it, we pull it back under the indexers, but it's the same pool of indexers. And so what that means is that you're either sizing for peak compute in which you're paying way too much money, or you're sizing for, trough, in which case when you do have a peak compute need, you're getting very bad performance. And there's Splunk Federated Search for S3, which is more dynamic. The compute does scale up and down. They cover for that by charging you for the amount quarried. But the issue with that is that it's a system that is designed for doing a small handful of queries, not a ton of queries. There's not good caching of the data. We send the query out to S3, we get the response back. My colleague, Eric was, telling us that we don't want to just query the same 100 terabytes of data over and over and over again out of S3, we want to build indexes as we query it, And then we want to hit those indexes the next time. We're not doing that with federated search for S3. So we need a different data layer. And you may have noticed by now, I'm calling this an AI talk. It's only kind of an AI talk. It's really a talk about how an awesome data layer is great. It's motivated by the fact that I think the need of agents for more and more context and data makes a great data layer more critical. You can't just but humans appreciate it. You know, humans would love this data if they could get it. You wanna be able to query without planning ahead. You wanna be able to scale, compute elastically. You wanna be able to search the full history without moving data first. So Loomi, as Eric was saying, as my colleague was saying, delivers this architecture. You can query without planning ahead. There's no need for summary indexes, predefined schemas, none of that. You can scale for bursts with decoupled virtual compute, and you can search on the full history in object storage. We can build indexes the first time we search it, and then we can leverage those indexes for follow-up searches on the same data. Loomi, as you heard from Eric, is a built it's a product built on the Apache JIRA technology, which you've been very familiar with and imply for many years. This version of the logo slide, the scale slide, these are scales that people are operating at with Apache Druid, in the Apache Druid community. Thousands of QPS, millions of queries a day, trillions of rows. So solid technology that we've been using in the columnar data warehouse space for some time, and we are now bringing to the logging space. A little bit about how the compute works. I mentioned you wanna have compute that scales to your workload. Our model is to have two pools of compute. One that we call a hot compute pool, that's designed to handle your detections, your always on searches. Generally, you would size that hot compute pool to locally cache the last, let's say thirty days of data, and to store that cache at all times to make sure queries on the last thirty days are always super fast at all times. And then you have Elastic Compute for investigations. That's the one that spins up on demand. When it spins up, doesn't take long, takes a couple minutes. If the data had been indexed, we just load the indexes and start querying them. If it hadn't been indexed, we index it on demand, save them for next time, use those. With Elastic Compute, once it's warmed up, it's just as fast as you're still getting, you know, searches, amazing real time level performance. The only, downside is the spin up time. But what you get from that is, it's not running when you're not using it. So you get all the, you get, you get a really great cost profile and the spin up time is not that bad. We're not talking days to rehydrate. We're talking minutes to turn on compute. You can keep Splunk and you can change the architecture underneath. BLUME is a shared security data layer. It's a multi language data layer. We speak SPL and can act as a Splunk indexer. We also speak SQL, we speak LogQL, so you can point Grafana at us. So it's a multi language system that, works with Splunk and other stuff. Bottom line, agents and humans, but you know, this is an AI talk, so agents will ask questions nobody planned for, they're gonna go across fields you haven't added to a data model, and your data architecture determines what questions you can ask during an investigation. And ideally, you'd wanna be able to ask any question. So I'm gonna do a little demo of, I'll describe what this is. I said we're not shipping an AI sock, we're not selling an AI sock. This is an AI sock, but it's not a product of ours. I wanted to show, we wanted to show how straightforward it is to build authentic stuff, when you have a nice data layer. My colleague Vadim, vibe coded this AI sock in one day with Claude code, that day was, this past Sunday. So this is an AI sock that has only been existence on this, on this earth for two days. And, you know, it works all right. You know, there's some detections, detections are real detections. They're actually happening on real data. And there's a real queue of investigations that this queue of unresolved detections that real agents are actually picking up and investigating. And what they do is, let me look at one that has resolved. You know, what they do is the detection fires. This is a query. I said we're a multi language system, SPL, SQL, LogQL. This is using SQL. So the detection fires, little table, then an agent starts investigating it. You get a little thought chain where it says, you know, little cute agent stuff. I wanna run a few more queries to firm things up. It's like, alright, well, great. I mean, these queries take two hundred milliseconds, two hundred milliseconds, and this is what you want. This is how long queries take once the compute layer is warmed up. If the compute layer wasn't warmed up when it started, if we were looking at data from six months ago, what you would see is one of these queries take about five minutes as the compute layer warmed up and then query same data until, you know, throughout the investigation, each subsequent query would take this two hundred milliseconds, three hundred milliseconds. So this whole investigation takes a few minutes and then it writes up a nice report. And like I said, the agent that does this is, you know, vibe coded by, by one guy in one day. So it's pretty basic, but, you know, it's honestly kind of a good start. I mean, these models are pretty good. And I think the one thing you gotta ask yourself is these AI sock companies, you know, how much of what they're doing is their own proprietary intelligence and how much is just like, yeah, models are pretty good. You know, Claude's pretty smart, ChatGPT is pretty smart. Try their own AI sock and find out for yourself. Thank you.
The Agentic SIEM: What comes next for security analytics
Gian Merlino · Co-Founder & CTO, Imply
Watch now
We'll start with Lumi. So Lumi, a little bit of background for us as a company. Imply has been around over ten years. We, started this, open source product called Apache Druid. We have, funding from VCs and all of this stuff, but we've introduced this Lumi product. It was initially announced at dot conf last year, and we're here to talk about kind of more of what we've done. So, what is Loomi? Loomi, you can think of it as the next generation engine for your SIEM. And when you think about your SIEM, you can think about it as multiple layers. You might be using Splunk Enterprise Security SIEM. You might be other SIMs. You might have some people in your company also that are using Kibana for some things, and using Enterprise Security for other things. And maybe you have some people using Datalog for some logs, And some people using CrowdStrike for stuff. Or maybe you don't have all of those things. But, generally speaking, there's a lot of different ways that people interact with data up at the interaction layer. There's a lot of different things that people can use up at the interaction layer. But your data layer is really what provides the data. It's what allows you to query. It's what allows you to actually interact with and investigate when something goes wrong in your environment and see what's going on. And that is where Imply Lumi sits. We are a data lake for your logs that allows you to integrate with a number of different services. But we're here, .conf over there, so we will talk about ImplyLumi for Splunk. How does it fit into the Splunk system? Basically, you can think of ImplyLumi as sitting alongside your Splunk cluster as another indexer, or the technically correct thing is that we exist as another federated Splunk cluster. To the Splunk search head, we look like a federated cluster, and you can just issue queries as if you were federating to another Splunk cluster. This is Imply Loomi. This is our integrations page. You can see a bunch of a bunch of stuff here, but there's this Splunk Federated Search that you can integrate with. Now, we have some data in this environment. Go to this Explore page. You can see there is data here. Now, I can also go to this Splunk search head and we have a federated provider. I'm issuing a Splunk query. This is SPL. We've got data. When we look over on the side, you can see the Splunk FederatedProvider is pulling data from this demo environment. Now, this is a pretty simple SPL statement, just index equals. You might do something where you want to do a wild card search for... How about platform beds? Why not? Platform. Let's do a wild card search for platform. You do a search, you get some results back. You can do your normal SPL popping in here and doing stats. Maybe you wanna do stats count, right? Now, I'm only looking at the last sixty minutes. It's not really that interesting. Let's pull it back and look at seven days. Okay. For this stats count, there were 56,091 things. Maybe you want to do something a bit more interesting. Look at the average bytes in, bytes out, distinct count, everything. It's just normal stats. You can do evals. I'll show you later some other things. But this is what I'm talking about. It's just normal Splunk. You're used to using search and reporting. You're used to typing in queries. That's it. You connect it up. You query. It's really no more difficult than that. What does that mean? If it just works with Splunk, all your dashboards, your artifacts in Splunk, they just keep working. You have a dashboard written in SPL with doing a bunch of queries. Great. That just works. Continue using it. Here's the dashboard. Right now, if I filter for everything, this is now loading the dashboard including data from both Splunk and Lumi. So there's some data in the Splunk indexers. There's some data in Lumi. It's loading it from both. You notice this sessions thing takes quite a long time. Annoying. Look at just the Lumi data. We look at just the Lumi data, it loads up and it's there quite a bit faster. Much quicker. So why use Lumi? What is the beneficial? One, we have our own compression technology that we built up specifically for compressing logs. And so it's a domain specific compression focused on logs that allows us to shrink the size of the data, saving bytes on disk, saving storage capacity, all of that. Next one, we have, on top of that compression format, our own indexing structures. And so, we've built I mentioned the company's been around for a long time. We've been working with data indexing in the column oriented database world for a very long time. And we have leveraged all of that indexing technology for the logs world in order to provide an extremely compressed log storage or index storage format that also optimizes analytical queries and makes them faster. And so, it speeds up queries and allows things to run. Now, these are much larger, more complex SPL statements. For those of you who can't see it, there's some RECs, there's some EVALs, there's STAT, some more EVALs. And on the left side, we have the data sitting in the Splunk indexers. On the right side, we have the same data, the same query, and everything. The only difference is that in the index name here, there is this word LUMI. And that is causing it to issue the query to LUMI. The data for this index is in LUMI versus the other. Now, let me hit this search button. It started. It's off to the races. We hit the search button and Loomy's done. Loomy's done but Splunk is still going. Still going. It got there. Okay. There's a difference, easily measured on a relatively complex query. Let's go to another one. You can guess that the conclusion of this is basically the same. You would be guessing correct. Complex SPL again. Here we go. Hit my search. It's off. Let's come over here. Let's hit my search. The Splunk one is still going. You can see that blue bar going. Oh, Lumi's done. The Splunk one is still going. Still going. Still going. Okay. Now, it's there. Same data, same setup. The comparison in queries is pretty much night and day. So we have improved search performance, especially for these kind of analytical style queries. And then there's another feature that we've added, which is called Log Lake. And this has been our big announcement this year. But it's connecting the dots on basically searching logs in place, searching it where they are. What we've seen is a lot of organizations, there's data sitting out there in S3 and a lot of times, the security team doesn't own that data. It's owned by some other team. Some other team controls where it goes, why it's there, how long it's there, but the security team needs it. And for that, you really want to just point at the data, make it queryable, make it quick. And so, there, we have introduced Ogg Lake. Okay. On the Loomi UI, we have this stores thing. And I'm gonna point you at this comma set. This is where we've set up a thing pointing at data out in s three. We've got some locations of some data sitting out in s three that is then queryable. And I have this set up. I have it set up as this federated comma set, data set. So this is using standard mode federated provider. So that's why it has the federated there. We support both transparent and standard mode, either one. For the demo, it's easier to do this standard. So we have that. I have this. I push my, query. Oh, wait. Let me do this as the head 100. Let's look at the data real quick. This data has a whole bunch of stuff. It's pretty complex nested JSON. There is this source dot process ID field. And let's just do a stats count on this data real quick. There's two zero two million lines here, as you can see from this result. 200 and 2000300 and 4700 and 93. And let's break that out by source dot process ID right here. Break that out. We got our results across the 200,000,000. Actually, we really want to have that sorted by count descending. So, let's do that. This is just your normal SPL running against where this was pointed at that data out in S3. It's running against this. The speed that's coming, this is the same speed that we had from the normal queries that are coming in. That's because of our indexing structures. Even when you point us at S3, when we go and process that data, we generate the indexing structures in order to allow for that next query and the next query and the next query to go faster. If you're constantly querying one terabyte, you're not paying a terabyte a query. You pay that up front once and then you're just getting speed, speed, speed, speed, speed. The data is just out there in S3. That's the only storage for it. Maybe a little snarky, but we believe that this is federated search done right. This is what federated search is supposed to be and what the promise of federated search has been for a long time. We believe that that is what we brought. And then, there's another thing because, you know, everyone loves AI. All the people are AI. AI is taking over the world. There's connectivity with AI here, but instead of going into this deeply so just a little bit of a summary, bringing it back. The the setup is simple. You go into the Splunk UI. You put in some API keys and point it at the Lumi endpoint. And you can query. That's really all it takes, under five minutes. A couple of case studies. We've had a number of case studies. The access to data speeds up. The size of data on disk got smaller. Overall cost got smaller. The cost of everything greatly reduced out there in production for, various companies. Lastly, the, logo slide of, you know, all of the logos that, you interact with. These lots of companies have trusted us with things. We know how to operate, production software and systems, and we would love to help you as well.
Federated Search, Done Right: build a security data lake without replacing Splunk
Eric Tschetter · Chief Architect, Imply
Watch now
expert-led sessions
Rising costs are forcing security teams to choose which logs are worth retaining and how long they can afford to keep them searchable.
Peter Marshall, Director of Developer Relations at Imply, joins Solutions Review for a live session on how enterprises are solving this with a security data lake built around their existing tools.
Oct 21, 2026 · 12:00 PM ET
About Imply
Organizations are rethinking how security data should be stored and analyzed. As data volumes continue to grow and AI increases the value of historical data, security teams are looking for new ways to retain more data, investigate faster, and control costs without disrupting the tools and workflows they already rely on. The sessions above show how organizations are retaining more data, investigating faster, and lowering costs without changing their existing Splunk workflows.