How Finix is Leaning Into Real-Time data with Apache Druid and Imply Polaris with Ross Morrow

Transcript

[00:00:00.000] – Reena Leone

Welcome to Tales at Scale, a podcast that cracks open the world of analytics projects. I’m your host, Reena from Imply, and I’m here to bring you stories from developers doing cool things with Apache Druid, real-time data and analytics, but way beyond your basic BI. We’re talking about analytics applications that are taking data and insights to a whole new level.

[00:00:18.470] – Reena Leone

Finix is a payment processor with a platform that is built to help companies quickly optimize revenue and manage their business operations. They’re on a mission to create the most accessible financial services ecosystem system in history. Payments are just the tip of the iceberg. They’re working towards a global operating system for fintech. One of the technologies helping them with this mission is Apache Druid via Imply Polaris. Joining me to talk me through the whole project is Ross Morrow, Software Engineer at Finix.

[00:00:47.270] – Reena Leone

Ross, welcome to the show.

[00:00:48.580] – Ross Morrow

Thank you very much, Reena. It’s a pleasure to be here.

[00:00:51.190] – Reena Leone

I always like to get a little background on my guests, how you got to where you are today. Tell me your story of how you ended up at Finix working in fintech.

[00:01:03.000] – Ross Morrow

How far back?

[00:01:03.950] – Reena Leone

As far back as you want to go.

[00:01:07.850] – Ross Morrow

I’m a little untraditional, but that’s not super uncommon in tech. So tech is my second career, I guess. I was an academic. I stayed in school for way too long. And then I was a professor for about five years. I mostly did research and teaching associated with computers and computer things, even though I wasn’t in the computer science department. About a decade ago, I left that career path. I had a bit of a realization that I was a little bit more of a geek than aligned with the academic career path. Then I got into tech. I’ve had the good fortune to to have a variety of different roles in the tech sector in different domains. And for about, I think about five years now, I’ve been in fintech, insurTech, areas, focusing primarily on software engineering related to the platform, DevOps, infrastructure development, developer experience, and how all that overlaps with data. That’s how I arrived here.

[00:02:33.170] – Reena Leone

Very cool. I love non-traditional career paths. As I have gone through that myself, didn’t think going to school for music would end up with me being a developer advocate. Here we are!

[00:02:44.600] – Reena Leone

Okay, so your company, Finix. Can you tell me a little bit about what your company does for those who aren’t familiar?

[00:02:50.250] – Ross Morrow

So we’re a payments facilitator and payments processor, basically. We offer a highly configurable and very flexible, but as I’ll probably discuss a little bit more, all in one business to business B2B payments platform. So we help money move, and we help money move in a variety of ways. And that’s not as simple as just actually moving the money because there’s a lot of regulatory compliance involved. There’s a lot of reporting and accounting involved and all sorts of other bells and whistles that are very important. So a couple of ways we do that, I don’t know the right non-reductive way to say this, right? But there are some very large legacy financial service providers that are a little bit… They don’t have to offer new technology because there’s something like an oligopoly, right? There are just a few of them. They’re the ones that can move the money. They’ve built the infrastructure, and they’re largely built up Their capabilities with mergers and acquisitions. So their offerings tend to be somewhat fragmented. And so we have products and we have customers that use us to offer a reasonable experience on top of those legacy providers.

[00:04:17.530] – Ross Morrow

More recently, we have become actually a payments processor. So we are actually connected to the payments trails, and we are migrating into the space that the several handful of legacy financial service providers for payments occupy. So directly connected to all the card networks, directly moving the money and so on within the same customer experience for our customers.

[00:04:45.270] – Reena Leone

So within your area of fintech, what sets you apart from other payment service and processing companies?

[00:04:52.090] – Ross Morrow

Well, again, we’re a highly configurable and lower cost platform for our customers that want to help, usually their customers, accept payments, accept payouts, and generally move money. So we are lower cost than some other providers, but we offer a lot of capabilities for our customers to also decide how they’re participating and what fees, dues, and assessments are actually happening when money moves. And that’s a big advantage we see as something we can offer to our customers. We have a really, really fantastic award-winning, modern dashboard for our customers to see all their transactions, all their disputes, all their refunds, with insights and graphs as well as all the detail, and that’s all based on APIs. It has gotten pretty common. Anything somebody can see in our dashboard, they can pretty much access through APIs. And that’s a really powerful enabler for what customers can do, because when there’s such a really nice award-winning dashboard, they have to know less technical detail sometimes. We’re providing a lot of services that are getting towards the low and no code spectrum, which opens up more opportunities for people that want to accept payments or send money to integrate with us.

[00:06:30.320] – Ross Morrow

We have really good customer service, so I don’t get the opportunity to work with customers directly very much, but I see all the kudos we get. So we have really good customer service, and that’s not the case. We’re called the payments providers.

[00:06:49.530] – Reena Leone

Yeah, I can imagine.

[00:06:50.870] – Ross Morrow

We’re payments facilitators. I think about maybe every month, I see a LinkedIn post about some other payment provider I won’t name, and what happened. So importantly, also, and I think I hinted at this before, we’re really trying to offer an all-in-one multifunctional platform that does not require somebody to integrate with a customer to integrate with a different part of our product suite to do something else. So in our one platform, we can handle multi-channel money movement in store and on online. We are international, at least US and Canada. We accept payments in the United States and Canada today, and we’re multidirectional. We can do payouts as well as accept pay-ins and do push-to-card transactions and other things. But the important part of that is the product is built to not have to… Not to require you to adopt a different part of our company’s product suite to do that.

[00:08:03.250] – Ross Morrow

It’s all in the same place.

[00:08:04.670] – Reena Leone

All right. Speaking of platforms, let’s shift gears and talk about databases, my favorite topic. For this project, what prompted your search for a new database?

[00:08:17.060] – Ross Morrow

Well, we’re really at a phase where we’re focusing on our ability to scale. So depending on what your bar is, for data volume, velocity, and so on. We’re not like [Cisco] ThousandEyes. We’re not Google scale, but it’s still big enough data that we have to think about how we work with it. And we’re also looking to the future with our current growth trends. We need to make sure we are scaling correctly for our customers. And one of the places we need to focus on that is in our data systems that supply not just our platform capabilities, but our customers’ understanding of what’s happening in our systems and how the money is moving and all of that. So in general, this adopting Druid has been part of our work to try to understand better how we’re not limited as we scale with our data. In particular, specifically to something like Druid, we deal with a lot of data that is probably more suited to OLAP than a OLTP. Our primary platform is built on transactional systems, and we use some transactional systems. And one of the potential roads for us to achieve some more scalability is identifying technologies that better suit the specific need for for analytics.

[00:10:00.930] – Ross Morrow

we’re looking for technologies that may be natively distributed because that may be an important part of not relying so much on traditional relational database management systems that work well up to a point, or work well up to the experience of the people and the investment the company is making in managing those systems. And we’re also very interested in event-driven computing. We, for all intents and purposes, pretty much have an event-driven computing system already. And we’re looking at opportunities, especially with [Apache] Kafka, to both do internal stuff, like messaging through our systems and as an ETL tool and maybe a change data capture tool. And so we’re interested in technologies that are natively compatible with Kafka, especially for data.

Along with the scaling concerns, 

[00:11:01.800] – Ross Morrow

We are not that big of a company. We have enough, but not that many engineers, and we have a lot of responsibilities, and we have to focus on things that differentiate us in our market. And that does not mean we get a lot of time to focus on database administration. So we need something, and we need technologies that help us reduce our potential costs for managing managed services, so to speak, and working with things like relational database management systems or data lakes or other related technologies.

[00:11:53.290] – Ross Morrow

And I think what’s critical for me is that there’s a really important distinction between immutable and mutable data. And there’s a lot of data we deal with and either do or can potentially do analytics on that just simply does not change. So there are a lot of activities that our customers do in interacting with our platform that we treat in an event-based way, and that data never changes. There’s a record of that occurring, and then it’s gone. And then Moving money is complicated and actually involves quite a bit of back and forth. Data back and forth representing the current state of a particular transaction on a particular card network for two particular parties. But still, there are a lot of pieces of that that are immutable, that we have to render as this event happened, this money moved in this direction. I think that recognition that there’s some data that never changes and isn’t going to be an important thing for us to have in mind as we’re looking for more scalable database alternatives, broadening our technology suite.

[00:13:11.060] – Reena Leone

So I heard you mention Apache Druid a couple times. Had you worked with Druid before or did it come up as you were evaluating database options?

[00:13:19.820] – Ross Morrow

I personally have worked with Druid before. So I have used Druid for a similar purpose. So basically, telemetry is the high-level data domain. Used it for storing basically GPS plus signals that came from real-world sensors in something like real-time, certainly real-time for the company I was working for.

[00:13:55.120] – Reena Leone

I feel like you covered this when we talk about the key features that you were needing. But I’m assuming with Fintech, things like security, data accuracy, high reliability, those are all probably on your list. But are there any we haven’t covered that you… Key features that you needed for this?

[00:14:11.600] – Ross Morrow

I mean, those are really important.

[00:14:14.830] – Reena Leone

Let me just summarize it for you real quick.

[00:14:19.090] – Reena Leone

Yeah, those are really important for us.

[00:14:22.570] – Reena Leone

You know what I forgot to ask you, though? What were you using before, Drew, before this project?

[00:14:27.370] – Ross Morrow

In terms of database technology?

[00:14:29.750] – Reena Leone

Yeah.

[00:14:30.670] – Ross Morrow

Well, we already use everything under the sun. So we use relational database management technology. I mean, that’s a critical component of, I feel like most systems, right? Something transactional is required. But we have highly schematized data, like highly structured data and highly unstructured data. But for our highly structured data that we really need very hot access to, relational databases make a ton of sense. We also use NoSQL databases. We also use KeyValueStores. We use KeyValueStore caching. We definitely use all of those things. In a couple of the use cases we’ve explored with Druid, we’ve used transactional schema-less data stores, and whatever works is probably appropriate, but we probably use these a little inappropriately. It’s like we never made updates. And there are a few other features we can talk about, we can talk about later. So we use a variety of technologies. And I think the issue for us in a couple of the use cases that we’ve moved over to Apache Druid is mostly maintenance, good data design, right? And costs. So our implementations of, say, an existing NoSQL-like data store were actually on the pricey side for us.

[00:16:18.020] – Reena Leone

Did you look at any other Druid-esque databases, any other OLAP databases when you were trying to evaluate?

[00:16:26.060] – Ross Morrow

I think in general, we’re aware of the land landscape for streaming compatible OLAP style databases. We have some experience and know what some of the alternatives are. But for us, we’re trying to move really fast. We develop really fast. We release features really fast. I’ve seen this a few times. While I’ve been at FInix. Sometimes look The entire experience really makes or breaks the ability to actually engage with the project. Sometimes, I think Druid is a fantastic technology. I like using it. I think it offers a lot of opportunity and options for managing real-time data, especially at scale, and serving insights to customers, internal or external. But sometimes the initial project is not what the company lands on and uses five years from now, but it can still open the doors and create the acknowledgement of an opportunity to use the technology. So I think in this case, we didn’t do any expansive multi-technology comparison proof of concept work because we don’t want to devote. We could have those resources, but we don’t want to devote them, right, specifically. We want to focus more on what’s differentiating us for our customers as a good payments platform.

[00:18:06.750] – Reena Leone

Let’s dive into your specific use case. What are you using Druid for?

[00:18:12.460] – Ross Morrow

I mentioned we effectively have an event-driven computing platform under the hood. So we are using Druid internally as a way to store all of the event data that’s passing around our systems to to describe money movements, to state certain directives about how money should move, to describe auditing events for various activities, as well as identify how we’re interacting with our customers. So the whole platform is based on a lot of these events that are moving through our system. So out of band, outside the critical path, thanks to Kafka and Druid, we are able to persist that data, and we are able to analyze that data. We are able to look into what it is saying about how money is moving in our systems in a dramatically easier way than with previous technologies. I can explain why, of course. We’re also using it for for, during our initial adoption, some relatively low risk but higher volume immutable data like logs that we have to store for our internal analysis and for visibility to our customers. So this feeds customer-facing features as well.

[00:19:50.290] – Reena Leone

You mentioned Kafka a couple of times. Can you walk me through where Druid sits in your data architecture?

[00:19:57.850] – Ross Morrow

Sure. So it sits, thanks to Kafka, outside the critical path. So we can asynchronously write out to Kafka, and then we can read that data into Druid via just native Kafka ingestion jobs in Druid. So it was a great opportunity for us to be able to change and advance our storage of some of these events and logs and so forth without any possibility of that really interfering with money movement activity or customer interactions. And that’s really, really important to us from an engineering standpoint and a product standpoint. So outside of that critical path, we still have all the data. It’s still very, very fast. And we can facilitate both ad hoc and API access to the data. So in particular, we haven’t mentioned Imply very much, but we’re using Imply.

[00:21:13.380] – Reena Leone

We’ll get there. We will get there. Don’t you worry haha

[00:21:16.900] – Reena Leone

So we can always go to the SQL workbench, and we can do whatever analysis we want to do. And this has been super helpful for us. And we also serve data back via API. And for that, we use just normal JDBC connections and queries with Avatica. So we send the data out of it. We see, say, events or an incoming API call up, or we send out a webhook, and we’re sending events into Kafka. That’s getting into Druid. And then depending on when someone calls our APIs to access that data. We return data via call to Druid.

[00:22:05.870] – Reena Leone

So for this particular project, you chose the path of doing Druid via Imply Polaris, which is our database as a service built for Druid. Why go with something like a managed service like Imply Polaris versus, say, open source Druid?

[00:22:23.700] – Ross Morrow

Well, I think I mentioned this before. We want to focus on what’s valuable to our customers. We want to focus on differentiating ourselves as a payments processor. And as cool as Druid and other database technology is, and as geeky as we are wanting to learn more about those technologies and operate them and operate them well, that’s not necessarily what differentiates us. So managed services exist on a spectrum. Some are really, really managed. Some are not-so-managed managed services. And we not only need to know that we don’t have to spend a lot of time and other resources to stand something up, but also not to get surprised by future maintenance expenses that distract us from being able to operate the best payments platform. Those are really important for us. If that’s an important consideration for us. And so working with Imply, specifically for Druid, we get the opportunity to work with the experts in Druid. We get to work with the people that know it best and are the most invested in it and have offered in Polaris a very accessible low code integration point to interact with Druid. So that’s really important.

[00:24:02.520] – Ross Morrow

And I think also having the tooling, the additional tooling that Imply has built around Druid purely as a database is really critical and useful for us. In some of the use cases we’ve moved into Druid, one of the shortcomings has been the access to the data. So our API systems have no issues accessing the data because we’ve written software to access them. But not every NoSQL database is the same, and they all have their peculiarities, and they all have their patterns of querying. So having something that natively supports SQL and has a nice dashboard for us to do exploratory data analysis on the data we’re sending through Druid without having to stand up and maintain and manage stuff ourselves to facilitate the democratized data access to whatever we’re sending through the system, even just internally, is really, really powerful opportunity for us.

[00:25:10.300] – Reena Leone

I mean, full transparency, how is that going? How are things? Because Polaris is still relatively new in terms of technology. Druid has been around for about a decade now, but how are things?

[00:25:25.610] – Ross Morrow

I think things are great.

[00:25:26.870] – Reena Leone

Okay.

[00:25:27.800] – Reena Leone

The dramatic pause!

[00:25:30.110] – Ross Morrow

The dramatic pause. I think things are great. I always feel a little bit conflicted about the build versus buy. As previously when I used Druid, I also worked with Imply, but we did the other thing, the non-Polaris thing. I don’t think Polaris existed at that point. We ran it ourselves. We ran Druid ourselves. And I’m a highly technical person, and I like the details, and I like to understand. So the build versus buy is always an internal tension for me. So I’m very curious about what makes optimized segments? How are we doing this? How’s this going to impact our query performance if we do X, Y, or 6. So from that perspective, like coming from that perspective, when I originally heard about Polaris, I thought, well, won’t we lose something? How well will it work? Do I lose a lot of capability to understand the system and to make sure the system is working to my benefit or my company’s benefit? What am I giving up for this? And I was really I really leaned on in exploring Polaris as an option, our need for minimal intervention and maintenance. We can start to take advantage of a new data technology for our company if it is not heavy on the overhead.

[00:27:05.590] – Ross Morrow

And the experience has honestly surprised me from that perspective. So I don’t feel like we’re losing much, if anything. I think the system generally works well. So we can do ClickOps to make tables and create our schemas and run ingestion jobs at the same time as the API is super clear. To put ingestion specs and table specs and other things in our version control systems and have a simple scripting or CI that helps us understand that things are working well. So it’s been a great experience, honestly.

[00:27:50.810] – Reena Leone

Were there any challenges that you’ve run into? How did you solve them? Actually, you know what? I’ll even broaden that, not just Polaris, but Druid in general.

[00:27:59.520] – Ross Morrow

Well, there’s some specific things, and there’s some general things. So one specific thing that’s about Druid is a database design. Both are about Druid as a database design. Druid’s got these inverted indexes, and indexes everything. So you have to think really carefully about the data model. You don’t have to think about your indexes like you have to do with other NoSQL database technologies. But you do have to think about what you’re doing. So we ran into some challenges, having a mentality that everything would really be free when we’re indexing everything. And we had some features embedded into our data models and ran into some minor limitations there. But it was really an interaction with data modeling and understanding what the database is actually, how the database is actually interpreting the data. I think in general, though, it’s a little bit of a mentality shift for engineers that are really, really, really used to relational database management systems in particular. So one of the first questions that comes up is, can we join? How do I join?

[00:29:20.920] – Reena Leone

I’ve talked so much about joins.

[00:29:24.070] – Ross Morrow

I know. I imagine, right? But that perspective, some of those perspectives are so ingrained, right? Or how do I update a record? And so, well, we’re dealing with classes of records that we don’t actually want to update. And maybe that’s a priority for us. And when you’re asking, do I want to update this? You’re signaling something is maybe a little funky with the data model. And none of those are critical challenges, but their mentality shifts, and depending on who you’re working with, they can be… People can think those are interesting or people can sometimes think those are deal breakers. They don’t fit into their mental model.

[00:30:15.990] – Ross Morrow

I’ll admit, I would love a code generating SDK to understand and generate code for for our tables to be able to use the encode, SQL-like, but more software-like interfaces. We are very, very used to that. And a lot of companies, I think, are really used to that and use ORMs or some variant of an ORM. And I think the developers, myself and other developers that have worked with our experience with Druid through Polaris, are comfortable writing SQL. And it’s definitely the right interface point.

[00:31:04.730] – Ross Morrow

But also it feels like we’re really missing something that lets us speak in like, speak queries, but in chained software operations with defined objects.

[00:31:20.230] – Reena Leone

You mentioned part of it is having a mentality shift, but do you have any other advice for people who may be just getting started? What should they keep in mind if they have identified a project to use Druid for?

[00:31:34.720] – Ross Morrow

I think one of the things that should come first in this, this is probably, it’s probably going to sound weird, but it’s analytics, right? Is doing things with the data. We have been interested in and needing to get to the really real-time data availability, right? But at least in terms of Druid, it’s also like the joins and the updates. Easy to think about another database technology as a record retrieval system or subset of records retrieval system where that misses the “A” in the OLAP. And there are really good opportunities to provide internal insights sites, external insights, customer-facing features. If you lean in from the beginning on doing the analytics portions. So both of the applications I’ve been a part of that have leveraged Druid have sat in the middle. And I think it is leaving some of the analytics value on the table because they try to both do data retrieval as well as analytics activities with the database. I think in general, though, I really like the experience that Druid offers. I really like the experience that Imply offers. And one of the things that’s been really compelling for me about this particular engagement is how easy it has been to get started.

[00:33:27.090] – Ross Morrow

Druid is a complicated system, and Imply has has made it very accessible, more accessible this year than three years ago. And I think that’s highly relevant to me as somebody that sometimes has to sit in an architecture role and somebody that’s thinking about the future scalability that our customers at Finix will depend on, because I’ve been around a lot of places that think like, Druid’s for Netflix, right? Druid’s for scale. And I also hear a lot of stories about people pushing their Postgres until it breaks, right? And so I’m really excited by the opportunity to engage with it, right? Kind of at a low price point, not just in terms of money, right? But the resources we have to expand to just start exploring if it’s a really strong long-term fit for our systems.

[00:34:22.350] – Reena Leone

I tell people that all the time, too, where they think that they need to be doing petabytes of data to use a system like this, and that they have to have these really robust infrastructure in place in all of this, to your point, like Netflix. But that’s the whole point of something like Polaris is to be able to get started and then have the ability to scale as your company grows, as your data volume grows versus having to start with all of that in the first place.

[00:34:50.230] – Ross Morrow

Yeah. I think some of those challenges I mentioned, can I join? Can I update? All of these things, they’re all also opportunities to reflect on the data companies actually working with and how the company wants to work with that data. So even when those are confrontational questions, I think those confrontations can result in some organizational value in recognizing what are we doing and how are we dealing with it and at what cost in terms of human resources and money and time and all of that. Question people should ask about real real-time. Our real-time data in Imply Polaris via Druid is sub-second, like 100%. Like sub-second queries, sub-second data availability. It’s a different order of magnitude than anything that’s not our hot relational database management systems. And so broadly, there are some important business questions about when and where do you need that? But it’s definitely Very somewhat straightforward to get it, right? If you use this technology stack we’ve talked about. And there are certain situations where you opening the opportunity and acknowledging that you could have something like that might change people’s perspectives on what the product can be and what the systems can look like that are important to address.

[00:36:26.420] – Reena Leone

Well, Ross, thank you so much for joining me today. This has been amazing. If you want to learn more about what Finix is up to, please visit go.finix.com. If you want to learn more about Apache Druid, please visit druid.apache.org. And if you want to learn more about Imply and what we’re doing with Imply Polaris, please visit imply.io. Until next time. Keep it real.