Overview
Paytm is India’s largest mobile payments, commerce, and financial services platform. With a user base of more than 300 million, Paytm is on a mission to bring half a billion underbanked individuals and businesses into the mainstream economy using its digital banking and payment products.
Paytm’s platform powers transactions for more than 20 million merchants, which translates to a large volume of data—roughly 5 billion events per day. The company’s Growth team uses this data to create new revenue opportunities through targeted customer acquisition, building credit score models, and cross-selling services. To facilitate these analyses, the Data team set out to build a more robust analytics application. And to support a modern, highly interactive analytics application, Paytm needed a high-powered database built to deliver real-time insights at scale.
Challenge
Initially, Paytm’s Data team focused on replacing their off-the-shelf analytics service by building a custom analytics application. The legacy solution created unnecessary complexity for engineers trying to build out new verticals for data ingestion. On top of that, query results would sometimes take minutes—or even hours—to load, hindering the Growth team’s ability to act on insights.
But once the team decided to build their own application, they faced another decision: which database would be cost-effective and easy to scale, while also providing the sub-second query speed users expected? Ultimately, Paytm chose Apache Druid for its flexible, distributed architecture and always-on operability.
“We realized we needed to build a solution that would meet our current requirements and allow us to scale,” explained Ravi Maurya, Technical Lead at Paytm. “We chose Druid because it’s open source, which enabled us to start building quickly, but also provided the performance, flexibility, and ease of development we were looking for.”
After deploying Druid in a self-hosted AWS environment, the team saw an immediate improvement in query performance. At first, this was enough to power the interactive experience Paytm’s Growth team needed to better understand customer trends. With sub-second, ad hoc queries, members from the Growth team were able to have live conversations with their data.
However, as Paytm’s customer base expanded, so did user interactions across its various Apps—which went from generating 3.5 billion to more than 5 billion events daily.
“As Paytm adoption grew with more customers across new verticals, our Druid deployment was required to handle billions of events every day,” Maurya said. “But in order to keep up with our Growth team’s more sophisticated analytic queries, DevOps continuously optimized and tuned the Druid cluster, limiting their ability to build new code.”
Solution
A key aspect of any open-source software’s performance is maintenance. Paytm quickly found the process of operating and tuning its Druid cluster consumed a significant number of engineering hours each week. In addition to reducing infrastructure costs, the Paytm team needed to minimize the maintenance and work required to manage a distributed system. Maurya said making the move from an open-source Druid deployment to Imply helped optimize Paytm’s Druid system, enabling the team to redirect valuable engineering resources toward other business initiatives.
“[Our engineers] can concentrate more on the other pipelines and spend more time thinking about where data will be flowing, rather than how data will be stored and how to optimize performance on the data store end,” Maurya said.
Paytm now runs the enterprise distribution of Apache Druid—delivered by Imply—which is consistently fast, reliable, and requires zero DevOps resources to manage. Backed by the deep Druid expertise and world-class support of Imply, the Paytm team is able to focus on scaling other aspects of their data pipeline.
“We don’t spend a single minute fixing anything related to Druid,” Maurya added. “No DevOps is involved in making any changes to the cluster, upgrading a node, or adding a property to a specific node. That has reduced a lot of effort from our side.”
Results
With Druid and Imply, Paytm has been able to drastically reduce their operational costs while enabling a dynamic, interactive data experience for the Growth team across petabytes of raw data.
Along with implementing Imply, Paytm engaged Imply Professional Services to recommend the best ways to maintain consistent performance on Druid while controlling costs. The Professional Services team quickly determined that Paytm’s AWS Virtual Private Cloud (VPC) resources were overprovisioned.
“With Imply’s expertise, we quickly resized our AWS VPC and reduced our infrastructure costs by half, more than covering the cost of Imply Enterprise,” Maurya said. “Even better, with Imply, cluster operations became point and click, requiring zero DevOps involvement, freeing up 12 engineering hours a week.”
Paytm also gained critical performance improvements, able to query data sets 10 times faster (and sometimes 35 times faster) than before, with latency now at 100 milliseconds or less. This allows the Growth team to explore data in real-time, and help propel Paytm’s presence in new verticals and use cases.
To learn more about modern analytics applications in the Financial Industry, watch the full webinar.