Meet the team: Abhishek Agarwal, engineering lead in India

You live in Bangalore and work for Imply in San Francisco. Please introduce yourself.

I come from a small town in the north of India, and I studied at the Indian Institute of Technology, Roorkee. I graduated with a degree in computer science.

You are Imply’s first engineer in India — was it a difficult decision to take that role?

To be honest, I was certainly anxious about how this would work out with me being alone here in a different time zone. I’ve always felt it was important to have face time with your colleagues. It was a slight shift of role for me as well, since I was to build a team.

But, I took the risk, as this was the kind of opportunity I had always been looking for. There is no other database company I know of that is hiring engineers to work on their core technology. I also felt that Jad, the head of engineering, and the other company leaders would have my back.

In hindsight, this was the best possible decision. The people at Imply are smart, humble and extremely good engineers. The talent I have seen here is unmatched. I have never felt excluded because of being in a different timezone.

What can you tell us about Imply’s plans for India?

Imply is going to deliver some very high-impact projects from India. This work will be instrumental in setting us apart from our competitors.

Bangalore has plenty of solid talent that we want to tap into. In the short term, that means more Druid specialists. We have already hired some really great engineers. In the long term — once the team is well integrated into Imply R&D — that’s when we’re going to think about how we scale this site laterally. This means growing the platform team, customer support, operations and more.

“Imply is going to deliver some very high-impact projects from India.”

What would you say to engineers in Bangalore thinking about joining Imply?

If you are passionate about building large scale systems or databases, then working for Imply is a golden opportunity. Many companies have big-data problems and centralized teams to address these problems. At Imply, you will be solving data problems for a wide range of domains. There is an opportunity of moving from building on top of a Druid-like project to building Druid itself. As an additional bonus, you get great exposure to opensource.

How has the market for IT talent developed in Bangalore?

Over the past decade or so, we’ve seen a domino effect on the India tech startup scene. Many engineers here started out working on large-scale problems at consumer internet startups. This exposure has led to growth of local talent that we did not have earlier in India. As the alumni from these startups created companies of their own, so the engineering talent and expertise have grown too. This has also led to a shift in the strategy around setting up development centers in India. Companies used to set up their Indian development centers very late in their growth cycle — pre- or immediately post-IPO — but now it happens much earlier. Early stage companies are now also setting up development centers in India to benefit from the country’s geo-distributed talent.

I studied computer science at Korea University in Seoul. My major was in distributed systems, an area I’ve always been interested in. It’s long been a dream of mine to develop my own distributed database.

About five years ago, I went to Singapore for an IT conference and attended a breakout session run by the founder and CEO of Imply, Fangjin Yang. Imply was a small startup back then and Fangjin was traveling around to get people interested in Druid. I had a lot of technical questions and he was very good at answering them. As I learned more, my interest in Druid only grew. I got really excited about all its cool features and the challenging problems it can solve.

When I returned to Korea, I contacted Fangjin and told him that I really wanted to join Imply. Even though I expected some challenges due to my limited English skills and being located in Korea, they brought me onboard anyway. I worked remotely from Korea for the first year before moving to California in 2018.

Tell me about your job and what problems you are trying to solve.

At its core, data analytics is about repeating query execution and result analysis. Data analysts run some queries, get some insights out of the results, and run more queries to get a deeper understanding.

The problem today is that data analysts are often unable to focus on their primary job — analyzing data — and instead need to spend a lot of time on tuning their queries. As they are not experts in the underlying system, it often becomes a very hard process for them that takes a long time. As a result, in the worst case, they could miss a golden business opportunity and even negatively impact their company.

This is the problem that we’re solving at Imply with Apache Druid — one of the fastest query-processing engines in the world.

“We are trying to make machines understand what humans need, so that we can make our customers happy.”

There are many companies trying to solve this problem, but a lot of the solutions out there still require too much manual work. What we are trying to do is automate the entire analytics process. I know this sounds impossible, but it’s our guiding vision. We are trying to make machines understand what humans need, so that we can make our customers happy.

Can you tell us about a specific project you’re working on?

Druid is an open-source project, so I also use my spare time to make Druid better. There is a side project that I’ve been working on about query speed.

One of the current challenges with Druid is that when a query has to read a lot of big data, Druid uses multiple threads to process that data faster. When you use multiple threads, you always need to think about how to synchronize and coordinate them for the greatest efficiency. So I have been trying to minimize the synchronization cost while maximizing the CPU utilization of those multiple threads. It’s very experimental work, but I’ve proven that my idea could generate speeds up to two times faster than what we have now. Now I’m working on making it more stable.

What’s it like to work with the team and how has it evolved?

About five years ago, when I joined, there were only four people in my team. Now we have 30.

We’re growing so fast that we’ve had to split the team into three smaller groups. All the new people are amazing and have really good backgrounds. The team is so welcoming and competent that people do not even need that much time to get adjusted — they just immediately start working on something cool.

“In the smartest team, you don’t need to be the smartest person — you just need to contribute with something that you’re an expert in.”

What we want to do is build the smartest engineering team in the world. This means thinking together, making decisions together, and growing together. In the smartest team, you don’t need to be the smartest person — you just need to contribute with something that you’re an expert in. This is the kind of team I want to be part of.

What can you say about relocating to the United States for Imply?

There is so much I could say — this company has really helped me a lot!

One of the things I appreciate the most is that everyone at Imply has always respected me. My English is still not perfect, and five years ago it was not as good as it is now. They could probably hardly understand me! But they always listened to me and gave me the time I needed to finish what I wanted to say or explain. Putting myself in their shoes, I’m not sure I would have had the same patience!

I’ve always been warmly welcomed to join social occasions outside of work too. It’s been a while now since I’ve seen everyone, as we’ve all been working at home during the pandemic. So I’m looking forward to meeting up again when the time is right.