Peter Marshall Pollfish, the “easiest and most affordable way to get real-time insights from real consumers”, delivers democratic, real-time insights with an innovative Apache Druid-powered pipeline that includes microservices leveraging an open-source Scala library for Apache Druid called Scruid. Anastasios Skarlatidis , Director of Data Engineering and Science at Pollfish, tells us more.
The Pollfish advanced data collection platform enables application developers to create mobile app and game surveys, reaching more than half-a-billion real consumers, helping global companies inform advertising, branding, market research, content, and product strategies with fresh and fast data.
“Whether understanding the time it will take to collect answers from their audience of interest, or breaking down results in real-time as respondents answer surveys, Druid delivers the high-performance time-series OLAP database we need to monitor, measure and explore data in real-time.”
Researchers extract and query statistical information on a plethora of dimensions, including income, gender, age distribution, country, state, city and geographical area over multiple time intervals with varying time granularities.
Real-time analytics dashboards show survey performance data: revenue, eCPM, funnel survey participation and more, live and direct from the stream.
The majority of data for analysis in this low-latency pipeline is passed through Apache Kafka, backed by a compute stack composed of Spark, Flink and Akka Streams. Micro-services are implemented mainly in Finatra / Finagle and provide RESTful and Thrift APIs. Cassandra and PostgreSQL live alongside Apache Druid and an HDFS-compatible data lake for raw data which is, for the most part, stored in Parquet and made accessible via Hive.
Enrichment processes (e.g., transformations, enrichments with demographic and geographic data, etc.) publish data to the Apache Kafka® stream, where Apache Druid acts a native sink powering Metabase and Looker for those ad-hoc queries, BI tasks and internal dashboards.
With support for Group By, Top N, Timeseries, Scan, and Search queries, Scruid enables geo-spatial queries, context adjustments per query, and can handle large payloads with streaming. The advanced client enables cached connection pools, load-balancing, and incorporates strategies to handle failures and authentication. Scruid is completely non-blocking / asynchronous, giving a rich high-level API for creating queries for Druid native queries, as well as for SQL.
“Apache Druid accelerates and simplifies the process of exploring the data. On the one hand, you have a system that ingests real-time data and on the other hand, the same system, can directly answer any analytical query.”
Pollfish have built an inspirational example of the democratic, real-time data revolution.
Come into the spotlight! Share your Apache Druid story with the whole community, too. Email community@imply.io and we will get the ball rolling!
Imply Lumi Product Preview: Removing the Cost–Performance Tradeoff in Observability
If you caught our recent product update, you’ve already seen the pace of development on Imply Lumi has been relentless. Last quarter, we delivered major performance and usability improvements to data...
Learn MoreImply Lumi product update: what’s new
Since releasing Imply Lumi in September 2025 as a decoupled data layer for observability, the Imply R&D team has been hard at work to make it easier and more economical to retain, query, and analyze observability...
Learn MoreThe Most-Read Imply Blogs of 2025 (and what they signal for 2026)
Before we take on 2026, let’s rewind. 2025 was the year observability teams stopped asking, “How do we reduce data?” and started asking the real question: “How do we build an architecture that can keep...
Learn More