Community Spotlight: Pollfish survey insights, powered by Apache Druid

Pollfish, the “easiest and most affordable way to get real-time insights from real consumers”, delivers democratic, real-time insights with an innovative Apache Druid-powered pipeline that includes microservices leveraging an open-source Scala library for Apache Druid called Scruid. Anastasios Skarlatidis , Director of Data Engineering and Science at Pollfish, tells us more.

The Pollfish advanced data collection platform enables application developers to create mobile app and game surveys, reaching more than half-a-billion real consumers, helping global companies inform advertising, branding, market research, content, and product strategies with fresh and fast data.

“Whether understanding the time it will take to collect answers from their audience of interest, or breaking down results in real-time as respondents answer surveys, Druid delivers the high-performance time-series OLAP database we need to monitor, measure and explore data in real-time.”

Researchers extract and query statistical information on a plethora of dimensions, including income, gender, age distribution, country, state, city and geographical area over multiple time intervals with varying time granularities.

Real-time analytics dashboards show survey performance data: revenue, eCPM, funnel survey participation and more, live and direct from the stream.

The majority of data for analysis in this low-latency pipeline is passed through Apache Kafka, backed by a compute stack composed of Spark, Flink and Akka Streams. Micro-services are implemented mainly in Finatra / Finagle and provide RESTful and Thrift APIs. Cassandra and PostgreSQL live alongside Apache Druid and an HDFS-compatible data lake for raw data which is, for the most part, stored in Parquet and made accessible via Hive.

Enrichment processes (e.g., transformations, enrichments with demographic and geographic data, etc.) publish data to the Apache Kafka® stream, where Apache Druid acts a native sink powering Metabase and Looker for those ad-hoc queries, BI tasks and internal dashboards.

With support for Group By, Top N, Timeseries, Scan, and Search queries, Scruid enables geo-spatial queries, context adjustments per query, and can handle large payloads with streaming. The advanced client enables cached connection pools, load-balancing, and incorporates strategies to handle failures and authentication. Scruid is completely non-blocking / asynchronous, giving a rich high-level API for creating queries for Druid native queries, as well as for SQL.

“Apache Druid accelerates and simplifies the process of exploring the data. On the one hand, you have a system that ingests real-time data and on the other hand, the same system, can directly answer any analytical query.”

Pollfish have built an inspirational example of the democratic, real-time data revolution.

Come into the spotlight! Share your Apache Druid story with the whole community, too. Email community@imply.io and we will get the ball rolling!