This fairy has never flown faster. Helping FairyGodBoss elevate millions of women at work, with the power of sub-second analytics.

by Fern Melone · Melissa Halfon · March 30, 2021

9 Million Users served by analytics
<95ms Average query response time
12x Capacity in events processed daily (compared before adopting imply)

Business Goals

Fairygodboss (FGB) is the largest career community for women. They are a place where job seekers can find opportunities at companies that are investing in diversity and equality initiatives and are committed to improving the workplace for women.

They needed an analytics solution that would serve both sides of the business: provide their internal team with real-time metrics to assess operations as well as support customer reporting. That meant being able to answer simple questions like "which SEO articles are driving the most user acquisition?" while also aggregating a slew of customer metrics for their Brand Strategy team to report key metrics back to clients.

“We wanted all of this data available in one place for ease of use.”

Melissa Halfon, Data Engineer, FGB

Technology Requirements

There are two data pipelines their engineering team manages: a streaming pipeline and a batch pipeline. The batch pipeline is responsible for feeding data for processing to hourly and daily Spark jobs orchestrated by Airflow. Think user notifications, reminders, retention emails. The streaming pipeline is where they leverage Imply.

FGB’s core product is a web application with a Node and MySQL backend built on top of a suite of AWS services (container deployment via ECR, database management via RDS, and many more). Their existing reliance on AWS as a cloud provider made it easy to choose AWS Kinesis as a data streaming provider to capture clickstream data as well as other user events emitted from the application. This was the data they most needed to expose to various teams for analysis.

They wrote a custom middleware in Node that serializes any event payload they wish to capture before streaming it to Kinesis. They model their data streams by business logic and set up numerous streams to consume metrics relevant to their business.

With application data flowing into Kinesis seamlessly, all that was left to do from there was set up a source for Druid ingestion.

“Luckily, Imply made the rest simple with support to ingest either Kafka or Kinesis streams to a Druid Datasource while guaranteeing exactly-once processing. We wrote up a JSON ingestion spec to define the dimensions, metrics, and transforms relevant to our analysis, and we were off to the races.”

Melissa Halfon, Data Engineer, FGB

Make or buy - Why Imply

In the early days with a lean engineering team, having Imply as a partner allowed FGB to roll out a real-time streaming pipeline without having to worry about managing infrastructure to stand up an end-to-end analytics system.

“We were able to spin something up in a fraction of the time we could independently.”

Melissa Halfon, Data Engineer, FGB

Business Outcomes

Enabling teams to improve their service by using Imply Pivot

The solution built with Imply provides insights into core tenants of their business: both on the user-facing side and the client-facing side. Many Fairygodboss employees are power users of Imply Pivot. From creating custom dashboards to enabling ad-hoc exploratory analysis, Pivot has been a backbone for their company for years.

Having Imply as a partner is a force multiplier

Melissa said: “The support provided by Imply has been a benefit as well. After nearly three years of our systems running in production, we began to hit some scale issues that required a re-architecture at Druid’s data ingestion layer.”

They observed there were some data enrichment computations applied at query time that were causing serious performance headaches, mostly in the form of query timeouts. With guidance from the Imply team, they were able to successfully perform a reindex of 100% of their historical data with zero downtime to their end-users. They moved computationally expensive transforms to apply at ingestion time rather than at query time, thus improving query performance significantly.

“Imply Clarity was a great tool for diagnosing slow queries and monitoring performance before and after the reindex.”

Melissa Halfon, Data Engineer, FGB

Back to blog

How can we help?