RixEngine

Background

RixEngine, an AI-driven advertising exchange platform incubated by Baidu Global, operates independently as a full-stack SaaS solution for programmatic advertising. This advanced platform requires robust analytics capabilities to facilitate real-time bidding and generate actionable insights for performance optimization.

Prior to adopting Imply Enterprise Hybrid for real-time analytics, RixEngine relied on Apache Druid® and Amazon Redshift as their core database technologies. For visualization, they used Metabase and an internal operations platform.

Challenge

Before switching to Imply Enterprise Hybrid, RixEngine relied heavily on open-source Druid as their primary database. Although Druid provided a performant foundation for real-time analytics, they faced several challenges that prompted them to upgrade to Imply Hybrid as a more robust and scalable solution:

• Unstable Query Performance: Initially, RixEngine attempted to improve query performance by adding more data nodes, but this approach proved to be costly and only provided marginal improvements. They tried tuning parameters for the Historical and Broker nodes, but couldn’t achieve the desired performance levels.
• Data Ingestion Bottlenecks: With daily data ingestion volumes reaching hundreds of terabytes, RixEngine frequently experienced data consumption delays. The ingestion pipeline felt overwhelmed, and the service was often fragile. Even minor changes, such as adding a new field, could lead to unexpected issues and disruptions.
• Operational Complexity: Managing and maintaining the Druid cluster required significant effort, and the system’s limitations made it difficult to scale efficiently or adapt to evolving business needs.
• Data Silos: Data was spread across various systems, including Kafka streams (ad request and reporting logs, blocked ad request logs, machine monitoring data), AWS S3, Google Cloud Storage (GCS), MySQL databases (operational platform data), and third-party platforms providing anti-fraud information. As each source had different urgency and importance profiles, the distributed landscape made comprehensive analysis difficult, slowed down reporting, and increased operational overhead.

Solution

Overall, Imply Hybrid provided stable and high-performance queries, efficient handling of large-scale data ingestion, and flexible, in-depth data exploration. Imply’s managed services also reduced operational overhead for the RixEngine team, allowing them to focus more on delivering value to the business. Beyond these core capabilities, Imply delivered two key improvements for RixEngine: (1) unified data analysis and (2) flexible data management.

(1) Unified data analysis: Imply helped RixEngine eliminate data siloes by centralizing data into a unified platform and improving data quality through cleansing, transformation, and enrichment. This consolidation replaced a fragmented architecture that spanned across multiple databases, BI tools and self-managed systems ー thereby reducing operational overhead and the risk of data errors. Imply provides a single source of truth via:

• Real-time Ingestion (Kafka): Imply’s robust Kafka integration allows RixEngine to ingest streaming data with low latency, which is crucial for near real-time dashboards, monitoring, and alerting on critical events.
• Batch Ingestion (MSQ): For less time-sensitive data in S3 and GCS, RixEngine leverages Imply’s MSQ (Multi-Stage Query engine) for efficient batch ingestion and reindexing. This lends flexibility to load historical data or update existing datasets.
• Lookups: Imply’s Lookup feature further enriches their data by augmenting event data with contextual information stored in an external system (MySQL).

(2) Flexible data management: Imply Hybrid’s flexibility allows RixEngine to tailor data management to their specific needs for optimal performance. To illustrate:

• Adaptive Data Compression: For data with low query frequency, they leverage Imply’s auto-compaction feature, minimizing storage costs without impacting performance. Conversely, for frequently queried data, they use MSQ to periodically compress data older than three hours. This ensures optimal query performance for recent data while maintaining cost-effective storage for historical data.
• Real-time Data Enrichment with Lookups: For contextual data or ID-to-name mappings stored in MySQL, Imply’s lookup functionality enables rapid implementation and enrichment of data, requiring minimal code changes and accelerating time to market for new features.
• Efficient Data Synchronization with MSQ: As RixEngine maintains a separate copy of critical data in Redshift and other databases, they use Imply’s MSQ to efficiently export key dimensions and metrics from various sources to S3, enabling seamless data synchronization between systems. This streamlines their data pipelines and ensures data consistency across their organization.

Impact

Switching to Imply Enterprise Hybrid helped RixEngine decrease machine costs by over 50%, while reducing query times from over 10 seconds to under 5 seconds. Their ability to reduce the total cost of infrastructure (TCO) by 50% stemmed from data node optimization (50% fewer nodes deployed for equivalent workloads with Imply vs. self-hosted Druid), bandwidth optimization across machines (co-developed a multi-stage aggregation pipeline with Imply, reducing bandwidth usage by 70%), tool consolidation, and operational efficiency gained from data replay incident reduction and lower DevOps overhead.

Additionally, Imply saved RixEngine’s engineering team 8 hours per week by improving data analysis, monitoring, visualization and support processes via:

• Enhanced Data Analysis: The data cube, alert, and dashboard modules have enriched their data analysis capabilities, enabling quick issue identification across nearly 100 dimensions. This has allowed them to pinpoint root causes, whether they originate from our service, network providers, or upstream/downstream users.
• Superior Support: The support team’s proactive standby during upgrades, scenario-based testing, and assistance in testing new features like window functions have enhanced RixEngine’s efficiency.
• Improved Monitoring with Clarity: Clarity has provided better service and query monitoring. RixEngine can observe which dimensions have the poorest query performance and consider adding them as partition keys to optimize performance.
• User-Friendly UI: The Pivot design in Imply’s UI is more intuitive and user-friendly compared to alternatives like Metabase and Superset, making it easier for users to perform queries and navigate the platform.

Moving forward, RixEngine plans to continue advancing along the path of data- and algorithm-driven development by combining Imply’s data analysis capabilities (e.g. more SQL quantile-related and window functions) with large language models. They’re also investigating how to make their visualizations more accessible via Pivot 一 The overall goal is to guide the team toward deeper data analysis and present it in an easily understandable way through Pivot, lowering the barrier for non-developers to use these functions.