Ingestion from Confluent Cloud and Kafka in Polaris
Apr 20, 2023
Timmy Freese
One of the foundational pillars of real-time data analytics is the ability to move data from one place to another quickly; Apache Kafka® is a widely adopted solution for this functionality. Confluent Cloud provides a fully-managed solution for Kafka. Whether you use Confluent Cloud, self-hosted Kafka, or alternative hosting platforms such as AWS MSK or Aiven MAK, Imply Polaris offers a solution for you to ingest data.
For Confluent Cloud users, Polaris offers a native, pull-based ingestion capability. Users only need to specify a few fields to establish a connection to their Confluent Cloud account and can ingest data in no time—every event enters the database and is queryable with sub-second latency. Additionally, this method offers exactly-once semantics and infers the schema from available data using Druid’s sampler.
For self-hosted and other Kafka solutions, Polaris offers an HTTP-based Kafka Connector which uses our Push API endpoint. This connector will automatically batch and compress your data and handles auth token renewal for you. Additionally, this solution can be used in tandem with Polaris’s PrivateLink offering to provide enhanced security.
Regardless of which solution you choose, you can expect high throughput and low latency results. We provide OLAP functionality with OLTP performance.
Imply Lumi Virtual Tier is now generally available
Lumi Virtual Tier is generally available today on Lumi Cloud and Lumi Enterprise. Customers are already using it to keep historical data searchable without always-on infrastructure. Compute spins up when a...
Imply Lumi Loglake vs. Splunk Machine Data Lake: Your Data Lake as a Splunk Index
Splunk customers want to keep more log data for longer without paying indexer prices for it. Splunk’s answer is Machine Data Lake (MDL), now generally available on Splunk Cloud Platform. Imply Lumi Loglake...
Search your Splunk DDSS archive without restoring it
Security teams often keep logs long after they have aged out of their SIEM platform’s searchable window. The data is there for investigations or audits, but accessing the data for insights can be complicated....