Ingestion from Confluent Cloud and Kafka in Polaris
Apr 20, 2023
Timmy Freese
One of the foundational pillars of real-time data analytics is the ability to move data from one place to another quickly; Apache Kafka® is a widely adopted solution for this functionality. Confluent Cloud provides a fully-managed solution for Kafka. Whether you use Confluent Cloud, self-hosted Kafka, or alternative hosting platforms such as AWS MSK or Aiven MAK, Imply Polaris offers a solution for you to ingest data.
For Confluent Cloud users, Polaris offers a native, pull-based ingestion capability. Users only need to specify a few fields to establish a connection to their Confluent Cloud account and can ingest data in no time—every event enters the database and is queryable with sub-second latency. Additionally, this method offers exactly-once semantics and infers the schema from available data using Druid’s sampler.
For self-hosted and other Kafka solutions, Polaris offers an HTTP-based Kafka Connector which uses our Push API endpoint. This connector will automatically batch and compress your data and handles auth token renewal for you. Additionally, this solution can be used in tandem with Polaris’s PrivateLink offering to provide enhanced security.
Regardless of which solution you choose, you can expect high throughput and low latency results. We provide OLAP functionality with OLTP performance.
A security team of four has run a major SIEM since February. Nobody on the team writes SPL. Nobody opens the console or looks at the dashboards. Every question goes through an AI agent, which writes the query,...
Why You Shouldn’t Have to Delete Your VPC Flow Logs
When a security incident happens, investigators almost always start with the same questions: Which systems communicated? Where did the traffic originate? What changed before the incident? Was data exfiltrated?...
Splunk Smartstore vs Lumi Loglake: Two Very Different Ways to Search Logs in Object Storage
One copies data back before it can be searched. The other queries it where it lives. Lumi Loglake lets Splunk teams query logs directly in object storage, including AWS S3, Delta Lake, Apache Iceberg, using...