What’s new in Imply Polaris – Q2 2023

Every week, we add new features and capabilities to Imply Polaris.  Over the past few months, we have focused on global expansion, enhanced security, flexible data ingestion, and improved visualization capabilities.

If you’re not familiar, let me give you a quick recap of what Polaris is all about. It starts with a Database-as-a-Service, built from Apache Druid^®, offering all the perks of a fully managed cloud service plus additional built-in capabilities for data ingestion and visualization.  In just minutes, you can start extracting insights from your data without worrying about setting up infrastructure. You can use the same cloud database to manage your data from start to scale, with automatic tuning and continuous upgrades that ensure the best performance at every stage of your application’s life.

Now for what’s new!

Global Expansion

Polaris in multiple regions

We’re excited to announce that Imply Polaris is now available in multiple regions, both in the United States and Europe. By expanding our reach across five AWS regions—us-east-1 (N.Virginia), us-west-2 (Oregon), eu-central-1 (Frankfurt), eu-west-1 (Ireland), and ap-south-1 (Mumbai)—we offer you the freedom to choose the region(s) that best align with your requirements.

Expanding across multiple regions helps ensure Polaris is available where your data is already located.  This approach optimizes network costs, resulting in efficient data transfer, reduced expenses, and maximum performance. For example, if you are using Amazon Managed Streaming for Apache Kafka (MSK) in Ireland, running Polaris in the same region minimizes data transfer costs and simplifies the setup of private connectivity.  Additionally, you can comply with data regulations and sovereignty requirements by processing and storing data in regions that follow specific guidelines. 

Enhanced Security

Ingest data into Polaris over PrivateLink and VPC Peering

Expanding options to use Apache Kafka^®, Polaris now has private networking options enabling ingesting data over AWS PrivateLink from your Kafka clusters in Confluent Cloud or AWS.  AWS PrivateLink ensures private and secure communication with AWS, eliminating the need for public IP addresses or internet traversal. With a private connection, you can ensure data privacy, reduce security threats, and simplify network configurations.

Additionally, you can choose VPC Peering for pull ingestion from AWS (Amazon MSK or open-source Kafka) to Polaris. VPC Peering enables direct network connectivity between VPCs within the same AWS account or different accounts. This feature eliminates data transfer costs over the internet, making it a cost-effective solution, especially for larger data volumes or frequent transfers.

AWS API Keys as credentials for JDBC connections in Polaris

Polaris allows you to authenticate and authorize JDBC connections using unique API keys instead of traditional credentials, like OAuth tokens.  These API keys act as secure identifiers for authentication, allowing your applications to establish connections to Polaris via JDBC. They offer enhanced security, simplified credential management, and granular access control, enabling fine-grained permissions and roles. By using API keys, you can connect and interact with Polaris securely using standard JDBC while enjoying improved security and access control.

Ingestion Made Easier

Automatically discover the schema on ingestion jobs

When you ingest data into Imply Polaris, Polaris automatically detects the schema of the input fields of your source data.  By default, Polaris ingests these fields into dimensions with matching names and data types, thanks to its schema auto-discovery feature. Even better, this capability extends to long-running streaming ingestion jobs, where new fields are automatically detected and seamlessly added to the table as events are consumed or published from the event stream.

Automatically change the table schema 

We also added a feature called flexible table mode, allowing automatic changes to the table schema when ingesting data from various sources. 

Here’s how it works: You get to choose between two modes. In the standard mode, the database enforces the table schema, making sure everything follows a predefined structure. But if you’re dealing with data that’s a bit wild and lacks a predefined structure or goes through frequent schema changes, then the new flexible mode is a great fit for you. In this mode, the data itself shapes the table schema, allowing you to adapt on the fly and handle diverse data sources without any pesky constraints holding you back.  This means no more worrying about rigid structures or struggling to fit your data into predefined boxes. 

Polaris now supports SQL Datatypes and all MSQ ingestion aggregators

Polaris now supports SQL datatypes and all MSQ ingestion aggregators, aligning with standard SQL databases and analytics. 

SQL Datatypes

Polaris now supports a wide range of SQL datatypes defined in the SQL standard. Think numeric types like integer and decimal, character types like char and varchar, datetime types, and more. This means you can work with your data in Polaris using familiar SQL datatypes, making it super easy to represent and manipulate your data effectively. You can seamlessly integrate Polaris with your existing SQL-based systems and tools, simplifying your data modeling and query processes. It’s all about compatibility and streamlining your workflows.

SQL MSQ Ingestion Aggregators

Polaris also supports all MSQ ingestion aggregators—Metric, String, and Quantity. These aggregators are commonly used in analytics and reporting scenarios, and now you can leverage them in Polaris. Perform operations like SUM(), COUNT(), MAX(), MIN(), and many more on your metric values, string values, and quantity values. With these standard aggregation functions at your disposal, you can accurately aggregate and analyze your data, gaining valuable insights for your business.

With support for various datatypes defined in the SQL standard, you can seamlessly integrate with existing systems and tools, simplifying data modeling and queries.  Additionally, Imply Polaris enables the use of common SQL aggregations (Metric, String, and Quantity) for accurate analysis and reporting, improving compatibility and facilitating seamless transformations.

Ingest data using SQL rather than the JSON-based Polaris APIs

In addition to JSON-based Polaris APIs, Polaris now also allows you to ingest data using standard SQL statements. This means you can leverage your existing SQL skills and familiarity with SQL syntax for data ingestion tasks, providing a more intuitive and efficient process.

With Polaris, you can write SQL statements like INSERT and REPLACE OVERWRITE to load your data directly into Polaris tables. It’s as simple as that! This approach simplifies the data loading process, letting you focus on what matters most—analyzing and extracting valuable insights from your data. No more getting tangled up in complicated workflows or wasting time on a steep learning curve. 

Furthermore, SQL-based ingestion in Polaris ensures compatibility with a wide range of ecosystem tools that support SQL connectivity. You can seamlessly integrate Polaris with popular database management tools, ETL frameworks, and other SQL-capable tools. This means effortless data transfer and integration with your preferred tooling, without the need for major modifications to your existing workflows. It’s all about keeping things smooth and easy, so you can leverage the advanced capabilities of Polaris while staying in your comfort zone

Ingest Kafka records and metadata

In Imply Polaris, you can ingest both event data and event metadata from an Apache Kafka topic, giving you a comprehensive view of your data.  This is a really nice feature, especially when your data lacks a timestamp. With Polaris, you can rely on the metadata, including the key, Kafka timestamp, and headers, to fill in the gaps. 

Let’s take an example: Confluent Cloud’s default data generator doesn’t provide timestamps. With Polaris, you can ingest data from Confluent Cloud and easily dive deep into advanced analysis and data exploration. When you’re able to leverage both event data and metadata, you get a complete picture of Kafka topics.

Estimate distributions with quantile sketches

For those times when you need to know more than just the average value in a stream, Polaris now supports Quantiles sketches from the Apache DataSketches library. This feature lets you dig deeper into the distribution of numeric values by giving you key insights like the 25th percentile, the median, and the 75th percentile values.  This gives you a reliable snapshot of the data’s distribution enabling you to quickly understand the big picture without simply relying on averages.

Efficient Visualization

Extend insights to your customers under your brand

With the latest enhancements in Polaris, you now have the ability to create a branded user experience through private-label branding and secure embedding links. Private-label branding lets you customize Polaris’ built-in visualization capabilities to match your brand identity. You can incorporate your logos, color schemes, and styling elements to create a consistent and engaging user experience within your analytics application. It’s all about strengthening brand recognition and ensuring your customers have an immersive and visually appealing experience.

Polaris also adds secure embedding links, which means you can seamlessly embed visualizations directly into your analytics applications. No more redirecting users to a separate interface—embed the visualizations within your application, allowing users to access and interact with the data right where they are. This streamlined and integrated user experience reduces friction and enhances data exploration.

Polaris also ensures the embedded visualization links are secure, protecting sensitive data and preserving privacy. You have full control over access, ensuring only authorized users can view and interact with the data. It’s all about keeping your data safe while delivering an engaging user experience.

Cost Optimization 

Smaller Project Sizes with tiered pricing

We want to make Imply Polaris more accessible to everyone, so we’ve introduced smaller project sizes and tiered pricing. Smaller project sizes allow you to experiment and test the waters, starting small and scaling up as needed.  Starting at just US$0.80 per hour, so you can explore Polaris without breaking the bank.

On top of that, our tiered data ingestion pricing offers a fair and transparent structure. You can choose pricing options that align perfectly with your data ingestion needs, paying only for the resources you require. It’s all about optimizing cost-efficiency and resource allocation.

Learn more and get started for free!

Ready to get started? Sign up for a free 30-day trial of Imply Polaris—no credit card is required! As always, we’re here to help—if you want to learn more or simply have questions, set up a demo with an Imply expert.