Have you ever wanted more visibility in your AWS network traffic? Your AWS VPC logs contain critical information about the health of your systems, and by collecting, analyzing, and visualizing these logs, you can better:
- Diagnose network bottlenecks
- Analyze infrastructure performance
- Find network security vulnerabilities
- Increase application throughput
This blog post is a “how-to” for analyzing AWS VPC logs with Imply. This blog assumes the reader is familiar with AWS and has the necessary security access to perform the tasks outlined in the referenced AWS articles.
In our setup, we will be using AWS Kinesis, AWS Cloudwatch, and Imply.
Collect your flow logs
If you are not already collecting your VPC flow logs, please first refer to the AWS’s docs to get started.
Set up Imply
- We will be leveraging Imply as the central analytics engine to store, analyze, and visualize flow logs. Follow these getting started instructions to install Imply.
druid-kinesis-indexing-serviceare included as extensions in your
common.runtime.properties, similar to below:
druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-kafka-indexing-service", "druid-parser-route", "imply-utility-belt"]
You may need to restart Imply processes after this change.
Set up AWS
- Create a Cloudwatch log group.
- Enable VPC flow logs for a VPC and send them to AWS Cloudwatch.
- Create a Kinesis data stream.
- Associate a log group to a Kinesis Stream.
Load live streaming data
- Go to Imply and assign Kinesis as a Data Source in Imply by selecting the Data tab and in the upper left and then
+ Load datain the top right.
- Paste in the ingestion spec (make sure to update it based on your access credentials in
- After you’ve updated the spec with your specific AWS information, select “Send” and then “Submit as supervisor spec”. After the initial overhead of connecting to Kinesis, data should start streaming in to Imply.
- You should now be able to load in the provided data cube and dashboard to see your data. Go to the settings page accessible from the user menu. Click
Import settings. Paste in the provided data cube and dashboard and click
- You should now be able to go to the
Visualstab and see your data in the provided data cube and dashboard.
As an example, the following dashboards show interesting views based on AWS VPC Logs. You can modify the provided dashboard and data cube (or create new ones) as you desire.
As you can see from the above, Pivot can be a powerful tool for understanding your AWS VPC log data and providing interesting insights for a number of operational use-cases in seconds. If you’d like to learn more about how Amazon AWS VPC logs can add value for your company, contact us.