Kinesis Data Firehose Integration
A fully managed service to help you load streaming data continuously to data stores and other destinations in near real time. With Site24x7's AWS integration you can monitor metrics on throughput, delivery, data transformation and API activity to make sure records are reaching their destination.
Setup and configuration
- If you haven't done it already, enable access to your AWS resource by creating Site24x7 as an IAM user or by creating a cross-account IAM role between your account and Site24x7's AWS account. Learn more.
- Next, In the Integrate AWS Account page, please make sure the Kinesis Firehose checkbox is selected in the Services to be discovered field. Learn more.
Policies and Permissions
Please make sure the following read level API actions are present in the IAM policy assigned to Site24x7 entity. learn more.
- "firehose:ListDeliveryStreams",
- "firehose:ListTagsForDeliveryStream",
- "firehose:DescribeDeliveryStream"
Polling frequency
Performance metrics are collected as per the poll frequency set (1 minute to a day), by querying the CloudWatch API. Learn more.
To view metrics
- Log in to the Site24x7 console, click on AWS > Monitored AWS account and select Kinesis Firehose from the drop down
- From the list of monitored delivery streams, choose the stream for which you want to access metrics for.
- The various graphs applicable for your delivery stream are displayed in the stream details tab.
Supported metrics
The following metrics are collected on a per delivery stream level:
Based on backup
| Attribute | Description | Data type | Statistics |
|---|---|---|---|
| BackupToS3.Bytes | Measures the number of bytes delivered to Amazon S3 for backup. | Bytes | Sum |
| BackupToS3.DataFreshness (Sum) | Measures the age of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. | Seconds | Sum |
| BackupToS3.DataFreshness (Maximum) | Measures the age of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. | Seconds | Maximum |
| BackupToS3.Records | Measures the number of records delivered to Amazon S3 for backup. | Count | Sum |
| BackupToS3.Success | Measures the sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. | Count | Sum |
Based on destination
| Attribute | Description | Data type | Statistics |
|---|---|---|---|
| DeliveryToElasticsearch.Bytes | Measures the number of bytes indexed to Amazon ES. | Bytes | Sum |
| DeliveryToElasticsearch.Records | Measures the number of records indexed to Amazon ES. | Count | Sum |
| DeliveryToElasticsearch.Success | Measures the sum of the successfully indexed records over the sum of records that were attempted. | Count | Sum |
| DeliveryToRedshift.Bytes | Measures the number of bytes copied to Amazon Redshift. | Bytes | Sum |
| DeliveryToRedshift.Records | Measures the number of records copied to Amazon Redshift. | Count | Sum |
| DeliveryToRedshift.Success | Measures the sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands. | Count | Sum |
| DeliveryToS3.Bytes | Measures the number of bytes delivered to Amazon S3 over the specified time period. | Bytes | Sum |
| DeliveryToS3.DataFreshness (Sum) | Measures age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. | Seconds | Sum |
| DeliveryToS3.DataFreshness (Maximum) | Measures age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. | Seconds | Maximum |
| DeliveryToS3.Records | Measures the number of records delivered to Amazon S3. | Count | Sum |
| DeliveryToS3.Success | Measures the sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. | Count | Sum |
| DeliveryToSplunk.Bytes | Measures the number of bytes delivered to Splunk. | Bytes | Sum |
| DeliveryToSplunk.DataFreshness (Sum) | Measures the age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to Splunk. | Seconds | Sum |
| DeliveryToSplunk.DataFreshness (Maximum) | Measures the age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to Splunk. | Seconds | Maximum |
| DeliveryToSplunk.Records | Measures the number of records delivered to Splunk | Count | Sum |
| DeliveryToSplunk.Success | Measures the sum of the successfully indexed records over the sum of records that were attempted. | Count | Sum |
| IncomingBytes | Measures the number of bytes ingested into the Kinesis Data Firehose stream. | Bytes | Sum |
| IncomingRecords | Measures the number of records ingested into the Kinesis Data Firehose stream. | Count | Sum |
API-level metrics
| Attribute | Description | Data type | Statistics |
|---|---|---|---|
| DescribeDeliveryStream.Latency | Measures the time taken per DescribeDeliveryStream operation. | Milliseconds | Average |
| DescribeDeliveryStream.Requests | Measures the total number of DescribeDeliveryStream requests. | Count | Sum |
| ListDeliveryStreams.Latency | Measures the time taken per ListDeliveryStream operation. | Milliseconds | Sum |
| ListDeliveryStreams.Requests | Measures the total number of ListFirehose requests. | Count | Sum |
| PutRecord.Bytes | Measures the number of bytes put to the Kinesis Data Firehose delivery stream. | Bytes | Sum |
| PutRecord.Latency | Measures the time taken per PutRecord operation. | Milliseconds | Sum |
| PutRecord.Requests | Measures the total number of PutRecord requests. | Count | Sum |
| PutRecordBatch.Bytes | Measures the number of bytes put to the Kinesis Data Firehose delivery stream using PutRecordBatch. | Bytes | Sum |
| PutRecordBatch.Latency | Measures the time taken per PutRecordBatch operation. | Milliseconds | Average |
| PutRecordBatch.Records | Measures the total number of records from PutRecordBatch operations. | Count | Sum |
| PutRecordBatch.Requests | Measures the total number of PutRecordBatch requests. | Count | Sum |
| UpdateDeliveryStream.Latency | Measures the time taken per UpdateDeliveryStream operation. | Milliseconds | Average |
| UpdateDeliveryStream.Requests | Measures the total number of UpdateDeliveryStream requests | Count | Sum |
Data transformation metrics
Data points for the following performance metrics will only be available if you've enabled data transformation.
| Attribute | Description | Data type | Statistics |
|---|---|---|---|
| ExecuteProcessing.Duration | Measures the time it takes for each Lambda function invocation performed by Kinesis Data Firehose. | Seconds | Sum |
| ExecuteProcessing.Success | Measures the sum of the successful Lambda function invocations over the sum of the total Lambda function invocations. | Count | Sum |
| SucceedProcessing.Records | Measures the number of successfully processed records over the specified time period. | Count | Sum |
| SucceedProcessing.Bytes | Measures the number of successfully processed bytes over the specified time period. | Bytes | Sum |
Configuration details
The following general configuration data is collected on a per delivery stream basis:
| Attribute | Description |
|---|---|
| Stream name | Displays the names for the delivery stream. |
| Stream ARN | Shows the Amazon Resource Name for the Delivery stream. |
| Stream status | Displays the current status for the delivery stream. |
| Stream Type | Displays the logical category or type of index. |
| Version ID | Shows the version number for the delivery stream |
| Delivery Start Time | Displays the time stamp when the record was first received by Firehose. |
| Source Stream ARN | Shows the Amazon Resource Name for the source - direct or Kinesis stream. |
| Source IAM Role | Displays the IAM Role attached to the source stream. |
| Destination Type | Displays the destination - ElasticSearch cluster, S3 bucket, RedShift or Splunk |