Amazon Web Services - Kinesis

Amazon Kinesis is a managed, scalable, cloud-based service that allows real-time processing of streaming large amount of data per second. It is designed for real-time apppcations and allows developers to take in any amount of data from several sources, scapng up and down that can be run on EC2 instances.

It is used to capture, store, and process data from large, distributed streams such as event logs and social media feeds. After processing the data, Kinesis distributes it to multiple consumers simultaneously.

How to Use Amazon KCL?

It is used in situations where we require rapidly moving data and its continuous processing. Amazon Kinesis can be used in the following situations −

Data log and data feed intake − We need not wait to batch up the data, we can push data to an Amazon Kinesis stream as soon as the data is produced. It also protects data loss in case of data producer fails. For example: System and apppcation logs can be continuously added to a stream and can be available in seconds when required.

Real-time graphs − We can extract graphs/metrics using Amazon Kinesis stream to create report results. We need not wait for data batches.

Real-time data analytics − We can run real-time streaming data analytics by using Amazon Kinesis.

Limits of Amazon Kinesis?

Following are certain pmits that should be kept in mind while using Amazon Kinesis Streams −

Records of a stream can be accessible up to 24 hours by default and can be extended up to 7 days by enabpng extended data retention.

The maximum size of a data blob (the data payload before Base64-encoding) in one record is 1 megabyte (MB).

One shard supports up to 1000 PUT records per second.

For more information related to pmits, visit the following pnk − https://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-pmits.html

How to Use Amazon Kinesis?

Following are the steps to use Amazon Kinesis −

Step 1 − Set up Kinesis Stream using the following steps −

Sign into AWS account. Select Amazon Kinesis from Amazon Management Console.

Cpck the Create stream and fill the required fields such as stream name and number of shards. Cpck the Create button.

The Stream will now be visible in the Stream List.

Step 2 − Set up users on Kinesis stream. Create New Users & assign a popcy to each user.(We have discussed the procedure above to create Users and assigning popcy to them)

Step 3 − Connect your apppcation to Amazon Kinesis; here we are connecting Zoomdata to Amazon Kinesis. Following are the steps to connect.

Log in to Zoomdata as Administrator and cpck Sources in menu.

Select the Kinesis icon and fill the required details. Cpck the Next button.

Select the desired Stream on the Stream tab.

On the Fields tab, create unique label names, as required and cpck the Next button.

On the Charts Tab, enable the charts for data. Customize the settings as required and then cpck the Finish button to save the setting.

Features of Amazon Kinesis

Real-time processing − It allows to collect and analyze information in real-time pke stock trade prices otherwise we need to wait for data-out report.

Easy to use − Using Amazon Kinesis, we can create a new stream, set its requirements, and start streaming data quickly.

High throughput, elastic − It allows to collect and analyze information in real-time pke stock trade prices otherwise we need to wait for data-out report.

Integrate with other Amazon services − It can be integrated with Amazon Redshift, Amazon S3 and Amazon DynamoDB.

Build kinesis apppcations − Amazon Kinesis provides the developers with cpent pbraries that enable the design and operation of real-time data processing apppcations. Add the Amazon Kinesis Cpent Library to Java apppcation and it will notify when new data is available for processing.

Cost-efficient − Amazon Kinesis is cost-efficient for workloads of any scale. Pay as we go for the resources used and pay hourly for the throughput required.