Streaming ingestion with Kinesis, Firehose and MSK
Streaming questions test whether you can match latency, management effort and consumer needs to the right service.
Kinesis Data Streams
Amazon Kinesis Data Streams is a durable, ordered stream. In provisioned mode you manage shards: each accepts about 1 MB/s or 1,000 records/s in and delivers 2 MB/s out. On-demand mode scales automatically. Retention is 24 hours by default and can be extended up to 365 days.
The partition key decides the shard. Ordering is guaranteed only within a shard. A skewed key creates a hot shard and throttling errors. Producers can use the SDK, the Kinesis Producer Library (batching and aggregation) or the Kinesis agent. Consumers use the Kinesis Client Library, Lambda event source mappings or Flink. Shared consumers split 2 MB/s per shard. Enhanced fan-out gives each registered consumer a dedicated 2 MB/s.
Amazon Data Firehose
Amazon Data Firehose (formerly Kinesis Data Firehose) is a managed delivery service. It needs no consumer code and loads to S3, Redshift, OpenSearch, Splunk and HTTP endpoints. It buffers by size or time, can transform records with Lambda, and can convert JSON to Parquet or ORC using a Glue table schema. Dynamic partitioning writes to S3 prefixes based on record content. Delivery is near real time, not real time.
Managed Service for Apache Flink
Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs Flink applications with SQL, Java or Python for windowed aggregations, joins and anomaly detection. It reads from Kinesis, MSK and other sources.
Amazon MSK
Amazon MSK runs Apache Kafka. Choose it when you already use Kafka APIs, need Kafka tooling, or want very long retention with tiered storage. MSK Serverless removes capacity planning. Glue streaming jobs, Lambda and Flink can read MSK topics. MSK Connect runs Kafka Connect connectors.
Quick comparison
| Need | Pick |
|---|---|
| Custom consumers, replay, ordering by key | Kinesis Data Streams |
| Load to S3 or Redshift with no code | Amazon Data Firehose |
| Real-time analytics on streams | Managed Service for Apache Flink |
| Existing Kafka workloads | Amazon MSK |
Exam traps
- Firehose is not real time and does not support replay. Data Streams does.
- Adding shards does not fix a hot partition key. Fix the key distribution first.
- Throughput limits apply per shard, not per stream.
- Firehose converts to Parquet or ORC, not to Avro or CSV.
- The old names (Kinesis Data Analytics, Kinesis Data Firehose) still appear in older material, so recognize both.
Practice questions
Choose an answer, then review the explanation to check your understanding.
1. Clickstream events must be delivered to S3 as Parquet with no custom consumer code, and a few minutes of delay is acceptable. Which service fits best?
Show answer
Answer: B.Amazon Data Firehose is fully managed, buffers records and can convert JSON to Parquet or ORC before writing to S3.
2. A Kinesis data stream in provisioned mode throws ProvisionedThroughputExceededException for one partition key. What is the likely cause and fix?
Show answer
Answer: B.Each shard handles about 1 MB/s or 1,000 records/s in. A skewed partition key overloads one shard.
3. A team needs windowed aggregations over a stream, with results written to another stream. Which service performs the processing?
Show answer
Answer: A.Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs stateful stream processing such as windowed aggregations.
4. Several independent applications must each read the same Kinesis stream at full speed without sharing throughput. What should be used?
Show answer
Answer: B.Enhanced fan-out gives each consumer its own dedicated 2 MB/s per shard.