Module Review: Streams & Lambda

[!NOTE] This module explores the core principles of Module Review: Streams & Lambda, deriving solutions from first principles and hardware constraints to build world-class, production-ready expertise.

1. Key Takeaways

  1. Event-Driven Architecture: DynamoDB Streams allow you to decouple your database from your downstream consumers (Search, Analytics, Notifications).
  2. Strict Ordering: Changes to a single Item (Partition Key) are strictly ordered.
  3. Lambda Integration: The Event Source Mapping (ESM) polls shards for you. You must tune BatchSize and BatchWindow for optimal throughput/latency.
  4. Error Handling: A single failed record can block the entire shard (“Poison Pill”). Always use BisectBatchOnFunctionError and return BatchItemFailures.
  5. Monitoring: Watch IteratorAge. If it grows, your consumers are too slow or failing.

2. Flashcards

Retention Period

How long does DynamoDB Streams keep data?

24 Hours

Data is deleted after 24 hours. For longer retention (up to 365 days), use Kinesis Data Streams.

IteratorAge

What does a high IteratorAge indicate?

Processing Lag

It means your consumer is processing old records. You are falling behind the write rate of the table.

BisectBatchOnFunctionError

What does this setting do?

Splits Failed Batches

If a batch fails, AWS splits it into two halves and retries them separately to isolate the bad record.

Stream View Type

Which view type gives you both the before and after state of an item?

NEW_AND_OLD_IMAGES

It provides the full item as it appeared before and after the modification.

Idempotency

Why must stream consumers be idempotent?

At-Least-Once Delivery

DynamoDB Streams guarantees at-least-once delivery. Retries may cause the same record to be delivered multiple times.

Concurrency Limit

How many Lambda instances can process a single shard at once?

Exactly One

One Lambda invocation per shard to guarantee strict ordering of records.

3. Production Checklist

Before going live with DynamoDB Streams, ensure you have configured the following:

4. Cheat Sheet

Concept Description Key Metric / Setting
Streams Time-ordered sequence of item changes. Retained for 24h. View Type (NEW_AND_OLD_IMAGES)
Lambda Integration ESM polls shards and invokes Lambda synchronously. BatchSize, BatchWindow
Error Handling Unhandled errors retry the whole batch indefinitely. BisectBatchOnFunctionError, DLQ
Kinesis Streams For higher scale, multiple consumers, longer retention. Shards, Kinesis Firehose

5. Quick Revision

  • DynamoDB Streams vs Kinesis Data Streams: Use DynamoDB Streams for simple reactive logic (1-2 consumers). Use Kinesis for analytics pipelines, long retention (up to 365 days), and fan-out (up to 20 consumers).
  • IteratorAge: The most critical metric. If it grows, your Lambda is falling behind the stream or is stuck in an error loop.
  • Idempotency: At-least-once delivery means your consumers must be idempotent (e.g., using UPSERT operations).
  • Partial Failures: Return BatchItemFailures from Lambda to only retry the failed records instead of the entire batch.

Review all key terms and definitions in the DynamoDB Glossary.

7. Next Steps

You have mastered the event-driven side of DynamoDB. In the final module, we will explore advanced optimization techniques, including Time To Live (TTL), Transactions, and Capacity Planning.

Module 5: Optimization & Best Practices →