Overview
Today I will present a sample approach to reduce the telemetry ingest costs by 84%. We will achieve that by properly designing the AWS infrastructure.
I will also help you understand the output of the AWS Pricing Calculator and AWS infrastructure cost factors.
Sample scenario
We have 100 devices constantly connected to the AWS IoT Core.
A single device sends 10 KB of telemetry data every 5 seconds.
Our task is to deliver telemetry data to S3 Bucket at a minimal cost.
Some math
Let’s start with some basic math to calculate the estimated load - we need this data as input for the AWS Pricing Calculator.
| Number of devices | Number of messages sent daily by single device | Average message size [KB] | Total number of daily messages | Total size of daily messages [KB] |
|---|---|---|---|---|
| 100 | 17 280* | 10 | 1 728 000 | 17 280 000 |
*) device sends a message every 5 seconds, 86 400 seconds in a day / 5 seconds = 17 280 daily messages
| Total number of monthly messages | Total size of monthly messages [KB] | Total size of monthly messages [GB] |
|---|---|---|
| 52 560 000** | 525 600 000 | 501.25 |
**) I assumed 43,800 minutes per month
Cost factors
The first step is to send telemetry data from our devices to the AWS IoT Core.
To reduce the telemetry transfer costs I suggest using the Basic Ingest. This way we can avoid the AWS IoT Core messaging costs.
Important: devices and applications can not subscribe to Basic Ingest reserved topics! We need to use the IoT Rule to handle received messages.
AWS IoT Core
Connectivity
Our devices send telemetry data every 5 seconds and they are constantly connected to the AWS IoT Core.
The estimated connectivity cost looks as follows (copied from the AWS Pricing Calculator with my notes):
43,800 minutes per month x 0.00000008 USD x 100 devices = 0.35 USD per month Note: we have 100 devices that are constantly connected to the IoT Core
Minutes of connection cost (monthly): 0.35 USD
Rule Engine
Since we are using the Basic Ingest, we need to invoke an IoT Rule to handle received data.
The estimated Rule Engine cost looks as follows (copied from the AWS Pricing Calculator with my notes):
10 KB per message / 5 KB message increment = 2 message(s) Note: Rules and actions are metered in 5 KB increments of the message size, which is why one 10 KB message is billed as two 5 KB messages.
RoundUp (2) = 2 metered message(s)
1 rule(s) executed per message x 525,600 message(s) x 2 metered message(s) x 100 devices = 105,120,000 total billable rules Note: this is the invocation of our Basic Ingest rule
1 action(s) triggered per rule x 105,120,000 total billable rules = 105,120,000 total billable actions Note: this is the invocation of our S3 action which stores telemetry in S3 bucket, actions are metered in 5 KB increments (and we send 10 KB messages) that is why this number is twice time bigger than actual S3 calls
105,120,000 Rules per month x 0.00000015 USD = 15.77 USD (Rules)
105,120,000 Actions per month x 0.00000015 USD = 15.77 USD (Actions)
15.77 USD + 15.77 USD = 31.54 USD per month
Rules Engine cost (monthly): 31.54 USD
Options
We have multiple options to store received telemetry data in S3 Bucket. I will discuss two approaches:
- direct write of telemetry data to S3 Bucket
- buffering of telemetry data before writing to S3 Bucket
Option 1 - direct write to S3 Bucket
This is the simplest option I could think of - our IoT Rule receives telemetry data and writes it directly to the S3 Bucket.
Because of its simplicity, it “should be” the cheapest option.
Let’s validate our intuition.
S3
Unit conversions
S3 Standard Average Object Size: 10 KB x 9.5367432e-7 GB in a KB = 0.0000095367432 GB Note: 10 KB is the size of a single telemetry message
Pricing calculations
52,560,000 PUT requests for S3 Standard Storage x 0.000005 USD per request = 262.80 USD (S3 Standard PUT requests cost) Note: this is the cost of invoking S3 API to store each telemetry message
S3 Standard cost (monthly): 262.80 USD
S3 Standard cost (upfront): 0.00 USD
Note: to keep things simple, I excluded the cost of S3 storage from my calculation.
Total monthly cost:
| IoT Core [USD] | S3 [USD] | Total [USD] |
|---|---|---|
| 31.54 | 262.80 | 294.34 |
Option 2 - buffering S3 API calls using Kinesis Data Firehose
In this option, we are introducing an additional AWS Service - Kinesis Data Firehose. Since we need to pay for the usage of every service, our monthly bill should be higher than in the first option.
Let’s do some math.
Kinesis Data Firehose
We use Kinesis Data Firehose to buffer telemetry messages received by the IoT Core before writing them to the S3 Bucket.
Unit conversions management events
Number of records for data ingestion: 12 per minute * (60 minutes in an hour x 730 hours in a month) = 525600 per month Note: each device sends a telemetry message every 5 seconds, which equals 12 messages per minute
Pricing calculations
525,600 monthly records x 100 unit multiplier = 52,560,000 total records per month Note: we use 100 devices which is why there is the 100 unit multiplier
RoundUp (10) = 10 KB Average record size (rounded to nearest 5 KB increment) Note: Kinesis Data Firehose ingestion pricing is tiered and billed per GB ingested in 5KB increments
52,560,000 records x 10 KB = 525,600,000.00 KB per month
525,600,000 KB / 1048576 KB in a GB = 501.251220703125 GB per month
Tiered price for: 501.251220703125 GB
501.251220703125 GB x 0.0310000000 USD = 15.54 USD
Total tier cost = 15.54 USD for data ingestion
Kinesis Data Firehose data ingestion cost (monthly): 15.54 USD
S3
Kinesis Data Firehose writes buffered data to S3 Bucket every 900 seconds. This way we reduce the number of S3 API invocations.
Unit conversions
S3 Standard Average Object Size: 1800 KB x 9.5367432e-7 GB in a KB = 0.001716613776 GB
Pricing calculations
2,920 PUT requests for S3 Standard Storage x 0.000005 USD per request = 0.0146 USD (S3 Standard PUT requests cost): Note: 43,800 minutes per month x 60 seconds / 900 seconds of Kinesis Data Firehose Buffer interval gives 2,920 uploads to S3 Bucket
S3 Standard cost (monthly): 0.01 USD
S3 Standard cost (upfront): 0.00 USD
Total monthly cost:
| IoT Core [USD] | Kinesis Data Firehose [USD] | S3 [USD] | Total [USD] |
|---|---|---|---|
| 31.54 | 15.54 | 0.01 | 47.09 |
The IoT Core cost of telemetry transfer is the same as in option 1.
We introduced the cost of Kinesis Data Firehose (15.54 USD) and reduced the S3 cost from 262.80 to 0.01.
By using the Kinesis Data Firehose we are saving 247.25 USD per month - that is an 84% cost reduction!
Summary
In this simplified example I wanted to demonstrate the importance of careful solution design. Sometimes the “simplest design” is not the best/cheapest one.
Pricing estimations are not easy and require knowledge of cost factors related to AWS services. Please mind that these kinds of estimations are never accurate, but they help to define the order of magnitude of infrastructure cost.
The same business case can be implemented using multiple different solutions, the cost estimation gives additional data to make an informed decision.
You can find my telemetry ingest costs calculation in AWS Pricing Calculator:
https://calculator.aws/#/estimate?id=d5f3ab8025dbe7809ccc2c3a45312aec639d69e1
Version exported to PDF: telemetry_ingest_costs_calculation.pdf
Please let me know if you would be interested in the AWS IoT costs estimation training.
Video
