The AWS Kinesis Node enables a Rayven workflow to ingest streaming data from an AWS Kinesis Data Stream. It connects using secure AWS credentials and reads data records from one or more shards, emitting parsed JSON into the workflow.
What It Does
This node connects to a Kinesis stream in a specified AWS region and retrieves records at defined intervals. It supports UID mapping, time-based filtering, and reset behavior for shard iterators. Each record is treated as an incoming event and can be processed through the workflow.
Step-by-Step: How to Configure the AWS Kinesis Node
-
Add the node
-
Drag the AWS Kinesis Node from the Inputs section to the canvas.
-
-
Open the configuration window
-
Double-click the node to configure credentials, stream access, and payload behavior.
-
-
Connect to downstream logic or visualization nodes
-
Link any workflow node that should receive records from Kinesis.
-
⚙️ Configuration Fields
AWS Credentials & Stream Info
| Field | Requirement | Description |
|---|---|---|
| Node Name* | Required | Internal name for the node in the workflow. |
| AWS Access Key ID* | Required | Access key associated with an AWS IAM user that has read access to Kinesis. |
| AWS Secret Access Key* | Required | Secret key paired with the access key. |
| Kinesis Stream Name* | Required | Name of the AWS Kinesis stream to subscribe to. |
| AWS Region* | Required | AWS region where the stream resides (e.g., ap-northeast-1, us-west-2). |
Polling Configuration
| Field | Description |
|---|---|
| Activation Time Request (minutes) | Optional. Time offset (in minutes) to start from. Use this to read from a point in the past. |
| Request Interval (minutes) | Required. How often to poll the stream for new records. Default: 5. |
Payload & Time Mapping
| Field | Description |
|---|---|
| UID JSON Key | Field in the Kinesis record that represents the device UID. Used for grouping and routing. |
| Reset Shard Iterator | If enabled, resets the shard pointer to the configured activation time. |
| Take Date Field from JSON | If enabled, extracts timestamp from a field within each record instead of using AWS ingestion time. |
| Timestamp JSON Key | Field in the record containing a timestamp value. |
| Timestamp Format | Format used to parse the timestamp (e.g., yyyy-MM-dd HH:mm:ss). |
| Time Zone | Time zone for interpreting the timestamp (e.g., UTC, UID time zone). |
Output Behavior
Each incoming record is parsed into JSON and emitted into the workflow. Timestamps and UIDs are extracted and normalized if configured.
Example Input Record:
{
"device_id": "sensor-01",
"temperature": 27.5,
"timestamp": "2025-07-15 14:35:00"
}
Output (with UID and timestamp parsing):
{
"uid": "sensor-01",
"temperature": 27.5,
"timestamp": "2025-07-15T14:35:00Z"
}
Best Practices
-
Use IAM access keys with permission limited to the target Kinesis stream.
-
If consuming from high-frequency streams, use short Request Intervals (e.g., 1–2 minutes).
-
Enable timestamp parsing only when the incoming data includes well-formed datetime fields.
-
Use the Reset Shard Iterator option during development or testing to replay older data.
-
If
Take Date Field from JSONis disabled, the system will use the AWS arrival timestamp instead.
Use Cases
-
Ingest IoT sensor data from AWS Kinesis for real-time analysis
-
Replay past telemetry data from shards for test scenarios
-
Process time-aligned event data streams with UID-based routing
-
Connect edge or cloud-based devices publishing to Kinesis into Rayven workflows
FAQ
Q: What happens if the shard iterator expires?
A: The node resets the iterator and may miss some records unless reset behavior is configured correctly.
Q: Can I connect to multiple streams at once?
A: No. Each AWS Kinesis node connects to one stream. Use multiple nodes for different streams.
Q: What if I don't define a UID?
A: The node will emit payloads without device grouping. This may affect downstream label-based visualization or logic.