Sink data from RisingWave to Amazon DynamoDB

This guide describes how to sink data from RisingWave to DynamoDB using the DynamoDB sink connector in RisingWave.

Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. It provides consistent, single-digit millisecond response times and offers built-in security, backup and restore, and in-memory caching.

PREMIUM EDITION FEATURE

This is a Premium Edition feature. All Premium Edition features are available out of the box without additional cost on RisingWave Cloud. For self-hosted deployments, users need to purchase a license key to access this feature. To purchase a license key, please contact sales team at [e-mail].

For a full list of Premium Edition features, see RisingWave Premium Edition.

PUBLIC PREVIEW

This feature is currently in public preview, meaning it is nearing the final product but may not yet be fully stable. If you encounter any issues or have feedback, please reach out to us via our Slack channel. Your input is valuable in helping us improve this feature. For more details, see our Public Preview Feature List.

Syntax

CREATE SINK [ IF NOT EXISTS ] sink_name
[FROM sink_from | AS select_query]
WITH (
   connector = 'dynamodb',
   connector_parameter = 'value', ...
)
FORMAT data_format ENCODE data_encode [ (
    format_parameter = 'value'
) ];

Parameters

Field	Note
table	Required. Name of the DynamoDB table where you want to write the data.
primary_key	Required. A pair of columns representing the partition key and sort key of DynamoDB, e.g., key1,key2, separated by comma.
aws.region	Required. AWS region where your DynamoDB table is hosted.
aws.endpoint_url	Optional. Custom endpoint URL for connecting to DynamoDB. Useful for testing with DynamoDB Local.
aws.credentials.access_key_id	Conditional. AWS access key ID for accessing DynamoDB. Must be specified when using access key ID and secret key.
aws.credentials.secret_access_key	Conditional. AWS secret access key for accessing DynamoDB. Must be specified when using access key ID and secret key.
aws.credentials.role.arn	Conditional. ARN of the IAM role to assume for accessing DynamoDB. Must be specified when using AssumeRole.
aws.credentials.role.external_id	Conditional. External ID for assuming the IAM role specified in aws.credentials.role.arn.
aws.profile	Optional. The name of the AWS CLI profile to use for accessing DynamoDB. If specified, it overrides the default profile.
dynamodb.max_batch_item_nums	Optional. The maximum number of items in the BatchWriteItem operation. It must be larger than 1, and less than or equal to 25. The default is 25.
dynamodb.max_future_send_nums	Optional. The maximum number of concurrent write futures in DynamoDB. It must be less than 360, and the default is 256. This is derived from user-defined max_parallelism_units (40000 by default). If the write throughput of RisingWave exceeds the max_parallelism_units set in DynamoDB, an error would be reported.

Partition key and sort key mapping

For the DynamoDB sink to function correctly, it is required that the source table in RisingWave has a composite primary key comprising two columns, corresponding to the partition key and sort key defined in the target DynamoDB table.

For example, if you are sinking data into a DynamoDB table named movies that is configured with a partition key title and a sort key year, you must ensure that the RisingWave table schema also defines a composite primary key with the same columns. Below is an example schema definition for the RisingWave table:

CREATE TABLE IF NOT EXISTS movies (
    year integer,
    title varchar,
    description varchar,
    primary key (year, title)
);

This makes sure that the data structure in RisingWave aligns with the key definition in DynamoDB, thus preventing any further potential errors.

Data type mapping

RisingWave Data Type	DynamoDB Data Type
boolean	Bool
smallint	number (N)
integer	number (N)
bigint	number (N)
numeric	number (N)
real	number (N)
double precision	number (N)
serial	number (N)
character varying (varchar)	string (S)
bytea	binary
date	string (S)
time without time zone	string (S)
timestamp without time zone	string (S)
timestamp with time zone	string (S)
interval	string (S)
struct	map (M)
array	list (L)
JSONB	string (S)

The struct datatype in RisingWave will map to map (M) in DynamoDB in a recursive way. Refer to source code for details.