-
Notifications
You must be signed in to change notification settings - Fork 479
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lambda support for DynamoDb Kinesis Stream processing #1554
Comments
Correction: It seems that using According to the docs here https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Services/DynamoDBv2/Generated/Model/StreamRecord.cs#L45-L52 the value should have second granularity, is this a bug in kinesis? As a work around for now I copied the |
@danny-zegel-zocdoc Thanks for submitting feature request. Please review if this issue is similar to existing issue #839 in our backlog. If yes, kindly close this issue to avoid duplicate issue. Thanks, |
@ashishdhingra the bug aspect of this issue does seem to be the same issue as #839 however the bug was not the primary aspect of this request. This request is primarily suggesting that there be a more prescribed/standardized way of parsing Kinesis records that come from DynamoDb being that what you need to do right now is not at all intuitive. |
Agreed. But changing the type in any of the nested properties would be a breaking change for existing customer as you mentioned. Creation of separate package needs to be reviewed by the team. |
Alternatively a new type could be created in the existing package 🤷. I'll leave it up to your team to decide if/how this would be done. Thank you for the consideration. |
I think adding a @ashishdhingra We should add this feature request to our backlog but to level set @danny-zegel-zocdoc I'm not sure when we will be able to get to this and in the Lambda space getting ready for .NET 8 is our priority. If you are interested in doing a PR I'm happy to take a look at it. |
Describe the feature
Given a DynamoDb table configured with a Kinesis Stream and a Lambda to process the Kinesis Stream, the
Amazon.Lambda.KinesisEvents
package providesKinesisEvent
as the lambda input object with the following structureIn the case of a DynamoDb Kinesis Stream the
Record.Data
MemoryStream
contains a non-encoded Json string which appears to match the typeAmazon.DynamoDBv2.Model.Record
from theAWSSDK.DynamoDBv2
package. Unfortunately the Json cannot be easily deserialized into aAmazon.DynamoDBv2.Model.Record
due to the nested propertyApproximateCreationDateTime
which is in the form of epoch time in the JSON (i.e. an integer) but is typed asDateTime
inAmazon.DynamoDBv2.Model.StreamRecord
. I'm also concerned that there might be breaking edge-cases when directly deserializingAttributeValue
types which are also nested insideAmazon.DynamoDBv2.Model.StreamRecord
.I seem to be able to deserialize
Record.Data
successfully using theAmazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer
serializer which is really not intuitive and disappointing to have to do.I'm wondering if it would be reasonable to either create a
KinesisDynamoDbEvent
type either inAmazon.Lambda.KinesisEvents
or another package which types the nestedData
property as a DynamoDbStreamRecord
instead of a MemoryStream, or perhaps provide an intuitive utility to deserialize theMemoryStream
appropriately.Use Case
When using Lambda to process a Kinesis Stream that is backed by a DynamoDb table it is difficult to deserialize the
Amazon.Kinesis.Model.Record
MemoryStream
property to a dynamodb record.Proposed Solution
No response
Other Information
No response
Acknowledgements
AWS .NET SDK and/or Package version used
Amazon.Lambda.KinesisEvents
Targeted .NET Platform
.NET 6 and up
Operating System and version
N/A
The text was updated successfully, but these errors were encountered: