Skip to content

Conversation

@mgagliardo
Copy link
Contributor

When the data put into kinesis streams is put into an SdkBytes byte format and not standard bytes or base64 format it needs to be decoded. This PR fixes that issue.

Fixes #3588

Steps to reproduce:

Java code:
src/main/java/com/test/PutDataExample.java

package com.test;

import java.nio.ByteBuffer;

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.kinesis.AmazonKinesisClient;
import com.amazonaws.services.kinesis.model.PutRecordRequest;
import com.amazonaws.services.kinesis.model.PutRecordResult;

public class PutDataExample {

  AmazonKinesisClient kinesisClient;

  PutDataExample() {
    String accessKey = "test";
    String secretKey = "test";
    AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
    kinesisClient = new AmazonKinesisClient(credentials);
    kinesisClient.setEndpoint("http://localhost:4566", "kinesis", "us-east-1");
    System.out.println("starting the PutDataExample Application");
  }

  public void putData(String fileContent, String session) throws Exception {
    final String myStreamName = "test";

    PutRecordRequest putRecordRequest = new PutRecordRequest();
    putRecordRequest.setStreamName(myStreamName);
    String putData = fileContent;
    putRecordRequest.setData(ByteBuffer.wrap(putData.getBytes()));
    putRecordRequest.setPartitionKey("session" + session);
    PutRecordResult putRecordResult = kinesisClient.putRecord(putRecordRequest);
    System.out.println("Put Record Successful, partition key : " + putRecordRequest.getPartitionKey() + ", ShardID : " + putRecordResult.getShardId());
    System.out.println(fileContent);
    System.out.println("Sequence Number: " + putRecordResult.getSequenceNumber());

  }

  public static void main(String[] args) throws Exception {
    new PutDataExample().putData("User has logged in successfully", "s1");
    new PutDataExample().putData("User started to bet ", "s2");
    new PutDataExample().putData("User has logged out", "s3");
    new PutDataExample().putData("User has exit from the system", "s4");
  }
}

pom.xml (I'm probably using more dependencies than expected):



<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <groupId>com.test.</groupId>
   <artifactId>my-app</artifactId>
   <version>1.0-SNAPSHOT</version>
   <properties>
      <maven.compiler.source>1.7</maven.compiler.source>
      <maven.compiler.target>1.7</maven.compiler.target>
   </properties>
   <dependencies>
      <dependency>
         <groupId>software.amazon.kinesis</groupId>
         <artifactId>amazon-kinesis-client</artifactId>
         <version>2.3.3</version>
      </dependency>
      <dependency>
         <groupId>com.amazonaws</groupId>
         <artifactId>aws-java-sdk</artifactId>
         <version>1.11.954</version>
      </dependency>
      <dependency>
         <groupId>com.amazonaws</groupId>
         <artifactId>aws-java-sdk-core</artifactId>
         <version>1.11.954</version>
      </dependency>
      <dependency>
         <groupId>software.amazon.awssdk</groupId>
         <artifactId>kinesis</artifactId>
         <version>2.15.81</version>
      </dependency>
   </dependencies>
   <build>
      <plugins>
         <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
               <archive>
                  <manifest>
                     <mainClass>com.test.PutDataExample</mainClass>
                  </manifest>
               </archive>
               <descriptorRefs>
                  <descriptorRef>jar-with-dependencies</descriptorRef>
               </descriptorRefs>
            </configuration>
         </plugin>
      </plugins>
   </build>
</project>
  1. Execute mvn clean compile assembly:single
  2. Create the stream:
awslocal kinesis create-stream --stream-name test --shard-count 1
  1. Then insert data into the stream: java -cp target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar com.test.PutDataExample
  2. Get the shard iterator:
awslocal kinesis get-shard-iterator --stream-name test --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON
  1. Insert data in base64 format with the following python code:
import base64
import boto3
import json
import random
import calendar
from datetime import datetime

thing_id = 'aa-bb'

payload = {
            'prop': str(random.randint(40, 120)),
            'timestamp': calendar.timegm(datetime.utcnow().timetuple()),
            'thing_id': thing_id
          }

client = boto3.client('kinesis', region_name='us-east-1', endpoint_url='http://localhost:4566', aws_access_key_id='test', aws_secret_access_key='test')

response = client.put_record(
  StreamName='test',
  Data=base64.b64encode(json.dumps(payload).encode('utf-8')),
  PartitionKey=thing_id
)
  1. Read data from it:
awslocal kinesis get-records --shard-iterator  $SHARD_ITERATOR_STR

Example output:

{
    "Records": [
        {
            "SequenceNumber": "49615513957620404616321850166192823801450482321451909122",
            "ApproximateArrivalTimestamp": "2021-02-14T21:12:49+01:00",
            "Data": "User has logged in successfully",
            "PartitionKey": "sessions1"
        },
        {
            "SequenceNumber": "49615513957620404616321850166194032727270096950626615298",
            "ApproximateArrivalTimestamp": "2021-02-14T21:12:49+01:00",
            "Data": "User started to bet ",
            "PartitionKey": "sessions2"
        },
        {
            "SequenceNumber": "49615513957620404616321850166195241653089711579801321474",
            "ApproximateArrivalTimestamp": "2021-02-14T21:12:49+01:00",
            "Data": "User has logged out",
            "PartitionKey": "sessions3"
        },
        {
            "SequenceNumber": "49615513957620404616321850166196450578909326208976027650",
            "ApproximateArrivalTimestamp": "2021-02-14T21:12:49+01:00",
            "Data": "User has exit from the system",
            "PartitionKey": "sessions4"
        },
        {
            "SequenceNumber": "49615513957620404616321850166197659504728943793088233474",
            "ApproximateArrivalTimestamp": "2021-02-14T21:13:32+01:00",
            "Data": "ZXlKd2NtOXdJam9nSWpjM0lpd2dJblJwYldWemRHRnRjQ0k2SURFMk1UTXpNek0yTURZc0lDSjBhR2x1WjE5cFpDSTZJQ0poWVMxaVlpSjk=",
            "PartitionKey": "aa-bb"
        }
    ],
    "NextShardIterator": "AAAAAAAAAAGpB9vwvdjBrklL9AR2RkFT9BdOh2yW52J2DEfzTDDzgbACyaLBJy2Vwkj6Uc4YoeGpxHiljGK5/YMh63MEQEwFxEKDND0AJckZZFlHaRnjYU4uOPZf7ade7fsU1ype77b7QQSzLrjFnmWqP+qZMOn3CmJpIIsin0QhSZT01rdUwk89qJ1JTDVL41zSUjEneEk=",
    "MillisBehindLatest": 0
}

@coveralls
Copy link

Coverage Status

Coverage increased (+0.05%) to 70.371% when pulling d8aaf8f on mgagliardo:fix/issue3588 into 01b898f on localstack:master.

@whummer
Copy link
Member

whummer commented Feb 15, 2021

Great, thanks for digging into this issue @mgagliardo . Do you think we can add this test also to our Java library integration tests? Thanks!

@whummer whummer merged commit 23a63ff into localstack:master Feb 15, 2021
@whummer
Copy link
Member

whummer commented Feb 15, 2021

@mgagliardo Just a quick note - is this something that appears only with AWS Java SDK v2, or also v1? Can we also verify that it is backwards compatible with v1? (We have tests for both v1 and v2 in the Java repo..) Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

localstack crashes when reading Kinesis record that has been produced with a JVM client - "Object of type bytearray is not JSON serializable"

3 participants