3 questions
-1
votes
1
answer
59
views
how to configure beam application with spark runner to use S3ACommitter?
I have a beam application and its running with spark runner. It encountered kind of data lost issue as this application save data to a S3 storage.
I looked into this page https://hadoop.apache.org/...
0
votes
2
answers
116
views
Spark on AWS EKS java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found when running in cluster mode
I am trying to run a spark job on an EKS cluster. When I run it in cluster mode I received the following
WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-...
-1
votes
1
answer
536
views
Why are S3A staging committers faster when overwriting data?
I tested the staging (directory, partitioned) committers versus the magic committer when overwriting data in an s3 compatible object store, and, for some reason, the staging committers are faster when ...
- 2,304