RasterRef should not read HDFS scheme with GDAL reader.#319
RasterRef should not read HDFS scheme with GDAL reader.#319vpipkt wants to merge 6 commits intolocationtech:developfrom
Conversation
Signed-off-by: Jason T. Brown <jason@astraea.earth>
Signed-off-by: Jason T. Brown <jason@astraea.earth>
core/src/main/scala/org/locationtech/rasterframes/ref/RasterSource.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/org/locationtech/rasterframes/ref/RasterSourceSpec.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Jason T. Brown <jason@astraea.earth>
|
Still having some class not found errors in travis. Unsure why. |
|
@vpipkt I think something's wrong with the implementation, because in this test case the scheme resolution should not be attempting to use the GDAL reader if it's not available, yet it's still selecting it over the default reader. https://travis-ci.org/locationtech/rasterframes/builds/580393491#L832-L866 |
metasim
left a comment
metasim
left a comment
There was a problem hiding this comment.
I think there's a bug causing GDAL reader to be selected when GDAL not available and JVM one sufficient.
|
@vpipkt Not 100% sure what's going on here, but don't want to be extra sure the JVM fallback reader still gets selected in non-GDAL scenarios. Wonder if we should leverage this sort of logic? rasterframes/.circleci/config.yml Line 126 in 36be700 Also odd: not seeing it here: |
|
Now you can see in Circle CI, it is passing. https://circleci.com/gh/s22s/rasterframes/tree/fix%2F314 |
Signed-off-by: Jason T. Brown <jason@astraea.earth>
| true | ||
| } catch { | ||
| case _: UnsatisfiedLinkError => | ||
| case _:UnsatisfiedLinkError | _:NoClassDefFoundError => |
There was a problem hiding this comment.
@metasim I suspect this may be a terrible idea.
There was a problem hiding this comment.
I don't understand why Travis isn't passing yet. CircleCI doesn't surprise me because GDAL is installed. I've not confirmed that it is in the Travis configuration.
|
@vpipkt Once this weirdness is resolved, should we be trying to use this (if memory serves, written by Azavea folk!)?: https://gdal.org/user/virtual_file_systems.html#vsihdfs-hadoop-file-system |
|
@metasim yep, by @jamesmcclain |
I am here to help if you run into any trouble with that. Please be advised that some users have run into issues when trying to access HDFS filesystems through GDAL from a JVM process. The problem appears to be related to the fact that GDAL spawns its own inferior JVM to interface with HDFS, and that is done from within a JVM process running native GDAL code (more here). EDIT: Specifically, please see this comment and surrounding comments. There does not appear to be a problem on OpenJDK 11, but there is a problem with OpenJDK 8. |
|
@jamesmcclain Thanks for the background color... sounds to me like we shouldn't go that route if we don't have to. |
You are welcome. |
metasim
force-pushed
the
develop
branch
2 times, most recently
from
844c524 to
a5c6446
Compare
4 participants
And enforce other logic like can't read a gdal only format (mrf, jp2) outside the gdal reader.
Needs a close look and maybe more extensive unit tests. Possibly also more extensive / detailed error messages.
For #314
This also relates, from the user perspective, to #267