Scala 2.12 and Spark 3 support#554
Merged
metasim merged 56 commits intolocationtech:developfrom Sep 16, 2021
Merged
Conversation
Not sure this is fully right
Filter is now sealed so Spatial filters can't be pushed down to GeoTrellis relation like so. This looks like a "nice to have" so going to worry about this later.
echeipesh
commented
Jun 1, 2021
core/src/test/scala/org/locationtech/rasterframes/encoders/DevSpec.scala
Outdated
Show resolved
Hide resolved
echeipesh
commented
Jun 1, 2021
core/src/test/scala/org/locationtech/rasterframes/encoders/DevSpec.scala
Outdated
Show resolved
Hide resolved
echeipesh
commented
Jun 1, 2021
core/src/test/scala/org/locationtech/rasterframes/encoders/DevSpec.scala
Outdated
Show resolved
Hide resolved
echeipesh
commented
Jun 1, 2021
core/src/test/scala/org/locationtech/rasterframes/encoders/DevSpec.scala
Outdated
Show resolved
Hide resolved
echeipesh
commented
Jun 1, 2021
core/src/test/scala/org/locationtech/rasterframes/encoders/DevSpec.scala
Outdated
Show resolved
Hide resolved
Has more flexible ExpressionEncoder constructor
This can't be registered right now and prevents testing, tmp disable
These are the changes required for the specs to succeed, but this is a bad idea. This version of tile encoder has inconsistent nesting. Committing for discussion.
this needs to be refactored out with static test data to get rid of the http client dependency.
The hope here is to replicate the encoder structure used by spark such that all the re-write rules apply to RasterFrame encoders when we need them.
- ProjectedRasterTile is now a case class - CRS, Bounds, Dimensions UDT added - LocalFunctionsSpec updated - RasterRefTile is no longer ProjectedRasterTile, just Tile
metasim
reviewed
Jun 14, 2021
core/src/test/scala/org/locationtech/rasterframes/functions/LocalFunctionsSpec.scala
Show resolved
Hide resolved
pomadchin
force-pushed
the
scala-2.12
branch
from
194d127 to
c2c0fa9
Compare
pomadchin
force-pushed
the
scala-2.12
branch
from
93e1ee8 to
e2c1c86
Compare
pomadchin
force-pushed
the
scala-2.12
branch
from
48d2ac3 to
89c6627
Compare
pomadchin
force-pushed
the
scala-2.12
branch
2 times, most recently
from
e8f4715 to
7e8a12e
Compare
pomadchin
force-pushed
the
scala-2.12
branch
from
7e8a12e to
05037ab
Compare
…oss hardware. Tagged predicate pushdown test as ignored for now.
…c-scala-2.12 * 'scala-2.12' of github.com:echeipesh/rasterframes: Uncomment the forgotten test Add scalafmt
Member
|
@echeipesh @metasim How about merging it as is without fixing the Python for now to unblock #551? |
Member
Let's go for it! I'll work the python fixes on another PR. |
Member
|
@echeipesh @pomadchin Congratulations!! 🎉 Amazing work!! |
Member
|
Closes #530 as well! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR bumps to Spark 3 and Scala 2.12, dropping support for Scala 2.11
Two major problems faced by this PR as:
ExpresisonEncodernow has expectations for what serializer and deserializer expressions look like and rewrites them on construction. Combined with getting rid offlatproperty this has introduced inconsistency in how RF encoders are treated when they're top of the column vs when they're member of a struct.Filteris now sealed class. It's no longer possible to push down spatial filter predicates to GeoTrellis relation. This is sad but probably not a huge deal in reality. Very much secondary issue.The PR core part is implemented and finalized by @pomadchin, it also incroporates: