Python: Replace SQLContext with SparkSession#424
Python: Replace SQLContext with SparkSession#424WeichenXu123 merged 2 commits intographframes:masterfrom ttc0419:spark-session
Conversation
|
Tagging @WeichenXu123, since you reviewed the most recent MR. |
|
@ttc0419 Thank you! I will take a look soon. |
|
The trait |
Spark 2.4.x does not support SparkSession.getActiveSession(). According to https://spark.apache.org/versioning-policy.html, spark 2.4.x is EOL and the last version was released 5 years ago. There's little meaning to support it.
I'm not familiar with Scala, probably need someone else and another MR to do that. |
We can do it in follow-up PR. |
|
@WeichenXu123 Anything else need to be done before merging this? |
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## master #424 +/- ##
==========================================
- Coverage 91.37% 91.19% -0.19%
==========================================
Files 18 18
Lines 870 829 -41
Branches 53 49 -4
==========================================
- Hits 795 756 -39
+ Misses 75 73 -2
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
@WeichenXu123 Do we need another reviewer to merge this? |
I can merge it. |
According to the official document,
SQLContextis deprecated since spark 2.0. UseSparkSessioninstead.Notes:
This MR also removes CI for spark 2.4.8 since it does not support
SparkSession.getActiveSession(). And according to versioning policy, spark 2.4.x is EOL and the last version was released in 5 years ago. There's little meaning to support it.Closes: #423