Compute cosine similarity spark java

Question

How to compute cosine similarity between 2 Spark Vector. I am using the new ml package.

Spark 2.1.1

EDIT:

Spark provide RowMatrix which can be used to compute similarity but it accepts mllib.vector not an ml.vector.

Is there a way to convert Vectors from the different packages? Is there an implementation that uses ml.vector?

You could create an UDF which takes the two vectors as input and make the calculations there. — Shaido
– Shaido, Commented May 19, 2017 at 15:29

Shaido · Accepted Answer · 2017-05-22 07:47:01Z

2

The easiest way to convert from an mllib vector to an ml vector is to use the Vectors.fromML method, see Vectors documentation. Example:

val mlVector = org.apache.spark.ml.linalg.Vectors.dense((Array(1.0,2.0,3.0)))
println(mlVector.getClass())

val mllibVector = org.apache.spark.mllib.linalg.Vectors.fromML(mlVector)
println(mllibVector.getClass())

Gives an output:

class org.apache.spark.ml.linalg.DenseVector
class org.apache.spark.mllib.linalg.DenseVector

answered May 22, 2017 at 7:47

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

LonsomeHell Over a year ago

That is what I needed thanks. And to go from mllib.Vector to ml.Vector just use asML() from the mllib.Vector instance directly.

Collectives™ on Stack Overflow

Compute cosine similarity spark java

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related