Skip to content

[QUESTION] Should VSIM filter expression consider missing json fields implicitly as null? #14499

@Jacopo47

Description

@Jacopo47

Let's use this expression as example: .name=="jacopo" || .year == 1995
In order to be considered by this expression an element MUST contain both the fields: name and year.

In case the field name or the field year are not explicitly set into the json the expression will not match even if name is equal to jacopo or year is equal to 1995.
Instead the expression will match if the json contains both the fields but eventually set at null. (eg. {"name": "jacopo", "year": null})

As far as I understood right now redis is "circuit breaking" by not matching the element if one of the fields required by the expression is missing into the json.

That the reason behind my question: should redis treat missing fields in attribute's json implicitly at null?

To reproduce

VADD test VALUES 3 1 1 1 a SETATTR '{"name": "jacopo", "year": 1995}'
VADD test VALUES 3 1 1 1 b SETATTR '{"name": "jacopo"}'
VADD test VALUES 3 1 1 1 c SETATTR '{"year": 1995}'

VSIM test VALUES 3 0 0 0 FILTER '.name=="jacopo" || .year == 1995'
> "a"

# setting year explicitly to null in b
VSETATTR test b '{"name": "jacopo", "year": null}'

VSIM test VALUES 3 0 0 0 FILTER '.name=="jacopo" || .year == 1995'
> "b"
> "a"

Expected behavior

Given the above set of elements for the vector test, in which some properties are implicitly at null:

VADD test VALUES 3 1 1 1 a SETATTR '{"name": "jacopo", "year": 1995}'
VADD test VALUES 3 1 1 1 b SETATTR '{"name": "jacopo"}'
VADD test VALUES 3 1 1 1 c SETATTR '{"year": 1995}'

Or given an equivalent set of elements for the vector test, in which the properties are explicitly at null:

VADD test VALUES 3 1 1 1 a SETATTR '{"name": "jacopo", "year": 1995}'
VADD test VALUES 3 1 1 1 b SETATTR '{"name": "jacopo", "year": null}'
VADD test VALUES 3 1 1 1 c SETATTR '{"name":   null  , "year": 1995}'

Should result in the same VSIM's output given this filter expression:

VSIM test VALUES 3 0 0 0 FILTER '.name=="jacopo" || .year == 1995'
> "c"
> "b"
> "a"

Additional information

In this issue my point of view starts from an assumption: a json field behaves in the same way either if it's explicitly set to null or when the field is not provided (implicitly set at null).
But my point of view misses an important piece: the context behind how redis treats the attributes' schema and maybe I'm approaching to attributes starting from a wrong assumption.

What I can share, in order to provide more context about the reason behind me facing this scenario, is that I'm writing a java layer that, when an object is provided, it serializes it as a json into the vector's element.
Nothing stops me to explicitly serialize nulls into attributes's json but, if possible, I would like to avoid it.

Right now few reasons comes into my mind but one is more important than others: data evolution. (I hope I'm not looking at it too vertically to my needs)
With data evolution I refer to a scenario in which an application at time t will start with just in mind a small set of data and fields (e.g. only the name).
Then at time t+1 the application evolves and it has the need to add other properties to the json (e.g. the year).
In a scenario in which redis consider json's properties implicitly at null will allow the application to write expressions like: .name=="jacopo" || .year == 1995
without the need to update all the t vector's elements in order to explicitly set the new property.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions