Skip to content

Waterline clarification #2119

@blp

Description

@blp

The new async merger code merges a lot faster than the old code and that causes a nondeterministic failure in the operator::dynamic::input::test::map_test_mt1 test (and probably related ones but that's the one I'm investigating at the moment).

The problem is here in input_map_updates2:

            vec![
                // Attempt to insert new value overwriting value below waterline - ignored.
                Tup2(1, Update::Insert(20)),
                // Overwrite value above waterline with a new value above waterline - accepted
                Tup2(2, Update::Insert(10)),
                // Create new value above waterline - accepted, try to overwrite it with a value
                // below waterline - ignored.
                Tup2(4, Update::Insert(15)),
                Tup2(4, Update::Insert(4)),
            ],

Just before this point in the test, a key-value pair (1,1) had been in the spine with weight 1, and the waterline had been such that any value less than 5 would be deleted (therefore this pair should be deleted).

The test assumes that (1,1) will not be deleted, and therefore the update inserting (1, 20) will get rejected. The expected output requires that. The original fueled spine and the old asynchronous spine never deleted it, so the test always passed. The new async spine sometimes does not delete it either.

The new async spine does sometimes delete (1,1), though. In that case, (1, 20) is inserted successfully, and the test fails.

@ryzhyk, can you help understand this case?

Metadata

Metadata

Assignees

Labels

DBSP coreRelated to the core DBSP librarybugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions