Matillion Grid Iterator Running Random Job order

Question

I have SQL in a "Query Result to Grid" Component similar to the following: -

SELECT JOB, BATCH_ID
FROM   ELT_JOBS
ORDER BY JOB, BATCH_ID ASC

The values are saved in a Grid Variable called "BATCHES" and despite the "ORDER BY" being applied correctly, when the jobs run it's in a random order: I'd expect batches 1 to 10 to run in sequence but instead they're run in a chaotic order like 5,3,6,10,1 etc.

To correct this I've added a Python Component between the "Query Result to Grid" component and the "Grid Iterator" component that uses the following code: -

l_BATCH_IDS = context.getGridVariable('BATCH_IDS')

l_BATCH_IDS.sort()

context.updateGridVariable('BATCH_IDS',l_BATCH_IDS)

which fixes the problem, but it shouldn't be occurring in the first place.

Has anyone else encountered this issue and can you suggest what the root problem is please?

peterb · Accepted Answer · 2021-04-15 20:30:28Z

0

According to documentation, "In Sequential mode, the iterations will always proceed in the same order that the data is held in the Grid Variable."

If you are running in concurrent mode, that may be the issue. Otherwise, I suspect this is a bug in the version of Matillion you are using

answered Apr 15, 2021 at 20:30

peterb

7373 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

SnakePliskin Over a year ago

Thanks for the reply. Yes, it defaults to sequential and is already set to this. Whatsmore even if it was in concurrent mode then the fix that's been made to force the Grid variable ordering wouldn't have worked since this is applied before the Grid Iterator. Given that the forced ordering works every time and that this is the only change then the problem must lie somewhere else. It seems that you're right though; that it is a Matillion bug.

Collectives™ on Stack Overflow

Matillion Grid Iterator Running Random Job order

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related