0

I was wondering if it is possible to centrally manage Python scripts in Matillion. I envision that a Python component I need in multiple Matillion pipelines (on multiple instances) can be managed with GIT. Matillion can then pull the latest changes to the script and consequently run it.

The manage external file sources with the synch file source seemed promising, but I believe it only works with dbt currently (using Matillion 1.72.4). The idea was mentioned on this forum post as well:

https://community.matillion.com/s/idea/0874G000000kBgRQAU/detail

If this is not (yet) possible, I am wondering if there is another workaround.

Many thanks in advance!

  • Read all Matillion documentation on the topic
  • Read Matillion forum posts
  • Tried to implement the synch component myself, but I could not get it to work (at least with Python)
Improve this question

1 Answer 1

Reset to default
1

As with most of my Matillion installs, I make everything data driven feeding grids right down to detect changes configs, and Python no exception. You can store the Python in Snowflake tables then use Query Results to Scalar to pull the appropriate script into a variable (v_pythonscript) and then just put ${v_pythonscript} into a Python Script component.

Most of my SQL components are just ${v_dynamicsqlquery} or such like with all the config stored nicely in the DB or generated on the fly.

Getting code into the config tables with the quotes " can be fun, but code seems to paste just fine into DBeaver (and honours indents etc in Python) so that works for us.

Easy to manage, and you can version control as you would any type 2 dimension with a latest timestamp for a particular script ID.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.