pycall.rb + pdm: call python from ruby - #60048

This PR is a continuation of @davidsbailey 's work in #60003 and allows Python code to be invoked easily from within Ruby. Ruby and Python have very similar object systems, and pycall.rb allows bridging the two. Invocations are very fast and can be generally treated as "function calls" in terms of performance.

pycall.rb permits a possible architecture where ruby and python code run in the same monolith. This should be considered an experiment for now, but if it proves reliable in practice, I would advocate this "side by side in the monolith" approach become the primary way we use python.

This PR uses pdm for python dependency management, virtualenv management, and python interpreter installation. pdm is similar to bundle + rbenv or yarn + nvm.

For more information on how we might structure python code in our repo, see: python/README.md

Example

Ruby code using the python math package:

pyimport 'math'
puts "Python says the square root of 25 is: #{math.sqrt(25)}"

Ruby runs python code from /python/pycdo/pycdo/test_module/test_func.py:

pyfrom 'pycdo.test_module`, import: :test_func
puts test_func()
# => "Ruby can call Python!"

See pycall_test.rb and python/README.md for more examples.

Motivation

Architectural advantages

As we've started to do AI work, we've discovered that python has more and better libraries of interest (compared to ruby). This has led to python being the obvious language for writing much of our AI code in, and we only expect the gap to grow over time.

As of 2024, we have decided not to use a microservice architecture, but in two cases pragmatism has necessitated auxiliary microservices. Both were needed for the same reason: support domain-specific must-use programming languages that would be too expensive to do in ruby. The javabuilder service was required in order to host javac, and AI services like aiproxy have been needed to utilize python's ai-adjacent packages.

Javabuilder is a small body of code, highly contained and not anticipated to grow, so it represents an exception rather than the rule. However, since we expect AI work to continue growing, and anticipate that we'll continue to see a gap between ruby and python in supporting AI libraries, I am concerned with a growing fork between "we write ruby here" and "we write python there".

With pycall.rb, we can keep our Majestic Monolith while easily taking advantage of amazing python libraries.

Everyday advantages

Directly calling python from ruby can make programming easier (vs multiple services):

1. Its easy to pass ruby callbacks into python
   • This also lowers the need for python to store state / access a DB. That in turn increases re-use of Rails model code. See: comment with a callback example
2. A single ruby+py PR lets you review and atomically deploy changes that require modification to both sides of the code.
3. The high-performance reduces the need for "fat python code": its cheap to do lots of invocations rather than shifting code into python. Wanna call python code in a for loop? No need to shift the loop to python, just call it from a ruby for loop.
   • This could reduce the amount of python we write to accomplish the same goal.
4. Its easy to casually take advantage of Python libraries, outside core AI, in cases where its only "helpful" not needed. Would scikit-learn make a random graph easier to generate? Use it!
   • This could increase the number of projects python helps us accomplish.

Considerations

Performance

CPU

On my system, you can do 4 million function calls/second from ruby to python:

Benchmark.measure {
  1_000_000.times { math.sqrt(5) }
}.total
# => 0.2446950000000007

Conclusion: function calls from ruby => python are only 10x slower than ruby => ruby function calls, they can be treated essentially the same way.

Resident RAM

1. load irb without our environment
2. ps -o pid,rss => irb process using 42.6 MB RSS
3. require ./lib/cdo/pycall.rb to load pycall
4. ps -o pid,rss => irb process using 49.7 MB RSS
5. PyCall.import_module('math') to force use of python
6. ps -o pid,rss => irb process using 56.2 MB RSS

56.2 - 42.6 => pycall + python take 13 MB of RAM (compare to python3 running by itself which takes 12.5MB of RAM).

Conclusion: we can expect each Ruby process to take a baseline extra 13 MB of RAM + ram taken by whatever python packages we want to use (e.g. with openai, numpy and boto3: 62MB)

Memory leaks

I tested on Linux with MALLOC_CHECK_=1 irb as well as valgrind's memcheck, played with importing a couple dozen python modules and doing lots of function calls. Did not get report of any memory leaks or other memory issues.

Thread Safety

We use threads in our puma configuration, and PyCall is not thread-safe. The good news is that its SO not thread safe that its very obvious if you make a mistake: it crashes instantly if ever used from a thread. I believe these issues should be addressed in a follow-up PR.

1. This is not an issue for our delayed_job ActiveJob workers in prod, which use processes instead of threads.

• This does not work with the :async workers we currently have set as the default in :development. On the one hand, the async workers are nice because they don't require launching bin/delayed_job start to use worker features. On the other hand, it'd be nice if our stock config worked with Python features without requiring a locals.yml tweak. Ideally, we could start delayed_jobs along with dashboard server. This is out of scope for this PR and would be an interesting subject for a future PR.

2. This means you cannot use PyCall from Rails controllers as-is.

• I have some work toward remedying this issue here: Run all python code within PyCallThread.run #60267 and https://github.com/snickell/pycall_thread, but it still has a critical bug that I need to find before we can use it.

pdm

pdm solves both the python interpreter version problem, the virtualenv problem, and the dependency versioning problem.

PDM:

• Functionality is similar to bundle:
  • bundle install => pdm install
    • pdm install will install a python interpreter if needed to fulfill the requires-python clause in pyproject.toml
    • pdm install will install packages into the venv, including deps of sub-packages like pycdo
  • bundle exec ____ => pdm run ____
  • pdm add or pdm add --dev can be used to add deps and dev-deps to the pyproject.toml
• permits local sub-packages to be defined, which allows dependencies to be specified in /python/subpackage/pyproject.toml files rather than all mixed together in a top-level /pyproject.toml.
• follows PEP 621 for its dependency metadata, which means future package managers should be compatible with it as well.
  • uses a lockfile pdm.lock, which allows the pyproject.toml dependencies to focus on what's actually compatible and not try to double as an imperfect lockfile (ala requirements.txt)

Changes in PR

• Adds the pycall gem
• Includes a basic test that we can invoke python code from within ruby, including accessing python libraries
• Uses pdm for managing the python virtualenv as well pip dependencies.
• lib/cdo/pycall.rb is called at rails init and uses pdm to direct pycall toward the correct python version/virtualenv
• Adds a /pyproject.toml for defining python our major python packages
  -/pyproject.toml also specifies a Python version: 3.12 so we start off as fresh as possible
• Extends our Dockerfile (including drone) to install pdm
  • Adds a new Chef cookbook: cdo-python
    • installs pdm to /usr/local/bin
• Extends rake install and rake build to run pdm install (equivalent to bundle install)
  • this will download (and build if necessary) a version of python matching /pyproject.toml
  • this will create a virtual environment and install deps in it
• Updates SETUP.md to include instructions for installing pdm.
• Creates a package structure in /python showing how to create sub-packages, and add dependencies to them.
  • Example package: /python/pycdo
• Extends lint.rake (lint everything) and lint.rb (lint git commit hooks) to run linting on Python code in /python using Ruff

Things for a followup PR:

• Fill in the sketch of how to use python started by /python:
  • System for running tests for all sub-packages of /python and laddering up the results
  • More extension /python/README.md, in particular advice on how to break work into packages
• Example of using python from a ruby controller/model

Post Merge: post on #developers with install instructions

Currently

• Mac users: brew install pdm
• Linux users:
  • sudo pip3 install --prefix=/usr/local --upgrade pdm
• Then both mac & linux users, from the project root:
  • pdm install --dev: should create a venv in .venv
  • pdm run python3 --version and confirm its 3.12.x

TODO

• verify shelling out to aws cli command works
• get a linux user to test the linux instructions
• get at least one more mac user to test mac instructions
• write chef dep install for pdm
• test chef install on adhoc
• write /python/README.md
• Right before merge: tag docker build with a version number, push to hub, and update .drone.yml with the version number (currently using tag codedotorg/code-dot-org:pycall-pdm), probably will use codedotorg/code-dot-org:1.10.0 depending on when we merge
• don't forget: send post merge instructions to #developers

Co-authored-by: dave@code.org