I'll look into this more when I have more time, but trying out this branch on Python3 with a basic script

#!/usr/bin/env python3

from spython.main import Client

Client.pull(
    'shub://mwiens91/shub-debug',
    pull_folder='/home/matt/temp',
    name_by_commit=True,
)

Resulted in

singularity pull --commit shub://mwiens91/shub-debug
Traceback (most recent call last):
  File "singularity_pull_test.py", line 8, in <module>
    name_by_commit=True,
  File "/home/matt/.virtualenvs/asdf/local/lib/python3.6/site-packages/spython-0.0.42-py3.6.egg/spython/main/pull.py", line 94, in pull
    final_image = os.path.join(pull_folder, name)
  File "/home/matt/.virtualenvs/asdf/lib/python3.6/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/matt/.virtualenvs/asdf/lib/python3.6/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

Regular pulling seems to work fine, though.

okay I think I fixed the issue! Here is what I just did:

from spython.main import Client

Client.pull(
    'shub://mwiens91/shub-debug',
    pull_folder='/tmp',
    name_by_commit=True,
)

## -- End pasted text --
singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /tmp/d13898a106f9678c695ab479335e285c634d3686.simg
/tmp/d13898a106f9678c695ab479335e285c634d3686.simg

Let me know if it works for you!

Hmm. I'm getting another error:

singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /home/matt/temp/d13898a106f9678c695ab479335e285c634d3686.simg
Progress |===================================| 100.0% 
Done. Container is at: /home/matt/temp/d13898a106f9678c695ab479335e285c634d3686.simg
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255

Traceback (most recent call last):
  File "singularity_pull_test.py", line 8, in <module>
    name_by_commit=True,
  File "/home/matt/.virtualenvs/asdf/local/lib/python3.6/site-packages/spython-0.0.42-py3.6.egg/spython/main/pull.py", line 103, in pull
    final_image = output.split('Container is at:')[-1].strip('\n').strip()
AttributeError: 'tuple' object has no attribute 'split'

It seems to be pulling twice?? Not sure what to make of it.

Hrm I am seeing the error from Singularity - what version of Singularity do you have? I think the issue is that there is a Singularity error that then returns output that is unexpected, and it can't be parsed to return the container name.

$ singularity --version
2.4.2-dist

This the version provided by default in Ubuntu 18.04.

I don't think ubuntu has a default version? Anyway, could you please update? That is a very old version (and one with security issues too, I think).

Yeah I'll try updating.

https://launchpad.net/ubuntu/+source/singularity-container

is what Ubuntu provides, btw.

Oh I see, maybe 2.5.2 isn't specified for your distribution? --> https://launchpad.net/ubuntu/+source/singularity-container/2.5.2-2

Hmmm. I upgraded to 2.5.2 and am still getting the same error.

singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /home/matt/temp/d13898a106f9678c695ab479335e285c634d3686.simg
Progress |===================================| 100.0% 
Done. Container is at: /home/matt/temp/d13898a106f9678c695ab479335e285c634d3686.simg
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255

Traceback (most recent call last):
  File "./singularity_pull_test.py", line 8, in <module>
    name_by_commit=True,
  File "/home/matt/.virtualenvs/asdf/local/lib/python3.6/site-packages/spython-0.0.42-py3.6.egg/spython/main/pull.py", line 103, in pull
    final_image = output.split('Container is at:')[-1].strip('\n').strip()
AttributeError: 'tuple' object has no attribute 'split'

Maybe I need to initialize my "config subsystem" (whatever that may mean)?

okay I'm going to update mine to the latest (2.6.0) to test this out.
https://www.sylabs.io/guides/2.6/user-guide/quick_start.html#installation

Make sure you install based on those instructions - there is no "non sudo" install, etc. This works for me:

$ singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /tmp/d13898a106f9678c695ab479335e285c634d3686.simg

Now in python! Note that I removed the old versions of spython first, otherwise you get weird conflicts.

rm -rf /home/vanessa/anaconda3/lib/python3.6/site-packages/spython*

This still works okay for me, see if it does for you with updated singularity and removing old versions of spython:

from spython.main import Client

Client.pull(
    'shub://mwiens91/shub-debug',
    pull_folder='/tmp',
    name_by_commit=True,
)

If that doesn't work, please open the file spython/main/pull.py and have it do a print for final_image
before it errors out so we can look at what the tuple is. I'll start looking into this config subsystem error, it might relate to your distribution (but I can't imagine why you get it just with python)

Interesting I'm looking at the differences in the command now, and it comes down to having capture=True (to get the image name)

output = self._run_command(cmd, capture=True)

vs

# capture is False
self._run_command(cmd, capture=capture)

If you report that regular pull works and this one doesn't, then this is the issue - capturing the output. If the above doesn't work I have a workaround I think :)

and I would bet we installed singularity differently to get the different functionality - maybe you did disable suid?

For Singularity 2.4.2 I just did a regular Ubuntu

sudo apt install singularity-container

with no other modification past that. Upgrading to Singularity 2.5.2, I used the package archive at Debian Neuroscience here:

http://neuro.debian.net/pkgs/singularity-container.html

and upgraded with apt again:

sudo apt upgrade

Again, with no further modification.

I haven't tried building from source on my main machine (package managers are nice 😄), although I believe I have done so on a few VMs.

okay I just a modification that should make the pull work as the one that you say is working... fingers crossed! It works for me still.

See 6d0d25d

The workaround is to pull (without capture) to a temporary folder, and then move to the final spot and clean up.

And I'm definitely terrible, I didn't even know we had an official package on there, heh :)

Ok great! I'm still getting the weird Singularity errors but it seems to be working.

singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /tmp/tmpuqycevfa/d13898a106f9678c695ab479335e285c634d3686.simg
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255

/home/matt/temp/d13898a106f9678c695ab479335e285c634d3686.simg

Naming by hash works great too:

singularity pull --hash shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /tmp/tmpr5tdyrv3/07299790f4a52fdfeb6e95aa6fa97eaf.simg
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255

/home/matt/temp/07299790f4a52fdfeb6e95aa6fa97eaf.simg

When do you expect these changes will be brought into the singularity-cli package on PyPI?

Also, as I mentioned in #48, this option becomes way more useful when this issue:

apptainer/singularity#1841

is fixed, but, as we discussed before, that's an issue with Singularity native, not spython.

Awesome! The singularity error is strange indeed! I think it's related possibly to the package that you installed (because I'm not getting it locally with my install from source). I can get this merged and on pypi tonight, but first let's open an issue on the Singularity board. I want to get a little more insight to why this might be happening.

Yeah it definitely is to do with the package. I switched one of my Travis builds from building Singularity from source to getting Singularity from the Debian Neuroscience package repository, and the error popped up after making that change (it was absent before).

Before:
https://travis-ci.com/mwiens91/saltant/jobs/143532296#L3274

After:
https://travis-ci.com/mwiens91/saltant/jobs/144609969#L1055

okay, I'm good integrating this and we will hope to resolve the neurodebian issue on the other thread. Merging!

Here you go @mwiens91 thanks for your help with this! https://pypi.org/project/spython/ https://pypi.org/project/spython/0.0.42/