to_bytes() fails with TypeError: Expected bytes, got <subclass> for subclasses of str/bytes #106

Description

Every API that goes through ssh.utils.to_bytes() (e.g. Session.options_set(), Session.userauth_password(), import_privkey_file()) raises a TypeError when it is passed an instance of a subclass of bytes, or a subclass of str whose encode() returns a bytes subclass.

to_bytes() is declared as cdef bytes to_bytes(_str). Cython enforces an exact bytes type on the return value of a function typed bytes, but the function returns its input unchanged in the bytes branch and returns the result of .encode() unchanged in the str branch:

# ssh/utils.pyx
cdef bytes to_bytes(_str):
    if isinstance(_str, bytes):
        return _str                    # bytes subclass returned as-is
    elif isinstance(_str, unicode):
        return _str.encode(ENCODING)   # may return a bytes subclass
    return _str

So both branches can return a bytes subclass, and the Cython return type check rejects it.

Real-world impact

This breaks Ansible NETCONF connections that use libssh (ansible.netcommon.netconf with use_libssh: true → ncclient libssh transport → ssh-python). Ansible passes host, username and password as AnsibleUnsafeText, a str subclass whose encode() returns AnsibleUnsafeBytes, and the connection fails with:

ansible.module_utils.connection.ConnectionError: Expected bytes, got AnsibleUnsafeBytes

The same setup works with the paramiko transport.

Minimal reproducer (no Ansible required)

from ssh.session import Session
from ssh.options import HOST

class MyBytes(bytes):
    pass

class MyStr(str):
    def encode(self, *args, **kwargs):
        return MyBytes(super().encode(*args, **kwargs))

s = Session()
s.options_set(HOST, "localhost")             # OK
s.options_set(HOST, b"localhost")            # OK
s.options_set(HOST, MyBytes(b"localhost"))   # TypeError: Expected bytes, got MyBytes
s.options_set(HOST, MyStr("localhost"))      # TypeError: Expected bytes, got MyBytes
s.userauth_password(MyStr("user"), MyStr("pw"))  # TypeError: Expected bytes, got MyBytes

Expected behaviour

Subclasses of str and bytes should be accepted like their base types, since they pass the isinstance() checks in to_bytes().

Suggested fix

Coerce to an exact bytes object before returning. For an exact bytes instance, bytes(x) returns the same object without copying, so there is no overhead in the common case:

cdef bytes to_bytes(_str):
    if isinstance(_str, bytes):
        return bytes(_str)
    elif isinstance(_str, unicode):
        return bytes(_str.encode(ENCODING))
    return _str

Environment

• ssh-python 1.2.0.post1 (PyPI wheel, Linux x86_64)
• Python 3.11 / 3.12
• Found via ncclient 0.7.1 (libssh transport) and ansible.netcommon 8.7.1 with ansible-core 2.18