zts_py_send() was using strlen() to determine the length of the
send(). This works fine with strings, but fails with binary
data.
To fix this, I have removed the string encoding code, and
converted to using the Buffer protocol as is done in the
Python socketmodule send() implementation. This does mean
that this send() implementation only takes byte-like objects.
The workaround for this could be at the python level rather
than the C++ level.
NOTE: This implementation has a bug in the exception handling
if a non-bytes-like object is passed. You get an exception,
but the exception is not accurate, it reports the TypeError,
but the actual raised exception is due to there being a return
value when the error indicator is set. I spent a few hours
trying to fix this but was unable to. I'm afraid I just couldn't
figure it out.
My SSH proxy was misbehaving because the second block
of data going from the client to the server had a NUL byte as
the first byte to send, so the send was returning 0 bytes
sent, but that was due to send() being told to send 0 bytes.
With this, my SSH proxy is now working, including able to run
an rsync of my boot initrd over SSH over ZeroTier entirely in
userspace.
There were some bugs in my blocking code that are fixed in this.
I had the setblocking() flag reversed, and wasn't passing the
flag along to the underlying code properly.
Many changes were based on conventions in the Python socketmodule.
Changed many of the docstrings to match the Python socket library
conventions and enhancing them. I can either remove the prototype
part or add it to other docstrings in the library, depending on
feedback I get.
Changed setblocking to use the flag argument instead of always
just setting NONBLOCK.
Added enable/disable threading around more lwip calls.
Implementing optional backlog on listen().
Removing a few seemingly unneeded Py_INCREF(Py_None) calls.
Moved getblocking function based on alpha sorting of names.
There were a few I'm attempting to fix in zts_py_recv():
Was allocating a static buffer of 4096, but taking whatever
length the user passed in. This change allocates a PyBytes
object of the size the user requests.
Was converting to a string without giving a size. Which probably
led to the UnicodeDecodeError I was seeing below, trying to
decode a character beyond the received bytes. Would also lead
to problems reading binary data that included embedded NULs.
Used the number of bytes received as the size of the returned data.
Variable "err" was being used, changed that to "bytes_read" as
that's what lwip_recv() returns, with <0 bytes read indicating
error.
Read data was being returned as a Unicode string, leading to this
response when I tried talking to my SSH server:
UnicodeDecodeError: 'utf-8' codec can't decode
byte 0xa1 in position 42: invalid start byte
My ssh banner is 40 bytes long, so I think position 42 was outside
the received buffer. Changed it to a PyBytes response as is
normal for network data. This code was cribbed from the Python
socketmodule
This was producing this error, as it was still returning the tuple:
SystemError: <built-in function zts_py_recv> returned a
result with an error set
This indicates that the UnicodeDecodeError had set an exception,
but a tuple was being returned instead of NULL.
In the case of a lwip_recv() error, the error response was not
being sent back to the wrapper code. Instead, a "return NULL;"
was done. I changed this case to return the tuple (err, None)
to the wrapper.
NOTE: this code built using "./build.sh host-python release", but
there was some sort of build problem I didn't understand which
produced a _libzt.so that I couldn't import, due to:
ImportError: dynamic module does not define module export
function (PyInit__libzt)
So I don't have a way to test these changes.
Sean Reifschneider
Joseph Henry
Sean Reifschneider