Removed the automatic window_size adjustments from
_libssh2_channel_read() and instead all channel readers must now make
sure to enlarge the window sizes properly themselves.
libssh2_channel_read_ex() - the public function, now grows the window
size according to the requested buffer size. Applications can still opt
to grow the window more on demand. Larger windows tend to give higher
performance.
sftp_read() now uses the read-ahead logic to figure out a window_size.
A returned READ packet that is short will now only reduce the
offset.
This is a temporary fix as it is slightly better than the previous
approach but still not very good.
When seeking to a new position, flush the packetlist and buffered data
to prevent already received or pending data to wrongly get used when
sftp-reading from the new offset within the file.
In the case where a read packet has been received from the server, but
the entire contents couldn't be copied to the user-buffer, the data is
instead buffered and copied to the user's buffer in the next invocation
of sftp_read(). When that "extra" copy is made, the 'offset' pointer was
not advanced accordingly.
The biggest impact of this flaw was that the 'already' variable at the
top of the function that figures out how much data "ahead" that has
already been asked for would slowly go more and more out of sync, which
could lead to the file not being read all the way to the end.
This problem was most noticable in cases where the application would
only try to read the exact file size amount, like curl does. In the
examples libssh2 provides the sftp read function is most often called
with a fixed size large buffer and then the bug would not appear as
easily.
This bug was introduced in the SFTP rewrite in 1.2.8.
Bug: http://curl.haxx.se/mail/lib-2011-08/0305.htmlhttp://www.libssh2.org/mail/libssh2-devel-archive-2011-08/0085.shtml
As pointed out in bug #206, if a second invoke of libssh2_sftp_read()
would shrink the buffer size, libssh2 would go nuts and send out read
requests like crazy. This was due to an unsigned variable turning
"negative" by some wrong math, and that value would be the amount of
data attempt to pre-buffer!
Bug: http://trac.libssh2.org/ticket/206
If asked to read data into a buffer and the buffer is too small to hold
the data, this function now returns an error instead of as previously
just copy as much as fits.
By using a new separate struct for incoming SFTP packets and not sharing
the generic packet struct, we can get rid of an unused field and add a
new one dedicated for holding the request_id for the incoming
package. As sftp_packet_ask() is called fairly often, a "mere" integer
comparison is MUCH faster than the previous memcmp() of (typically) 5
bytes.
Make sure that we cleanup remainders when the handle is closed and when
the subsystem is shutdown.
Existing flaw: if a single handle sends packets that haven't been
replied to yet at the time when the handle is closed, those packets will
arrive later and end up in the generic packet brigade queue and they
will remain in there until flushed. They will use unnecessary memory,
make things slower and they will ruin the SFTP handling if the
request_id counter ever wraps (highly unlikely to every happen).
The SFTP read function now does transfers the same way the SFTP write
function was made to recently: it creates a list of many outgoing
FXP_READ packets that each asks for a small data chunk. The code then
tries to keep sending read request while collecting the acks for the
previous requests and returns the received data.
The new SFTP write code caused a regression as the seek function no
longer worked as it didn't set the write position properly.
It should be noted that seeking is STRONGLY PROHIBITED during upload, as
the upload magic uses two different offset positions and the multiple
outstanding packets etc make them sensitive to change in the midst of
operations.
This functionality was just verified with the new example code
sftp_append. This bug was filed as bug #202:
Bug: http://trac.libssh2.org/ticket/202
The SFTP handle struct now buffers number of acked bytes that haven't
yet been returned. The way this is used is as following:
1. sftp_write() gets called with a buffer of let say size 32000. We
split 32000 into 8 smaller packets and send them off one by one. One of
them gets acked before the function returns so 4000 is returned.
2. sftp_write() gets called again a short while after the previous one,
now with a much smaller size passed in to the function. Lets say 8000.
In the mean-time, all of the remaining packets from the previous call
have been acked (7*4000 = 28000). This function then returns 8000 as all
data passed in are already sent and it can't return any more than what
it got passed in. But we have 28000 bytes acked. We now store the
remaining 20000 in the handle->u.file.acked struct field to add up in
the next call.
3. sftp_write() gets called again, and now there's a backlogged 20000
bytes to return as fine and that will get skipped from the beginning
of the buffer that is passed in.
This function now only returns EAGAIN if a lower layer actually returned
EAGAIN to it. If nothing was acked and no EAGAIN was received, it will
now instead return 0.
SFTP packets come as [32 bit length][payload] and the code didn't
previously handle that the initial 32 bit field was read only partially
when it was read.
There were some chances that they would cause -1 to get returned by
public functions and as we're hunting down all such occurances and since
the underlying functions do return valuable information the code now
passes back proper return codes better.
The sftp_write function shouldn't assume that the buffer pointer will be
the same in subsequent calls, even if it assumes that the data already
passed in before haven't changed.
The sftp structs are now moved to sftp.h (which I forgot to add before)
sftp_write was rewritten to split up outgoing data into multiple packets
and deal with the acks in a more asynchronous manner. This is meant to
help overcome latency and round-trip problems with the SFTP protocol.
Neither _libssh2_channel_write nor sftp_write now have the 32500 size
limit anymore and instead the channel writing function now has its own
logic to send data in multiple calls until everything is sent.
The new function takes two data areas, combines them and sends them as a
single SSH packet. This allows several functions to allocate and copy
less data.
I also found and fixed a mixed up use of the compression function
arguments that I introduced in my rewrite in a recent commit.
sftp_write() now limits how much data it gets at a time even more
than before. Since this function creates a complete outgoing
packet based on what gets passed to it, it is crucial that it
doesn't create too large packets.
With this method, there's also no longer any problem to use very
large buffers in your application and feed that to libssh2. I've
done numerous tests now with uploading data over SFTP using 100K
buffers and I've had no problems with that.
If an application accidentally provides a NULL handle pointer to
the channel or sftp public functions, they now return an error
instead of segfaulting.
Alexander Lamaison filed bug #172
(http://trac.libssh2.org/ticket/172), and pointed out that SFTP
init would do bad if the session isn't yet authenticated at the
time of the call, so we now check for this situation and returns
an error if detected. Calling sftp_init() at this point is bad
usage to start with.
As the long-term goal is to get rid of the extensive set of
macros from the API we can just as well start small by not adding
new macros when we add new functions. Therefore we let the
function be libssh2_sftp_statvfs() plainly without using an _ex
suffix.
I also made it use size_t instead of unsigned int for the string
length as that too is a long-term goal for the API.
