The function sftp_read never return more then 2000 bytes (as it should
when I asked Daniel). I increased the MAX_SFTP_READ_SIZE to 30000 but
didn't get the same speed as a sftp read in SecureSSH. I analyzed the
code and found that a return always was dona when a chunk has been read.
I changed it to a sliding buffer and worked on all available chunks. I
got an increase in speed and non of the test I have done has failed
(both local net and over Internet). Please review and test. I think
30000 is still not the optimal MAX_SFTP_READ_SIZE, my next goal is to
make an API to enable changing this value (The SecureSSH sftp_read has
more complete filled packages when comparing the network traffic)
Fixes issue arising when server does not support statfvs and or fstatvfs
extensions. sftp_statvfs() and sftp_fstatvfs() after this patch will
handle the case when SSH_FXP_STATUS is returned from server.
The new libssh2_sftp_fsync API causes data and metadata in the
currently open file to be committed to disk at the server.
This is an OpenSSH extension to the SFTP protocol. See:
https://bugzilla.mindrot.org/show_bug.cgi?id=1798
sftp_packet_add takes ownership of the packet passed to it and (now that we
handle zombies) might free the packet. sftp_packet_read uses the packet type
byte as its return code but by this point sftp_packet_add might have freed
it. This change fixes the problem by caching the packet type before calling
sftp_packet_add.
I don't understand why sftp_packet_read uses the packet type as its return
code. A future change might get rid of this entirely.
As this function is called when the SFTP session is closed, it needs to
also kill all zombies left in the SFTP session to avoid leaking memory
just in case some zombie would still be in there.
When flushing the packetlist, we must only add the request as a zombie
if no response has already been received. Otherwise we could wrongly
make it a zombie even though the response was already received and then
we'd get a zombie stuck there "forever"...
Since the sftp_packetlist_flush() function will move all the existing
FXP_READ requests in this handle to the zombie list we must first remove
this just received packet as it is clearly not a zombie.
Exactly as the comment in the code said, checking the return code from
sftp_packet_read() with <= was wrong and it should be < 0. With the new
filtering on incoming packets that are "zombies" we can now see this
getting zero returned.
In order to be fast, sftp_read sends many read requests at once. With a small
file, this can mean that when EOF is received back, many of these requests are
still outstanding. Responses arriving after we close the file and abandon the
file handle are queued in the SFTP packet queue and never collected. This
causes transfer speed to drop as a progressively longer queue must be searched
for every packet.
This change introduces a zombie request-ID list in the SFTP session that is
used to recognise these outstanding requests and prevent them being added to
the queue.
When calling _libssh2_channel_receive_window_adjust() internally, we now
always use the 'force' option to prevent libssh2 to avoid sending the
update if the update isn't big enough.
It isn't fully analyzed but we have seen corner cases which made a
necessary window update not get send due to this and then the other side
doesn't send data our side then sits waiting for forever.
The commit in 7194a9bd7ba45 wasn't complete. This change makes sure
variables are initialized properly before used in the EAGAIN and window
adjust cases.
Lots of places in the code translated the original error into the more
generic LIBSSH2_ERROR_SOCKET_TIMEOUT but this turns out to distort the
original error reason a lot and makes tracking down the real origin of a
problem really hard. This change makes the original error code be
preserved to a larger extent when return up to the parent function.
Commit 03ca9020756 tried to simplify the window sizing logic but broke
SFTP readdir as there was no window sizing code left there so large
directory listings no longer worked.
This change introduces window sizing logic to the sftp_packet_read()
function so that it now tells the remote about the local size having a
window size that suffice when it is about to ask for directory data.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2012-03/0069.shtml
Reported by: Eric
Whenever we have acked data and is about to call a function that *MAY*
return EAGAIN we must return the number now and wait to get called
again. Our API only allows data *or* EAGAIN and we must never try to get
both.
Removed the total_read variable that originally must have tracked how
much data had been written to the buffer. With non-blocking reads, we
must return straight away once we have read data into the buffer so this
variable served not purpose.
I think it was still hanging around in case the initial processing of
'leftover' data meant we wrote to the buffer but this case, like the
others, must return immediately. Now that it does, the last remaining
need for the variable is gone.
Whenever we have data and is about to call a function that *MAY* return
EAGAIN we must return the data now and wait to get called again. Our API
only allows data *or* EAGAIN and we must never try to get both.
Some SFTP servers send SFTP packets larger than 40000. Since the limit
is only present to avoid insane sizes anyway, we can easily bump it.
The define was formerly in the public header libssh2_sftp.h but served
no external purpose and was moved into the source dir.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2011-11/0004.shtml
Reported by: Michael Harris
Set read_state back to idle before trying to send anything so that if
the state somehow is wrongly set.
Also, avoid such a case of confusion by resetting the read_state when an
sftp handle is closed.
Removed the automatic window_size adjustments from
_libssh2_channel_read() and instead all channel readers must now make
sure to enlarge the window sizes properly themselves.
libssh2_channel_read_ex() - the public function, now grows the window
size according to the requested buffer size. Applications can still opt
to grow the window more on demand. Larger windows tend to give higher
performance.
sftp_read() now uses the read-ahead logic to figure out a window_size.
A returned READ packet that is short will now only reduce the
offset.
This is a temporary fix as it is slightly better than the previous
approach but still not very good.
When seeking to a new position, flush the packetlist and buffered data
to prevent already received or pending data to wrongly get used when
sftp-reading from the new offset within the file.
In the case where a read packet has been received from the server, but
the entire contents couldn't be copied to the user-buffer, the data is
instead buffered and copied to the user's buffer in the next invocation
of sftp_read(). When that "extra" copy is made, the 'offset' pointer was
not advanced accordingly.
The biggest impact of this flaw was that the 'already' variable at the
top of the function that figures out how much data "ahead" that has
already been asked for would slowly go more and more out of sync, which
could lead to the file not being read all the way to the end.
This problem was most noticable in cases where the application would
only try to read the exact file size amount, like curl does. In the
examples libssh2 provides the sftp read function is most often called
with a fixed size large buffer and then the bug would not appear as
easily.
This bug was introduced in the SFTP rewrite in 1.2.8.
Bug: http://curl.haxx.se/mail/lib-2011-08/0305.htmlhttp://www.libssh2.org/mail/libssh2-devel-archive-2011-08/0085.shtml
As pointed out in bug #206, if a second invoke of libssh2_sftp_read()
would shrink the buffer size, libssh2 would go nuts and send out read
requests like crazy. This was due to an unsigned variable turning
"negative" by some wrong math, and that value would be the amount of
data attempt to pre-buffer!
Bug: http://trac.libssh2.org/ticket/206
