The number of people has been steadily increasing who read our source
code with an editor that thinks tab stops are 4 spaces apart, as
opposed to the traditional tty-derived 8 that the PuTTY code expects.
So I've been wondering for ages about just fixing it, and switching to
a spaces-only policy throughout the code. And I recently found out
about 'git blame -w', which should make this change not too disruptive
for the purposes of source-control archaeology; so perhaps now is the
time.
While I'm at it, I've also taken the opportunity to remove all the
trailing spaces from source lines (on the basis that git dislikes
them, and is the only thing that seems to have a strong opinion one
way or the other).
Apologies to anyone downstream of this code who has complicated patch
sets to rebase past this change. I don't intend it to be needed again.
Coverity pointed out a call to sftp_pkt_free(pktin) straight after an
unconditional return statement, which is obviously silly.
Fortunately, it was benign: pktin was freed anyway by the function
being called _by_ the return statement, so the unreachability of this
free operation prevented a double-free rather than allowing a memory
leak. So the right fix is to just remove the code, rather than
arranging for it to be run.
Those were a reasonable abbreviation when the code almost never had to
deal with little-endian numbers, but they've crept into enough places
now (e.g. the ECC formatting) that I think I'd now prefer that every
use of the integer read/write macros was clearly marked with its
endianness.
So all uses of GET_??BIT and PUT_??BIT are now qualified. The special
versions in x11fwd.c, which used variable endianness because so does
the X11 protocol, are suffixed _X11 to make that clear, and where that
pushed line lengths over 80 characters I've taken the opportunity to
name a local variable to remind me of what that extra parameter
actually does.
I noticed a few of these in the course of preparing the previous
commit. I must have been writing that idiom out by hand for _ages_
before it became totally habitual to #define it as 'lenof' in every
codebase I touch. Now I've gone through and replaced all the old
verbosity with nice terse lenofs.
My normal habit these days, in new code, is to treat int and bool as
_almost_ completely separate types. I'm still willing to use C's
implicit test for zero on an integer (e.g. 'if (!blob.len)' is fine,
no need to spell it out as blob.len != 0), but generally, if a
variable is going to be conceptually a boolean, I like to declare it
bool and assign to it using 'true' or 'false' rather than 0 or 1.
PuTTY is an exception, because it predates the C99 bool, and I've
stuck to its existing coding style even when adding new code to it.
But it's been annoying me more and more, so now that I've decided C99
bool is an acceptable thing to require from our toolchain in the first
place, here's a quite thorough trawl through the source doing
'boolification'. Many variables and function parameters are now typed
as bool rather than int; many assignments of 0 or 1 to those variables
are now spelled 'true' or 'false'.
I managed this thorough conversion with the help of a custom clang
plugin that I wrote to trawl the AST and apply heuristics to point out
where things might want changing. So I've even managed to do a decent
job on parts of the code I haven't looked at in years!
To make the plugin's work easier, I pushed platform front ends
generally in the direction of using standard 'bool' in preference to
platform-specific boolean types like Windows BOOL or GTK's gboolean;
I've left the platform booleans in places they _have_ to be for the
platform APIs to work right, but variables only used by my own code
have been converted wherever I found them.
In a few places there are int values that look very like booleans in
_most_ of the places they're used, but have a rarely-used third value,
or a distinction between different nonzero values that most users
don't care about. In these cases, I've _removed_ uses of 'true' and
'false' for the return values, to emphasise that there's something
more subtle going on than a simple boolean answer:
- the 'multisel' field in dialog.h's list box structure, for which
the GTK front end in particular recognises a difference between 1
and 2 but nearly everything else treats as boolean
- the 'urgent' parameter to plug_receive, where 1 vs 2 tells you
something about the specific location of the urgent pointer, but
most clients only care about 0 vs 'something nonzero'
- the return value of wc_match, where -1 indicates a syntax error in
the wildcard.
- the return values from SSH-1 RSA-key loading functions, which use
-1 for 'wrong passphrase' and 0 for all other failures (so any
caller which already knows it's not loading an _encrypted private_
key can treat them as boolean)
- term->esc_query, and the 'query' parameter in toggle_mode in
terminal.c, which _usually_ hold 0 for ESC[123h or 1 for ESC[?123h,
but can also hold -1 for some other intervening character that we
don't support.
In a few places there's an integer that I haven't turned into a bool
even though it really _can_ only take values 0 or 1 (and, as above,
tried to make the call sites consistent in not calling those values
true and false), on the grounds that I thought it would make it more
confusing to imply that the 0 value was in some sense 'negative' or
bad and the 1 positive or good:
- the return value of plug_accepting uses the POSIXish convention of
0=success and nonzero=error; I think if I made it bool then I'd
also want to reverse its sense, and that's a job for a separate
piece of work.
- the 'screen' parameter to lineptr() in terminal.c, where 0 and 1
represent the default and alternate screens. There's no obvious
reason why one of those should be considered 'true' or 'positive'
or 'success' - they're just indices - so I've left it as int.
ssh_scp_recv had particularly confusing semantics for its previous int
return value: its call sites used '<= 0' to check for error, but it
never actually returned a negative number, just 0 or 1. Now the
function and its call sites agree that it's a bool.
In a couple of places I've renamed variables called 'ret', because I
don't like that name any more - it's unclear whether it means the
return value (in preparation) for the _containing_ function or the
return value received from a subroutine call, and occasionally I've
accidentally used the same variable for both and introduced a bug. So
where one of those got in my way, I've renamed it to 'toret' or 'retd'
(the latter short for 'returned') in line with my usual modern
practice, but I haven't done a thorough job of finding all of them.
Finally, one amusing side effect of doing this is that I've had to
separate quite a few chained assignments. It used to be perfectly fine
to write 'a = b = c = TRUE' when a,b,c were int and TRUE was just a
the 'true' defined by stdbool.h, that idiom provokes a warning from
gcc: 'suggest parentheses around assignment used as truth value'!
This commit includes <stdbool.h> from defs.h and deletes my
traditional definitions of TRUE and FALSE, but other than that, it's a
100% mechanical search-and-replace transforming all uses of TRUE and
FALSE into the C99-standardised lowercase spellings.
No actual types are changed in this commit; that will come next. This
is just getting the noise out of the way, so that subsequent commits
can have a higher proportion of signal.
The annoying int64.h is completely retired, since C99 guarantees a
64-bit integer type that you can actually treat like an ordinary
integer. Also, I've replaced the local typedefs uint32 and word32
(scattered through different parts of the crypto code) with the
standard uint32_t.
Unlike the traditional Unix SSH server organisation, the SFTP server
is built into the same process as all the rest of the code. sesschan.c
spots a subsystem request for "sftp", and responds to it by
instantiating an SftpServer object and swapping out its own vtable for
one that talks to it.
(I rather like the idea of an object swapping its own vtable for a
different one in the middle of its lifetime! This is one of those
tricks that would be absurdly hard to implement in a 'proper' OO
language, but when you're doing vtables by hand in C, it's no more
difficult than any other piece of ordinary pointer manipulation. As
long as the methods in both vtables expect the same physical structure
layout, it doesn't cause a problem.)
The SftpServer object doesn't deal directly with SFTP packet formats;
it implements the SFTP server logic in a more abstract way, by having
a vtable method for each SFTP request type with an appropriate
parameter list. It sends its replies by calling methods in another
vtable called SftpReplyBuilder, which in the normal case will write an
SFTP reply packet to send back to the client. So SftpServer can focus
more or less completely on the details of a particular filesystem API
- and hence, the implementation I've got lives in the unix source
directory, and works directly with file descriptors and struct stat
and the like.
(One purpose of this abstraction layer is that I may well want to
write a second dummy implementation, for test-suite purposes, with
completely controllable behaviour, and now I have a handy place to
plug it in in place of the live filesystem.)
In between sesschan's parsing of the byte stream into SFTP packets and
the SftpServer object, there's a layer in the new file sftpserver.c
which does the actual packet decoding and encoding: each request
packet is passed to that, which pulls the fields out of the request
packet and calls the appropriate method of SftpServer. It also
provides the default SftpReplyBuilder which makes the output packet.
I've moved some code out of the previous SFTP client implementation -
basic packet construction code, and in particular the BinarySink/
BinarySource marshalling fuinction for fxp_attrs - into sftpcommon.c,
so that the two directions can share as much as possible.
It's not used anywhere, but this would make it one step easier to add
a mode argument to PSFTP's mkdir command, if anyone needs it. Mostly
the point is to get rid of the FIXME comment in fxp_mkdir_send itself.
This is the first major piece of code converted to the new
unmarshalling system, and allows me to remove all the sftp_pkt_get*
functions in sftp.c that were previously duplicating standard decode
logic.
This wraps up a (pointer, length) pair into a convenient struct that
lets me return it by value from a function, and also pass it through
to other functions in one go.
Ideally quite a lot of this code base could be switched over to using
ptrlen in place of separate pointer and length variables or function
parameters. (In fact, in my personal ideal conception of C, the usual
string type would be of this form, and all the string.h functions
would operate on ptrlens instead of zero-terminated 'char *'.)
For the moment, I'm just introducing it to make some upcoming
refactoring less inconvenient. Bulk migration of existing code to
ptrlen is a project for another time.
Along with the type itself, I've provided a convenient system of
including the contents of a ptrlen in a printf; a constructor function
that wraps up a pointer and length so you can make a ptrlen on the fly
in mid-expression; a function to compare a ptrlen against an ordinary
C string (which I mostly expect to use with string literals); and a
function 'mkstr' to make a dynamically allocated C string out of one.
That last function replaces a function of the same name in sftp.c,
which I'm promoting to a whole-codebase facility and adjusting its
API.
This is a cleanup I started to notice a need for during the BinarySink
work. It removes a lot of faffing about casting things to char * or
unsigned char * so that some API will accept them, even though lots of
such APIs really take a plain 'block of raw binary data' argument and
don't care what C thinks the signedness of that data might be - they
may well reinterpret it back and forth internally.
So I've tried to arrange for all the function call APIs that ought to
have a void * (or const void *) to have one, and those that need to do
pointer arithmetic on the parameter internally can cast it back at the
top of the function. That saves endless ad-hoc casts at the call
sites.
xfer_download_data could return actuallen as either 0 or -1 to
indicate EOF. Now it's always 0, and scp_recv_filedata actually checks
for that case and reports an error.
It is possible for SSH_FXP_CLOSE requests to fail. This can happen if the
server buffers writes and an error occurs flushing the data to disk while
processing the SSH_FXP_CLOSE request. If the close fails, sftp_put_file now
returns an error as well.
The id member of the sftp_request structure is of type unsigned int.
This type is also used in the sftp_reqfind callback. In
sftp_find_request we thus need to pass a pointer to unsigned int to
find234. Before this commit, sftp_find_request was passing a pointer
to unsigned long to find234, which causes the lookup to fail on
big-endian platforms where sizeof(unsigned int) != sizeof(unsigned
long), e.g. ppc64.
Formerly PuTTY's SFTP code would transmit (or buffer) a megabyte of data
before even starting to look for acknowledgements, but wouldn't allow
there to be more than a megabyte of unacknowledged data at a time. Now,
instead, it pays attention to whether the transmit path is blocked, and
transmits iff it isn't.
This should mean that SFTP goes faster over long fat pipes, and also
doesn't end up buffering so much over thin ones.
I practice, I tend to run into other performance limitations (such as
TCP or SSH-2 windows) before this enhancement looks particularly good,
but with an artificial lag of 250 ms on the loopback interface this
patch almost doubles my upload speed, so I think it's worthwhile.
Having found a lot of unfixed constness issues in recent development,
I thought perhaps it was time to get proactive, so I compiled the
whole codebase with -Wwrite-strings. That turned up a huge load of
const problems, which I've fixed in this commit: the Unix build now
goes cleanly through with -Wwrite-strings, and the Windows build is as
close as I could get it (there are some lingering issues due to
occasional Windows API functions like AcquireCredentialsHandle not
having the right constness).
Notable fallout beyond the purely mechanical changing of types:
- the stuff saved by cmdline_save_param() is now explicitly
dupstr()ed, and freed in cmdline_run_saved.
- I couldn't make both string arguments to cmdline_process_param()
const, because it intentionally writes to one of them in the case
where it's the argument to -pw (in the vain hope of being at least
slightly friendly to 'ps'), so elsewhere I had to temporarily
dupstr() something for the sake of passing it to that function
- I had to invent a silly parallel version of const_cmp() so I could
pass const string literals in to lookup functions.
- stripslashes() in pscp.c and psftp.c has the annoying strchr nature
of the GET_32BIT macros and then used as length fields. Missing bounds
checks against zero have been added, and also I've introduced a helper
function toint() which casts from unsigned to int in such a way as to
avoid C undefined behaviour, since I'm not sure I trust compilers any
more to do the obviously sensible thing.
[originally from svn r9918]
effect of handling it, but they do not free it if it isn't a packet
they recognise as part of their upload/download. Invent a return value
that specifically signals this, and consistently free pktin at every
call site if that return value comes back. Also, ensure that that
return value also always comes with something meaningful in fxp_error.
[originally from svn r9915]
places we simply enforce by assertion that it will match the request
we sent out a moment ago: in fact it can also return NULL, so it makes
more sense to report a proper error message if it doesn't return the
expected value, and while we're at it, have that error message
whatever message was helpfully left in fxp_error() by
sftp_find_request when it failed.
To do this, I've written a centralised function in psftp.c called
sftp_wait_for_reply, which is handed a request that's just been sent
out and deals with the mechanics of waiting for its reply, returning
the reply when it arrives, and aborting with a sensible error if
anything else arrives instead. The numerous sites in psftp.c which
called sftp_find_request have all been rewritten to do this instead,
and as a side effect they now look more sensible. The only other uses
of sftp_find_request were in xfer_*load_gotpkt, which had to be
tweaked in its own way.
While I'm here, also fix memory management in sftp_find_request, which
was freeing its input packet on some but not all error return paths.
[originally from svn r9894]
using one for the length field and one for the rest of the packet
contents. Since sftp_senddata() has no queuing or deferral mechanism
but instead constructs and sends an SSH2_MSG_CHANNEL_DATA message
immediately, this change has the effect of ceasing to split every SFTP
packet across two SSH messages.
[originally from svn r9603]
I think I have to consider this to be a separate but related change to
the wishlist item 'pscp-filemodes'; that was written before the Unix
port existed, and referred to the ability to configure the permissions
used for files copied from Windows to Unix - which is still not done.
[originally from svn r9260]
encountered part way through transfer. In particular, this caused
psftp to hang (waiting for FXP_READ replies which had already
arrived) if you try `get' (without -r) on a remote directory.
[originally from svn r5005]
sftp.c, and psftp.c now uses that instead of going it alone. Should
in principle be easily installed in PSCP as well, but I haven't done
it yet; also it only handles downloads, not uploads, and finally it
doesn't yet properly calculate the correct number of parallel
requests to queue. Still, it's a start, and in my own tests it
seemed to perform as expected (download speed suddenly became
roughly what you'd expect from the available bandwidth, and
decreased by roughly the expected number of round-trip times).
[originally from svn r3468]
... here's a Unix port of PSFTP. Woo. (Oddly PSCP looks to be
somewhat harder; there's more Windows code interleaved than there
was in PSFTP.)
[originally from svn r3419]
has been split into a send half and a receive half, so that callers
can set several requests in motion at a time and deal with the
responses in whatever order they arrive.
[originally from svn r3318]
malloc functions, which automatically cast to the same type they're
allocating the size of. Should prevent any future errors involving
mallocing the size of the wrong structure type, and will also make
life easier if we ever need to turn the PuTTY core code from real C
into C++-friendly C. I haven't touched the Mac frontend in this
checkin because I couldn't compile or test it.
[originally from svn r3014]
absent, and also (I think) all the frontend request functions (such
as request_resize) take a context pointer, so that multiple windows
can be handled sensibly. I wouldn't swear to this, but I _think_
that only leaves the Unicode stuff as the last stubborn holdout.
[originally from svn r2147]
meaning that PSFTP couldn't meaningfully be debugged using
Minefield. That's what I get for developing it under Unix and
forgetting to port it properly :-/
[originally from svn r1383]
scp1 if it can't. Currently not very tested - I checked it in as
soon as it completed a successful recursive copy in both directions.
Also, one known bug: you can't specify a remote wildcard, because by
the nature of SFTP we'll need to implement the wildcard engine on
the client side. I do intend to do this (and use the same wildcard
engine in PSFTP as well) but I haven't got round to it yet.
[originally from svn r1208]
Simon Tatham
Tim Kosse
Ben Harris
Jacob Nevins