manual: Revise crypt.texi.

This is a major rewrite of the description of 'crypt', 'getentropy',
and 'getrandom'.

A few highlights of the content changes:

 - Throughout the manual, public headers, and user-visible messages,
   I replaced the term "password" with "passphrase", the term
   "password database" with "user database", and the term
   "encrypt(ion)" with "(one-way) hashing" whenever it was applied to
   passphrases.  I didn't bother making this change in internal code
   or tests.  The use of the term "password" in ruserpass.c survives,
   because that refers to a keyword in netrc files, but it is adjusted
   to make this clearer.

   There is a note in crypt.texi explaining that they were
   traditionally called passwords but single words are not good enough
   anymore, and a note in users.texi explaining that actual passphrase
   hashes are found in a "shadow" database nowadays.

 - There is a new short introduction to the "Cryptographic Functions"
   section, explaining how we do not intend to be a general-purpose
   cryptography library, and cautioning that there _are_, or have
   been, legal restrictions on the use of cryptography in many
   countries, without getting into any kind of detail that we can't
   promise to keep up to date.

 - I added more detail about what a "one-way function" is, and why
   they are used to obscure passphrases for storage.  I removed the
   paragraph saying that systems not connected to a network need no
   user authentication, because that's a pretty rare situation
   nowadays.  (It still says "sometimes it is necessary" to
   authenticate the user, though.)

 - I added documentation for all of the hash functions that glibc
   actually supports, but not for the additional hash functions
   supported by libxcrypt.  If we're going to keep this manual section
   around after the transition is more advanced, it would probably
   make sense to add them then.

 - There is much more detailed discussion of how to generate a salt,
   and the failure behavior for crypt is documented.  (Returning an
   invalid hash on failure is what libxcrypt does; Solar Designer's
   notes say that this was done "for compatibility with old programs
   that assume crypt can never fail".)

 - As far as I can tell, the header 'crypt.h' is entirely a GNU
   invention, and never existed on any other Unix lineage.  The
   function 'crypt', however, was in Issue 1 of the SVID and is now
   in the XSI component of POSIX.  I tried to make all of the
   @standards annotations consistent with this, but I'm not sure I got
   them perfectly right.

 - The genpass.c example has been improved to use getentropy instead
   of the current time to generate the salt, and to use a SHA-256 hash
   instead of MD5. It uses more random bytes than is strictly
   necessary because I didn't want to complicate the code with proper
   base64 encoding.

 - The testpass.c example has three hardwired hashes now, to
   demonstrate that different one-way functions produce different
   hashes for the same input.  It also demonstrates how DES hashing
   only pays attention to the first eight characters of the input.

 - There is new text explaining in more detail how a CSPRNG differs
   from a regular random number generator, and how
   getentropy/getrandom are not exactly a CSPRNG.  I tried not to make
   specific falsifiable claims here.  I also tried to make the
   blocking/cancellation/error behavior of both getentropy and
   getrandom clearer.

18 files changed, 426 insertions, 226 deletions

diff --git a/ChangeLog b/ChangeLog
index 338dbc3fc5..9b6edb1f1f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,31 @@
 2018-06-29  Zack Weinberg  <zackw@panix.com>
 
+	* crypt/crypt.h, posix/unistd.h: Update comments and
+	prototypes for crypt and crypt_r.
+
+	* manual/crypt.texi (Cryptographic Functions): New initial
+	exposition.
+	(crypt): Section renamed to 'Passphrase Storage'.  Full rewrite.
+	(Unpredictable Bytes): Improve initial exposition.  Clarify error
+	behavior of getentropy and getrandom.
+	* manual/examples/genpass.c: Generate a salt using getentropy
+	instead of the current time. Use hash $5$ (SHA-2-256).
+	* manual/examples/testpass.c: Demonstrate validation against
+	hashes generated with three different one-way functions.
+
+	* manual/intro.texi: crypt.texi does not need an overview
+	anymore.
+
+	* manual/nss.texi, manual/memory.texi, manual/socket.texi
+	* manual/terminal.texi: Consistently refer to "passphrases"
+	* instead of "passwords", and to the "user database" instead
+	* of the "password database".
+	* manual/users.texi: Similarly.  Add notes about how actual
+	passphrase hashes are now stored in the shadow database.
+	Remove 20-year-old junk todo note.
+
+2018-06-29  Zack Weinberg  <zackw@panix.com>
+
 	* manual/crypt.texi: Use a normal top-level @node declaration.
 	Move most of the introductory text to the 'crypt' section.
 	Move the example programs below the @deftypefun for 'crypt_r'.
diff --git a/crypt/crypt.h b/crypt/crypt.h
index 3cb18de14d..ebe8607452 100644
--- a/crypt/crypt.h
+++ b/crypt/crypt.h
@@ -28,13 +28,18 @@
 
 __BEGIN_DECLS
 
-/* Encrypt at most 8 characters from KEY using salt to perturb DES.  */
-extern char *crypt (const char *__key, const char *__salt)
+/* One-way hash PHRASE, returning a string suitable for storage in the
+   user database.  SALT selects the one-way function to use, and
+   ensures that no two users' hashes are the same, even if they use
+   the same passphrase.  The return value points to static storage
+   which will be overwritten by the next call to crypt.  */
+extern char *crypt (const char *__phrase, const char *__salt)
      __THROW __nonnull ((1, 2));
 
 #ifdef __USE_GNU
-/* Reentrant version of 'crypt'.  The additional argument
-   points to a structure where the results are placed in.  */
+
+/* This structure provides scratch and output buffers for 'crypt_r'.
+   Its contents should not be accessed directly.  */
 struct crypt_data
   {
     char keysched[16 * 8];
@@ -49,7 +54,13 @@ struct crypt_data
     int  direction, initialized;
   };
 
-extern char *crypt_r (const char *__key, const char *__salt,
+/* Thread-safe version of 'crypt'.
+   DATA must point to a 'struct crypt_data' allocated by the caller.
+   Before the first call to 'crypt_r' with a new 'struct crypt_data',
+   that object must be initialized to all zeroes.  The pointer
+   returned, if not NULL, will point within DATA.  (It will still be
+   overwritten by the next call to 'crypt_r' with the same DATA.)  */
+extern char *crypt_r (const char *__phrase, const char *__salt,
 		      struct crypt_data * __restrict __data)
      __THROW __nonnull ((1, 2, 3));
 #endif
diff --git a/inet/ruserpass.c b/inet/ruserpass.c
index 5b2747bc50..4fa6520c1a 100644
--- a/inet/ruserpass.c
+++ b/inet/ruserpass.c
@@ -177,7 +177,7 @@ next:
 			    fstat64(fileno(cfile), &stb) >= 0 &&
 			    (stb.st_mode & 077) != 0) {
 	warnx(_("Error: .netrc file is readable by others."));
-	warnx(_("Remove password or make file unreadable by others."));
+	warnx(_("Remove 'password' line or make file unreadable by others."));
 				goto bad;
 			}
 			if (token() && *apass == 0) {
diff --git a/manual/contrib.texi b/manual/contrib.texi
index dd28e9293b..6191f46f31 100644
--- a/manual/contrib.texi
+++ b/manual/contrib.texi
@@ -129,7 +129,7 @@ Martin Galvan for contributing gdb pretty printer support to glibc and adding
 an initial set of pretty printers for structures in the POSIX Threads library.
 
 @item
-Michael Glad for the DES encryption function @code{crypt} and related
+Michael Glad for the passphrase-hashing function @code{crypt} and related
 functions.
 
 @item
diff --git a/manual/crypt.texi b/manual/crypt.texi
index 0f04ee9899..c41b911c8f 100644
--- a/manual/crypt.texi
+++ b/manual/crypt.texi
@@ -1,121 +1,200 @@
 @node Cryptographic Functions, Debugging Support, System Configuration, Top
 @chapter Cryptographic Functions
-@c %MENU% Password storage and strongly unpredictable bytes
+@c %MENU% Passphrase storage and strongly unpredictable bytes.
+
+@Theglibc{} includes only a few special-purpose cryptographic
+functions: one-way hash functions for passphrase storage, and access
+to a cryptographic randomness source, if one is provided by the
+operating system.  Programs that need general-purpose cryptography
+should use a dedicated cryptography library, such as
+@uref{https://www.gnu.org/software/libgcrypt/,,libgcrypt}.
+
+Many countries place legal restrictions on the import, export,
+possession, or use of cryptographic software.  We deplore these
+restrictions, but we must still warn you that @theglibc{} may be
+subject to them, even if you do not use the functions in this chapter
+yourself.  The restrictions vary from place to place and are changed
+often, so we cannot give any more specific advice than this warning.
 
 @menu
-* crypt::                       A one-way function for passwords.
-* Unpredictable Bytes::         Randomness for cryptography purposes.
+* Passphrase Storage::          One-way hashing for passphrases.
+* Unpredictable Bytes::         Randomness for cryptographic purposes.
 @end menu
 
-@node crypt
-@section Encrypting Passwords
+@node Passphrase Storage
+@section Passphrase Storage
+@cindex passphrase hashing
+@cindex one-way hashing
+@cindex hashing, passphrase
 
-On many systems, it is unnecessary to have any kind of user
-authentication; for instance, a workstation which is not connected to a
-network probably does not need any user authentication, because to use
-the machine an intruder must have physical access.
-
-Sometimes, however, it is necessary to be sure that a user is authorized
+Sometimes it is necessary to be sure that a user is authorized
 to use some service a machine provides---for instance, to log in as a
 particular user id (@pxref{Users and Groups}).  One traditional way of
-doing this is for each user to choose a secret @dfn{password}; then, the
-system can ask someone claiming to be a user what the user's password
-is, and if the person gives the correct password then the system can
-grant the appropriate privileges.
-
-If all the passwords are just stored in a file somewhere, then this file
-has to be very carefully protected.  To avoid this, passwords are run
-through a @dfn{one-way function}, a function which makes it difficult to
-work out what its input was by looking at its output, before storing in
-the file.
-
-@Theglibc{} provides a one-way function that is compatible with
-the behavior of the @code{crypt} function introduced in FreeBSD 2.0.
-It supports two one-way algorithms: one based on the MD5
-message-digest algorithm that is compatible with modern BSD systems,
-and the other based on the Data Encryption Standard (DES) that is
-compatible with Unix systems.
-
-@deftypefun {char *} crypt (const char *@var{key}, const char *@var{salt})
-@standards{BSD, crypt.h}
-@standards{SVID, crypt.h}
+doing this is for each user to choose a secret @dfn{passphrase}; then, the
+system can ask someone claiming to be a user what the user's passphrase
+is, and if the person gives the correct passphrase then the system can
+grant the appropriate privileges.  (Traditionally, these were called
+``passwords,'' but nowadays a single word is too easy to guess.)
+
+Programs that handle passphrases must take special care not to reveal
+them to anyone, no matter what.  It is not enough to keep them in a
+file that is only accessible with special privileges.  The file might
+be ``leaked'' via a bug or misconfiguration, and system administrators
+shouldn't learn everyone's passphrase even if they have to edit that
+file for some reason.  To avoid this, passphrases should also be
+converted into @dfn{one-way hashes}, using a @dfn{one-way function},
+before they are stored.
+
+A one-way function is easy to compute, but there is no known way to
+compute its inverse.  This means the system can easily check
+passphrases, by hashing them and comparing the result with the stored
+hash.  But an attacker who discovers someone's passphrase hash can
+only discover the passphrase it corresponds to by guessing and
+checking.  The one-way functions are designed to make this process
+impractically slow, for all but the most obvious guesses.  (Do not use
+a word from the dictionary as your passphrase.)
+
+@Theglibc{} provides an interface to four one-way functions, based on
+the SHA-2-512, SHA-2-256, MD5, and DES cryptographic primitives.  New
+passphrases should be hashed with either of the SHA-based functions.
+The others are too weak for newly set passphrases, but we continue to
+support them for verifying old passphrases.  The DES-based hash is
+especially weak, because it ignores all but the first eight characters
+of its input.
+
+@deftypefun {char *} crypt (const char *@var{phrase}, const char *@var{salt})
+@standards{X/Open, unistd.h}
+@standards{GNU, crypt.h}
 @safety{@prelim{}@mtunsafe{@mtasurace{:crypt}}@asunsafe{@asucorrupt{} @asulock{} @ascuheap{} @ascudlopen{}}@acunsafe{@aculock{} @acsmem{}}}
 @c Besides the obvious problem of returning a pointer into static
 @c storage, the DES initializer takes an internal lock with the usual
-@c set of problems for AS- and AC-Safety.  The FIPS mode checker and the
-@c NSS implementations of may leak file descriptors if canceled.  The
+@c set of problems for AS- and AC-Safety.
+@c The NSS implementations may leak file descriptors if cancelled.
 @c The MD5, SHA256 and SHA512 implementations will malloc on long keys,
 @c and NSS relies on dlopening, which brings about another can of worms.
 
-The @code{crypt} function takes a password, @var{key}, as a string, and
-a @var{salt} character array which is described below, and returns a
-printable ASCII string which starts with another salt.  It is believed
-that, given the output of the function, the best way to find a @var{key}
-that will produce that output is to guess values of @var{key} until the
-original value of @var{key} is found.
-
-The @var{salt} parameter does two things.  Firstly, it selects which
-algorithm is used, the MD5-based one or the DES-based one.  Secondly, it
-makes life harder for someone trying to guess passwords against a file
-containing many passwords; without a @var{salt}, an intruder can make a
-guess, run @code{crypt} on it once, and compare the result with all the
-passwords.  With a @var{salt}, the intruder must run @code{crypt} once
-for each different salt.
-
-For the MD5-based algorithm, the @var{salt} should consist of the string
-@code{$1$}, followed by up to 8 characters, terminated by either
-another @code{$} or the end of the string.  The result of @code{crypt}
-will be the @var{salt}, followed by a @code{$} if the salt didn't end
-with one, followed by 22 characters from the alphabet
-@code{./0-9A-Za-z}, up to 34 characters total.  Every character in the
-@var{key} is significant.
-
-For the DES-based algorithm, the @var{salt} should consist of two
-characters from the alphabet @code{./0-9A-Za-z}, and the result of
-@code{crypt} will be those two characters followed by 11 more from the
-same alphabet, 13 in total.  Only the first 8 characters in the
-@var{key} are significant.
-
-The MD5-based algorithm has no limit on the useful length of the
-password used, and is slightly more secure.  It is therefore preferred
-over the DES-based algorithm.
-
-When the user enters their password for the first time, the @var{salt}
-should be set to a new string which is reasonably random.  To verify a
-password against the result of a previous call to @code{crypt}, pass
-the result of the previous call as the @var{salt}.
+The function @code{crypt} converts a passphrase string, @var{phrase},
+into a one-way hash suitable for storage in the user database.  The
+string that it returns will consist entirely of printable ASCII
+characters.  It will not contain whitespace, nor any of the characters
+@samp{:}, @samp{;}, @samp{*}, @samp{!}, or @samp{\}.
+
+The @var{salt} parameter controls which one-way function is used, and
+it also ensures that the output of the one-way function is different
+for every user, even if they have the same passphrase.  This makes it
+harder to guess passphrases from a large user database.  Without salt,
+the attacker could make a guess, run @code{crypt} on it once, and
+compare the result with all the hashes.  Salt forces the attacker to
+make separate calls to @code{crypt} for each user.
+
+To verify a passphrase, pass the previously hashed passphrase as the
+@var{salt}.  To hash a new passphrase for storage, set @var{salt} to a
+string consisting of a prefix plus a sequence of randomly chosen
+characters, according to this table:
+
+@multitable @columnfractions .2 .1 .3
+@headitem One-way function @tab Prefix @tab Random sequence
+@item SHA-2-512
+@tab @samp{$6$}
+@tab 16 characters
+@item SHA-2-256
+@tab @samp{$5$}
+@tab 16 characters
+@item MD5
+@tab @samp{$1$}
+@tab 8 characters
+@item DES
+@tab @samp{}
+@tab 2 characters
+@end multitable
+
+In all cases, the random characters should be chosen from the alphabet
+@code{./0-9A-Za-z}.
+
+With all of the hash functions @emph{except} DES, @var{phrase} can be
+arbitrarily long, and all eight bits of each byte are significant.
+With DES, only the first eight characters of @var{phrase} affect the
+output, and the eighth bit of each byte is also ignored.
+
+@code{crypt} can fail.  Some implementations return @code{NULL} on
+failure, and others return an @emph{invalid} hashed passphrase, which
+will begin with a @samp{*} and will not be the same as @var{salt}.  In
+either case, @code{errno} will be set to indicate the problem.  Some
+of the possible error codes are:
+
+@table @code
+@item EINVAL
+@var{salt} is invalid; neither a previously hashed passphrase, nor a
+well-formed new salt for any of the supported hash functions.
+
+@item EPERM
+The system configuration forbids use of the hash function selected by
+@var{salt}.
+
+@item ENOMEM
+Failed to allocate internal scratch storage.
+
+@item ENOSYS
+@itemx EOPNOTSUPP
+Hashing passphrases is not supported at all, or the hash function
+selected by @var{salt} is not supported.  @Theglibc{} does not use
+these error codes, but they may be encountered on other operating
+systems.
+@end table
+
+@code{crypt} uses static storage for both internal scratchwork and the
+string it returns.  It is not safe to call @code{crypt} from multiple
+threads simultaneously, and the string it returns will be overwritten
+by any subsequent call to @code{crypt}.
+
+@code{crypt} is specified in the X/Open Portability Guide and is
+present on nearly all historical Unix systems.  However, the XPG does
+not specify any one-way functions.
+
+@code{crypt} is declared in @file{unistd.h}.  @Theglibc{} also
+declares this function in @file{crypt.h}.
 @end deftypefun
 
-@deftypefun {char *} crypt_r (const char *@var{key}, const char *@var{salt}, {struct crypt_data *} @var{data})
+@deftypefun {char *} crypt_r (const char *@var{phrase}, const char *@var{salt}, struct crypt_data *@var{data})
 @standards{GNU, crypt.h}
 @safety{@prelim{}@mtsafe{}@asunsafe{@asucorrupt{} @asulock{} @ascuheap{} @ascudlopen{}}@acunsafe{@aculock{} @acsmem{}}}
+@tindex struct crypt_data
 @c Compared with crypt, this function fixes the @mtasurace:crypt
 @c problem, but nothing else.
 
-The @code{crypt_r} function does the same thing as @code{crypt}, but
-takes an extra parameter which includes space for its result (among
-other things), so it can be reentrant.  @code{data@w{->}initialized} must be
-cleared to zero before the first time @code{crypt_r} is called.
-
-The @code{crypt_r} function is a GNU extension.
+The function @code{crypt_r} is a thread-safe version of @code{crypt}.
+Instead of static storage, it uses the memory pointed to by its
+@var{data} argument for both scratchwork and the string it returns.
+It can safely be used from multiple threads, as long as different
+@var{data} objects are used in each thread.  The string it returns
+will still be overwritten by another call with the same @var{data}.
+
+@var{data} must point to a @code{struct crypt_data} object allocated
+by the caller.  All of the fields of @code{struct crypt_data} are
+private, but before one of these objects is used for the first time,
+it must be initialized to all zeroes, using @code{memset} or similar.
+After that, it can be reused for many calls to @code{crypt_r} without
+erasing it again.  @code{struct crypt_data} is very large, so it is
+best to allocate it with @code{malloc} rather than as a local
+variable.  @xref{Memory Allocation}.
+
+@code{crypt_r} is a GNU extension.  It is declared in @file{crypt.h},
+as is @code{struct crypt_data}.
 @end deftypefun
 
-The @code{crypt} and @code{crypt_r} functions are prototyped in the
-header @file{crypt.h}.
-
-The following short program is an example of how to use @code{crypt} the
-first time a password is entered.  Note that the @var{salt} generation
-is just barely acceptable; in particular, it is not unique between
-machines, and in many applications it would not be acceptable to let an
-attacker know what time the user's password was last set.
+The following program shows how to use @code{crypt} the first time a
+passphrase is entered.  It uses @code{getentropy} to make the salt as
+unpredictable as possible; @pxref{Unpredictable Bytes}.
 
 @smallexample
 @include genpass.c.texi
 @end smallexample
 
-The next program shows how to verify a password.  It prompts the user
-for a password and prints ``Access granted.'' if the user types
-@code{GNU libc manual}.
+The next program demonstrates how to verify a passphrase.  It checks a
+hash hardcoded into the program, because looking up real users' hashed
+passphrases may require special privileges (@pxref{User Database}).
+It also shows that different one-way functions produce different
+hashes for the same passphrase.
 
 @smallexample
 @include testpass.c.texi
@@ -123,93 +202,121 @@ for a password and prints ``Access granted.'' if the user types
 
 @node Unpredictable Bytes
 @section Generating Unpredictable Bytes
-
-Some cryptographic applications (such as session key generation) need
-unpredictable bytes.
-
-In general,