status: add status serialization mechanism

Teach STATUS to optionally serialize the results of a
status computation to a file.

Teach STATUS to optionally read an existing serialization
file and simply print the results, rather than actually
scanning.

This is intended for immediate status results on extremely
large repos and assumes the use of a service/daemon to
maintain a fresh current status snapshot.

2021-10-30: packet_read() changed its prototype in ec9a37d (pkt-line.[ch]:
remove unused packet_read_line_buf(), 2021-10-14).

2021-10-30: sscanf() now does an extra check that "%d" goes into an "int"
and complains about "uint32_t". Replacing with "%u" fixes the compile-time
error.

2021-10-30: string_list_init() was removed by abf897b (string-list.[ch]:
remove string_list_init() compatibility function, 2021-09-28), so we need to
initialize manually.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

Documentation/config/status.txt

@@ -75,3 +75,9 @@ status.submoduleSummary::
 	the --ignore-submodules=dirty command-line option or the 'git
 	submodule summary' command, which shows a similar output but does
 	not honor these settings.
+
+status.deserializePath::
+	EXPERIMENTAL, Pathname to a file containing cached status results
+	generated by `--serialize`.  This will be overridden by
+	`--deserialize=<path>` on the command line.  If the cache file is
+	invalid or stale, git will fall-back and compute status normally.

Documentation/git-status.txt

@@ -149,6 +149,19 @@ ignored, then the directory is not shown, but all contents are shown.
 	threshold.
 	See also linkgit:git-diff[1] `--find-renames`.
 
+--serialize[=<version>]::
+	(EXPERIMENTAL) Serialize raw status results to stdout in a
+	format suitable for use by `--deserialize`.  Valid values for
+	`<version>` are "1" and "v1".
+
+--deserialize[=<path>]::
+	(EXPERIMENTAL) Deserialize raw status results from a file or
+	stdin rather than scanning the worktree.  If `<path>` is omitted
+	and `status.deserializePath` is unset, input is read from stdin.
+--no-deserialize::
+	(EXPERIMENTAL) Disable implicit deserialization of status results
+	from the value of `status.deserializePath`.
+
 <pathspec>...::
 	See the 'pathspec' entry in linkgit:gitglossary[7].
 
@@ -421,6 +434,26 @@ quoted as explained for the configuration variable `core.quotePath`
 (see linkgit:git-config[1]).
 
 
+SERIALIZATION and DESERIALIZATION (EXPERIMENTAL)
+------------------------------------------------
+
+The `--serialize` option allows git to cache the result of a
+possibly time-consuming status scan to a binary file.  A local
+service/daemon watching file system events could use this to
+periodically pre-compute a fresh status result.
+
+Interactive users could then use `--deserialize` to simply
+(and immediately) print the last-known-good result without
+waiting for the status scan.
+
+The binary serialization file format includes some worktree state
+information allowing `--deserialize` to reject the cached data
+and force a normal status scan if, for example, the commit, branch,
+or status modes/options change.  The format cannot, however, indicate
+when the cached data is otherwise stale -- that coordination belongs
+to the task driving the serializations.
+
+
 CONFIGURATION
 -------------
 

Documentation/technical/status-serialization-format.txt

@@ -0,0 +1,107 @@
+Git status serialization format
+===============================
+
+Git status serialization enables git to dump the results of a status scan
+to a binary file.  This file can then be loaded by later status invocations
+to print the cached status results.
+
+The file contains the essential fields from:
+() the index
+() the "struct wt_status" for the overall results
+() the contents of "struct wt_status_change_data" for tracked changed files
+() the list of untracked and ignored files
+
+Version 1 Format:
+=================
+
+The V1 file begins with a required header section followed by optional
+sections for each type of item (changed, untracked, ignored).  Individual
+item sections are only present if necessary.  Each item section begins
+with an item-type header with the number of items in the section.
+
+Each "line" in the format is encoded using pkt-line with a final LF.
+Flush packets are used to terminate sections.
+
+-----------------
+PKT-LINE("version" SP "1")
+<v1-header-section>
+[<v1-changed-item-section>]
+[<v1-untracked-item-section>]
+[<v1-ignored-item-section>]
+-----------------
+
+
+V1 Header
+---------
+
+The v1-header-section fields are taken directly from "struct wt_status".
+Each field is printed on a separate pkt-line.  Lines for NULL string
+values are omitted.  All integers are printed with "%d".  OIDs are
+printed in hex.
+
+v1-header-section    = <v1-index-headers>
+		       <v1-wt-status-headers>
+		       PKT-LINE(<flush>)
+
+v1-index-headers     = PKT-LINE("index_mtime" SP <sec> SP <nsec> LF)
+
+v1-wt-status-headers = PKT-LINE("is_initial" SP <integer> LF)
+		       [ PKT-LINE("branch" SP <branch-name> LF) ]
+		       [ PKT-LINE("reference" SP <reference-name> LF) ]
+		       PKT-LINE("show_ignored_files" SP <integer> LF)
+		       PKT-LINE("show_untracked_files" SP <integer> LF)
+		       PKT-LINE("show_ignored_directory" SP <integer> LF)
+		       [ PKT-LINE("ignore_submodule_arg" SP <string> LF) ]
+		       PKT-LINE("detect_rename" SP <integer> LF)
+		       PKT-LINE("rename_score" SP <integer> LF)
+		       PKT-LINE("rename_limit" SP <integer> LF)
+		       PKT-LINE("detect_break" SP <integer> LF)
+		       PKT-LINE("sha1_commit" SP <oid> LF)
+		       PKT-LINE("committable" SP <integer> LF)
+		       PKT-LINE("workdir_dirty" SP <integer> LF)
+
+
+V1 Changed Items
+----------------
+
+The v1-changed-item-section lists all of the changed items with one
+item per pkt-line.  Each pkt-line contains:  a binary block of data
+from "struct wt_status_serialize_data_fixed" in a fixed header where
+integers are in network byte order and OIDs are in raw (non-hex) form.
+This is followed by one or two raw pathnames (not c-quoted) with NUL
+terminators (both NULs are always present even if there is no rename).
+
+v1-changed-item-section = PKT-LINE("changed" SP <count> LF)
+			  [ PKT-LINE(<changed_item> LF) ]+
+			  PKT-LINE(<flush>)
+
+changed_item = <byte[4] worktree_status>
+	       <byte[4] index_status>
+	       <byte[4] stagemask>
+	       <byte[4] score>
+	       <byte[4] mode_head>
+	       <byte[4] mode_index>
+	       <byte[4] mode_worktree>
+	       <byte[4] dirty_submodule>
+	       <byte[4] new_submodule_commits>
+	       <byte[20] oid_head>
+	       <byte[20] oid_index>
+	       <byte[*] path>
+	       NUL
+	       [ <byte[*] src_path> ]
+	       NUL
+
+
+V1 Untracked and Ignored Items
+------------------------------
+
+These sections are simple lists of pathnames.  They ARE NOT
+c-quoted.
+
+v1-untracked-item-section = PKT-LINE("untracked" SP <count> LF)
+			    [ PKT-LINE(<pathname> LF) ]+
+			    PKT-LINE(<flush>)
+
+v1-ignored-item-section = PKT-LINE("ignored" SP <count> LF)
+			  [ PKT-LINE(<pathname> LF) ]+
+			  PKT-LINE(<flush>)

Makefile

@@ -1115,6 +1115,8 @@ LIB_OBJS += wrapper.o
 LIB_OBJS += write-or-die.o
 LIB_OBJS += ws.o
 LIB_OBJS += wt-status.o
+LIB_OBJS += wt-status-deserialize.o
+LIB_OBJS += wt-status-serialize.o
 LIB_OBJS += xdiff-interface.o
 LIB_OBJS += zlib.o
 

builtin/commit.c

@@ -160,6 +160,70 @@ static int opt_parse_porcelain(const struct option *opt, const char *arg, int un
 	return 0;
 }
 
+static int do_serialize = 0;
+static int do_implicit_deserialize = 0;
+static int do_explicit_deserialize = 0;
+static char *deserialize_path = NULL;
+
+/*
+ * --serialize | --serialize=1 | --serialize=v1
+ *
+ * Request that we serialize our output rather than printing in
+ * any of the established formats.  Optionally specify serialization
+ * version.
+ */
+static int opt_parse_serialize(const struct option *opt, const char *arg, int unset)
+{
+	enum wt_status_format *value = (enum wt_status_format *)opt->value;
+	if (unset || !arg)
+		*value = STATUS_FORMAT_SERIALIZE_V1;
+	else if (!strcmp(arg, "v1") || !strcmp(arg, "1"))
+		*value = STATUS_FORMAT_SERIALIZE_V1;
+	else
+		die("unsupported serialize version '%s'", arg);
+
+	if (do_explicit_deserialize)
+		die("cannot mix --serialize and --deserialize");
+	do_implicit_deserialize = 0;
+
+	do_serialize = 1;
+	return 0;
+}
+
+/*
+ * --deserialize | --deserialize=<path> |
+ * --no-deserialize
+ *
+ * Request that we deserialize status data from some existing resource
+ * rather than performing a status scan.
+ *
+ * The input source can come from stdin or a path given here -- or be
+ * inherited from the config settings.
+ */
+static int opt_parse_deserialize(const struct option *opt, const char *arg, int unset)
+{
+	if (unset) {
+		do_implicit_deserialize = 0;
+		do_explicit_deserialize = 0;
+	} else {
+		if (do_serialize)
+			die("cannot mix --serialize and --deserialize");
+		if (arg) {
+			/* override config or stdin */
+			free(deserialize_path);
+			deserialize_path = xstrdup(arg);
+		}
+		if (deserialize_path && *deserialize_path
+		    && (access(deserialize_path, R_OK) != 0))
+			die("cannot find serialization file '%s'",
+			    deserialize_path);
+
+		do_explicit_deserialize = 1;
+	}
+
+	return 0;
+}
+
 static int opt_parse_m(const struct option *opt, const char *arg, int unset)
 {
 	struct strbuf *buf = opt->value;
@@ -1152,6 +1216,8 @@ static void handle_untracked_files_arg(struct wt_status *s)
 		s->show_untracked_files = SHOW_NORMAL_UNTRACKED_FILES;
 	else if (!strcmp(untracked_files_arg, "all"))
 		s->show_untracked_files = SHOW_ALL_UNTRACKED_FILES;
+	else if (!strcmp(untracked_files_arg,"complete"))
+		s->show_untracked_files = SHOW_COMPLETE_UNTRACKED_FILES;
 	/*
 	 * Please update $__git_untracked_file_modes in
 	 * git-completion.bash when you add new options
@@ -1437,6 +1503,19 @@ static int git_status_config(const char *k, const char *v, void *cb)
 		s->relative_paths = git_config_bool(k, v);
 		return 0;
 	}
+	if (!strcmp(k, "status.deserializepath")) {
+		/*
+		 * Automatically assume deserialization if this is
+		 * set in the config and the file exists.  Do not
+		 * complain if the file does not exist, because we
+		 * silently fall back to normal mode.
+		 */
+		if (v && *v && access(v, R_OK) == 0) {
+			do_implicit_deserialize = 1;
+			deserialize_path = xstrdup(v);
+		}
+		return 0;
+	}
 	if (!strcmp(k, "status.showuntrackedfiles")) {
 		if (!v)
 			return config_error_nonbool(k);
@@ -1477,7 +1556,8 @@ int cmd_status(int argc, const char **argv, const char *prefix)
 	static const char *rename_score_arg = (const char *)-1;
 	static struct wt_status s;
 	unsigned int progress_flag = 0;
-	int fd;
+	int try_deserialize;
+	int fd = -1;
 	struct object_id oid;
 	static struct option builtin_status_options[] = {
 		OPT__VERBOSE(&verbose, N_("be verbose")),
@@ -1492,6 +1572,12 @@ int cmd_status(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, "porcelain", &status_format,
 		  N_("version"), N_("machine-readable output"),
 		  PARSE_OPT_OPTARG, opt_parse_porcelain),
+		{ OPTION_CALLBACK, 0, "serialize", &status_format,
+		  N_("version"), N_("serialize raw status data to stdout"),
+		  PARSE_OPT_OPTARG | PARSE_OPT_NONEG, opt_parse_serialize },
+		{ OPTION_CALLBACK, 0, "deserialize", NULL,
+		  N_("path"), N_("deserialize raw status data from file"),
+		  PARSE_OPT_OPTARG, opt_parse_deserialize },
 		OPT_SET_INT(0, "long", &status_format,
 			    N_("show status in long format (default)"),
 			    STATUS_FORMAT_LONG),
@@ -1536,10 +1622,26 @@ int cmd_status(int argc, const char **argv, const char *prefix)
 	    s.show_untracked_files == SHOW_NO_UNTRACKED_FILES)
 		die(_("Unsupported combination of ignored and untracked-files arguments"));
 
+	if (s.show_untracked_files == SHOW_COMPLETE_UNTRACKED_FILES &&
+	    s.show_ignored_mode == SHOW_NO_IGNORED)
+		die(_("Complete Untracked only supported with ignored files"));
+
 	parse_pathspec(&s.pathspec, 0,
 		       PATHSPEC_PREFER_FULL,
 		       prefix, argv);
 
+	/*
+	 * If we want to try to deserialize status data from a cache file,
+	 * we need to re-order the initialization code.  The problem is that
+	 * this makes for a very nasty diff and causes merge conflicts as we
+	 * carry it forward.  And it easy to mess up the merge, so we
+	 * duplicate some code here to hopefully reduce conflicts.
+	 */
+	try_deserialize = (!do_serialize &&
+			   (do_implicit_deserialize || do_explicit_deserialize));
+	if (try_deserialize)
+		goto skip_init;
+
 	enable_fscache(0);
 	if (status_format != STATUS_FORMAT_PORCELAIN &&
 	    status_format != STATUS_FORMAT_PORCELAIN_V2)
@@ -1554,6 +1656,7 @@ int cmd_status(int argc, const char **argv, const char *prefix)
 	else
 		fd = -1;
 
+skip_init:
 	s.is_initial = get_oid(s.reference, &oid) ? 1 : 0;
 	if (!s.is_initial)
 		oidcpy(&s.oid_commit, &oid);
@@ -1570,6 +1673,24 @@ int cmd_status(int argc, const char **argv, const char *prefix)
 			s.rename_score = parse_rename_score(&rename_score_arg);
 	}
 
+	if (try_deserialize) {
+		if (s.relative_paths)
+			s.prefix = prefix;
+
+		if (wt_status_deserialize(&s, deserialize_path) == DESERIALIZE_OK)
+			return 0;
+
+		/* deserialize failed, so force the initialization we skipped above. */
+		enable_fscache(1);
+		read_cache_preload(&s.pathspec);
+		refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL);
+
+		if (use_optional_locks())
+			fd = hold_locked_index(&index_lock, 0);
+		else
+			fd = -1;
+	}
+
 	wt_status_collect(&s);
 
 	if (0 <= fd)

contrib/completion/git-completion.bash

@@ -1655,7 +1655,7 @@ _git_clone ()
 	esac
 }
 
-__git_untracked_file_modes="all no normal"
+__git_untracked_file_modes="all no normal complete"
 
 _git_commit ()
 {

pkt-line.c

@@ -225,7 +225,7 @@ static int do_packet_write(const int fd_out, const char *buf, size_t size,
 	return 0;
 }
 
-static int packet_write_gently(const int fd_out, const char *buf, size_t size)
+int packet_write_gently(const int fd_out, const char *buf, size_t size)
 {
 	struct strbuf err = STRBUF_INIT;
 	if (do_packet_write(fd_out, buf, size, &err)) {

pkt-line.h

@@ -31,6 +31,7 @@ void packet_write(int fd_out, const char *buf, size_t size);
 void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
 int packet_flush_gently(int fd);
 int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
+int packet_write_gently(const int fd_out, const char *buf, size_t size);
 int write_packetized_from_fd_no_flush(int fd_in, int fd_out);
 int write_packetized_from_buf_no_flush(const char *src_in, size_t len, int fd_out);
 

t/t7522-serialized-status.sh

@@ -0,0 +1,141 @@
+#!/bin/sh
+
+test_description='git serialized status tests'
+
+. ./test-lib.sh
+
+# This file includes tests for serializing / deserializing
+# status data. These tests cover two basic features:
+#
+# [1] Because users can request different types of untracked-file
+#     and ignored file reporting, the cache data generated by
+#     serialize must use either the same untracked and ignored
+#     parameters as the later deserialize invocation; otherwise,
+#     the deserialize invocation must disregard the cached data
+#     and run a full scan itself.
+#
+#     To increase the number of cases where the cached status can
+#     be used, we have added a "--untracked-file=complete" option
+#     that reports a superset or union of the results from the
+#     "-u normal" and "-u all".  We combine this with a filter in
+#     deserialize to filter the results.
+#
+#     Ignored file reporting is simpler in that is an all or
+#     nothing; there are no subsets.
+#
+#     The tests here (in addition to confirming that a cache
+#     file can be generated and used by a subsequent status
+#     command) need to test this untracked-file filtering.
+#
+# [2] ensuring the status calls are using data from the status
+#     cache as expected.  This includes verifying cached data
+#     is used when appropriate as well as falling back to
+#     performing a new status scan when the data in the cache
+#     is insufficient/known stale.
+
+test_expect_success 'setup' '
+	git branch -M main &&
+	cat >.gitignore <<-\EOF &&
+	*.ign
+	ignored_dir/
+	EOF
+
+	mkdir tracked ignored_dir &&
+	touch tracked_1.txt tracked/tracked_1.txt &&
+	git add . &&
+	test_tick &&
+	git commit -m"Adding original file." &&
+	mkdir untracked &&
+	touch ignored.ign ignored_dir/ignored_2.txt \
+	      untracked_1.txt untracked/untracked_2.txt untracked/untracked_3.txt
+'
+
+test_expect_success 'verify untracked-files=complete with no conversion' '
+	test_when_finished "rm serialized_status.dat new_change.txt output" &&
+	cat >expect <<-\EOF &&
+	? expect
+	? serialized_status.dat
+	? untracked/
+	? untracked/untracked_2.txt
+	? untracked/untracked_3.txt
+	? untracked_1.txt
+	! ignored.ign
+	! ignored_dir/
+	EOF
+
+	git status --untracked-files=complete --ignored=matching --serialize >serialized_status.dat &&
+	touch new_change.txt &&
+
+	git status --porcelain=v2 --untracked-files=complete --ignored=matching --deserialize=serialized_status.dat >output &&
+	test_cmp expect output
+'
+
+test_expect_success 'verify untracked-files=complete to untracked-files=normal conversion' '
+	test_when_finished "rm serialized_status.dat new_change.txt output" &&
+	cat >expect <<-\EOF &&
+	? expect
+	? serialized_status.dat
+	? untracked/
+	? untracked_1.txt
+	EOF
+
+	git status --untracked-files=complete --ignored=matching --serialize >serialized_status.dat &&
+	touch new_change.txt &&
+
+	git status --porcelain=v2 --deserialize=serialized_status.dat >output &&
+	test_cmp expect output
+'
+
+test_expect_success 'verify untracked-files=complete to untracked-files=all conversion' '
+	test_when_finished "rm serialized_status.dat new_change.txt output" &&
+	cat >expect <<-\EOF &&
+	? expect
+	? serialized_status.dat
+	? untracked/untracked_2.txt
+	? untracked/untracked_3.txt
+	? untracked_1.txt
+	! ignored.ign
+	! ignored_dir/
+	EOF
+
+	git status --untracked-files=complete --ignored=matching --serialize >serialized_status.dat &&
+	touch new_change.txt &&
+
+	git status --porcelain=v2 --untracked-files=all --ignored=matching --deserialize=serialized_status.dat >output &&
+	test_cmp expect output
+'
+
+test_expect_success 'verify serialized status with non-convertible ignore mode does new scan' '
+	test_when_finished "rm serialized_status.dat new_change.txt output" &&
+	cat >expect <<-\EOF &&
+	? expect
+	? new_change.txt
+	? output
+	? serialized_status.dat
+	? untracked/
+	? untracked_1.txt
+	! ignored.ign
+	! ignored_dir/
+	EOF
+
+	git status --untracked-files=complete --ignored=matching --serialize >serialized_status.dat &&
+	touch new_change.txt &&
+
+	git status --porcelain=v2 --ignored --deserialize=serialized_status.dat >output &&
+	test_cmp expect output
+'
+
+test_expect_success 'verify serialized status handles path scopes' '
+	test_when_finished "rm serialized_status.dat new_change.txt output" &&
+	cat >expect <<-\EOF &&
+	? untracked/
+	EOF
+
+	git status --untracked-files=complete --ignored=matching --serialize >serialized_status.dat &&
+	touch new_change.txt &&
+
+	git status --porcelain=v2 --deserialize=serialized_status.dat untracked >output &&
+	test_cmp expect output
+'
+
+test_done

t/t7523-status-complete-untracked.sh

@@ -0,0 +1,39 @@
+#!/bin/sh
+
+test_description='git status untracked complete tests'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	cat >.gitignore <<-\EOF &&
+	*.ign
+	ignored_dir/
+	EOF
+
+	mkdir tracked ignored_dir &&
+	touch tracked_1.txt tracked/tracked_1.txt &&
+	git add . &&
+	test_tick &&
+	git commit -m"Adding original file." &&
+	mkdir untracked &&
+	touch ignored.ign ignored_dir/ignored_2.txt \
+	      untracked_1.txt untracked/untracked_2.txt untracked/untracked_3.txt
+'
+
+test_expect_success 'verify untracked-files=complete' '
+	cat >expect <<-\EOF &&
+	? expect
+	? output
+	? untracked/
+	? untracked/untracked_2.txt
+	? untracked/untracked_3.txt
+	? untracked_1.txt
+	! ignored.ign
+	! ignored_dir/
+	EOF
+
+	git status --porcelain=v2 --untracked-files=complete --ignored >output &&
+	test_cmp expect output
+'
+
+test_done

wt-status-deserialize.c

@@ -0,0 +1,600 @@
+#include "cache.h"
+#include "wt-status.h"
+#include "pkt-line.h"
+#include "trace.h"
+
+static struct trace_key trace_deserialize = TRACE_KEY_INIT(DESERIALIZE);
+
+enum deserialize_parse_strategy {
+	DESERIALIZE_STRATEGY_AS_IS,
+	DESERIALIZE_STRATEGY_SKIP,
+	DESERIALIZE_STRATEGY_NORMAL,
+	DESERIALIZE_STRATEGY_ALL
+};
+
+static int check_path_contains(const char *out, int out_len, const char *in, int in_len)
+{
+	return (out_len > 0 &&
+		out_len < in_len &&
+		(out[out_len - 1] == '/') &&
+		!memcmp(out, in, out_len));
+}
+
+static const char *my_packet_read_line(int fd, int *line_len)
+{
+	static char buf[LARGE_PACKET_MAX];
+
+	*line_len = packet_read(fd, buf, sizeof(buf),
+				PACKET_READ_CHOMP_NEWLINE |
+				PACKET_READ_GENTLE_ON_EOF);
+	return (*line_len > 0) ? buf : NULL;
+}
+
+/*
+ * mtime_reported contains the mtime of the index when the
+ * serialization snapshot was computed.
+ *
+ * mtime_observed_on_disk contains the mtime of the index now.
+ *
+ * If these 2 times are different, then the .git/index has
+ * changed since the serialization cache was created and we
+ * must reject the cache because anything could have changed.
+ *
+ * If they are the same, we continue trying to use the cache.
+ */
+static int my_validate_index(const struct cache_time *mtime_reported)
+{
+	const char *path = get_index_file();
+	struct stat st;
+	struct cache_time mtime_observed_on_disk;
+
+	if (lstat(path, &st)) {
+		trace_printf_key(&trace_deserialize, "could not stat index");
+		return DESERIALIZE_ERR;
+	}
+	mtime_observed_on_disk.sec = st.st_mtime;
+	mtime_observed_on_disk.nsec = ST_MTIME_NSEC(st);
+	if ((mtime_observed_on_disk.sec != mtime_reported->sec) ||
+	    (mtime_observed_on_disk.nsec != mtime_reported->nsec)) {
+		trace_printf_key(&trace_deserialize, "index mtime changed [des %d.%d][obs %d.%d]",
+			     mtime_reported->sec, mtime_reported->nsec,
+			     mtime_observed_on_disk.sec, mtime_observed_on_disk.nsec);
+		return DESERIALIZE_ERR;
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int wt_deserialize_v1_header(struct wt_status *s, int fd)
+{
+	struct cache_time index_mtime;
+	int line_len, nr_fields;
+	const char *line;
+	const char *arg;
+
+	/*
+	 * parse header lines up to the first flush packet.
+	 */
+	while ((line = my_packet_read_line(fd, &line_len))) {
+
+		if (skip_prefix(line, "index_mtime ", &arg)) {
+			nr_fields = sscanf(arg, "%u %u",
+					   &index_mtime.sec,
+					   &index_mtime.nsec);
+			if (nr_fields != 2) {
+				trace_printf_key(&trace_deserialize, "invalid index_mtime (%d) '%s'",
+					     nr_fields, line);
+				return DESERIALIZE_ERR;
+			}
+			continue;
+		}
+
+		if (skip_prefix(line, "is_initial ", &arg)) {
+			s->is_initial = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "branch ", &arg)) {
+			s->branch = xstrdup(arg);
+			continue;
+		}
+		if (skip_prefix(line, "reference ", &arg)) {
+			s->reference = xstrdup(arg);
+			continue;
+		}
+		/* pathspec */
+		/* verbose */
+		/* amend */
+		if (skip_prefix(line, "whence ", &arg)) {
+			s->whence = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		/* nowarn */
+		/* use_color */
+		/* no_gettext */
+		/* display_comment_prefix */
+		/* relative_paths */
+		/* submodule_summary */
+		if (skip_prefix(line, "show_ignored_mode ", &arg)) {
+			s->show_ignored_mode = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "show_untracked_files ", &arg)) {
+			s->show_untracked_files = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "ignore_submodule_arg ", &arg)) {
+			s->ignore_submodule_arg = xstrdup(arg);
+			continue;
+		}
+		/* color_palette */
+		/* colopts */
+		/* null_termination */
+		/* commit_template */
+		/* show_branch */
+		/* show_stash */
+		if (skip_prefix(line, "hints ", &arg)) {
+			s->hints = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "detect_rename ", &arg)) {
+			s->detect_rename = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "rename_score ", &arg)) {
+			s->rename_score = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "rename_limit ", &arg)) {
+			s->rename_limit = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		/* status_format */
+		if (skip_prefix(line, "sha1_commit ", &arg)) {
+			if (get_oid_hex(arg, &s->oid_commit)) {
+				trace_printf_key(&trace_deserialize, "invalid sha1_commit");
+				return DESERIALIZE_ERR;
+			}
+			continue;
+		}
+		if (skip_prefix(line, "committable ", &arg)) {
+			s->committable = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		if (skip_prefix(line, "workdir_dirty ", &arg)) {
+			s->workdir_dirty = (int)strtol(arg, NULL, 10);
+			continue;
+		}
+		/* prefix */
+
+		trace_printf_key(&trace_deserialize, "unexpected line '%s'", line);
+		return DESERIALIZE_ERR;
+	}
+
+	return my_validate_index(&index_mtime);
+}
+
+/*
+ * Build a string-list of (count) <changed-item> lines from the input.
+ */
+static int wt_deserialize_v1_changed_items(struct wt_status *s, int fd, int count)
+{
+	struct wt_status_serialize_data *sd;
+	char *p;
+	int line_len;
+	const char *line;
+	struct string_list_item *item;
+
+	memset(&s->change, 0, sizeof(s->change));
+	s->change.strdup_strings = 1;
+
+	/*
+	 * <wt_status_change_data_fields>+
+	 * <flush>
+	 *
+	 * <fixed_part><path> NUL [<head_path>] NUL
+	 */
+	while ((line = my_packet_read_line(fd, &line_len))) {
+		struct wt_status_change_data *d = xcalloc(1, sizeof(*d));
+		sd = (struct wt_status_serialize_data *)line;
+
+		d->worktree_status = ntohl(sd->fixed.worktree_status);
+		d->index_status = ntohl(sd->fixed.index_status);
+		d->stagemask = ntohl(sd->fixed.stagemask);
+		d->rename_score = ntohl(sd->fixed.rename_score);
+		d->mode_head = ntohl(sd->fixed.mode_head);
+		d->mode_index = ntohl(sd->fixed.mode_index);
+		d->mode_worktree = ntohl(sd->fixed.mode_worktree);
+		d->dirty_submodule = ntohl(sd->fixed.dirty_submodule);
+		d->new_submodule_commits = ntohl(sd->fixed.new_submodule_commits);
+		oidcpy(&d->oid_head, &sd->fixed.oid_head);
+		oidcpy(&d->oid_index, &sd->fixed.oid_index);
+
+		p = sd->variant;
+		item = string_list_append(&s->change, p);
+		p += strlen(p) + 1;
+		if (*p)
+			d->rename_source = xstrdup(p);
+		item->util = d;
+
+		trace_printf_key(
+			&trace_deserialize,
+			"change: %d %d %d %d %o %o %o %d %d %s %s '%s' '%s'",
+			d->worktree_status,
+			d->index_status,
+			d->stagemask,
+			d->rename_score,
+			d->mode_head,
+			d->mode_index,
+			d->mode_worktree,
+			d->dirty_submodule,
+			d->new_submodule_commits,
+			oid_to_hex(&d->oid_head),
+			oid_to_hex(&d->oid_index),
+			item->string,
+			(d->rename_source ? d->rename_source : ""));
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int wt_deserialize_v1_untracked_items(struct wt_status *s,
+					     int fd,
+					     int count,
+					     enum deserialize_parse_strategy strategy)
+{
+	int line_len;
+	const char *line;
+	char *out = NULL;
+	int out_len = 0;
+
+	memset(&s->untracked, 0, sizeof(s->untracked));
+	s->untracked.strdup_strings = 1;
+
+	/*
+	 * <pathname>+
+	 * <flush>
+	 */
+	while ((line = my_packet_read_line(fd, &line_len))) {
+		if (strategy == DESERIALIZE_STRATEGY_AS_IS)
+			string_list_append(&s->untracked, line);
+		if (strategy == DESERIALIZE_STRATEGY_SKIP)
+			continue;
+		if (strategy == DESERIALIZE_STRATEGY_NORMAL) {
+
+			/* Only add "normal" entries to list */
+			if (out &&
+				check_path_contains(out, out_len, line, line_len)) {
+				continue;
+			}
+			else {
+				out = string_list_append(&s->untracked, line)->string;
+				out_len = line_len;
+			}
+		}
+		if (strategy == DESERIALIZE_STRATEGY_ALL) {
+			/* Only add "all" entries to list */
+			if (line[line_len - 1] != '/')
+				string_list_append(&s->untracked, line);
+		}
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int wt_deserialize_v1_ignored_items(struct wt_status *s,
+					   int fd,
+					   int count,
+					   enum deserialize_parse_strategy strategy)
+{
+	int line_len;
+	const char *line;
+
+	memset(&s->ignored, 0, sizeof(s->ignored));
+	s->ignored.strdup_strings = 1;
+
+	/*
+	 * <pathname>+
+	 * <flush>
+	 */
+	while ((line = my_packet_read_line(fd, &line_len))) {
+		if (strategy == DESERIALIZE_STRATEGY_AS_IS)
+			string_list_append(&s->ignored, line);
+		else
+			continue;
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int validate_untracked_files_arg(enum untracked_status_type cmd,
+					enum untracked_status_type des,
+					enum deserialize_parse_strategy *strategy)
+{
+	*strategy = DESERIALIZE_STRATEGY_AS_IS;
+
+	if (cmd == des) {
+		*strategy = DESERIALIZE_STRATEGY_AS_IS;
+	} else if (cmd == SHOW_NO_UNTRACKED_FILES) {
+		*strategy = DESERIALIZE_STRATEGY_SKIP;
+	} else if (des == SHOW_COMPLETE_UNTRACKED_FILES) {
+		if (cmd == SHOW_ALL_UNTRACKED_FILES)
+			*strategy = DESERIALIZE_STRATEGY_ALL;
+		else if (cmd == SHOW_NORMAL_UNTRACKED_FILES)
+			*strategy = DESERIALIZE_STRATEGY_NORMAL;
+	} else {
+		return DESERIALIZE_ERR;
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int validate_ignored_files_arg(enum show_ignored_type cmd,
+				      enum show_ignored_type des,
+				      enum deserialize_parse_strategy *strategy)
+{
+	*strategy = DESERIALIZE_STRATEGY_AS_IS;
+
+	if (cmd == SHOW_NO_IGNORED) {
+		*strategy = DESERIALIZE_STRATEGY_SKIP;
+	}
+	else if (cmd != des) {
+		return DESERIALIZE_ERR;
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int wt_deserialize_v1(const struct wt_status *cmd_s, struct wt_status *s, int fd)
+{
+	int line_len;
+	const char *line;
+	const char *arg;
+	int nr_changed = 0;
+	int nr_untracked = 0;
+	int nr_ignored = 0;
+
+	enum deserialize_parse_strategy ignored_strategy = DESERIALIZE_STRATEGY_AS_IS, untracked_strategy = DESERIALIZE_STRATEGY_AS_IS;
+
+	if (wt_deserialize_v1_header(s, fd) == DESERIALIZE_ERR)
+		return DESERIALIZE_ERR;
+
+	/*
+	 * We now have the header parsed. Look at the command args (as passed in), and see how to parse
+	 * the serialized data
+	*/
+	if (validate_untracked_files_arg(cmd_s->show_untracked_files, s->show_untracked_files, &untracked_strategy)) {
+		trace_printf_key(&trace_deserialize, "reject: show_untracked_file: command: %d, serialized : %d",
+				cmd_s->show_untracked_files,
+				s->show_untracked_files);
+		return DESERIALIZE_ERR;
+	}
+
+	if (validate_ignored_files_arg(cmd_s->show_ignored_mode, s->show_ignored_mode, &ignored_strategy)) {
+		trace_printf_key(&trace_deserialize, "reject: show_ignored_mode: command: %d, serialized: %d",
+				cmd_s->show_ignored_mode,
+				s->show_ignored_mode);
+		return DESERIALIZE_ERR;
+	}
+
+	/*
+	 * [<changed-header> [<changed-item>+] <flush>]
+	 * [<untracked-header> [<untracked-item>+] <flush>]
+	 * [<ignored-header> [<ignored-item>+] <flush>]
+	 */
+	while ((line = my_packet_read_line(fd, &line_len))) {
+		if (skip_prefix(line, "changed ", &arg)) {
+			nr_changed = (int)strtol(arg, NULL, 10);
+			if (wt_deserialize_v1_changed_items(s, fd, nr_changed)
+			    == DESERIALIZE_ERR)
+				return DESERIALIZE_ERR;
+			continue;
+		}
+		if (skip_prefix(line, "untracked ", &arg)) {
+			nr_untracked = (int)strtol(arg, NULL, 10);
+			if (wt_deserialize_v1_untracked_items(s, fd, nr_untracked, untracked_strategy)
+			    == DESERIALIZE_ERR)
+				return DESERIALIZE_ERR;
+			continue;
+		}
+		if (skip_prefix(line, "ignored ", &arg)) {
+			nr_ignored = (int)strtol(arg, NULL, 10);
+			if (wt_deserialize_v1_ignored_items(s, fd, nr_ignored, ignored_strategy)
+			    == DESERIALIZE_ERR)
+				return DESERIALIZE_ERR;
+			continue;
+		}
+		trace_printf_key(&trace_deserialize, "unexpected line '%s'", line);
+		return DESERIALIZE_ERR;
+	}
+
+	return DESERIALIZE_OK;
+}
+
+static int wt_deserialize_parse(const struct wt_status *cmd_s, struct wt_status *s, int fd)
+{
+	int line_len;
+	const char *line;
+	const char *arg;
+
+	memset(s, 0, sizeof(*s));
+
+	if ((line = my_packet_read_line(fd, &line_len)) &&
+	    (skip_prefix(line, "version ", &arg))) {
+		int version = (int)strtol(arg, NULL, 10);
+		if (version == 1)
+			return wt_deserialize_v1(cmd_s, s, fd);
+	}
+	trace_printf_key(&trace_deserialize, "missing/unsupported version");
+	return DESERIALIZE_ERR;
+}
+
+static inline int my_strcmp_null(const char *a, const char *b)
+{
+	const char *alt_a = (a) ? a : "";
+	const char *alt_b = (b) ? b : "";
+
+	return strcmp(alt_a, alt_b);
+}
+
+static int wt_deserialize_fd(const struct wt_status *cmd_s, struct wt_status *des_s, int fd)
+{
+	/*
+	 * Check the path spec on the current command
+	 */
+	if (cmd_s->pathspec.nr > 1) {
+		trace_printf_key(&trace_deserialize, "reject: multiple pathspecs");
+		return DESERIALIZE_ERR;
+	}
+
+	/*
+	 * If we have a pathspec, but it maches the root (e.g. no filtering)
+	 * then this is OK.
+	 */
+	if (cmd_s->pathspec.nr == 1 &&
+		my_strcmp_null(cmd_s->pathspec.items[0].match, "")) {
+		trace_printf_key(&trace_deserialize, "reject: pathspec");
+		return DESERIALIZE_ERR;
+	}
+
+	/*
+	 * Deserialize cached status
+	 */
+	if (wt_deserialize_parse(cmd_s, des_s, fd) == DESERIALIZE_ERR)
+		return DESERIALIZE_ERR;
+
+	/*
+	 * Compare fields in cmd_s with those observed in des_s and
+	 * complain if they are incompatible (such as different "-u"
+	 * or "--ignored" settings).
+	 */
+	if (cmd_s->is_initial != des_s->is_initial) {
+		trace_printf_key(&trace_deserialize, "reject: is_initial");
+		return DESERIALIZE_ERR;
+	}
+	if (my_strcmp_null(cmd_s->branch, des_s->branch)) {
+		trace_printf_key(&trace_deserialize, "reject: branch");
+		return DESERIALIZE_ERR;
+	}
+	if (my_strcmp_null(cmd_s->reference, des_s->reference)) {
+		trace_printf_key(&trace_deserialize, "reject: reference");
+		return DESERIALIZE_ERR;
+	}
+	/* verbose */
+	/* amend */
+	if (cmd_s->whence != des_s->whence) {
+		trace_printf_key(&trace_deserialize, "reject: whence");
+		return DESERIALIZE_ERR;
+	}
+	/* nowarn */
+	/* use_color */
+	/* no_gettext */
+	/* display_comment_prefix */
+	/* relative_paths */
+	/* submodule_summary */
+
+	/* show_ignored_files - already validated */
+	/* show_untrackes_files - already validated */
+
+	/*
+	 * Submodules are not supported by status serialization.
+	 * The status will not be serialized if it contains submodules,
+	 * and so this check is not needed.
+	 *
+	 * if (my_strcmp_null(cmd_s->ignore_submodule_arg, des_s->ignore_submodule_arg)) {
+	 *	trace_printf_key(&trace_deserialize, "reject: ignore_submodule_arg");
+	 * 	return DESERIALIZE_ERR;
+	 * }
+	 */
+
+	/* color_palette */
+	/* colopts */
+	/* null_termination */
+	/* commit_template */
+	/* show_branch */
+	/* show_stash */
+	/* hints */
+	if (cmd_s->detect_rename != des_s->detect_rename) {
+		trace_printf_key(&trace_deserialize, "reject: detect_rename");
+		return DESERIALIZE_ERR;
+	}
+	if (cmd_s->rename_score != des_s->rename_score) {
+		trace_printf_key(&trace_deserialize, "reject: rename_score");
+		return DESERIALIZE_ERR;
+	}
+	if (cmd_s->rename_limit != des_s->rename_limit) {
+		trace_printf_key(&trace_deserialize, "reject: rename_limit");
+		return DESERIALIZE_ERR;
+	}
+	/* status_format */
+	if (!oideq(&cmd_s->oid_commit, &des_s->oid_commit)) {
+		trace_printf_key(&trace_deserialize, "reject: sha1_commit");
+		return DESERIALIZE_ERR;
+	}
+
+	/*
+	 * Copy over display-related fields from the current command.
+	 */
+	des_s->verbose = cmd_s->verbose;
+	/* amend */
+	/* whence */
+	des_s->nowarn = cmd_s->nowarn;
+	des_s->use_color = cmd_s->use_color;
+	des_s->no_gettext = cmd_s->no_gettext;
+	des_s->display_comment_prefix = cmd_s->display_comment_prefix;
+	des_s->relative_paths = cmd_s->relative_paths;
+	des_s->submodule_summary = cmd_s->submodule_summary;
+	memcpy(des_s->color_palette, cmd_s->color_palette,
+	       sizeof(char)*WT_STATUS_MAXSLOT*COLOR_MAXLEN);
+	des_s->colopts = cmd_s->colopts;
+	des_s->null_termination = cmd_s->null_termination;
+	/* commit_template */
+	des_s->show_branch = cmd_s->show_branch;
+	des_s->show_stash = cmd_s->show_stash;
+	/* hints */
+	des_s->status_format = cmd_s->status_format;
+	des_s->fp = cmd_s->fp;
+	if (cmd_s->prefix && *cmd_s->prefix)
+		des_s->prefix = xstrdup(cmd_s->prefix);
+
+	return DESERIALIZE_OK;
+}
+
+
+/*
+ * Read raw serialized status data from the given file
+ *
+ * Verify that the args specified in the current command
+ * are compatible with the deserialized data (such as "-uno").
+ *
+ * Copy display-related fields from the current command
+ * into the deserialized data (so that the user can request
+ * long or short as they please).
+ */
+int wt_status_deserialize(const struct wt_status *cmd_s,
+			  const char *path)
+{
+	struct wt_status des_s;
+	int result;
+
+	if (path && *path && strcmp(path, "0")) {
+		int fd = xopen(path, O_RDONLY);
+		if (fd == -1) {
+			trace_printf_key(&trace_deserialize, "could not read '%s'", path);
+			return DESERIALIZE_ERR;
+		}
+		trace_printf_key(&trace_deserialize, "reading serialization file '%s'", path);
+		result = wt_deserialize_fd(cmd_s, &des_s, fd);
+		close(fd);
+	} else {
+		trace_printf_key(&trace_deserialize, "reading stdin");
+		result = wt_deserialize_fd(cmd_s, &des_s, 0);
+	}
+
+	if (result == DESERIALIZE_OK) {
+		wt_status_get_state(cmd_s->repo, &des_s.state, des_s.branch &&
+				    !strcmp(des_s.branch, "HEAD"));
+		wt_status_print(&des_s);
+	}
+
+	return result;
+}

wt-status-serialize.c

@@ -0,0 +1,213 @@
+#include "cache.h"
+#include "wt-status.h"
+#include "pkt-line.h"
+
+static struct trace_key trace_serialize = TRACE_KEY_INIT(SERIALIZE);
+
+/*
+ * Write V1 header fields.
+ */
+static void wt_serialize_v1_header(struct wt_status *s, int fd)
+{
+	/*
+	 * Write select fields from the current index to help
+	 * the deserializer recognize a stale data set.
+	 */
+	packet_write_fmt(fd, "index_mtime %d %d\n",
+			 s->repo->index->timestamp.sec,
+			 s->repo->index->timestamp.nsec);
+
+	/*
+	 * Write data from wt_status to qualify this status report.
+	 * That is, if this run specified "-uno", the consumer of
+	 * our serialization should know that.
+	 */
+	packet_write_fmt(fd, "is_initial %d\n", s->is_initial);
+	if (s->branch)
+		packet_write_fmt(fd, "branch %s\n", s->branch);
+	if (s->reference)
+		packet_write_fmt(fd, "reference %s\n", s->reference);
+	/* pathspec */
+	/* verbose */
+	/* amend */
+	packet_write_fmt(fd, "whence %d\n", s->whence);
+	/* nowarn */
+	/* use_color */
+	/* no_gettext */
+	/* display_comment_prefix */
+	/* relative_paths */
+	/* submodule_summary */
+	packet_write_fmt(fd, "show_ignored_mode %d\n", s->show_ignored_mode);
+	packet_write_fmt(fd, "show_untracked_files %d\n", s->show_untracked_files);
+	if (s->ignore_submodule_arg)
+		packet_write_fmt(fd, "ignore_submodule_arg %s\n", s->ignore_submodule_arg);
+	/* color_palette */
+	/* colopts */
+	/* null_termination */
+	/* commit_template */
+	/* show_branch */
+	/* show_stash */
+	packet_write_fmt(fd, "hints %d\n", s->hints);
+	packet_write_fmt(fd, "detect_rename %d\n", s->detect_rename);
+	packet_write_fmt(fd, "rename_score %d\n", s->rename_score);
+	packet_write_fmt(fd, "rename_limit %d\n", s->rename_limit);
+	/* status_format */
+	packet_write_fmt(fd, "sha1_commit %s\n", oid_to_hex(&s->oid_commit));
+	packet_write_fmt(fd, "committable %d\n", s->committable);
+	packet_write_fmt(fd, "workdir_dirty %d\n", s->workdir_dirty);
+	/* prefix */
+	packet_flush(fd);
+}
+
+/*
+ * Print changed/unmerged items.
+ * We write raw (not c-quoted) pathname(s).  The rename_source is only
+ * set when status computed a rename/copy.
+ *
+ * We ALWAYS write a final LF to the packet-line (for debugging)
+ * even though Linux pathnames allow LFs.
+ */
+static inline void wt_serialize_v1_changed(struct wt_status *s, int fd,
+					   struct string_list_item *item)
+{
+	struct wt_status_change_data *d = item->util;
+	struct wt_status_serialize_data sd;
+	char *begin;
+	char *end;
+	char *p;
+	int len_path, len_rename_source;
+
+	trace_printf_key(&trace_serialize,
+		"change: %d %d %d %d %o %o %o %d %d %s %s '%s' '%s'",
+		d->worktree_status,
+		d->index_status,
+		d->stagemask,
+		d->rename_score,
+		d->mode_head,
+		d->mode_index,
+		d->mode_worktree,
+		d->dirty_submodule,
+		d->new_submodule_commits,
+		oid_to_hex(&d->oid_head),
+		oid_to_hex(&d->oid_index),
+		item->string,
+		(d->rename_source ? d->rename_source : ""));
+
+	sd.fixed.worktree_status       = htonl(d->worktree_status);
+	sd.fixed.index_status          = htonl(d->index_status);
+	sd.fixed.stagemask             = htonl(d->stagemask);
+	sd.fixed.rename_score          = htonl(d->rename_score);
+	sd.fixed.mode_head             = htonl(d->mode_head);
+	sd.fixed.mode_index            = htonl(d->mode_index);
+	sd.fixed.mode_worktree         = htonl(d->mode_worktree);
+	sd.fixed.dirty_submodule       = htonl(d->dirty_submodule);
+	sd.fixed.new_submodule_commits = htonl(d->new_submodule_commits);
+	oidcpy(&sd.fixed.oid_head,  &d->oid_head);
+	oidcpy(&sd.fixed.oid_index, &d->oid_index);
+
+	begin = (char *)&sd;
+	end = begin + sizeof(sd);
+
+	p = sd.variant;
+
+	/*
+	 * Write <path> NUL [<rename_source>] NUL LF at the end of the buffer.
+	 */
+	len_path = strlen(item->string);
+	len_rename_source = d->rename_source ? strlen(d->rename_source) : 0;
+
+	/*
+	 * This is a bit of a hack, but I don't want to split the
+	 * status detail record across multiple pkt-lines.
+	 */
+	if (p + len_path + 1 + len_rename_source + 1 + 1 >= end)
+		BUG("path to long to serialize '%s'", item->string);
+
+	memcpy(p, item->string, len_path);
+	p += len_path;
+	*p++ = '\0';
+
+	if (len_rename_source) {
+		memcpy(p, d->rename_source, len_rename_source);
+		p += len_rename_source;
+	}
+	*p++ = '\0';
+	*p++ = '\n';
+
+	if (packet_write_gently(fd, begin, (p - begin)))
+		BUG("cannot serialize '%s'", item->string);
+}
+
+/*
+ * Write raw (not c-quoted) pathname for an untracked item.
+ * We ALWAYS write a final LF to the packet-line (for debugging)
+ * even though Linux pathnames allows LFs.  That is, deserialization
+ * should use the packet-line length and omit the final LF.
+ */
+static inline void wt_serialize_v1_untracked(struct wt_status *s, int fd,
+					     struct string_list_item *item)
+{
+	packet_write_fmt(fd, "%s\n", item->string);
+}
+
+/*
+ * Write raw (not c-quoted) pathname for an ignored item.
+ * We ALWAYS write a final LF to the packet-line (for debugging)
+ * even though Linux pathnames allows LFs.
+ */
+static inline void wt_serialize_v1_ignored(struct wt_status *s, int fd,
+					   struct string_list_item *item)
+{
+	packet_write_fmt(fd, "%s\n", item->string);
+}
+
+/*
+ * Serialize the list of changes to stdout.  The goal of this
+ * is to just serialize the key fields in wt_status so that a
+ * later command can rebuilt it and do the printing.
+ *
+ * We DO NOT include the contents of wt_status_state NOR
+ * current branch info.  This info easily gets stale and
+ * is relatively quick for the status consumer to compute
+ * as necessary.
+ */
+void wt_status_serialize_v1(struct wt_status *s)
+{
+	int fd = 1; /* we always write to stdout */
+	struct string_list_item *iter;
+	int k;
+
+	/*
+	 * version header must be first line.
+	 */
+	packet_write_fmt(fd, "version 1\n");
+
+	wt_serialize_v1_header(s, fd);
+
+	if (s->change.nr > 0) {
+		packet_write_fmt(fd, "changed %"PRIuMAX"\n", (uintmax_t)s->change.nr);
+		for (k = 0; k < s->change.nr; k++) {
+			iter = &(s->change.items[k]);
+			wt_serialize_v1_changed(s, fd, iter);
+		}
+		packet_flush(fd);
+	}
+
+	if (s->untracked.nr > 0) {
+		packet_write_fmt(fd, "untracked %"PRIuMAX"\n", (uintmax_t)s->untracked.nr);
+		for (k = 0; k < s->untracked.nr; k++) {
+			iter = &(s->untracked.items[k]);
+			wt_serialize_v1_untracked(s, fd, iter);
+		}
+		packet_flush(fd);
+	}
+
+	if (s->ignored.nr > 0) {
+		packet_write_fmt(fd, "ignored %"PRIuMAX"\n", (uintmax_t)s->ignored.nr);
+		for (k = 0; k < s->ignored.nr; k++) {
+			iter = &(s->ignored.items[k]);
+			wt_serialize_v1_ignored(s, fd, iter);
+		}
+		packet_flush(fd);
+	}
+}

wt-status.c

@@ -768,6 +768,9 @@ static void wt_status_collect_untracked(struct wt_status *s)
 	if (s->show_untracked_files != SHOW_ALL_UNTRACKED_FILES)
 		dir.flags |=
 			DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;
+	if (s->show_untracked_files == SHOW_COMPLETE_UNTRACKED_FILES)
+		dir.flags |= DIR_KEEP_UNTRACKED_CONTENTS;
+
 	if (s->show_ignored_mode) {
 		dir.flags |= DIR_SHOW_IGNORED_TOO;
 
@@ -2533,6 +2536,9 @@ void wt_status_print(struct wt_status *s)
 	case STATUS_FORMAT_LONG:
 		wt_longstatus_print(s);
 		break;
+	case STATUS_FORMAT_SERIALIZE_V1:
+		wt_status_serialize_v1(s);
+		break;
 	}
 
 	trace2_region_leave("status", "print", s->repo);

wt-status.h

@@ -4,6 +4,7 @@
 #include "string-list.h"
 #include "color.h"
 #include "pathspec.h"
+#include "pkt-line.h"
 #include "remote.h"
 
 struct repository;
@@ -25,7 +26,8 @@ enum color_wt_status {
 enum untracked_status_type {
 	SHOW_NO_UNTRACKED_FILES,
 	SHOW_NORMAL_UNTRACKED_FILES,
-	SHOW_ALL_UNTRACKED_FILES
+	SHOW_ALL_UNTRACKED_FILES,
+	SHOW_COMPLETE_UNTRACKED_FILES,
 };
 
 enum show_ignored_type {
@@ -73,6 +75,7 @@ enum wt_status_format {
 	STATUS_FORMAT_SHORT,
 	STATUS_FORMAT_PORCELAIN,
 	STATUS_FORMAT_PORCELAIN_V2,
+	STATUS_FORMAT_SERIALIZE_V1,
 
 	STATUS_FORMAT_UNSPECIFIED
 };
@@ -182,4 +185,51 @@ int require_clean_work_tree(struct repository *repo,
 			    int ignore_submodules,
 			    int gently);
 
+#define DESERIALIZE_OK  0
+#define DESERIALIZE_ERR 1
+
+struct wt_status_serialize_data_fixed
+{
+	uint32_t worktree_status;
+	uint32_t index_status;
+	uint32_t stagemask;
+	uint32_t rename_score;
+	uint32_t mode_head;
+	uint32_t mode_index;
+	uint32_t mode_worktree;
+	uint32_t dirty_submodule;
+	uint32_t new_submodule_commits;
+	struct object_id oid_head;
+	struct object_id oid_index;
+};
+
+/*
+ * Consume the maximum amount of data possible in a
+ * packet-line record.  This is overkill because we
+ * have at most 2 relative pathnames, but means we
+ * don't need to allocate a variable length structure.
+ */
+struct wt_status_serialize_data
+{
+	struct wt_status_serialize_data_fixed fixed;
+	char variant[LARGE_PACKET_DATA_MAX
+		     - sizeof(struct wt_status_serialize_data_fixed)];
+};
+
+/*
+ * Serialize computed status scan results using "version 1" format
+ * to the given file.
+ */
+void wt_status_serialize_v1(struct wt_status *s);
+
+/*
+ * Deserialize existing status results from the given file and
+ * populate a (new) "struct wt_status".  Use the contents of "cmd_s"
+ * (computed from the command line arguments) to verify that the
+ * cached data is compatible and overlay various display-related
+ * fields.
+ */
+int wt_status_deserialize(const struct wt_status *cmd_s,
+			  const char *path);
+
 #endif /* STATUS_H */