Measure code coverage between tests
Go has code coverage tooling for test mode, which temporarily rewrites
the source code to insert annotations which will activate during the
test run and track progress of executed code. Then, upon process
completion, that information is dumped into a coverage report.
We can't use this approach for hub, at least not without substantial
changes. First of all, hub's test coverage is mostly "from the outside",
utilizing Cucumber to invoke the binary with different arguments and
inspect the outputs and result. There are some tests in go, but they are
minimal compared to the cukes.
Second, hub frequently aborts the process on errors via `os.Exit(1)`,
and those scenarios need to be tested too. However, if the process exits
prematurely, the code coverage report will never be generated.
To work around this, I first used the go tool that annotates the source:
go tool cover -mode=set -var=LiveCoverage myfile.go
This injects `LiveCoverage.Count[pos] = 1` lines at appropriate places
all over the source code, and generates a mapping of line/column
positions in the original source.
Then I rewrite those lines to become a method invocation:
coverage.Record(LiveCoverage, pos)
The new `Record` method will immediately append the information to a
code coverage report file as soon as it's invoked. This ensures that
there is coverage information even if the process gets aborted.
This approach works the same for go tests as well as for cukes. They all
append to the same file. Finally, the rest of Go tooling is used to
generate an HTML report of code coverage:
go tool cover -html=cover.out
2016-09-12 22:18:52 +03:00
|
|
|
#!/bin/bash
|
|
|
|
set -e
|
|
|
|
|
|
|
|
source_files() {
|
|
|
|
script/build files | grep -vE '^\./(coverage|fixtures)/'
|
|
|
|
}
|
|
|
|
|
|
|
|
prepare() {
|
2018-06-07 17:07:20 +03:00
|
|
|
local changed_files="$(source_files | xargs git diff --name-only --)"
|
|
|
|
if [ -n "$changed_files" ]; then
|
|
|
|
echo "Aborted: please commit the following files before continuing" >&2
|
|
|
|
cat <<<"$changed_files" >&2
|
Measure code coverage between tests
Go has code coverage tooling for test mode, which temporarily rewrites
the source code to insert annotations which will activate during the
test run and track progress of executed code. Then, upon process
completion, that information is dumped into a coverage report.
We can't use this approach for hub, at least not without substantial
changes. First of all, hub's test coverage is mostly "from the outside",
utilizing Cucumber to invoke the binary with different arguments and
inspect the outputs and result. There are some tests in go, but they are
minimal compared to the cukes.
Second, hub frequently aborts the process on errors via `os.Exit(1)`,
and those scenarios need to be tested too. However, if the process exits
prematurely, the code coverage report will never be generated.
To work around this, I first used the go tool that annotates the source:
go tool cover -mode=set -var=LiveCoverage myfile.go
This injects `LiveCoverage.Count[pos] = 1` lines at appropriate places
all over the source code, and generates a mapping of line/column
positions in the original source.
Then I rewrite those lines to become a method invocation:
coverage.Record(LiveCoverage, pos)
The new `Record` method will immediately append the information to a
code coverage report file as soon as it's invoked. This ensures that
there is coverage information even if the process gets aborted.
This approach works the same for go tests as well as for cukes. They all
append to the same file. Finally, the rest of Go tooling is used to
generate an HTML report of code coverage:
go tool cover -html=cover.out
2016-09-12 22:18:52 +03:00
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
|
|
|
|
local n=0
|
|
|
|
for f in $(source_files); do
|
|
|
|
go tool cover -mode=set -var="LiveCoverage$((++n))" "$f" > "$f"~
|
|
|
|
sed -E '
|
|
|
|
/^package /a\
|
|
|
|
import "github.com/github/hub/coverage"
|
2018-06-07 17:59:43 +03:00
|
|
|
s/(LiveCoverage[0-9]+)\.Count\[([0-9]+)\][^;]+/coverage.Record(\1, \2)/g
|
Measure code coverage between tests
Go has code coverage tooling for test mode, which temporarily rewrites
the source code to insert annotations which will activate during the
test run and track progress of executed code. Then, upon process
completion, that information is dumped into a coverage report.
We can't use this approach for hub, at least not without substantial
changes. First of all, hub's test coverage is mostly "from the outside",
utilizing Cucumber to invoke the binary with different arguments and
inspect the outputs and result. There are some tests in go, but they are
minimal compared to the cukes.
Second, hub frequently aborts the process on errors via `os.Exit(1)`,
and those scenarios need to be tested too. However, if the process exits
prematurely, the code coverage report will never be generated.
To work around this, I first used the go tool that annotates the source:
go tool cover -mode=set -var=LiveCoverage myfile.go
This injects `LiveCoverage.Count[pos] = 1` lines at appropriate places
all over the source code, and generates a mapping of line/column
positions in the original source.
Then I rewrite those lines to become a method invocation:
coverage.Record(LiveCoverage, pos)
The new `Record` method will immediately append the information to a
code coverage report file as soon as it's invoked. This ensures that
there is coverage information even if the process gets aborted.
This approach works the same for go tests as well as for cukes. They all
append to the same file. Finally, the rest of Go tooling is used to
generate an HTML report of code coverage:
go tool cover -html=cover.out
2016-09-12 22:18:52 +03:00
|
|
|
' < "$f"~ > "$f"
|
|
|
|
rm "$f"~
|
|
|
|
done
|
|
|
|
|
|
|
|
rm -rf "$HUB_COVERAGE"
|
|
|
|
mkdir -p "${HUB_COVERAGE%/*}"
|
|
|
|
}
|
|
|
|
|
|
|
|
generate() {
|
|
|
|
source_files | xargs git checkout --
|
|
|
|
|
|
|
|
echo 'mode: count' > "$HUB_COVERAGE"~
|
|
|
|
sed -E 's!^.+/(github.com/github/hub/)!\1!' "$HUB_COVERAGE" | awk '
|
|
|
|
{ a[substr($0, 0, length()-2)] += $(NF) }
|
|
|
|
END { for (k in a) print k, a[k] }
|
|
|
|
' >> "$HUB_COVERAGE"~
|
|
|
|
|
|
|
|
go tool cover -func="$HUB_COVERAGE"~ > "${HUB_COVERAGE%.out}.func"
|
|
|
|
if [ -z "$CI" ]; then
|
|
|
|
go tool cover -html="$HUB_COVERAGE"~ -o "${HUB_COVERAGE%.out}.html"
|
|
|
|
fi
|
|
|
|
|
|
|
|
awk '/^total:/ { print $(NF) }' "${HUB_COVERAGE%.out}.func"
|
|
|
|
}
|
|
|
|
|
2018-06-09 19:10:52 +03:00
|
|
|
summarize() {
|
|
|
|
local total_coverage
|
|
|
|
local min_coverage="${1?}"
|
|
|
|
total_coverage="$(generate)"
|
|
|
|
echo "Code coverage: $total_coverage"
|
|
|
|
local result="$(bc <<<"${total_coverage%\%} < $min_coverage")"
|
|
|
|
if [ "$result" -eq 1 ]; then
|
|
|
|
echo "Error: coverage dropped below the minimum treshold of ${min_coverage}%!"
|
|
|
|
if [ -n "$CI" ]; then
|
|
|
|
html_result="${HUB_COVERAGE%.out}.html"
|
|
|
|
html_result="${html_result#$PWD/}"
|
|
|
|
printf 'Please run `script/test --coverage` locally and open `%s` to analyze the results.\n' "$html_result"
|
|
|
|
fi
|
|
|
|
return 1
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
|
|
|
cmd="${1?}"
|
|
|
|
shift 1
|
|
|
|
|
|
|
|
case "$cmd" in
|
|
|
|
prepare | generate | summarize )
|
|
|
|
"$cmd" "$@"
|
Measure code coverage between tests
Go has code coverage tooling for test mode, which temporarily rewrites
the source code to insert annotations which will activate during the
test run and track progress of executed code. Then, upon process
completion, that information is dumped into a coverage report.
We can't use this approach for hub, at least not without substantial
changes. First of all, hub's test coverage is mostly "from the outside",
utilizing Cucumber to invoke the binary with different arguments and
inspect the outputs and result. There are some tests in go, but they are
minimal compared to the cukes.
Second, hub frequently aborts the process on errors via `os.Exit(1)`,
and those scenarios need to be tested too. However, if the process exits
prematurely, the code coverage report will never be generated.
To work around this, I first used the go tool that annotates the source:
go tool cover -mode=set -var=LiveCoverage myfile.go
This injects `LiveCoverage.Count[pos] = 1` lines at appropriate places
all over the source code, and generates a mapping of line/column
positions in the original source.
Then I rewrite those lines to become a method invocation:
coverage.Record(LiveCoverage, pos)
The new `Record` method will immediately append the information to a
code coverage report file as soon as it's invoked. This ensures that
there is coverage information even if the process gets aborted.
This approach works the same for go tests as well as for cukes. They all
append to the same file. Finally, the rest of Go tooling is used to
generate an HTML report of code coverage:
go tool cover -html=cover.out
2016-09-12 22:18:52 +03:00
|
|
|
;;
|
|
|
|
* )
|
|
|
|
exit 1
|
|
|
|
;;
|
|
|
|
esac
|