Merge branch 'golang:master' into patch-2
This commit is contained in:
Ankit Verma
GitHub
Коммит
8914d0f4f3
|
|
@ -0,0 +1,109 @@
|
|||
---
|
||||
title: Go 1.20 is released!
|
||||
date: 2023-02-01
|
||||
by:
|
||||
- Robert Griesemer, on behalf of the Go team
|
||||
summary: Go 1.20 brings PGO, faster builds, and various tool, language, and library improvements.
|
||||
---
|
||||
|
||||
Today the Go team is thrilled to release Go 1.20,
|
||||
which you can get by visiting the [download page](/dl/).
|
||||
|
||||
Go 1.20 benefited from an extended development phase,
|
||||
made possible by earlier broad testing and improved overall stability
|
||||
of the code base.
|
||||
|
||||
We're particularly excited to launch a preview of [profile-guided optimization](/doc/pgo)
|
||||
(PGO), which enables the compiler to perform application- and
|
||||
workload-specific optimizations based on run-time profile information.
|
||||
Providing a profile to `go build` enables the compiler to speed up typical
|
||||
applications by around 3–4%, and we expect future releases to benefit even more
|
||||
from PGO.
|
||||
Since this is a preview release of PGO support, we encourage folks to try it out,
|
||||
but there are still rough edges which may preclude production use.
|
||||
|
||||
Go 1.20 also includes a handful of language changes,
|
||||
many improvements to tooling and the library, and better overall performance.
|
||||
|
||||
## Language changes
|
||||
|
||||
- The predeclared [`comparable`](/ref/spec#Type_constraints) constraint is
|
||||
now also [satisfied](/ref/spec#Satisfying_a_type_constraint) by
|
||||
ordinary [comparable types](/ref/spec#Comparison_operators), such as interfaces,
|
||||
which will simplify generic code.
|
||||
- The functions `SliceData`, `String`, and `StringData` have been added
|
||||
to package [`unsafe`](/ref/spec#Package_unsafe). They complete the set of functions
|
||||
for implementation-independent slice and string manipulation.
|
||||
- Go's type conversion rules have been extended to permit direct conversion
|
||||
[from a slice to an array](/ref/spec#Conversions_from_slice_to_array_or_array_pointer).
|
||||
- The language specification now defines the exact order in which array elements
|
||||
and struct fields are [compared](/ref/spec#Comparison_operators).
|
||||
This clarifies what happens in case of panics during comparisons.
|
||||
|
||||
## Tool improvements
|
||||
|
||||
- The [`cover` tool](/testing/coverage) now can collect coverage profiles of whole programs,
|
||||
not just of unit tests.
|
||||
- The [`go` tool](/cmd/go) no longer relies on pre-compiled standard library
|
||||
package archives in the `$GOROOT/pkg` directory, and they are no longer
|
||||
shipped with the distribution, resulting in smaller downloads.
|
||||
Instead, packages in the standard library are built as needed and cached
|
||||
in the build cache, like other packages.
|
||||
- The implementation of `go test -json` has been improved
|
||||
to make it more robust in the presence of stray writes to `stdout`.
|
||||
- The `go build`, `go install`, and other build-related
|
||||
commands now accept a `-pgo` flag enabling profile-guided optimizations
|
||||
as well as a `-cover` flag for whole-program coverage analysis.
|
||||
- The `go` command now disables `cgo` by default on systems without a C toolchain.
|
||||
Consequently, when Go is installed on a system without a C compiler, it will
|
||||
now use pure Go builds for packages in the standard library that optionally use cgo,
|
||||
instead of using pre-distributed package archives (which have been removed,
|
||||
as noted above).
|
||||
- The [`vet` tool](/cmd/vet) reports more loop variable reference mistakes
|
||||
that may occur in tests running in parallel.
|
||||
|
||||
## Standard library additions
|
||||
|
||||
- The new [`crypto/ecdh`](/pkg/crypto/ecdh) package provides explicit support for
|
||||
Elliptic Curve Diffie-Hellman key exchanges over NIST curves and Curve25519.
|
||||
- The new function [`errors.Join`](/pkg/errors#Join) returns an error wrapping a list of errors
|
||||
which may be obtained again if the error type implements the `Unwrap() []error` method.
|
||||
- The new [`http.ResponseController`](/pkg/net/http#ResponseController) type
|
||||
provides access to extended per-request functionality not handled by the
|
||||
[`http.ResponseWriter`](/pkg/net/http#ResponseWriter) interface.
|
||||
- The [`httputil.ReverseProxy`](/pkg/net/http/httputil#ReverseProxy)
|
||||
forwarding proxy includes a new `Rewrite` hook function, superseding the
|
||||
previous `Director` hook.
|
||||
- The new [`context.WithCancelCause`](/pkg/context#WithCancelCause) function
|
||||
provides a way to cancel a context with a given error.
|
||||
That error can be retrieved by calling the new
|
||||
[`context.Cause`](/pkg/context#Cause) function.
|
||||
- The new [`os/exec.Cmd`](/pkg/os/exec#Cmd) fields [`Cancel`](/pkg/os/exec#Cmd.Cancel)
|
||||
and [`WaitDelay`](/pkg/os/exec#Cmd.WaitDelay) specify the behavior of the
|
||||
`Cmd` when its associated `Context` is canceled or its process exits.
|
||||
|
||||
## Improved performance
|
||||
|
||||
- Compiler and garbage collector improvements have reduced memory overhead
|
||||
and improved overall CPU performance by up to 2%.
|
||||
- Work specifically targeting
|
||||
compilation times led to build improvements by up to 10%.
|
||||
This brings build speeds back in line with Go 1.17.
|
||||
|
||||
When [building a Go release from source](/doc/install/source),
|
||||
Go 1.20 requires a Go 1.17.13 or newer release.
|
||||
In the future, we plan to move the bootstrap toolchain forward approximately
|
||||
once a year.
|
||||
Also, starting with Go 1.21, some older operating systems will no longer be supported:
|
||||
this includes Windows 7, 8, Server 2008 and Server 2012,
|
||||
macOS 10.13 High Sierra, and 10.14 Mojave.
|
||||
On the other hand, Go 1.20 adds experimental support for FreeBSD on RISC-V.
|
||||
|
||||
For a complete and more detailed list of all changes see the [full release notes](/doc/go1.20).
|
||||
|
||||
Thanks to everyone who contributed to this release by writing code, filing bugs,
|
||||
sharing feedback, and testing the release candidates. Your efforts helped
|
||||
to ensure that Go 1.20 is as stable as possible.
|
||||
As always, if you notice any problems, please [file an issue](/issue/new).
|
||||
|
||||
Enjoy Go 1.20!
|
||||
|
|
@ -0,0 +1,218 @@
|
|||
---
|
||||
title: Profile-guided optimization preview
|
||||
date: 2023-02-08
|
||||
by:
|
||||
- Michael Pratt
|
||||
summary: Introduction to profile-guided optimization, available as a preview in Go 1.20.
|
||||
---
|
||||
|
||||
When you build a Go binary, the Go compiler performs optimizations to try to generate the best performing binary it can.
|
||||
For example, constant propagation can evaluate constant expressions at compile time, avoiding runtime evaluation cost.
|
||||
Escape analysis avoids heap allocations for locally-scoped objects, avoiding GC overheads.
|
||||
Inlining copies the body of simple functions into callers, often enabling further optimization in the caller (such as additional constant propagation or better escape analysis).
|
||||
|
||||
Go improves optimizations from release to release, but this is not always an easy task.
|
||||
Some optimizations are tunable, but the compiler can't just "turn it up to 11" on every function because overly aggressive optimizations can actually hurt performance or cause excessive build times.
|
||||
Other optimizations require the compiler to make a judgment call about what the "common" and "uncommon" paths in a function are.
|
||||
The compiler must make a best guess based on static heuristics because it can't know which cases will be common at run time.
|
||||
|
||||
Or can it?
|
||||
|
||||
With no definitive information about how the code is used in a production environment, the compiler can operate only on the source code of packages.
|
||||
But we do have a tool to evaluate production behavior: [profiling](https://go.dev/doc/diagnostics#profiling).
|
||||
If we provide a profile to the compiler, it can make more informed decisions: more aggressively optimizing the most frequently used functions, or more accurately selecting common cases.
|
||||
|
||||
Using profiles of application behavior for compiler optimization is known as _Profile-Guided Optimization (PGO)_ (also known as Feedback-Directed Optimization (FDO)).
|
||||
|
||||
Go 1.20 includes initial support for PGO as a preview.
|
||||
See the [profile-guided optimization user guide](https://go.dev/doc/pgo) for complete documentation.
|
||||
There are still some rough edges that may prevent production use, but we would love for you to try it out and [send us any feedback or issues you encounter](https://go.dev/issue/new).
|
||||
|
||||
## Example
|
||||
|
||||
Let's build a service that converts Markdown to HTML: users upload Markdown source to `/render`, which returns the HTML conversion.
|
||||
We can use [`gitlab.com/golang-commonmark/markdown`](https://pkg.go.dev/gitlab.com/golang-commonmark/markdown) to implement this easily.
|
||||
|
||||
### Set up
|
||||
|
||||
```
|
||||
$ go mod init example.com/markdown
|
||||
$ go get gitlab.com/golang-commonmark/markdown@bf3e522c626a
|
||||
```
|
||||
|
||||
In `main.go`:
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
_ "net/http/pprof"
|
||||
|
||||
"gitlab.com/golang-commonmark/markdown"
|
||||
)
|
||||
|
||||
func render(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != "POST" {
|
||||
http.Error(w, "Only POST allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
|
||||
src, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
log.Printf("error reading body: %v", err)
|
||||
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
md := markdown.New(
|
||||
markdown.XHTMLOutput(true),
|
||||
markdown.Typographer(true),
|
||||
markdown.Linkify(true),
|
||||
markdown.Tables(true),
|
||||
)
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := md.Render(&buf, src); err != nil {
|
||||
log.Printf("error converting markdown: %v", err)
|
||||
http.Error(w, "Malformed markdown", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := io.Copy(w, &buf); err != nil {
|
||||
log.Printf("error writing response: %v", err)
|
||||
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
http.HandleFunc("/render", render)
|
||||
log.Printf("Serving on port 8080...")
|
||||
log.Fatal(http.ListenAndServe(":8080", nil))
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
Build and run the server:
|
||||
|
||||
```
|
||||
$ go build -o markdown.nopgo.exe
|
||||
$ ./markdown.nopgo.exe
|
||||
2023/01/19 14:26:24 Serving on port 8080...
|
||||
```
|
||||
|
||||
Let's try sending some Markdown from another terminal.
|
||||
We can use the README from the Go project as a sample document:
|
||||
|
||||
```
|
||||
$ curl -o README.md -L "https://raw.githubusercontent.com/golang/go/c16c2c49e2fa98ae551fc6335215fadd62d33542/README.md"
|
||||
$ curl --data-binary @README.md http://localhost:8080/render
|
||||
<h1>The Go Programming Language</h1>
|
||||
<p>Go is an open source programming language that makes it easy to build simple,
|
||||
reliable, and efficient software.</p>
|
||||
...
|
||||
```
|
||||
|
||||
### Profiling
|
||||
|
||||
Now that we have a working service, let's collect a profile and rebuild with PGO to see if we get better performance.
|
||||
|
||||
In `main.go`, we imported [net/http/pprof](https://pkg.go.dev/net/http/pprof) which automatically adds a `/debug/pprof/profile` endpoint to the server for fetching a CPU profile.
|
||||
|
||||
Normally you want to collect a profile from your production environment so that the compiler gets a representative view of behavior in production.
|
||||
Since this example doesn't have a "production" environment, we will create a simple program to generate load while we collect a profile.
|
||||
Copy the source of [this program](https://go.dev/play/p/yYH0kfsZcpL) to `load/main.go` and start the load generator (make sure the server is still running!).
|
||||
|
||||
```
|
||||
$ go run example.com/markdown/load
|
||||
```
|
||||
|
||||
While that is running, download a profile from the server:
|
||||
|
||||
```
|
||||
$ curl -o cpu.pprof "http://localhost:8080/debug/pprof/profile?seconds=30"
|
||||
```
|
||||
|
||||
Once this completes, kill the load generator and the server.
|
||||
|
||||
### Using the profile
|
||||
|
||||
We can ask the Go toolchain to build with PGO using the `-pgo` flag to `go build`.
|
||||
`-pgo` takes either the path to the profile to use, or `auto`, which will use the `default.pgo` file in the main package directory.
|
||||
|
||||
We recommending commiting `default.pgo` profiles to your repository.
|
||||
Storing profiles alongside your source code ensures that users automatically have access to the profile simply by fetching the repository (either via the version control system, or via `go get`) and that builds remain reproducible.
|
||||
In Go 1.20, `-pgo=off` is the default, so users still need to add `-pgo=auto`, but a future version of Go is expected to change the default to `-pgo=auto`, automatically giving anyone that builds the binary the benefit of PGO.
|
||||
|
||||
Let's build:
|
||||
|
||||
```
|
||||
$ mv cpu.pprof default.pgo
|
||||
$ go build -pgo=auto -o markdown.withpgo.exe
|
||||
```
|
||||
|
||||
### Evaluation
|
||||
|
||||
We will use a Go benchmark version of the load generator to evaluate the effect of PGO on performance.
|
||||
Copy [this benchmark](https://go.dev/play/p/6FnQmHfRjbh) to `load/bench_test.go`.
|
||||
|
||||
First, we will benchmark the server without PGO. Start that server:
|
||||
|
||||
```
|
||||
$ ./markdown.nopgo.exe
|
||||
```
|
||||
|
||||
While that is running, run several benchmark iterations:
|
||||
|
||||
```
|
||||
$ go test example.com/markdown/load -bench=. -count=20 -source ../README.md > nopgo.txt
|
||||
```
|
||||
|
||||
Once that completes, kill the original server and start the version with PGO:
|
||||
|
||||
```
|
||||
$ ./markdown.withpgo.exe
|
||||
```
|
||||
|
||||
While that is running, run several benchmark iterations:
|
||||
|
||||
```
|
||||
$ go test example.com/markdown/load -bench=. -count=20 -source ../README.md > withpgo.txt
|
||||
```
|
||||
|
||||
Once that completes, let's compare the results:
|
||||
|
||||
```
|
||||
$ go install golang.org/x/perf/cmd/benchstat@latest
|
||||
$ benchstat nopgo.txt withpgo.txt
|
||||
goos: linux
|
||||
goarch: amd64
|
||||
pkg: example.com/markdown/load
|
||||
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
|
||||
│ nopgo.txt │ withpgo.txt │
|
||||
│ sec/op │ sec/op vs base │
|
||||
Load-12 393.8µ ± 1% 383.6µ ± 1% -2.59% (p=0.000 n=20)
|
||||
```
|
||||
|
||||
The new version is around 2.6% faster!
|
||||
In Go 1.20, workloads typically get between 2% and 4% CPU usage improvements from enabling PGO.
|
||||
Profiles contain a wealth of information about application behavior and Go 1.20 just begins to crack the surface by using this information for inlining.
|
||||
Future releases will continue improving performance as more parts of the compiler take advantage of PGO.
|
||||
|
||||
## Next steps
|
||||
|
||||
In this example, after collecting a profile, we rebuilt our server using the exact same source code used in the original build.
|
||||
In a real-world scenario, there is always ongoing development.
|
||||
So we may collect a profile from production, which is running last week's code, and use it to build with today's source code.
|
||||
That is perfectly fine!
|
||||
PGO in Go can handle minor changes to source code without issue.
|
||||
|
||||
For much more information on using PGO, best practices and caveats to be aware of, please see the [profile-guided optimization user guide](https://go.dev/doc/pgo).
|
||||
|
||||
Please send us your feedback!
|
||||
PGO is still in preview and we'd love to hear about anything that is difficult to use, doesn't work correctly, etc.
|
||||
Please file issues at https://go.dev/issue/new.
|
||||
|
|
@ -131,8 +131,8 @@ class DownloadsController {
|
|||
|
||||
// get version number.
|
||||
parseVersionNumber(string) {
|
||||
const rx = /(\d+\.)(\d+\.)(\d+)/g;
|
||||
const matches = rx.exec(string)
|
||||
const rx = /(\d+\.)(\d+)(\.\d+)?/g;
|
||||
const matches = rx.exec(string);
|
||||
if (matches?.[0]) {
|
||||
return matches[0];
|
||||
} else {
|
||||
|
|
|
|||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
|
|
@ -130,6 +130,11 @@ Main documentation page for Go fuzzing.
|
|||
Main documentation page for coverage testing of Go applications.
|
||||
</p>
|
||||
|
||||
<h3 id="pgo"><a href="/doc/pgo">Profile-guided optimization</a></h3>
|
||||
<p>
|
||||
Main documentation page for profile-guided optimization (PGO) of Go applications.
|
||||
</p>
|
||||
|
||||
<h3 id="data-access">Accessing databases</h3>
|
||||
|
||||
<h4 id="data-access-tutorial"><a href="/doc/tutorial/database-access">Tutorial: Accessing a relational database</a></h4>
|
||||
|
|
|
|||
|
|
@ -523,8 +523,8 @@ with either a comma or a pipe.
|
|||
|
||||
Go modules are frequently developed and distributed on version control servers
|
||||
and module proxies that aren’t available on the public internet. You can set the
|
||||
`GOPRIVATE` environment variables. You can set the `GOPRIVATE` environment variable
|
||||
to configure the `go` command to download and build modules from private sources.
|
||||
`GOPRIVATE` environment variable to configure the `go` command
|
||||
to download and build modules from private sources.
|
||||
Then the go command can download and build modules from private sources.
|
||||
|
||||
The `GOPRIVATE` or `GONOPROXY` environment variables may be set to lists of glob
|
||||
|
|
|
|||
|
|
@ -0,0 +1,271 @@
|
|||
---
|
||||
title: Profile-guided optimization
|
||||
layout: article
|
||||
---
|
||||
|
||||
Beginning in Go 1.20, the Go compiler supports profile-guided optimization (PGO) to further optimize builds.
|
||||
|
||||
_Note: As of Go 1.20, PGO is in public preview.
|
||||
We encourage folks to try it out, but there are still rough edges (noted below) which may preclude production use.
|
||||
Please report issues you experience to https://go.dev/issue/new. We expect PGO to be generally available in a future release._
|
||||
|
||||
Table of Contents:
|
||||
|
||||
[Overview](#overview)\
|
||||
[Collecting profiles](#collecting-profiles)\
|
||||
[Building with PGO](#building)\
|
||||
[Notes](#notes)\
|
||||
[Frequently Asked Questions](#faq)\
|
||||
[Appendix: alternative profile sources](#alternative-sources)
|
||||
|
||||
# Overview {#overview}
|
||||
|
||||
Profile-guided optimization (PGO), also known as feedback-directed optimization (FDO), is a compiler optimization technique that feeds information (a profile) from representative runs of the application back into to the compiler for the next build of the application, which uses that information to make more informed optimization decisions.
|
||||
For example, the compiler may decide to more aggressively inline functions which the profile indicates are called frequently.
|
||||
|
||||
In Go, the compiler uses CPU pprof profiles as the input profile, such as from [runtime/pprof](https://pkg.go.dev/runtime/pprof) or [net/http/pprof](https://pkg.go.dev/net/http/pprof).
|
||||
|
||||
As of Go 1.20, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-4%.
|
||||
We expect performance gains to generally increase over time as additional optimizations take advantage of PGO in future versions of Go.
|
||||
|
||||
|
||||
# Collecting profiles {#collecting-profiles}
|
||||
|
||||
The Go compiler expects a CPU pprof profile as the input to PGO.
|
||||
Profiles generated by the Go runtime (such as from [runtime/pprof](https://pkg.go.dev/runtime/pprof) and [net/http/pprof](https://pkg.go.dev/net/http/pprof)) can be used directly as the compiler input.
|
||||
It may also be possible to use/convert profiles from other profiling systems. See [the appendix](#alternative-sources) for additional information.
|
||||
|
||||
For best results, it is important that profiles are _representative_ of actual behavior in the application’s production environment.
|
||||
Using an unrepresentative profile is likely to result in a binary with little to no improvement in production.
|
||||
Thus, collecting profiles directly from the production environment is recommended, and is the primary method that Go’s PGO is designed for.
|
||||
|
||||
The typical workflow is as follows:
|
||||
|
||||
1. Build and release an initial binary (without PGO).
|
||||
2. Collect profiles from production.
|
||||
3. When it's time to release an updated binary, build from the latest source and provide the production profile.
|
||||
4. GOTO 2
|
||||
|
||||
Go PGO is generally robust to skew between the profiled version of an application and the version building with the profile, as well as to building with profiles collected from already-optimized binaries.
|
||||
This is what makes this iterative lifecycle possible.
|
||||
See the [AutoFDO](#autofdo) section for additional details about this workflow.
|
||||
|
||||
If it is difficult or impossible to collect from the production environment (e.g., a command-line tool distributed to end users), it is also possible to collect from a representative benchmark.
|
||||
Note that constructing representative benchmarks is often quite difficult (as is keeping them representative as the application evolves).
|
||||
In particular, _microbenchmarks are usually bad candidates for PGO profiling_, as they only exercise a small part of the application, which yields small gains when applied to the whole program.
|
||||
|
||||
# Building with PGO {#building}
|
||||
|
||||
The `go build -pgo` flag controls PGO profile selection.
|
||||
Setting this flag to anything other than `-pgo=off` enables PGO optimizations.
|
||||
|
||||
The standard approach is to store a pprof CPU profile with filename `default.pgo` in the main package directory of the profiled binary, and build with `go build -pgo=auto`, which will pick up `default.pgo` files automatically.
|
||||
|
||||
Commiting profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds.
|
||||
Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source.
|
||||
|
||||
_Note: In Go 1.20, the default is `-pgo=off`.
|
||||
A future version is likely to change the default to `-pgo=auto` to automatically build any binary with `default.pgo` with PGO._
|
||||
|
||||
_Note: In Go 1.20, `-pgo=auto` only works with a single main package.
|
||||
Attempting to build multiple main packages (`go build -pgo=auto ./cmd/foo ./cmd/bar`) will result in a build error.
|
||||
This is https://go.dev/issue/58099._
|
||||
|
||||
For more complex scenarios (e.g., different profiles for different scenarios of one binary, unable to store profile with source, etc), you may directly pass a path to the profile to use (e.g., `go build -pgo=/tmp/foo.pprof`).
|
||||
|
||||
_Note: A path passed to `-pgo` applies to all main packages.
|
||||
e.g., `go build -pgo=/tmp/foo.pprof ./cmd/foo ./cmd/bar` applies `foo.pprof` to both binaries `foo` and `bar`, which is often not what you want.
|
||||
Usually different binaries should have different profiles, passed via separate `go build` invocations._
|
||||
|
||||
# Notes {#notes}
|
||||
|
||||
## Collecting representative profiles from production
|
||||
|
||||
Your production environment is the best source of representative profiles for your application, as described in [Collecting profiles](#collecting-profiles).
|
||||
|
||||
The simplest way to start with this is to add [net/http/pprof](https://pkg.go.dev/net/http/pprof) to your application and then fetch `/debug/pprof/profile?seconds=30` from an arbitrary instance of your service.
|
||||
This is a great way to get started, but there are ways that this may be unrepresentative:
|
||||
|
||||
* This instance may not be doing anything at the moment it gets profiled, even though it is usually busy.
|
||||
|
||||
* Traffic patterns may change throughout the day, making behavior change throughout the day.
|
||||
|
||||
* Instances may perform long-running operations (e.g., 5 minutes doing operation A, then 5 minutes doing operation B, etc).
|
||||
A 30s profile will likely only cover a single operation type.
|
||||
|
||||
* Instances may not receive fair distributions of requests (some instances receive more of one type of request than others).
|
||||
|
||||
A more robust strategy is collecting multiple profiles at different times from different instances to limit the impact of differences between individual instance profiles.
|
||||
Multiple profiles may then be [merged](#merging-profiles) into a single profile for use with PGO.
|
||||
|
||||
Many organizations run “continuous profiling” services that perform this kind of fleet-wide sampling profiling automatically, which could then be used as a source of profiles for PGO.
|
||||
|
||||
## Merging profiles {#merging-profiles}
|
||||
|
||||
The pprof tool can merge multiple profiles like this:
|
||||
|
||||
```
|
||||
$ go tool pprof -proto a.pprof b.pprof > merged.pprof
|
||||
```
|
||||
|
||||
This merge is effectively a straightforward sum of samples in the input, regardless of wall duration of the profile.
|
||||
As a result, when profiling a small time slice of an application (e.g., a server that runs indefinitely), you likely want to ensure that all profiles have the same wall duration (i.e., all profiles are collected for 30s).
|
||||
Otherwise, profiles with longer wall duration will be overrepresented in the merged profile.
|
||||
|
||||
## AutoFDO {#autofdo}
|
||||
|
||||
Go PGO is designed to support an “[AutoFDO](https://research.google/pubs/pub45290/)” style workflow.
|
||||
|
||||
Let's take a closer look at the workflow described in [Collecting profiles](#collecting-profiles):
|
||||
|
||||
1. Build and release an initial binary (without PGO).
|
||||
2. Collect profiles from production.
|
||||
3. When it's time to release an updated binary, build from the latest source and provide the production profile.
|
||||
4. GOTO 2
|
||||
|
||||
This sounds deceptively simple, but there are a few important properties to note here:
|
||||
|
||||
* Development is always ongoing, so the source code of the profiled version of the binary (step 2) is likely slightly different from the latest source code getting built (step 3).
|
||||
Go PGO is designed to be robust to this, which we refer to as _source stability_.
|
||||
|
||||
* This is a closed loop.
|
||||
That is, after the first iteration the profiled version of the binary is already PGO-optimized with a profile from a previous iteration.
|
||||
Go PGO is also designed to be robust to this, which we refer to as _iterative stability_.
|
||||
|
||||
_Source stability_ is achieved using heuristics to match samples from the profile to the compiling source.
|
||||
As a result, many changes to source code, such as adding new functions, have no impact on matching existing code.
|
||||
When the compiler is not able to match changed code, some optimizations are lost, but note that this is a _graceful degradation_.
|
||||
A single function failing to match may lose out on optimization opportunities, but overall PGO benefit is usually spread across many functions. See the [source stability](#source-stability) section for more details about matching and degradation.
|
||||
|
||||
_Iterative stability_ is the prevention of cycles of variable performance in successive PGO builds (e.g., build #1 is fast, build #2 is slow, build #3 is fast, etc).
|
||||
We use CPU profiles to identify hot functions to target with optimizations.
|
||||
In theory, a hot function could be sped up so much by PGO that it no longer appears hot in the next profile and does not get optimized, making it slow again.
|
||||
The Go compiler takes a conservative approach to PGO optimizations, which we believe prevents significant variance.
|
||||
If you do observe this kind of instability, please file an issue at https://go.dev/issue/new.
|
||||
|
||||
Together, source and iterative stability eliminate the requirement for two-stage builds where a first, unoptimized build is profiled as a canary, and then rebuilt with PGO for production (unless absolutely peak performance is required).
|
||||
|
||||
## Source stability and refactoring {#source-stability}
|
||||
|
||||
As described in above, Go’s PGO makes a best-effort attempt to continue matching samples from older profiles to the current source code.
|
||||
Specifically, Go uses line offsets within functions (e.g., call on 5th line of function foo).
|
||||
|
||||
Many common changes will not break matching, including:
|
||||
|
||||
* Changes in a file outside of a hot function (adding/changing code above or below the function).
|
||||
|
||||
* Moving a function to another file in the same package (the compiler ignores source filenames altogether).
|
||||
|
||||
Some changes that may break matching:
|
||||
|
||||
* Changes within a hot function (may affect line offsets).
|
||||
|
||||
* Renaming the function (and/or type for methods) (changes symbol name).
|
||||
|
||||
* Moving the function to another package (changes symbol name).
|
||||
|
||||
If the profile is relatively recent, then differences likely only affect a small number of hot functions, limiting the impact of missed optimizations in functions that fail to match.
|
||||
Still, degradation will slowly accumulate over time since code is rarely refactored _back_ to its old form, so it is important to collect new profiles regularly to limit source skew from production.
|
||||
|
||||
One situation where profile matching may significantly degrade is a large-scale refactor that renames many functions or moves them between packages.
|
||||
In this case, you may take a short-term performance hit until a new profile shows the new structure.
|
||||
|
||||
For rote renames, an existing profile could theoretically be rewritten to change the old symbol names to the new names.
|
||||
[github.com/google/pprof/profile](https://pkg.go.dev/github.com/google/pprof/profile) contains the primitives required to rewrite a pprof profile in this way, but as of writing no off-the-shelf tool exists for this.
|
||||
|
||||
## Performance of new code
|
||||
|
||||
When adding new code or enabling new code paths with a flag flip, that code will not be present in the profile on the first build, and thus won't receive PGO optimizations until a new profile reflecting the new code is collected.
|
||||
Keep in mind when evaluating the rollout of new code that the initial release will not represent its steady state performance.
|
||||
|
||||
# Frequently Asked Questions {#faq}
|
||||
|
||||
## Is it possible to optimize Go standard library packages with PGO?
|
||||
|
||||
Yes.
|
||||
PGO in Go applies to the entire program.
|
||||
All packages are rebuilt to consider potential profile-guided optimizations, including standard library packages.
|
||||
|
||||
## Is it possible to optimize packages in dependent modules with PGO?
|
||||
|
||||
Yes.
|
||||
PGO in Go applies to the entire program.
|
||||
All packages are rebuilt to consider potential profile-guided optimizations, including packages in dependencies.
|
||||
This means that the unique way your application uses a dependency impacts the optimizations applied to that dependency.
|
||||
|
||||
## Will PGO with an unrepresentative profile make my program slower than no PGO?
|
||||
|
||||
It should not.
|
||||
While a profile that is not representative of production behavior will result in optimizations in cold parts of the application, it should not make hot parts of the application slower.
|
||||
If you encounter a program where PGO results in worse performance than disabling PGO, please file an issue at https://go.dev/issue/new.
|
||||
|
||||
## Can I use the same profile for different GOOS/GOARCH builds?
|
||||
|
||||
Yes.
|
||||
The format of the profiles is equivalent across OS and architecture configurations, so they may be used across different configurations.
|
||||
For example, a profile collected from a linux/arm64 binary may be used in a windows/amd64 build.
|
||||
|
||||
That said, the source stability caveats discussed [above](#autofdo) apply here as well.
|
||||
Any source code that differs across these configurations will not be optimized.
|
||||
For most applications, the vast majority of code is platform-independent, so degradation of this form is limited.
|
||||
|
||||
As a specific example, the internals of file handling in package `os` differ between Linux and Windows.
|
||||
If these functions are hot in the Linux profile, the Windows equivalents will not get PGO optimizations because they do not match the profiles.
|
||||
|
||||
You may merge profiles of different GOOS/GOARCH builds. See the next question for the tradeoffs of doing so.
|
||||
|
||||
## How should I handle a single binary used for different workload types?
|
||||
|
||||
There is no obvious choice here.
|
||||
A single binary used for different types of workloads (e.g., a database used in a read-heavy way in one service, and write-heavy in another service) may have different hot components, which benefit from different optimizations.
|
||||
|
||||
There are three options:
|
||||
|
||||
1. Build different versions of the binary for each workload: use profiles from each workload to build multiple workload-specific builds of the binary.
|
||||
This will provide the best performance for each workload, but may add operational complexity with regard to handling multiple binaries and profile sources.
|
||||
|
||||
2. Build a single binary using only profiles from the “most important” workload: select the “most important” workload (largest footprint, most performance sensitive), and build using profiles only from that workload.
|
||||
This provides the best performance for the selected workload, and likely still modest performance improvements for other workloads from optimizations to common code shared across workloads.
|
||||
|
||||
3. Merge profiles across workloads: take profiles from each workload (weighted by total footprint) and merge them into a single “fleet-wide” profile used to build a single common profile used to build.
|
||||
This likely provides modest performance improvements for all workloads.
|
||||
|
||||
## How does PGO affect build time?
|
||||
|
||||
Enabling PGO builds should cause measurable, but small, increases in package build times.
|
||||
Likely more noticeable than individual package build times is that PGO profiles apply to all packages in a binary, meaning that the first use of a profile requires a rebuild of every package in the dependency graph.
|
||||
These builds are cached like any other, so subsequent incremental builds using the same profile do not require complete rebuilds.
|
||||
|
||||
If you experience extreme increases in build time, please file an issue at https://go.dev/issue/new.
|
||||
|
||||
_Note: In Go 1.20, profile parsing adds significant overhead, particularly for large profiles, which can significantly increase build times.
|
||||
This is tracked by https://go.dev/issue/58102 and will be addressed in a future release._
|
||||
|
||||
## How does PGO affect binary size?
|
||||
|
||||
PGO can result in slightly larger binaries due to additional function inlining.
|
||||
|
||||
# Appendix: alternative profile sources {#alternative-sources}
|
||||
|
||||
CPU profiles generated by the Go runtime (via [runtime/pprof](https://pkg.go.dev/runtime/pprof), etc) are already in the correct format for direct use as PGO inputs.
|
||||
However, organizations may have alternative preferred tooling (e.g., Linux perf), or existing fleet-wide continuous profiling systems which they wish to use with Go PGO.
|
||||
|
||||
Profiles from alternative source may be used with Go PGO if converted to the [pprof format](https://github.com/google/pprof/tree/main/proto), provided they follow these general requirements:
|
||||
|
||||
* Sample index 0 should be type “cpu” and unit “count”.
|
||||
|
||||
* Samples should represent samples of CPU time at the sample location.
|
||||
|
||||
* The profile must be symbolized ([Function.name](https://github.com/google/pprof/blob/76d1ae5aea2b3f738f2058d17533b747a1a5cd01/proto/profile.proto#L208) must be set).
|
||||
|
||||
* Samples must contain stack frames for inlined functions.
|
||||
If inlined functions are omitted, Go will not be able to maintain iterative stability.
|
||||
|
||||
* [Function.start_line](https://github.com/google/pprof/blob/76d1ae5aea2b3f738f2058d17533b747a1a5cd01/proto/profile.proto#L215) must be set.
|
||||
This is the line number of the start of the function.
|
||||
i.e., the line containing the `func` keyword.
|
||||
The Go compiler uses this field to compute line offsets of samples (`Location.Line.line - Function.start_line`).
|
||||
**Note that many existing pprof converters omit this field.**
|
||||
|
||||
_Note: In Go 1.20, DWARF metadata omits function start lines (`DW_AT_decl_line`), which may make it difficult for tools to determine the start line.
|
||||
This is tracked by https://go.dev/issue/57308, and is expected to be fixed in Go 1.21._
|
||||
|
|
@ -259,7 +259,7 @@ func main() {
|
|||
</p>
|
||||
<div class="GettingStartedGo-ctas">
|
||||
<a class="GettingStartedGo-primaryCta" href="/learn/">Get Started</a>
|
||||
<a href="/dl">Download Go</a>
|
||||
<a href="/doc/install/">Download Go</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="GettingStartedGo-resourcesSection">
|
||||
|
|
|
|||
|
|
@ -93,6 +93,8 @@ footer:
|
|||
children:
|
||||
- name: Standard Library
|
||||
url: /pkg/
|
||||
- name: About Go Packages
|
||||
url: https://pkg.go.dev/about
|
||||
|
||||
- name: About
|
||||
url: /project
|
||||
|
|
|
|||
|
|
@ -94,7 +94,7 @@ We pride ourselves on being meticulous; no issue is too small.
|
|||
<p>
|
||||
Security-related issues should be reported to
|
||||
<a href="mailto:security@golang.org">security@golang.org</a>.<br>
|
||||
See the <a href="/security">security policy</a> for more details.
|
||||
See the <a href="/security/policy">security policy</a> for more details.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
|
|
|
|||
|
|
@ -52,9 +52,9 @@
|
|||
iconDark: /images/icons/gear-dark.svg
|
||||
iconName: Sphere
|
||||
packages:
|
||||
- title: opentracing/opentracing-go
|
||||
url: https://github.com/opentracing/opentracing-go
|
||||
- title: open-telemetry/opentelemetry-go
|
||||
url: https://github.com/open-telemetry/opentelemetry-go
|
||||
- title: istio/istio
|
||||
url: https://github.com/istio/istio
|
||||
- title: urfave/cli
|
||||
url: https://github.com/urfave/cli
|
||||
url: https://github.com/urfave/cli
|
||||
|
|
|
|||
|
|
@ -145,9 +145,9 @@ Go’s garbage collector means DevOps/SRE teams don’t have to worry about memo
|
|||
- title: Monitoring and tracing
|
||||
viewMoreUrl: https://pkg.go.dev/search?q=tracing
|
||||
items:
|
||||
- text: opentracing/opentracing-go
|
||||
url: https://pkg.go.dev/github.com/opentracing/opentracing-go?tab=overview
|
||||
desc: Vendor-neutral APIs and instrumentation for distributed tracing
|
||||
- text: open-telemetry/opentelemetry-go
|
||||
url: https://pkg.go.dev/go.opentelemetry.io/otel
|
||||
desc: Vendor-neutral APIs and instrumentation for monitoring and distributed tracing
|
||||
- text: jaegertracing/jaeger-client-go
|
||||
url: https://pkg.go.dev/github.com/jaegertracing/jaeger-client-go?tab=overview
|
||||
desc: An open source distributed tracing system developed by Uber formats
|
||||
|
|
|
|||
|
|
@ -66,7 +66,7 @@ Tigran Bayburtsyan, Co-Founder and CTO at Hexact Inc., summarizes five key reaso
|
|||
- text: Building a new router for gov.uk
|
||||
url: https://technology.blog.gov.uk/2013/12/05/building-a-new-router-for-gov-uk/
|
||||
- text: Using Go in government
|
||||
url: https://technology.blog.gov.uk/2013/12/05/building-a-new-router-for-gov-uk/
|
||||
url: https://technology.blog.gov.uk/2014/11/14/using-go-in-government/
|
||||
- company: Hugo
|
||||
url: http://gohugo.io/
|
||||
logoSrc: hugo.svg
|
||||
|
|
|
|||
|
|
@ -171,7 +171,7 @@ safe for concurrent use!
|
|||
* Where to Go from here...
|
||||
|
||||
#appengine: You can get started by
|
||||
#appengine: [[/dl/][installing Go]].
|
||||
#appengine: [[/doc/install/][installing Go]].
|
||||
|
||||
#appengine: Once you have Go installed, the
|
||||
The
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
steps:
|
||||
- name: gcr.io/cloud-builders/git
|
||||
args: ["clone", "--branch=${_GO_REF}", "--depth=1", "https://go.googlesource.com/go", "_gotmp"]
|
||||
args: ["clone", "--branch=release-branch.go1.20", "--depth=1", "https://go.googlesource.com/go", "_gotmp"]
|
||||
- name: gcr.io/cloud-builders/git
|
||||
args: ["archive", "--format=zip", "--output=../_goroot.zip", "HEAD"]
|
||||
dir: _gotmp
|
||||
|
|
|
|||
|
|
@ -288,7 +288,7 @@ func (s *server) codewalkFileprint(w http.ResponseWriter, r *http.Request, f str
|
|||
io.WriteString(w, "</pre>")
|
||||
}
|
||||
|
||||
// addrToByte evaluates the given address starting at offset start in data.
|
||||
// addrToByteRange evaluates the given address starting at offset start in data.
|
||||
// It returns the lo and hi byte offset of the matched region within data.
|
||||
// See https://9p.io/sys/doc/sam/sam.html Table II
|
||||
// for details on the syntax.
|
||||
|
|
|
|||
|
|
@ -14,6 +14,9 @@ import "html/template"
|
|||
//
|
||||
// The table is sorted by date, breaking ties with newer versions first.
|
||||
var Releases = []*Release{
|
||||
{
|
||||
Date: Date{2023, 2, 1}, Version: Version{1, 20, 0},
|
||||
},
|
||||
{
|
||||
Date: Date{2023, 1, 10}, Version: Version{1, 19, 5},
|
||||
Bug: &FixSummary{
|
||||
|
|
|
|||
|
|
@ -155,10 +155,7 @@ func firstIdent(x []byte) string {
|
|||
|
||||
// Comment formats the given documentation comment as HTML.
|
||||
func (p *Page) Comment(comment string) template.HTML {
|
||||
// TODO: After Go 1.20 is out, this can be simplified to:
|
||||
//return template.HTML(p.PDoc.HTML(comment))
|
||||
// While deleting the go118.go and go119.go files.
|
||||
return template.HTML(docPackageHTML(p.PDoc, comment))
|
||||
return template.HTML(p.PDoc.HTML(comment))
|
||||
}
|
||||
|
||||
// sanitize sanitizes the argument src by replacing newlines with
|
||||
|
|
|
|||
|
|
@ -1,19 +0,0 @@
|
|||
// Copyright 2022 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !go1.19
|
||||
// +build !go1.19
|
||||
|
||||
package pkgdoc
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"go/doc"
|
||||
)
|
||||
|
||||
func docPackageHTML(_ *doc.Package, text string) []byte {
|
||||
var buf bytes.Buffer
|
||||
doc.ToHTML(&buf, text, nil)
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
// Copyright 2022 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build go1.19
|
||||
// +build go1.19
|
||||
|
||||
package pkgdoc
|
||||
|
||||
import "go/doc"
|
||||
|
||||
var docPackageHTML = (*doc.Package).HTML
|
||||
|
|
@ -99,6 +99,9 @@ func (s *siteDir) locate(verb, file string, arg ...interface{}) (before, text, a
|
|||
if cfg.Line == -1 {
|
||||
cfg.Line = 1
|
||||
}
|
||||
lines := strings.SplitAfter(text, "\n")
|
||||
filterOmit(lines)
|
||||
text = strings.Join(lines, "")
|
||||
case 1:
|
||||
var n int
|
||||
before, text, after, n = s.oneLine(file, text, arg[0])
|
||||
|
|
@ -153,8 +156,12 @@ func (s *Site) oneLine(file, body string, arg interface{}) (before, text, after
|
|||
if !isInt {
|
||||
line = match(file, 0, lines, pattern)
|
||||
}
|
||||
filterOmit(lines)
|
||||
line--
|
||||
return strings.Join(lines[:line], ""), lines[line], strings.Join(lines[line+1:], ""), line
|
||||
return strings.Join(lines[:line], ""),
|
||||
lines[line],
|
||||
strings.Join(lines[line+1:], ""),
|
||||
line
|
||||
}
|
||||
|
||||
// multipleLines returns the text generated by a three-argument code invocation.
|
||||
|
|
@ -170,15 +177,12 @@ func (s *Site) multipleLines(file, body string, arg1, arg2 interface{}) (before,
|
|||
} else if line2 < line1 {
|
||||
log.Panicf("lines out of order for %q: %d %d", file, line1, line2)
|
||||
}
|
||||
for k := line1 - 1; k < line2; k++ {
|
||||
if strings.HasSuffix(lines[k], "OMIT\n") {
|
||||
lines[k] = ""
|
||||
}
|
||||
}
|
||||
filterOmit(lines)
|
||||
line1--
|
||||
return strings.Join(lines[:line1], ""),
|
||||
strings.Join(lines[line1:line2], ""),
|
||||
strings.Join(lines[line2:], ""), line1
|
||||
strings.Join(lines[line2:], ""),
|
||||
line1
|
||||
}
|
||||
|
||||
// parseArg returns the integer or string value of the argument and tells which it is.
|
||||
|
|
@ -223,3 +227,12 @@ func match(file string, start int, lines []string, pattern string) int {
|
|||
log.Panicf("unrecognized pattern: %q", pattern)
|
||||
return 0
|
||||
}
|
||||
|
||||
func filterOmit(lines []string) {
|
||||
for i, s := range lines {
|
||||
if strings.HasSuffix(s, "OMIT\n") {
|
||||
lines[i] = ""
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -63,6 +63,97 @@ func TestMarkdown(t *testing.T) {
|
|||
testServeBody(t, site, "/doc/test2", "<em>template</em>")
|
||||
}
|
||||
|
||||
func TestCode(t *testing.T) {
|
||||
site := NewSite(fstest.MapFS{
|
||||
"site.tmpl": {Data: []byte(`{{.Content}}`)},
|
||||
"doc/code.md": {Data: []byte(`
|
||||
# hi
|
||||
whole file
|
||||
{{code "_code/prog.go"}}
|
||||
one line
|
||||
{{code "_code/prog.go" "/func main/"}}
|
||||
multiple lines
|
||||
{{code "_code/prog.go" "/START/" "/END/"}}
|
||||
following lines
|
||||
{{code "_code/prog.go" "/START/" "$"}}
|
||||
play
|
||||
{{play "_code/prog.go" "/START/" "/END/"}}
|
||||
play with numbers
|
||||
{{play "_code/prog.go" "/START/" "/END/" 0}}
|
||||
`)},
|
||||
"doc/_code/prog.go": {Data: []byte(`
|
||||
// +build OMIT
|
||||
|
||||
package main
|
||||
|
||||
// START OMIT
|
||||
func main() { fmt.Println("hi") }
|
||||
// END OMIT
|
||||
|
||||
func foo() {}
|
||||
`)},
|
||||
})
|
||||
|
||||
testServeBody(t, site, "/doc/code", `<h1 id="hi">hi</h1>
|
||||
<p>whole file</p>
|
||||
<div class="code">
|
||||
<pre>package main
|
||||
|
||||
func main() { fmt.Println("hi") }
|
||||
|
||||
func foo() {}
|
||||
</pre>
|
||||
</div>
|
||||
<p>one line</p>
|
||||
<div class="code">
|
||||
<pre>func main() { fmt.Println("hi") }
|
||||
</pre>
|
||||
</div>
|
||||
<p>multiple lines</p>
|
||||
<div class="code">
|
||||
<pre>func main() { fmt.Println("hi") }
|
||||
</pre>
|
||||
</div>
|
||||
<p>following lines</p>
|
||||
<div class="code">
|
||||
<pre>func main() { fmt.Println("hi") }
|
||||
|
||||
func foo() {}
|
||||
</pre>
|
||||
</div>
|
||||
<p>play</p>
|
||||
<div class="playground">
|
||||
<pre style="display: none"><span>
|
||||
|
||||
package main
|
||||
|
||||
</span>
|
||||
</pre>
|
||||
<pre contenteditable="true" spellcheck="false">func main() { fmt.Println("hi") }
|
||||
</pre>
|
||||
<pre style="display: none"><span>
|
||||
func foo() {}
|
||||
</span>
|
||||
</pre>
|
||||
</div>
|
||||
<p>play with numbers</p>
|
||||
<div class="playground">
|
||||
<pre style="display: none"><span>
|
||||
|
||||
package main
|
||||
|
||||
</span>
|
||||
</pre>
|
||||
<pre contenteditable="true" spellcheck="false"><span class="number"> 5 </span>func main() { fmt.Println("hi") }
|
||||
<span class="number"> 6 </span>
|
||||
</pre>
|
||||
<pre style="display: none"><span>
|
||||
func foo() {}
|
||||
</span>
|
||||
</pre>
|
||||
</div>`)
|
||||
}
|
||||
|
||||
func TestTypeScript(t *testing.T) {
|
||||
exampleOut, err := os.ReadFile("testdata/example.js")
|
||||
if err != nil {
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче