This adds support for the passthrough on build, push, login, and search.
Revamp the integration test to cover these cases and make it more
robust.
Use backticks instead of quoted strings for backslash-heavy string
contstands.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Moving all strings to the errors package wasn't a good idea after all.
Our custom implementation of Go errors predates everything that's nice
and good about working with errors in Go. Take as an example what we
have to do to get an error message:
```go
func GetErrorMessage(err error) string {
switch err.(type) {
case errcode.Error:
e, _ := err.(errcode.Error)
return e.Message
case errcode.ErrorCode:
ec, _ := err.(errcode.ErrorCode)
return ec.Message()
default:
return err.Error()
}
}
```
This goes against every good practice for Go development. The language already provides a simple, intuitive and standard way to get error messages, that is calling the `Error()` method from an error. Reinventing the error interface is a mistake.
Our custom implementation also makes very hard to reason about errors, another nice thing about Go. I found several (>10) error declarations that we don't use anywhere. This is a clear sign about how little we know about the errors we return. I also found several error usages where the number of arguments was different than the parameters declared in the error, another clear example of how difficult is to reason about errors.
Moreover, our custom implementation didn't really make easier for people to return custom HTTP status code depending on the errors. Again, it's hard to reason about when to set custom codes and how. Take an example what we have to do to extract the message and status code from an error before returning a response from the API:
```go
switch err.(type) {
case errcode.ErrorCode:
daError, _ := err.(errcode.ErrorCode)
statusCode = daError.Descriptor().HTTPStatusCode
errMsg = daError.Message()
case errcode.Error:
// For reference, if you're looking for a particular error
// then you can do something like :
// import ( derr "github.com/docker/docker/errors" )
// if daError.ErrorCode() == derr.ErrorCodeNoSuchContainer { ... }
daError, _ := err.(errcode.Error)
statusCode = daError.ErrorCode().Descriptor().HTTPStatusCode
errMsg = daError.Message
default:
// This part of will be removed once we've
// converted everything over to use the errcode package
// FIXME: this is brittle and should not be necessary.
// If we need to differentiate between different possible error types,
// we should create appropriate error types with clearly defined meaning
errStr := strings.ToLower(err.Error())
for keyword, status := range map[string]int{
"not found": http.StatusNotFound,
"no such": http.StatusNotFound,
"bad parameter": http.StatusBadRequest,
"conflict": http.StatusConflict,
"impossible": http.StatusNotAcceptable,
"wrong login/password": http.StatusUnauthorized,
"hasn't been activated": http.StatusForbidden,
} {
if strings.Contains(errStr, keyword) {
statusCode = status
break
}
}
}
```
You can notice two things in that code:
1. We have to explain how errors work, because our implementation goes against how easy to use Go errors are.
2. At no moment we arrived to remove that `switch` statement that was the original reason to use our custom implementation.
This change removes all our status errors from the errors package and puts them back in their specific contexts.
IT puts the messages back with their contexts. That way, we know right away when errors used and how to generate their messages.
It uses custom interfaces to reason about errors. Errors that need to response with a custom status code MUST implementent this simple interface:
```go
type errorWithStatus interface {
HTTPErrorStatusCode() int
}
```
This interface is very straightforward to implement. It also preserves Go errors real behavior, getting the message is as simple as using the `Error()` method.
I included helper functions to generate errors that use custom status code in `errors/errors.go`.
By doing this, we remove the hard dependency we have eeverywhere to our custom errors package. Yes, you can use it as a helper to generate error, but it's still very easy to generate errors without it.
Please, read this fantastic blog post about errors in Go: http://dave.cheney.net/2014/12/24/inspecting-errors
Signed-off-by: David Calavera <david.calavera@gmail.com>
This feels like it's where it belongs and it makes it exported
again (which is needed for libcompose that was using it before 1.10).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
filepath.Clean converts filenames to filenames with native path
separators. Use ToSlash to normalize.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Currently, daemonbuilder package (part of daemon) implemented the
builder backend. However, it was a very thin wrapper around daemon
methods and caused an implementation dependency for api/server build
endpoint. api/server buildrouter should only know about the backend
implementing the /build API endpoint.
Removing daemonbuilder involved moving build specific methods to
respective files in the daemon, where they fit naturally.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Currently builder.Backend is implemented by daemonbuilder.Docker{} for
the daemon. This registration happens in the API/server code. However,
this is too implementation specific. Ideally we should be able to specify
that docker daemon (or any other) is implementing the Backend and abstract
the implementation details. So we should remove package daemonbuilder
dependency in build_routes.go
With this change, daemonbuilder.Docker is nothing more than the daemon.
A follow on change will remove the daemonbuilder package and move relevant
methods under daemon, so that API only knows about the backend.
Also cleanup code in api/client/build.go. docker cli always performs build
context tar download for remoteURLs and sends an empty remoteContext. So
remove relevant dead code.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Merge was used by builder and daemon. With this commit, the builder
call has been inlined and the function moved to the daemon package,
which is the only other caller.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
dockerfile.Config is almost redundant with ImageBuildOptions.
Unify the two so that the latter can be removed. This also
helps build's API endpoint code to be less dependent on package
dockerfile.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
The parse.go file is used almost exclusively in the client. The few small
functions that are used outside of the client could easily be copied out
when the client is extracted, allowing this runconfig/opts package to
move to the client.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
- Make the API client library completely standalone.
- Move windows partition isolation detection to the client, so the
driver doesn't use external types.
Signed-off-by: David Calavera <david.calavera@gmail.com>
This is a very docker concept that nobody elses need.
We only maintain it to keep the API backwards compatible.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Right now, the quiet (-q, --quiet) flag ignores the output
generated from within the container.
However, it ought to be quiet in a way that all kind
of diagnostic output should be ignored, unless the build
process fails.
This patch makes the quiet flag behave in the following way:
1. If the build process succeeds, stdout contains the image ID
and stderr is empty.
2. If the build process fails, stdout is empty and stderr
has the error message and the diagnostic output of that process.
If the quiet flag is not set, then everything goes to stdout
and error messages, if there are any, go to stderr.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
So other packages don't need to import the daemon package when they
want to use this struct.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
Add distribution package for managing pulls and pushes. This is based on
the old code in the graph package, with major changes to work with the
new image/layer model.
Add v1 migration code.
Update registry, api/*, and daemon packages to use the reference
package's types where applicable.
Update daemon package to use image/layer/tag stores instead of the graph
package
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
- Optional "--shm-size=" was added to the sub-command(run, create,and build).
- The size of /dev/shm in the container can be changed
when container is made.
- Being able to specify is a numerical value that applies number,
b, k, m, and g.
- The default value is 64MB, when this option is not set.
- It deals with both native and lxc drivers.
Signed-off-by: NIWA Hideyuki <niwa.hiedyuki@jp.fujitsu.com>
Closes#17781
This allows for env vars in EXPOSE to be parsed for spaces so that each
"word" is then treated independently instead of as a single word/arg.
Signed-off-by: Doug Davis <dug@us.ibm.com>
Fixes#17290
Fixes following issues:
- Cache checksums turning off while walking a broken symlink.
- Cache checksums were taken from symlinks while targets were actually copied.
- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The LXC driver was deprecated in Docker 1.8.
Following the deprecation rules, we can remove a deprecated feature
after two major releases. LXC won't be supported anymore starting on Docker 1.10.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Side effects:
- Decouple daemon and container to start containers.
- Decouple daemon and container to copy files.
Signed-off-by: David Calavera <david.calavera@gmail.com>
This patch creates interfaces in builder/ for building Docker images.
It is a first step in a series of patches to remove the daemon
dependency on builder and later allow a client-side Dockerfile builder
as well as potential builder plugins.
It is needed because we cannot remove the /build API endpoint, so we
need to keep the server-side Dockerfile builder, but we also want to
reuse the same Dockerfile parser and evaluator for both server-side and
client-side.
builder/dockerfile/ and api/server/builder.go contain implementations
of those interfaces as a refactoring of the current code.
Signed-off-by: Tibor Vass <tibor@docker.com>
Although having a request ID available throughout the codebase is very
valuable, the impact of requiring a Context as an argument to every
function in the codepath of an API request, is too significant and was
not properly understood at the time of the review.
Furthermore, mixing API-layer code with non-API-layer code makes the
latter usable only by API-layer code (one that has a notion of Context).
This reverts commit de41640435, reversing
changes made to 7daeecd42d.
Signed-off-by: Tibor Vass <tibor@docker.com>
Conflicts:
api/server/container.go
builder/internals.go
daemon/container_unix.go
daemon/create.go
This reverts commit ff92f45be4, reversing
changes made to 80e31df3b6.
Reverting to make the next revert easier.
Signed-off-by: Tibor Vass <tibor@docker.com>
Avoid creating a global context object that will be used while the daemon is running.
Not only this object won't ever be garbage collected, but it won't ever be used for anything else than creating other contexts in each request. I think it's a bad practive to have something like this sprawling aroud the code.
This change removes that global object and initializes a context in the cases we don't have already one, like shutting down the server.
This also removes a bunch of context arguments from functions that did nothing with it.
Signed-off-by: David Calavera <david.calavera@gmail.com>
This PR adds a "request ID" to each event generated, the 'docker events'
stream now looks like this:
```
2015-09-10T15:02:50.000000000-07:00 [reqid: c01e3534ddca] de7c5d4ca927253cf4e978ee9c4545161e406e9b5a14617efb52c658b249174a: (from ubuntu) create
```
Note the `[reqID: c01e3534ddca]` part, that's new.
Each HTTP request will generate its own unique ID. So, if you do a
`docker build` you'll see a series of events all with the same reqID.
This allow for log processing tools to determine which events are all related
to the same http request.
I didn't propigate the context to all possible funcs in the daemon,
I decided to just do the ones that needed it in order to get the reqID
into the events. I'd like to have people review this direction first, and
if we're ok with it then I'll make sure we're consistent about when
we pass around the context - IOW, make sure that all funcs at the same level
have a context passed in even if they don't call the log funcs - this will
ensure we're consistent w/o passing it around for all calls unnecessarily.
ping @icecrime @calavera @crosbymichael
Signed-off-by: Doug Davis <dug@us.ibm.com>
- The build-time variables are passed as environment-context for command(s)
run as part of the RUN primitve. These variables are not persisted in environment of
intermediate and final images when passed as context for RUN. The build environment
is prepended to the intermediate continer's command string for aiding cache lookups.
It also helps with build traceability. But this also makes the feature less secure from
point of view of passing build time secrets.
- The build-time variables also get used to expand the symbols used in certain
Dockerfile primitves like ADD, COPY, USER etc, without an explicit prior definiton using a
ENV primitive. These variables get persisted in the intermediate and final images
whenever they are expanded.
- The build-time variables are only expanded or passed to the RUN primtive if they
are defined in Dockerfile using the ARG primitive or belong to list of built-in variables.
HTTP_PROXY, HTTPS_PROXY, http_proxy, https_proxy, FTP_PROXY and NO_PROXY are built-in
variables that needn't be explicitly defined in Dockerfile to use this feature.
Signed-off-by: Madhav Puri <madhav.puri@gmail.com>
This is the first step in converting out static strings into well-defined
error types. This shows just a few examples of it to get a feel for how things
will look. Once we agree on the basic outline we can then work on converting
the rest of the code over.
Signed-off-by: Doug Davis <dug@us.ibm.com>
In particular I want to make sure that calling getEnv() when the same
var name appears more than once in the env list that we only pick up
the first one. PR #15182 counts on this
Signed-off-by: Doug Davis <dug@us.ibm.com>
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Windows: add support for images stored in alternate location.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Some structures use int for sizes and UNIX timestamps. On some
platforms, int is 32 bits, so this can lead to the year 2038 issues and
overflows when dealing with large containers or layers.
Consistently use int64 to store sizes and UNIX timestamps in
api/types/types.go. Update related to code accordingly (i.e.
strconv.FormatInt instead of strconv.Itoa).
Use int64 in progressreader package to avoid integer overflow when
dealing with large quantities. Update related code accordingly.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
If there's an error while unpacking the build context then we weren't erasing
the tmp dir created to persist the context.
Signed-off-by: Doug Davis <dug@us.ibm.com>
David Calavera
Tibor Vass
Tomasz Kopczynski
allencloud
Alexander Morozov
Sebastiaan van Stijn
Evan Hazlett
Aaron Lehmann
Jessica Frazelle
Antonio Murdaca
Qiang Huang
Stefan Weil
Vincent Demeester
John Howard
Brian Goff
Anusha Ragunathan
Arnaud Porterie
Lukas Waslowski
Doug Davis
Daniel Nephin
Boaz Shuster
Antonio Murdaca
Vincent Demeester
Tibor Vass
Justas Brazauskas
Morgan Bauer
mqliang
Tonis Tiigi
NIWA Hideyuki
Doug Davis
perhapszzy@sina.com
Darren Stahl
Stefan J. Wernli
Daehyeok Mun
Lei Jitang
Jess Frazelle
Soshi Katsuta
Madhav Puri
Alexander Morozov
Shijiang Wei
Soshi Katsuta
Veres Lajos
Alexey Guskov
Ma Shimiao