This commit is contained in:
Michael Taufen 2017-05-12 11:45:19 -07:00
Родитель d67c3ee48b
Коммит 3ed57711ac
9 изменённых файлов: 44 добавлений и 43 удалений

2
Godeps/Godeps.json сгенерированный
Просмотреть файл

@ -1,5 +1,5 @@
{
"ImportPath": "k8s.io/non-masquerade-daemon",
"ImportPath": "k8s.io/ip-masq-agent",
"GoVersion": "go1.7",
"GodepVersion": "v79",
"Packages": [

Просмотреть файл

@ -13,10 +13,10 @@
# limitations under the License.
# The binary to build (just the basename).
BIN := non-masquerade-daemon
BIN := ip-masq-agent
# This repo's root import path (under GOPATH).
PKG := k8s.io/non-masquerade-daemon
PKG := k8s.io/ip-masq-agent
# Where to push the docker image.
REGISTRY ?= gcr.io/google-containers

Просмотреть файл

@ -1,53 +1,53 @@
# non-masquerade-daemon
# ip-masq-agent
The non-masquerade-daemon configures `iptables` rules to `MASQUERADE` traffic outside link-local (optional, enabled by default) and additional arbitrary IP ranges.
The ip-masq-agent configures `iptables` rules to `MASQUERADE` traffic outside link-local (optional, enabled by default) and additional arbitrary IP ranges.
It creates an `iptables` chain called `NON-MASQUERADE-DAEMON`, which contains match rules for link local (`169.254.0.0/16`) and each of the user-specified IP ranges. It also creates a rule in `POSTROUTING` that jumps to this chain for any traffic not bound for a `LOCAL` destination.
It creates an `iptables` chain called `IP-MASQ-AGENT`, which contains match rules for link local (`169.254.0.0/16`) and each of the user-specified IP ranges. It also creates a rule in `POSTROUTING` that jumps to this chain for any traffic not bound for a `LOCAL` destination.
IPs that match the rules (except for the final rule) in `NON-MASQUERADE-DAEMON` are *not* subject to `MASQUERADE` via the `NON-MASQUERADE-DAEMON` chain (they `RETURN` early from the chain). The final rule in the `NON-MASQUERADE-DAEMON` chain will `MASQUERADE` any non-`LOCAL` traffic.
IPs that match the rules (except for the final rule) in `IP-MASQ-AGENT` are *not* subject to `MASQUERADE` via the `IP-MASQ-AGENT` chain (they `RETURN` early from the chain). The final rule in the `IP-MASQ-AGENT` chain will `MASQUERADE` any non-`LOCAL` traffic.
`RETURN` in `NON-MASQUERADE-DAEMON` resumes rule processing at the next rule the calling chain, `POSTROUTING`. Take care to avoid creating additional rules in `POSTROUTING` that cause packets bound for your configured ranges to undergo `MASQUERADE`.
`RETURN` in `IP-MASQ-AGENT` resumes rule processing at the next rule the calling chain, `POSTROUTING`. Take care to avoid creating additional rules in `POSTROUTING` that cause packets bound for your configured ranges to undergo `MASQUERADE`.
## Launching the daemon
This repo includes an example yaml file that can be used to launch the non-masquerade-daemon as a DaemonSet in a Kubernetes cluster.
## Launching the agent as a DaemonSet
This repo includes an example yaml file that can be used to launch the ip-masq-agent as a DaemonSet in a Kubernetes cluster.
```
kubectl create -f non-masquerade-daemon.yaml
kubectl create -f ip-masq-agent.yaml
```
The spec in `non-masquerade-daemon.yaml` specifies the `kube-system` namespace for the daemon pods.
The spec in `ip-masq-agent.yaml` specifies the `kube-system` namespace for the DaemonSet Pods.
## Configuring the daemon
## Configuring the agent
Important: You should not attempt to run this daemon in a cluster where the Kubelet is also configuring a non-masquerade CIDR. You can pass `--non-masquerade-cidr=0.0.0.0/0` to the Kubelet to nullify its rule, which will prevent the Kubelet from interfering with this daemon.
Important: You should not attempt to run this agent in a cluster where the Kubelet is also configuring a non-masquerade CIDR. You can pass `--non-masquerade-cidr=0.0.0.0/0` to the Kubelet to nullify its rule, which will prevent the Kubelet from interfering with this agent.
By default, the daemon is configured to treat the three private IP ranges specified by [RFC 1918](https://tools.ietf.org/html/rfc1918) as non-masquerade CIDRs. These ranges are `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16`. The daemon will also treat link-local (`169.254.0.0/16`) as a non-masquerade CIDR by default.
By default, the agent is configured to treat the three private IP ranges specified by [RFC 1918](https://tools.ietf.org/html/rfc1918) as non-masquerade CIDRs. These ranges are `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16`. The agent will also treat link-local (`169.254.0.0/16`) as a non-masquerade CIDR by default.
By default, the daemon is configured to reload its configuration from the `/etc/config/non-masquerade-daemon` file in its container every 60 seconds.
By default, the agent is configured to reload its configuration from the `/etc/config/ip-masq-agent` file in its container every 60 seconds.
The daemon configuration file should be written in yaml or json syntax, and may contain three optional keys:
The agent configuration file should be written in yaml or json syntax, and may contain three optional keys:
- `nonMasqueradeCIDRs []string`: A list strings in CIDR notation that specify the non-masquerade ranges.
- `linkLocal bool`: Whether to use `169.254.0.0/16` as a non-masquerade CIDR. True by default.
- `resyncInterval string`: The interval at which the daemon attempts to reload config from disk. The syntax is any format accepted by Go's [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration) function.
- `resyncInterval string`: The interval at which the agent attempts to reload config from disk. The syntax is any format accepted by Go's [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration) function.
The daemon will look for a config file in its container at `/etc/config/non-masquerade-daemon`. This file can be provided via a `ConfigMap`, plumbed into the container via a `ConfigMapVolumeSource`. As a result, the daemon can be reconfigured in a live cluster by creating or editing this `ConfigMap`.
The agent will look for a config file in its container at `/etc/config/ip-masq-agent`. This file can be provided via a `ConfigMap`, plumbed into the container via a `ConfigMapVolumeSource`. As a result, the agent can be reconfigured in a live cluster by creating or editing this `ConfigMap`.
This repo includes a directory-representation of a `ConfigMap` that can configure the daemon (the `daemon-config` directory). To use this directory to create the `ConfigMap` in your cluster:
This repo includes a directory-representation of a `ConfigMap` that can configure the agent (the `agent-config` directory). To use this directory to create the `ConfigMap` in your cluster:
```
kubectl create configmap non-masquerade-daemon --from-file=daemon-config --namespace=kube-system
kubectl create configmap ip-masq-agent --from-file=agent-config --namespace=kube-system
```
Note that we created the `ConfigMap` in the same namespace as the daemon pods, and named the `ConfigMap` to match the spec in `non-masquerade-daemon.yaml`. This is necessary for the `ConfigMap` to appear in the daemon pods' filesystems.
Note that we created the `ConfigMap` in the same namespace as the DaemonSet Pods, and named the `ConfigMap` to match the spec in `ip-masq-agent.yaml`. This is necessary for the `ConfigMap` to appear in the Pods' filesystems.
## Rationale
(from the [incubator proposal](https://gist.github.com/mtaufen/253309166e7d5aa9e9b560600a438447))
This daemon solves the problem of configuring the CIDR ranges for non-masquerade in a cluster (via iptables rules). Today, this is accomplished by passing a `--non-masquerade-cidr` flag to the Kubelet, which only allows one CIDR to be configured as non-masquerade. [RFC 1918](https://tools.ietf.org/html/rfc1918), however, defines three ranges (`10/8`, `172.16/12`, `192.168/16`) for the private IP address space.
This agent solves the problem of configuring the CIDR ranges for non-masquerade in a cluster (via iptables rules). Today, this is accomplished by passing a `--non-masquerade-cidr` flag to the Kubelet, which only allows one CIDR to be configured as non-masquerade. [RFC 1918](https://tools.ietf.org/html/rfc1918), however, defines three ranges (`10/8`, `172.16/12`, `192.168/16`) for the private IP address space.
Some users will want to communicate between these ranges without masquerade - for instance, if an organization's existing network uses the `10/8` range, they may wish to run their cluster and `Pod`s in `192.168/16` to avoid IP conflicts. They will also want these `Pod`s to be able to communicate efficiently (no masquerade) with each-other *and* with their existing network resources in `10/8`. This requires that every node in their cluster skips masquerade for both ranges.
We are trying to eliminate networking code from the Kubelet, so rather than extend the Kubelet to accept multiple CIDRs, [mtaufen/non-masquerade-daemon](https://github.com/mtaufen/non-masquerade-daemon) allows you to run a DaemonSet that configures a list of CIDRs as non-masquerade.
We are trying to eliminate networking code from the Kubelet, so rather than extend the Kubelet to accept multiple CIDRs, ip-masq-agent allows you to run a DaemonSet that configures a list of CIDRs as non-masquerade.
## Incubator
@ -64,12 +64,12 @@ See [RELEASE](RELEASE.md).
## Developing
Clone the repo to `$GOPATH/src/k8s.io/non-masquerade-daemon`.
Clone the repo to `$GOPATH/src/k8s.io/ip-masq-agent`.
The build tooling is based on [thockin/go-build-template](https://github.com/thockin/go-build-template).
Run `make` or `make build` to compile the non-masquerade-daemon. This will use a Docker image
to build the daemon, with the current directory volume-mounted into place. This
Run `make` or `make build` to compile the ip-masq-agent. This will use a Docker image
to build the agent, with the current directory volume-mounted into place. This
will store incremental state for the fastest possible build. Run `make
all-build` to build for all architectures.

Просмотреть файл

@ -1,6 +1,6 @@
# Release Process
The daemon is released on an as-needed basis. The process is as follows:
The ip-masq-agent is released on an as-needed basis. The process is as follows:
1. Someone must file an issue proposing a new release with a changelog since the last release.
1. All [OWNERS](OWNERS) must LGTM this release.
@ -9,4 +9,4 @@ The daemon is released on an as-needed basis. The process is as follows:
1. Runs `make push`, to build and push the container image for the release to `google_containers`.
1. Pushes the tag with `git push vx.x.x`.
1. The release issue is closed.
1. An announcement email is sent to `kubernetes-dev@googlegroups.com` with the subject `[ANNOUNCE] non-masquerade-daemon vx.x.x is released`.
1. An announcement email is sent to `kubernetes-dev@googlegroups.com` with the subject `[ANNOUNCE] ip-masq-agent vx.x.x is released`.

Просмотреть файл

@ -2,4 +2,5 @@ nonMasqueradeCIDRs:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
linkLocal: true
resyncInterval: 60s

Просмотреть файл

@ -28,21 +28,21 @@ import (
utilyaml "k8s.io/apimachinery/pkg/util/yaml"
"k8s.io/apiserver/pkg/util/logs"
"k8s.io/ip-masq-agent/cmd/ip-masq-agent/testing/fakefs"
utildbus "k8s.io/kubernetes/pkg/util/dbus"
utilexec "k8s.io/kubernetes/pkg/util/exec"
utiliptables "k8s.io/kubernetes/pkg/util/iptables"
"k8s.io/kubernetes/pkg/version/verflag"
"k8s.io/non-masquerade-daemon/cmd/non-masquerade-daemon/testing/fakefs"
"github.com/golang/glog"
)
const (
// name of nat chain for iptables masquerade rules
masqChain = utiliptables.Chain("NON-MASQUERADE-DAEMON")
masqChain = utiliptables.Chain("IP-MASQ-AGENT")
linkLocalCIDR = "169.254.0.0/16"
// path to a yaml or json file
configPath = "/etc/config/non-masquerade-daemon"
configPath = "/etc/config/ip-masq-agent"
)
// config object
@ -273,7 +273,7 @@ func (m *MasqDaemon) syncMasqRules() error {
// Feel free to dig around in iptables and see if you can figure out exactly why; I haven't had time to fully trace how it parses and handle subcommands.
// If you want to investigate, get the source via `git clone git://git.netfilter.org/iptables.git`, `git checkout v1.4.21` (the version I've seen this issue on,
// though it may also happen on others), and start with `git grep XT_EXTENSION_MAXNAMELEN`.
const postroutingJumpComment = "non-masquerade-daemon: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom " + string(masqChain) + " chain"
const postroutingJumpComment = "ip-masq-agent: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom " + string(masqChain) + " chain"
func (m *MasqDaemon) ensurePostroutingJump() error {
if _, err := m.iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting,
@ -285,7 +285,7 @@ func (m *MasqDaemon) ensurePostroutingJump() error {
return nil
}
const nonMasqRuleComment = `-m comment --comment "non-masquerade-daemon: cluster-local traffic should not be subject to MASQUERADE"`
const nonMasqRuleComment = `-m comment --comment "ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE"`
func writeNonMasqRule(lines *bytes.Buffer, cidr string) {
writeRule(lines, utiliptables.Append, masqChain,
@ -293,7 +293,7 @@ func writeNonMasqRule(lines *bytes.Buffer, cidr string) {
"-m", "addrtype", "!", "--dst-type", "LOCAL", "-d", cidr, "-j", "RETURN")
}
const masqRuleComment = `-m comment --comment "non-masquerade-daemon: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches)"`
const masqRuleComment = `-m comment --comment "ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches)"`
func writeMasqRule(lines *bytes.Buffer) {
writeRule(lines, utiliptables.Append, masqChain,

Просмотреть файл

@ -25,9 +25,9 @@ import (
"testing"
"time"
"k8s.io/ip-masq-agent/cmd/ip-masq-agent/testing/fakefs"
utiliptables "k8s.io/kubernetes/pkg/util/iptables"
iptest "k8s.io/kubernetes/pkg/util/iptables/testing"
"k8s.io/non-masquerade-daemon/cmd/non-masquerade-daemon/testing/fakefs"
)
// turn off glog logging during tests to avoid clutter in output
@ -235,7 +235,7 @@ func TestWriteNonMasqRule(t *testing.T) {
lines := bytes.NewBuffer(nil)
cidr := "10.0.0.0/8"
want := string(utiliptables.Append) + " " + string(masqChain) +
` -m comment --comment "non-masquerade-daemon: cluster-local traffic should not be subject to MASQUERADE"` +
` -m comment --comment "ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE"` +
" -m addrtype ! --dst-type LOCAL -d " + cidr + " -j RETURN\n"
writeNonMasqRule(lines, cidr)

Просмотреть файл

@ -1,17 +1,17 @@
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: non-masquerade-daemon
name: ip-masq-agent
namespace: kube-system
spec:
template:
metadata:
labels:
k8s-app: non-masquerade-daemon
k8s-app: ip-masq-agent
spec:
hostNetwork: true
containers:
- name: non-masquerade-daemon
- name: ip-masq-agent
image: gcr.io/google-containers/non-masquerade-daemon-amd64:v0.3.1
securityContext:
privileged: true
@ -22,9 +22,9 @@ spec:
- name: config
configMap:
# Note this ConfigMap must be created in the same namespace as the daemon pods - this spec uses kube-system
name: non-masquerade-daemon
name: ip-masq-agent
optional: true
items:
# The daemon looks for its config in a YAML file at /etc/config/non-masquerade-daemon
# The daemon looks for its config in a YAML file at /etc/config/ip-masq-agent
- key: config
path: non-masquerade-daemon
path: ip-masq-agent