This commit is contained in:
bartr 2021-02-17 18:16:15 -06:00
Родитель 251c7784d1
Коммит 6012316dab
9 изменённых файлов: 567 добавлений и 37 удалений

Просмотреть файл

@ -0,0 +1,9 @@
{
"name": "k8s-dev-cluster",
// Install extensions
"extensions": [
"davidanson.vscode-markdownlint",
"ms-azuretools.vscode-docker"
]
}

24
.github/ISSUE_TEMPLATE/bug.md поставляемый Normal file
Просмотреть файл

@ -0,0 +1,24 @@
---
name: Bug report
about: For reporting an issue in code or documentation for improvement
labels: Bug
---
# Bug Report
## Description
- A clear description of the bug
## Expected Behavior
- A clear description of what you expected
## Reproduce
1. Go to '...'
2. See error
## Additional Context

22
.github/ISSUE_TEMPLATE/feature.md поставляемый Normal file
Просмотреть файл

@ -0,0 +1,22 @@
---
name: Feature Request
about: Suggest an idea for this project
labels: Design Review, Enhancement
---
# Problem Statement
-
## Proposed Solution
-
## Alternative Proposals
-
## Additional Context
-

18
.github/PULL_REQUEST_TEMPLATE.md поставляемый Normal file
Просмотреть файл

@ -0,0 +1,18 @@
# Purpose of PR
## Type of PR
- [ ] Documentation changes
- [ ] Code changes
- [ ] Test changes
- [ ] CI-CD changes
## Validation
- [ ] Unit tests updated and ran successfully
- [ ] Documentation updated
## Issues Closed or Referenced
- Closes #issue_number (this will automatically close the issue when the PR closes)
- References #issue_number (this references the issue but does not close with PR)

74
CONTRIBUTING.md Normal file
Просмотреть файл

@ -0,0 +1,74 @@
# Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit <https://cla.microsoft.com>.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
- [Code of Conduct](#coc)
- [Issues and Bugs](#issue)
- [Feature Requests](#feature)
- [Submission Guidelines](#submit)
## Code of Conduct
Help us keep this project open and inclusive. Please read and follow our [Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
## Found an Issue
If you find a bug in the source code or a mistake in the documentation, you can help us by
[submitting an issue](#submit-issue) to the GitHub Repository. Even better, you can
[submit a Pull Request](#submit-pr) with a fix.
## Want a Feature
You can *request* a new feature by [submitting an issue](#submit-issue) to the GitHub
Repository. If you would like to *implement* a new feature, please submit an issue with
a proposal for your work first, to be sure that we can use it.
- **Small Features** can be crafted and directly [submitted as a Pull Request](#submit-pr).
## Submission Guidelines
### Submitting an Issue
Before you submit an issue, search the archive, maybe your question was already answered.
If your issue appears to be a bug, and hasn't been reported, open a new issue.
Help us to maximize the effort we can spend fixing issues and adding new
features, by not reporting duplicate issues. Providing the following information will increase the
chances of your issue being dealt with quickly:
- **Overview of the Issue** - if an error is being thrown a non-minified stack trace helps
- **Version** - what version is affected (e.g. 0.1.2)
- **Motivation for or Use Case** - explain what are you trying to do and why the current behavior is a bug for you
- **Browsers and Operating System** - is this a problem with all browsers?
- **Reproduce the Error** - provide a live example or a unambiguous set of steps
- **Related Issues** - has a similar issue been reported before?
- **Suggest a Fix** - if you can't fix the bug yourself, perhaps you can point to what might be
causing the problem (line of code or commit)
You can file new issues by providing the above information at the corresponding repository's issues link: <https://github.com/[organization-name]/[repository-name]/issues/new]>
### Submitting a Pull Request (PR)
Before you submit your Pull Request (PR) consider the following guidelines:
- Search the repository <https://github.com/[organization-name]/[repository-name]/pulls> for an open or closed PR that relates to your submission. You don't want to duplicate effort.
- Make your changes in a new git fork:
- Commit your changes using a descriptive commit message
- Push your fork to GitHub:
- In GitHub, create a pull request
- If we suggest changes then:
- Make the required updates.
- Rebase your fork and force push to your GitHub repository (this will update your Pull Request):
```shell
git rebase master -i
git push -f
```
That's it! Thank you for your contribution!

211
README.md
Просмотреть файл

@ -1,33 +1,204 @@
# Project
# Creating a Kubernetes Dev Cluster
> This repo has been populated by an initial template to help get you started. Please
> make sure to update the content to build a great experience for community-building.
> For information on setting up a production Kubernetes cluster on Azure please see [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/)
As the maintainer of this project, please make a few updates:
This script sets up a single-node Kubernetes development cluster on an Azure VM. While this is not intended to be a production cluster ([AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/) is a more complete solution). The approach is similar to using [minkube](https://minikube.sigs.k8s.io/docs/) or [kind](https://kind.sigs.k8s.io/docs/) but it's a complete Kubernetes deployment using [kubeadm](https://kubernetes.io/docs/tasks/tools/).
- Improving this README.MD file to provide a great experience
- Updating SUPPORT.MD with content about this project's support experience
- Understanding the security reporting process in SECURITY.MD
- Remove this section from the README
We have found that the `kubeadm` approach helps engineers learn more about what is happening under the covers with Kubernetes and AKS and it's a great next step from `minikube` or `kind`. It is also a great way for developers to debug applications as they have full access to Kubernetes and can quickly experiment and debug. There are also potential cost savings as a developer can run a dedicated Kubernetes "cluster" on a single VM.
> The scripts and instructions will work with other VM hosts with minimal changes
## More Information
- Explanation of the steps in this [script](https://github.com/retaildevcrews/k8s-quickstart/tree/main/02-Dev-Cluster-Setup)
- Kubernetes [best practices](https://kubernetes.io/docs/setup/best-practices/)
- Bootstrapping clusters with [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/)
- Azure Kubernetes Service [(AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/)
## Prerequisites
- Bash or Windows cmd shell
- Azure CLI ([download](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest))
> Visual Studio Code Remote is [highly recommended](https://code.visualstudio.com/docs/remote/ssh)
## Host VM Requirements
- tested on `Ubuntu 18.04 LTS`
- minimum 2 cores with 2 GB RAM
## Setup
### Login to Azure
```bash
az account list -o table
# login to Azure (if necessary)
az login
# select subscription (if necesary)
az account set -s YourSubscriptionName
```
### Installation (bash)
> From a bash terminal
```bash
# change your resource group name and location if desired
export AKDC_LOC=centralus
export AKDC_RG=akdc
# Create a resource group
az group create -l $AKDC_LOC -n $AKDC_RG
# download setup script
# replace user name
curl https://raw.githubusercontent.com/retaildevcrews/akdc/main/scripts/auto.sh | sed s/ME=akdc/ME=$USER/ > akdc.sh
# create an Ubuntu VM and install k8s
# save IP address into the AKDC_IP env var
export AKDC_IP=$(az vm create \
-g $AKDC_RG \
--admin-username $USER \
-n akdc \
--size standard_d2s_v3 \
--nsg-rule SSH \
--image Canonical:UbuntuServer:18.04-LTS:latest \
--os-disk-size-gb 128 \
--generate-ssh-keys \
--query publicIpAddress -o tsv \
--custom-data akdc.sh)
rm akdc.sh
echo $AKDC_IP
# (optional) open NodePort range on NSG
az network nsg rule create -g $AKDC_RG \
--nsg-name akdcNSG --access allow \
--description "AKDC Ports" \
--destination-port-ranges 30000-32767 \
--protocol tcp \
-n AkdcPorts --priority 1200
# SSH into the VM
ssh ${AKDC_IP}
```
### Installation (Windows cmd)
> From a Windows cmd prompt
```bash
# change your resource group name and location if desired
set AKDC_LOC=centralus
set AKDC_RG=akdc
# Create a resource group
az group create -l %AKDC_LOC% -n %AKDC_RG%
# download setup script
# replace user name
curl https://raw.githubusercontent.com/retaildevcrews/akdc/main/scripts/auto.sh | sed s/ME=akdc/ME=%USERNAME%/ > akdc.sh
# create an Ubuntu VM and install k8s
# save IP address into the AKDC_IP env var
for /f %f in (' ^
az vm create ^
-g %AKDC_RG% ^
--admin-username %USERNAME% ^
-n akdc ^
--size standard_d2s_v3 ^
--nsg-rule SSH ^
--image Canonical:UbuntuServer:18.04-LTS:latest ^
--os-disk-size-gb 128 ^
--generate-ssh-keys ^
--query publicIpAddress -o tsv ^
--custom-data akdc.sh') ^
do set AKDC_IP=%f
del akdc.sh
echo %AKDC_IP%
# (optional) open NodePort range on NSG
az network nsg rule create -g %AKDC_RG% ^
--nsg-name akdcNSG --access allow ^
--description "AKDC Ports" ^
--destination-port-ranges 30000-32767 ^
--protocol tcp ^
-n AkdcPorts --priority 1200
ssh %AKDC_IP%
```
## Validation
> From a bash shell in the VM via SSH
The first time you SSH into the VM, you might get the below error - it is safe to ignore.
- Command 'kubectl' not found, but can be installed with:
- sudo snap install kubectl
```bash
# this will tell you when the user data script is done
tail -f status
# (optional) install oh-my-bash kubectl aliases
sed -i "s/^plugins=($/plugins=(\n kubectl/g" .bashrc
source .bashrc
# make sure everything is up to date
sudo apt update
sudo apt dist-upgrade -y
# your single-node k8s dev cluster is now ready
kubectl get all --all-namespaces
```
## Reset cluster to start over
You can usually reset your k8s cluster to a clean install with `kubeadm reset` If reset fails, you will need to delete the VM and create a new one.
> From a bash shell in the VM via SSH
```bash
curl https://raw.githubusercontent.com/retaildevcrews/akdc/main/scripts/reset.sh > reset.sh
chmod +x reset.sh
# reset your cluster
./reset.sh
```
## Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [Microsoft Contributor License Agreement](https://cla.opensource.microsoft.com).
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
This project may contain trademarks or logos for projects, products, or services.
Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.

Просмотреть файл

@ -1,25 +1,11 @@
# TODO: The maintainer of this repo has not yet edited this file
**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?
- **No CSS support:** Fill out this template with information about how to file issues and get help.
- **Yes CSS support:** Fill out an intake form at [aka.ms/spot](https://aka.ms/spot). CSS will work with/help you to determine next steps. More details also available at [aka.ms/onboardsupport](https://aka.ms/onboardsupport).
- **Not sure?** Fill out a SPOT intake as though the answer were "Yes". CSS will help you decide.
*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*
# Support
## How to file issues and get help
This project uses GitHub Issues to track bugs and feature requests. Please search the existing
issues before filing new issues to avoid duplicates. For new issues, file your bug or
feature request as a new Issue.
This project uses GitHub Issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your bug or feature request as a new Issue.
For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE
FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
CHANNEL. WHERE WILL YOU HELP PEOPLE?**.
For help and questions about using this project, please open an issue.
## Microsoft Support Policy
Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
Support for this project is limited to the resources listed above.

209
scripts/auto.sh Normal file
Просмотреть файл

@ -0,0 +1,209 @@
#!/bin/bash
##################################
# automatically replaced with $USER (bash) or %USERNAME% (Windows)
export ME=akdc
##################################
# make some directories we will need
mkdir -p /home/${ME}/.ssh
mkdir -p /home/${ME}/.kube
mkdir -p /home/${ME}/bin
mkdir -p /home/${ME}/.local/bin
mkdir -p /home/${ME}/.k9s
mkdir -p /etc/containerd
mkdir -p /etc/systemd/system/docker.service.d
mkdir -p /etc/docker
cd /home/${ME}
echo "starting (1/15)" > status
cp /usr/share/zoneinfo/America/Chicago /etc/localtime
# create / add to groups
groupadd docker
usermod -aG sudo ${ME}
usermod -aG admin ${ME}
usermod -aG docker ${ME}
gpasswd -a ${ME} sudo
echo "${ME} ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/90-cloud-init-users
# oh my bash
git clone --depth=1 https://github.com/ohmybash/oh-my-bash.git .oh-my-bash
cp .oh-my-bash/templates/bashrc.osh-template .bashrc
# add to .bashrc
echo "" >> .bashrc
echo "alias k='kubectl'" >> .bashrc
echo "alias kga='kubectl get all'" >> .bashrc
echo "alias kgaa='kubectl get all --all-namespaces'" >> .bashrc
echo "alias kaf='kubectl apply -f'" >> .bashrc
echo "alias kdelf='kubectl delete -f'" >> .bashrc
echo "alias kl='kubectl logs'" >> .bashrc
echo "alias kccc='kubectl config current-context'" >> .bashrc
echo "alias kcgc='kubectl config get-contexts'" >> .bashrc
echo "export GO111MODULE=on" >> .bashrc
echo "alias ipconfig='ip -4 a show eth0 | grep inet | sed \"s/inet//g\" | sed \"s/ //g\" | cut -d / -f 1'" >> .bashrc
echo 'export PIP=$(ipconfig | tail -n 1)' >> .bashrc
echo 'export PATH="$PATH:$HOME/.dotnet/tools:$HOME/go/bin"' >> .bashrc
echo 'source /usr/share/bash-completion/bash_completion' >> .bashrc
echo 'source <(kubectl completion bash)' >> .bashrc
echo 'complete -F __start_kubectl k' >> .bashrc
# change ownership of home directory
chown -R ${ME}:${ME} /home/${ME}
# set the permissions on .ssh
chmod 700 /home/${ME}/.ssh
chmod 600 /home/${ME}/.ssh/*
# set the IP address
export PIP=$(ip -4 a show eth0 | grep inet | sed "s/inet//g" | sed "s/ //g" | cut -d '/' -f 1 | tail -n 1)
echo "updating (2/15)" >> status
apt-get update
echo "install base (3/15)" >> status
apt-get install -y apt-utils dialog apt-transport-https ca-certificates curl software-properties-common
echo "add repos (4/15)" >> status
# add Docker repo
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# add dotnet repo
echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$(lsb_release -cs)-prod $(lsb_release -cs) main" > /etc/apt/sources.list.d/dotnetdev.list
# add Azure CLI repo
curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/microsoft.asc.gpg
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/azure-cli.list
# add kubenetes repo
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update
echo "install utils (5/15)" >> status
apt-get install -y git wget nano jq zip unzip httpie dnsutils
echo "install libs (6/15)" >> status
apt-get install -y libssl-dev libffi-dev python-dev build-essential lsb-release gnupg-agent bash-completion
echo "install Azure CLI (7/15)" >> status
apt-get install -y azure-cli
echo " (optional) you can run az login and az account set -s YourSubscriptionName now" >> status
echo "install k8s (8/15)" >> status
apt-get install -y containerd.io kubectl kubelet kubeadm kubernetes-cni
# Set up the Docker daemon to use systemd
cat <<'EOF' > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# Setup required sysctl params
cat <<EOF >> /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# set network for containerd
cat <<EOF >> /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
# config crictl to use containerd
cat <<EOF >> /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: false
pull-image-on-create: true
EOF
# Apply sysctl params
sysctl --system
# apply network changes
modprobe overlay
modprobe br_netfilter
# Configure containerd
containerd config default > /etc/containerd/config.toml
# Restart containerd
systemctl restart containerd
echo "pulling images (9/15)" >> status
kubeadm config images pull
echo "kubeadm init (10/15)" >> status
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address $PIP --cri-socket /run/containerd/containerd.sock
echo "k8s setup (11/15)" >> status
# copy config file
cp -i /etc/kubernetes/admin.conf /home/${ME}/.kube/config
# add flannel network overlay
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml --namespace=kube-system
# add the taint to schedule normal pods on the control plane
# this let you run a "one node" cluster for development
kubectl taint nodes --all node-role.kubernetes.io/master-
# patch kube-proxy for metal LB
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
sed -e 's/mode: ""/mode: "ipvs"/' | \
kubectl apply -f - -n kube-system
# change ownership
chown -R ${ME}:${ME} /home/${ME}
echo "install docker (12/15)" >> status
apt-get install -y docker-ce docker-ce-cli
# upgrade Ubuntu
echo "upgrade (13/15)" >> status
apt-get dist-upgrade -y
apt-mark hold kubelet kubeadm kubectl
# CLI for CRI-compatible container runtimes
echo "install crictl (14/15)" >> status
VERSION=$(curl -i https://github.com/kubernetes-sigs/cri-tools/releases/latest | grep "location: https://github.com/" | rev | cut -f 1 -d / | rev | sed 's/\r//')
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
tar -zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz
echo "install tools (15/15)" >> status
VERSION=$(curl -i https://github.com/derailed/k9s/releases/latest | grep "location: https://github.com/" | rev | cut -f 1 -d / | rev | sed 's/\r//')
wget https://github.com/derailed/k9s/releases/download/$VERSION/k9s_Linux_x86_64.tar.gz
tar -zxvf k9s_Linux_x86_64.tar.gz -C /usr/local/bin
rm -f k9s_Linux_x86_64.tar.gz
# kubectl auto complete
kubectl completion bash > /etc/bash_completion.d/kubectl
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
complete -F __start_kubectl k
# install jp (jmespath)
VERSION=$(curl -i https://github.com/jmespath/jp/releases/latest | grep "location: https://github.com/" | rev | cut -f 1 -d / | rev | sed 's/\r//')
wget https://github.com/jmespath/jp/releases/download/$VERSION/jp-linux-amd64 -O /usr/local/bin/jp
chmod +x /usr/local/bin/jp
echo "done" >> status

17
scripts/reset.sh Normal file
Просмотреть файл

@ -0,0 +1,17 @@
#!/bin/bash
sudo kubeadm reset -f
sudo kubeadm config images pull
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address $PIP --cri-socket /run/containerd/containerd.sock
# copy config file
sudo rm ~/.kube/config
sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config
sudo chown -R ${USER}:${USER} ~/.kube
# add flannel network overlay
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml --namespace=kube-system
# add the taint to schedule normal pods on the control plane
# this let you run a "one node" cluster for development
kubectl taint nodes --all node-role.kubernetes.io/master-