feat: allow custom os images from external subs via Shared Image Galleries (#913)

* chore: allow custom os images from alternate subs

* fix: wrong sub used for creation

* chore: switch from raw subs to shared image gallery

* fix: typo in types

* fix: imageVersion -> version

* chore: sync bits for debug

* fix: bug in resourceID for VMSS

* test: unit test shared images

* chore: allow custom os images from alternate subs

* fix: wrong sub used for creation

* chore: switch from raw subs to shared image gallery

* fix: typo in types

* fix: imageVersion -> version

* chore: sync bits for debug

* fix: bug in resourceID for VMSS

* test: unit test shared images

* doc: shared image galleries

* test: conversions with custom agent pool

* fix: spelling typo

* chore: allow custom os images from alternate subs

* fix: wrong sub used for creation

* chore: switch from raw subs to shared image gallery

* fix: typo in types

* fix: imageVersion -> version

* chore: sync bits for debug

* fix: bug in resourceID for VMSS

* test: unit test shared images

* doc: shared image galleries

* test: coverage

* fix: style

* test: coverage

* fix: typo

* fix: test either condition failing
This commit is contained in:
Ace Eldeib 2019-07-03 16:25:18 -07:00 коммит произвёл Azure Kubernetes Service Bot
Родитель 54155a3971
Коммит 59103c3e8e
14 изменённых файлов: 916 добавлений и 19 удалений

Просмотреть файл

@ -512,6 +512,9 @@ Custom YAML specifications can be configured for kube-scheduler, kube-controller
| vnetCidr | no | Specifies the VNET cidr when using a custom VNET ([bring your own VNET examples](../../examples/vnet)). This VNET cidr should include both the master and the agent subnets. |
| imageReference.name | no | The name of the Linux OS image. Needs to be used in conjunction with resourceGroup, below |
| imageReference.resourceGroup | no | Resource group that contains the Linux OS image. Needs to be used in conjunction with name, above |
| imageReference.subscriptionId | no | ID of subscription containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario. |
| imageReference.gallery | no | Name of Shared Image Gallery containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario. |
| imageReference.version | no | Version containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario. |
| distro | no | Specifies the masters' Linux distribution. Currently supported values are: `ubuntu`, `ubuntu-18.04`, `aks-ubuntu-16.04` (previously `aks`), `aks-ubuntu-18.04`, and `coreos` (CoreOS support is currently experimental - [Example of CoreOS Master with CoreOS Agents](../../examples/coreos/kubernetes-coreos.json)). For Azure Public Cloud and Azure China Cloud, defaults to `aks-ubuntu-16.04`. For other Sovereign Clouds, the default is `ubuntu-16.04` (There is a [known issue](https://github.com/Azure/aks-engine/issues/761) with `ubuntu-18.04` + Azure CNI). `aks-ubuntu-16.04` is a custom image based on `ubuntu-16.04` that comes with pre-installed software necessary for Kubernetes deployments (Azure Public Cloud and Azure China Cloud only for now). |
| customFiles | no | The custom files to be provisioned to the master nodes. Defined as an array of json objects with each defined as `"source":"absolute-local-path", "dest":"absolute-path-on-masternodes"`.[See examples](../../examples/customfiles) |
| availabilityProfile | no | Supported values are `AvailabilitySet` (default) and `VirtualMachineScaleSets` (still under development: upgrade not supported; requires Kubernetes clusters version 1.10+ and agent pool availabilityProfile must also be `VirtualMachineScaleSets`). When MasterProfile is using `VirtualMachineScaleSets`, to SSH into a master node, you need to use `ssh -p 50001` instead of port 22. |

Просмотреть файл

@ -10,6 +10,7 @@
|Kata Containers Runtime|Alpha|`vlabs`|[kubernetes-kata-containers.json](../../examples/kubernetes-kata-containers.json)|[Description](#feat-kata-containers)|
|Private Cluster|Alpha|`vlabs`|[kubernetes-private-cluster.json](../../examples/kubernetes-config/kubernetes-private-cluster.json)|[Description](#feat-private-cluster)|
|Azure Key Vault Encryption|Alpha|`vlabs`|[kubernetes-keyvault-encryption.json](../../examples/kubernetes-config/kubernetes-keyvault-encryption.json)|[Description](#feat-keyvault-encryption)|
|Shared Image Gallery images|Alpha|`vlabs`|[custom-shared-image.json](../../examples/custom-shared-image.json)|[Description](#feat-shared-image-gallery)|
<a name="feat-kubernetes-msi"></a>
@ -454,3 +455,61 @@ To get `objectId` of the service principal:
```console
az ad sp list --spn <YOUR SERVICE PRINCIPAL appId>
```
<a name="feat-shared-image-gallery"></a>
## Use a Shared Image Gallery image
This is possible by specifying `imageReference` under `masterProfile` or on a given `agentPoolProfile`. It also requires setting the distro to an appropriate value (`ubuntu` or `coreos`). When using `imageReference` with Shared Image Galleries, provide an image name and version, as well as the resource group, subscription, and name of the gallery. Example:
```json
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"masterProfile": {
"imageReference": {
"name": "linuxvm",
"resourceGroup": "sig",
"subscriptionID": "00000000-0000-0000-0000-000000000000",
"gallery": "siggallery",
"version": "0.0.1"
},
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v3"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"imageReference": {
"name": "linuxvm",
"resourceGroup": "sig",
"subscriptionID": "00000000-0000-0000-0000-000000000000",
"gallery": "siggallery",
"version": "0.0.1"
},
"vmSize": "Standard_D2_v3",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}
}
}
```

Просмотреть файл

@ -0,0 +1,50 @@
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"masterProfile": {
"imageReference": {
"name": "linuxvm",
"resourceGroup": "sig",
"subscriptionID": "00000000-0000-0000-0000-000000000000",
"gallery": "siggallery",
"version": "0.0.1"
},
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v3"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"imageReference": {
"name": "linuxvm",
"resourceGroup": "sig",
"subscriptionID": "00000000-0000-0000-0000-000000000000",
"gallery": "siggallery",
"version": "0.0.1"
},
"vmSize": "Standard_D2_v3",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}
}
}

Просмотреть файл

@ -930,6 +930,9 @@ func convertMasterProfileToVLabs(api *MasterProfile, vlabsProfile *vlabs.MasterP
vlabsProfile.ImageRef = &vlabs.ImageReference{}
vlabsProfile.ImageRef.Name = api.ImageRef.Name
vlabsProfile.ImageRef.ResourceGroup = api.ImageRef.ResourceGroup
vlabsProfile.ImageRef.SubscriptionID = api.ImageRef.SubscriptionID
vlabsProfile.ImageRef.Gallery = api.ImageRef.Gallery
vlabsProfile.ImageRef.Version = api.ImageRef.Version
}
vlabsProfile.AvailabilityProfile = api.AvailabilityProfile
vlabsProfile.AgentSubnet = api.AgentSubnet
@ -1050,6 +1053,9 @@ func convertAgentPoolProfileToVLabs(api *AgentPoolProfile, p *vlabs.AgentPoolPro
p.ImageRef = &vlabs.ImageReference{}
p.ImageRef.Name = api.ImageRef.Name
p.ImageRef.ResourceGroup = api.ImageRef.ResourceGroup
p.ImageRef.SubscriptionID = api.ImageRef.SubscriptionID
p.ImageRef.Gallery = api.ImageRef.Gallery
p.ImageRef.Version = api.ImageRef.Version
}
p.Role = vlabs.AgentPoolProfileRole(api.Role)
}

Просмотреть файл

@ -639,6 +639,13 @@ func getDefaultContainerService() *ContainerService {
FQDN: "blueorange.westus2.com",
OSType: "Linux",
Subnet: "sampleSubnet",
ImageRef: &ImageReference{
Name: "testImage",
ResourceGroup: "testRg",
SubscriptionID: "testSub",
Gallery: "testGallery",
Version: "0.0.1",
},
},
},
},

Просмотреть файл

@ -941,6 +941,9 @@ func convertVLabsMasterProfile(vlabs *vlabs.MasterProfile, api *MasterProfile) {
api.ImageRef = &ImageReference{}
api.ImageRef.Name = vlabs.ImageRef.Name
api.ImageRef.ResourceGroup = vlabs.ImageRef.ResourceGroup
api.ImageRef.SubscriptionID = vlabs.ImageRef.SubscriptionID
api.ImageRef.Gallery = vlabs.ImageRef.Gallery
api.ImageRef.Version = vlabs.ImageRef.Version
}
api.AvailabilityProfile = vlabs.AvailabilityProfile
@ -1067,6 +1070,9 @@ func convertVLabsAgentPoolProfile(vlabs *vlabs.AgentPoolProfile, api *AgentPoolP
api.ImageRef = &ImageReference{}
api.ImageRef.Name = vlabs.ImageRef.Name
api.ImageRef.ResourceGroup = vlabs.ImageRef.ResourceGroup
api.ImageRef.SubscriptionID = vlabs.ImageRef.SubscriptionID
api.ImageRef.Gallery = vlabs.ImageRef.Gallery
api.ImageRef.Version = vlabs.ImageRef.Version
}
api.Role = AgentPoolProfileRole(vlabs.Role)
}

Просмотреть файл

@ -846,6 +846,21 @@ func TestConvertVLabsContainerService(t *testing.T) {
FQDN: "blueorange.westus2.azureapp.com",
OSType: "Linux",
},
{
Name: "sampleAgent-public",
Count: 2,
VMSize: "sampleVM",
DNSPrefix: "blueorange",
FQDN: "blueorange.westus2.com",
OSType: "Linux",
ImageRef: &vlabs.ImageReference{
Name: "testImage",
ResourceGroup: "testRg",
SubscriptionID: "testSub",
Gallery: "testGallery",
Version: "0.0.1",
},
},
},
MasterProfile: &vlabs.MasterProfile{
Count: 1,

Просмотреть файл

@ -487,8 +487,11 @@ type MasterProfile struct {
// ImageReference represents a reference to an Image resource in Azure.
type ImageReference struct {
Name string `json:"name,omitempty"`
ResourceGroup string `json:"resourceGroup,omitempty"`
Name string `json:"name,omitempty"`
ResourceGroup string `json:"resourceGroup,omitempty"`
SubscriptionID string `json:"subscriptionId,omitempty"`
Gallery string `json:"gallery,omitempty"`
Version string `json:"version,omitempty"`
}
// ExtensionProfile represents an extension definition
@ -1196,6 +1199,16 @@ func (p *Properties) GetMasterFQDN() string {
return p.MasterProfile.FQDN
}
// HasImageRef returns true if the customer brought os image
func (m *MasterProfile) HasImageRef() bool {
return m.ImageRef != nil && len(m.ImageRef.Name) > 0 && len(m.ImageRef.ResourceGroup) > 0
}
// HasImageGallery returns true if the customer brought os image from Shared Image Gallery
func (m *MasterProfile) HasImageGallery() bool {
return m.ImageRef != nil && len(m.ImageRef.SubscriptionID) > 0 && len(m.ImageRef.Gallery) > 0 && len(m.ImageRef.Version) > 0
}
// IsCustomVNET returns true if the customer brought their own VNET
func (m *MasterProfile) IsCustomVNET() bool {
return len(m.VnetSubnetID) > 0
@ -1313,6 +1326,18 @@ func (m *MasterProfile) GetCosmosEndPointURI() string {
return ""
}
// HasImageRef returns true if the customer brought os image
func (a *AgentPoolProfile) HasImageRef() bool {
imageRef := a.ImageRef
return imageRef != nil && len(imageRef.Name) > 0 && len(imageRef.ResourceGroup) > 0
}
// HasImageGallery returns true if the customer brought os image from Shared Image Gallery
func (a *AgentPoolProfile) HasImageGallery() bool {
imageRef := a.ImageRef
return imageRef != nil && len(imageRef.SubscriptionID) > 0 && len(imageRef.Gallery) > 0 && len(imageRef.Version) > 0
}
// IsCustomVNET returns true if the customer brought their own VNET
func (a *AgentPoolProfile) IsCustomVNET() bool {
return len(a.VnetSubnetID) > 0

Просмотреть файл

@ -1005,6 +1005,130 @@ func TestPropertiesIsHostedMasterProfile(t *testing.T) {
}
}
func TestPropertiesMasterCustomOS(t *testing.T) {
cases := []struct {
name string
p Properties
expectedRef bool
expectedGallery bool
}{
{
name: "valid shared gallery image",
p: Properties{
MasterProfile: &MasterProfile{
ImageRef: &ImageReference{
Name: "testImage",
ResourceGroup: "testRg",
SubscriptionID: "testSub",
Gallery: "testGallery",
Version: "0.0.1",
},
},
},
expectedRef: true,
expectedGallery: true,
},
{
name: "valid resource group image",
p: Properties{
MasterProfile: &MasterProfile{
ImageRef: &ImageReference{
Name: "testImage",
ResourceGroup: "testRg",
},
},
},
expectedRef: true,
expectedGallery: false,
},
{
name: "valid no custom image",
p: Properties{
MasterProfile: &MasterProfile{
ImageRef: nil,
},
},
expectedRef: false,
expectedGallery: false,
},
}
for _, c := range cases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
if c.p.MasterProfile.HasImageRef() != c.expectedRef || c.p.MasterProfile.HasImageGallery() != c.expectedGallery {
t.Fatalf("expected HasImageRef() to return %t but instead returned %t, HasImageGallery() expected: %t but actual: %t", c.expectedRef, c.p.MasterProfile.HasImageRef(), c.p.MasterProfile.HasImageGallery(), c.expectedGallery)
}
})
}
}
func TestPropertiesAgentCustomOS(t *testing.T) {
cases := []struct {
name string
p Properties
expectedRef bool
expectedGallery bool
}{
{
name: "valid shared gallery image",
p: Properties{
AgentPoolProfiles: []*AgentPoolProfile{
{
ImageRef: &ImageReference{
Name: "testImage",
ResourceGroup: "testRg",
SubscriptionID: "testSub",
Gallery: "testGallery",
Version: "0.0.1",
},
},
},
},
expectedRef: true,
expectedGallery: true,
},
{
name: "valid resource group image",
p: Properties{
AgentPoolProfiles: []*AgentPoolProfile{
{
ImageRef: &ImageReference{
Name: "testImage",
ResourceGroup: "testRg",
},
},
},
},
expectedRef: true,
expectedGallery: false,
},
{
name: "valid no custom image",
p: Properties{
AgentPoolProfiles: []*AgentPoolProfile{
{
ImageRef: nil,
},
},
},
expectedRef: false,
expectedGallery: false,
},
}
for _, c := range cases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
if c.p.AgentPoolProfiles[0].HasImageRef() != c.expectedRef || c.p.AgentPoolProfiles[0].HasImageGallery() != c.expectedGallery {
t.Fatalf("expected HasImageRef() to return %t but instead returned %t, HasImageGallery() expected: %t but actual: %t", c.expectedRef, c.p.AgentPoolProfiles[0].HasImageRef(), c.expectedGallery, c.p.AgentPoolProfiles[0].HasImageGallery())
}
})
}
}
func TestMasterAvailabilityProfile(t *testing.T) {
cases := []struct {
name string

Просмотреть файл

@ -404,8 +404,11 @@ type MasterProfile struct {
// ImageReference represents a reference to an Image resource in Azure.
type ImageReference struct {
Name string `json:"name,omitempty"`
ResourceGroup string `json:"resourceGroup,omitempty"`
Name string `json:"name,omitempty"`
ResourceGroup string `json:"resourceGroup,omitempty"`
SubscriptionID string `json:"subscriptionId,omitempty"`
Gallery string `json:"gallery,omitempty"`
Version string `json:"version,omitempty"`
}
// ExtensionProfile represents an extension definition

Просмотреть файл

@ -149,7 +149,6 @@ func CreateMasterVM(cs *api.ContainerService) VirtualMachineARM {
storageProfile := &compute.StorageProfile{}
imageRef := cs.Properties.MasterProfile.ImageRef
useMasterCustomImage := imageRef != nil && len(imageRef.Name) > 0 && len(imageRef.ResourceGroup) > 0
etcdSizeGB, _ := strconv.Atoi(kubernetesConfig.EtcdDiskSizeGB)
if !cs.Properties.MasterProfile.HasCosmosEtcd() {
dataDisk := compute.DataDisk{
@ -168,8 +167,12 @@ func CreateMasterVM(cs *api.ContainerService) VirtualMachineARM {
}
}
imgReference := &compute.ImageReference{}
if useMasterCustomImage {
imgReference.ID = to.StringPtr("[resourceId(parameters('osImageResourceGroup'), 'Microsoft.Compute/images', parameters('osImageName'))]")
if cs.Properties.MasterProfile.HasImageRef() {
if cs.Properties.MasterProfile.HasImageGallery() {
imgReference.ID = to.StringPtr(fmt.Sprintf("[concat('/subscriptions/', '%s', '/resourceGroups/', parameters('osImageResourceGroup'), '/providers/Microsoft.Compute/galleries/', '%s', '/images/', parameters('osImageName'), '/versions/', '%s')]", imageRef.SubscriptionID, imageRef.Gallery, imageRef.Version))
} else {
imgReference.ID = to.StringPtr("[resourceId(parameters('osImageResourceGroup'), 'Microsoft.Compute/images', parameters('osImageName'))]")
}
} else {
imgReference.Offer = to.StringPtr("[parameters('osImageOffer')]")
imgReference.Publisher = to.StringPtr("[parameters('osImagePublisher')]")
@ -475,10 +478,15 @@ func createAgentAvailabilitySetVM(cs *api.ContainerService, profile *api.AgentPo
} else {
imageRef := profile.ImageRef
useAgentCustomImage := imageRef != nil && len(imageRef.Name) > 0 && len(imageRef.ResourceGroup) > 0
if useAgentCustomImage {
storageProfile.ImageReference = &compute.ImageReference{
ID: to.StringPtr(fmt.Sprintf("[resourceId(variables('%[1]sosImageResourceGroup'), 'Microsoft.Compute/images', variables('%[1]sosImageName'))]", profile.Name)),
if profile.HasImageRef() {
if profile.HasImageGallery() {
storageProfile.ImageReference = &compute.ImageReference{
ID: to.StringPtr(fmt.Sprintf("[concat('/subscriptions/', '%s', '/resourceGroups/', parameters('%sosImageResourceGroup'), '/providers/Microsoft.Compute/galleries/', '%s', '/images/', parameters('%sosImageName'), '/versions/', '%s')]", imageRef.SubscriptionID, profile.Name, imageRef.Gallery, profile.Name, imageRef.Version)),
}
} else {
storageProfile.ImageReference = &compute.ImageReference{
ID: to.StringPtr(fmt.Sprintf("[resourceId(variables('%[1]sosImageResourceGroup'), 'Microsoft.Compute/images', variables('%[1]sosImageName'))]", profile.Name)),
}
}
} else {
storageProfile.ImageReference = &compute.ImageReference{

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

Просмотреть файл

@ -221,7 +221,6 @@ func CreateMasterVMSS(cs *api.ContainerService) VirtualMachineScaleSetARM {
storageProfile := compute.VirtualMachineScaleSetStorageProfile{}
imageRef := masterProfile.ImageRef
useMasterCustomImage := imageRef != nil && len(imageRef.Name) > 0 && len(imageRef.ResourceGroup) > 0
etcdSizeGB, _ := strconv.Atoi(k8sConfig.EtcdDiskSizeGB)
dataDisk := compute.VirtualMachineScaleSetDataDisk{
CreateOption: compute.DiskCreateOptionTypesEmpty,
@ -232,8 +231,12 @@ func CreateMasterVMSS(cs *api.ContainerService) VirtualMachineScaleSetARM {
dataDisk,
}
imgReference := &compute.ImageReference{}
if useMasterCustomImage {
imgReference.ID = to.StringPtr("[resourceId(parameters('osImageResourceGroup'), 'Microsoft.Compute/images', parameters('osImageName'))]")
if masterProfile.HasImageRef() {
if masterProfile.HasImageGallery() {
imgReference.ID = to.StringPtr(fmt.Sprintf("[concat('/subscriptions/', '%s', '/resourceGroups/', parameters('osImageResourceGroup'), '/providers/Microsoft.Compute/galleries/', '%s', '/images/', parameters('osImageName'), '/versions/', '%s')]", imageRef.SubscriptionID, imageRef.Gallery, imageRef.Version))
} else {
imgReference.ID = to.StringPtr("[resourceId(parameters('osImageResourceGroup'), 'Microsoft.Compute/images', parameters('osImageName'))]")
}
} else {
imgReference.Offer = to.StringPtr("[parameters('osImageOffer')]")
imgReference.Publisher = to.StringPtr("[parameters('osImagePublisher')]")
@ -577,11 +580,17 @@ func CreateAgentVMSS(cs *api.ContainerService, profile *api.AgentPoolProfile) Vi
}
vmssStorageProfile.DataDisks = getVMSSDataDisks(profile)
} else {
imageRef := profile.ImageRef
useAgentCustomImage := imageRef != nil && len(imageRef.Name) > 0 && len(imageRef.ResourceGroup) > 0
if useAgentCustomImage {
vmssStorageProfile.ImageReference = &compute.ImageReference{
ID: to.StringPtr(fmt.Sprintf("[resourceId(variables('%[1]sosImageResourceGroup'), 'Microsoft.Compute/images', variables('%[1]sosImageName'))]", profile.Name)),
if profile.HasImageRef() {
imageRef := profile.ImageRef
if profile.HasImageGallery() {
v := fmt.Sprintf("[concat('/subscriptions/', '%s', '/resourceGroups/', variables('%sosImageResourceGroup'), '/providers/Microsoft.Compute/galleries/', '%s', '/images/', variables('%sosImageName'), '/versions/', '%s')]", imageRef.SubscriptionID, profile.Name, imageRef.Gallery, profile.Name, imageRef.Version)
vmssStorageProfile.ImageReference = &compute.ImageReference{
ID: to.StringPtr(v),
}
} else {
vmssStorageProfile.ImageReference = &compute.ImageReference{
ID: to.StringPtr(fmt.Sprintf("[resourceId(variables('%[1]sosImageResourceGroup'), 'Microsoft.Compute/images', variables('%[1]sosImageName'))]", profile.Name)),
}
}
} else {
vmssStorageProfile.ImageReference = &compute.ImageReference{

Различия файлов скрыты, потому что одна или несколько строк слишком длинны