Azure
/
aks-engine
зеркало из https://github.com/Azure/aks-engine.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

feat: Azure Spot VMSS (#2547) 

* Azure Spot VMs support

* add doc

* Add the new field in vlabs and related unit tests

* fix unit test

* coding optimize

* Add missing parameters to make the configurable

* test: use spot instead of low pri in E2E tests

Co-authored-by: Jack Francis <jackfrancis@gmail.com>
This commit is contained in:
Frank Gao 2020-01-13 12:08:48 -08:00 коммит произвёл Azure Kubernetes Service Bot
Родитель 1110ee3de8
Коммит a5d625ea00
45 изменённых файлов: 289 добавлений и 59 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

4
cmd/generate_test.go
Просмотреть файл

@ -586,8 +586,8 @@ func TestExampleAPIModels(t *testing.T) {
setArgs: defaultSet,
},
{
name: "vmss low priority",
apiModelPath: "../examples/kubernetes-vmss-low-priority/kubernetes.json",
name: "vmss spot",
apiModelPath: "../examples/kubernetes-vmss-spot/kubernetes.json",
setArgs: defaultSet,
},
{

5
docs/topics/clusterdefinitions.md
Просмотреть файл

@ -565,8 +565,9 @@ A cluster can have 0 to 12 agent pool profiles. Agent Pool Profiles are used for
| count | yes | Describes the node count |
| [availabilityZones](../../examples/kubernetes-zones/README.md) | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. Check out [Availability Zones README](../../examples/kubernetes-zones/README.md) for more details. |
| singlePlacementGroup | no | Supported values are `true` (default) and `false`. A value of `true`: A VMSS with a single placement group and has a range of 0-100 VMs. A value of `false`: A VMSS with multiple placement groups and has a range of 0-1,000 VMs. For more information, check out [virtual machine scale sets placement groups](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-placement-groups). This configuration is only valid on an agent pool with an `"availabilityProfile"` value of `"VirtualMachineScaleSets"` |
| scaleSetPriority | no | Supported values are `Regular` (default) and `Low`. This configuration is only valid on an agent pool with an `"availabilityProfile"` value of `"VirtualMachineScaleSets"`. Enables the usage of [Low-priority VMs on Scale Sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-low-priority). |
| scaleSetEvictionPolicy | no | Supported values are `Delete` (default) and `Deallocate`. This configuration is only valid on an agent pool with an `"availabilityProfile"` value of `"VirtualMachineScaleSets"` and a `"scaleSetPriority"` value of `"Low"`. |
| scaleSetPriority | no | Supported values are `Regular` (default), `Low` (DEPRECATED) and `Spot`. This configuration is only valid on an agent pool with an `"availabilityProfile"` value of `"VirtualMachineScaleSets"`. Enables the usage of [Low-priority VMs on Scale Sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-low-priority) or [Azure Spot VMs](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/use-spot). |
| scaleSetEvictionPolicy | no | Supported values are `Delete` (default) and `Deallocate`. This configuration is only valid on an agent pool with an `"availabilityProfile"` value of `"VirtualMachineScaleSets"` and a `"scaleSetPriority"` value of `"Low"` or `"Spot"`. |
| spotMaxPrice | no | Supported values are `-1` (default) or any decimal value greater than zero. Only valid on an agent pool with an `"availabilityProfile"` value of `"VirtualMachineScaleSets"` and `scaleSetPriority` value of `"Spot"`. Specifies the maximum price you are willing to pay for a Azure Spot VM/VMSS. This price is in US Dollars. This price will be compared with the current Azure Spot price for the VM size. Also, the prices are compared at the time of create/update of Azure Spot VM/VMSS and the operation will only succeed if the maxPrice is greater than the current Azure Spot price. The maxPrice will also be used for evicting a Azure Spot VM/VMSS if the current Azure Spot price goes beyond the maxPrice after creation of VM/VMSS. You can set the maxPrice to -1 to indicate that the Azure Spot VM/VMSS should not be evicted for price reasons. Also, the default max price is -1 if it is not provided by you |
| diskSizesGB | no | Describes an array of up to 4 attached disk sizes. Valid disk size values are between 1 and 1024 |
| dnsPrefix | Required if agents are to be exposed publically with a load balancer | The dns prefix that forms the FQDN to access the loadbalancer for this agent pool. This must be a unique name among all agent pools. Not supported for Kubernetes clusters |
| name | yes | This is the unique name for the agent pool profile. The resources of the agent pool profile are derived from this name |

3
examples/kubernetes-vmss-low-priority/kubernetes.json → examples/kubernetes-vmss-spot/kubernetes.json
Просмотреть файл

@ -19,8 +19,9 @@
"count": 1,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scaleSetPriority": "Low",
"scaleSetPriority": "Spot",
"scaleSetEvictionPolicy": "Delete",
"spotMaxPrice": 88,
"storageProfile": "ManagedDisks"
}
],

26
parts/agentparams.t
Просмотреть файл

@ -40,6 +40,32 @@
"type": "string"
},
{{end}}
{{if .IsSpotScaleSet}}
"{{.Name}}ScaleSetPriority": {
"allowedValues":[
"Spot",
"Regular",
""
],
"defaultValue": "{{.ScaleSetPriority}}",
"metadata": {
"description": "The priority for the VM Scale Set. This value can be Spot or Regular."
},
"type": "string"
},
"{{.Name}}ScaleSetEvictionPolicy": {
"allowedValues":[
"Delete",
"Deallocate",
""
],
"defaultValue": "{{.ScaleSetEvictionPolicy}}",
"metadata": {
"description": "The Eviction Policy for a Spot VM Scale Set."
},
"type": "string"
},
{{end}}
"{{.Name}}VMSize": {
{{GetKubernetesAllowedVMSKUs}}
"defaultValue": "{{.VMSize}}",

2
pkg/api/const.go
Просмотреть файл

@ -98,6 +98,8 @@ const (
ScaleSetPriorityRegular = "Regular"
// ScaleSetPriorityLow means the ScaleSet will use Low-priority VMs
ScaleSetPriorityLow = "Low"
// ScaleSetPrioritySpot means the ScaleSet will use Spot VMs
ScaleSetPrioritySpot = "Spot"
// ScaleSetEvictionPolicyDelete is the default Eviction Policy for Low-priority VM ScaleSets
ScaleSetEvictionPolicyDelete = "Delete"
// ScaleSetEvictionPolicyDeallocate means a Low-priority VM ScaleSet will deallocate, rather than delete, VMs.

1
pkg/api/converterfromapi.go
Просмотреть файл

@ -516,6 +516,7 @@ func convertAgentPoolProfileToVLabs(api *AgentPoolProfile, p *vlabs.AgentPoolPro
p.AvailabilityProfile = api.AvailabilityProfile
p.ScaleSetPriority = api.ScaleSetPriority
p.ScaleSetEvictionPolicy = api.ScaleSetEvictionPolicy
p.SpotMaxPrice = api.SpotMaxPrice
p.StorageProfile = api.StorageProfile
p.DiskSizesGB = []int{}
p.DiskSizesGB = append(p.DiskSizesGB, api.DiskSizesGB...)

1
pkg/api/convertertoapi.go
Просмотреть файл

@ -561,6 +561,7 @@ func convertVLabsAgentPoolProfile(vlabs *vlabs.AgentPoolProfile, api *AgentPoolP
api.AvailabilityProfile = vlabs.AvailabilityProfile
api.ScaleSetPriority = vlabs.ScaleSetPriority
api.ScaleSetEvictionPolicy = vlabs.ScaleSetEvictionPolicy
api.SpotMaxPrice = vlabs.SpotMaxPrice
api.StorageProfile = vlabs.StorageProfile
api.DiskSizesGB = []int{}
api.DiskSizesGB = append(api.DiskSizesGB, vlabs.DiskSizesGB...)

8
pkg/api/defaults.go
Просмотреть файл

@ -627,9 +627,15 @@ func (p *Properties) setAgentProfileDefaults(isUpgrade, isScale bool) {
profile.AvailabilityProfile = VirtualMachineScaleSets
}
if profile.AvailabilityProfile == VirtualMachineScaleSets {
if profile.ScaleSetEvictionPolicy == "" && profile.ScaleSetPriority == ScaleSetPriorityLow {
if profile.ScaleSetEvictionPolicy == "" && (profile.ScaleSetPriority == ScaleSetPriorityLow || profile.ScaleSetPriority == ScaleSetPrioritySpot) {
profile.ScaleSetEvictionPolicy = ScaleSetEvictionPolicyDelete
}
if profile.ScaleSetPriority == ScaleSetPrioritySpot && profile.SpotMaxPrice == nil {
var maximumValueFlag float64 = -1
profile.SpotMaxPrice = &maximumValueFlag
}
if profile.VMSSOverProvisioningEnabled == nil {
profile.VMSSOverProvisioningEnabled = to.BoolPtr(DefaultVMSSOverProvisioningEnabled && !isUpgrade && !isScale)
}

25
pkg/api/defaults_test.go
Просмотреть файл

@ -1483,6 +1483,31 @@ func TestAgentPoolProfile(t *testing.T) {
t.Fatalf("AgentPoolProfile[0].ScaleSetEvictionPolicy did not have the expected configuration, got %s, expected %s",
properties.AgentPoolProfiles[0].ScaleSetEvictionPolicy, ScaleSetEvictionPolicyDelete)
}
properties.AgentPoolProfiles[0].ScaleSetPriority = ScaleSetPrioritySpot
mockCS.SetPropertiesDefaults(PropertiesDefaultsParams{
IsScale: false,
IsUpgrade: false,
PkiKeySize: helpers.DefaultPkiKeySize,
})
if properties.AgentPoolProfiles[0].ScaleSetEvictionPolicy != ScaleSetEvictionPolicyDelete {
t.Fatalf("AgentPoolProfile[0].ScaleSetEvictionPolicy did not have the expected configuration, got %s, expected %s",
properties.AgentPoolProfiles[0].ScaleSetEvictionPolicy, ScaleSetEvictionPolicyDelete)
}
if *properties.AgentPoolProfiles[0].SpotMaxPrice != float64(-1) {
t.Fatalf("AgentPoolProfile[0].SpotMaxPrice did not have the expected value, got %g, expected %g",
*properties.AgentPoolProfiles[0].SpotMaxPrice, float64(-1))
}
properties.AgentPoolProfiles[0].SpotMaxPrice = to.Float64Ptr(float64(88))
mockCS.SetPropertiesDefaults(PropertiesDefaultsParams{
IsScale: false,
IsUpgrade: false,
PkiKeySize: helpers.DefaultPkiKeySize,
})
if *properties.AgentPoolProfiles[0].SpotMaxPrice != float64(88) {
t.Fatalf("AgentPoolProfile[0].SpotMaxPrice did not have the expected value, got %g, expected %g",
*properties.AgentPoolProfiles[0].SpotMaxPrice, float64(88))
}
}
// TestDistroDefaults covers tests for setMasterProfileDefaults and setAgentProfileDefaults

12
pkg/api/types.go
Просмотреть файл

@ -578,6 +578,7 @@ type AgentPoolProfile struct {
AvailabilityProfile string `json:"availabilityProfile"`
ScaleSetPriority string `json:"scaleSetPriority,omitempty"`
ScaleSetEvictionPolicy string `json:"scaleSetEvictionPolicy,omitempty"`
SpotMaxPrice *float64 `json:"spotMaxPrice,omitempty"`
StorageProfile string `json:"storageProfile,omitempty"`
DiskSizesGB []int `json:"diskSizesGB,omitempty"`
VnetSubnetID string `json:"vnetSubnetID,omitempty"`
@ -1217,10 +1218,10 @@ func (p *Properties) HasAvailabilityZones() bool {
return hasZones
}
// HasLowPriorityScaleset returns true if any one node pool has a low-priority scaleset configuration
func (p *Properties) HasLowPriorityScaleset() bool {
// HasNonRegularPriorityScaleset returns true if any one node pool has a low or spot priority scaleset configuration
func (p *Properties) HasNonRegularPriorityScaleset() bool {
for _, agentPoolProfile := range p.AgentPoolProfiles {
if agentPoolProfile.IsLowPriorityScaleSet() {
if agentPoolProfile.IsLowPriorityScaleSet() || agentPoolProfile.IsSpotScaleSet() {
return true
}
}
@ -1516,6 +1517,11 @@ func (a *AgentPoolProfile) IsLowPriorityScaleSet() bool {
return a.AvailabilityProfile == VirtualMachineScaleSets && a.ScaleSetPriority == ScaleSetPriorityLow
}
// IsSpotScaleSet returns true if the VMSS is Spot Scale Set
func (a *AgentPoolProfile) IsSpotScaleSet() bool {
return a.AvailabilityProfile == VirtualMachineScaleSets && a.ScaleSetPriority == ScaleSetPrioritySpot
}
// IsManagedDisks returns true if the customer specified disks
func (a *AgentPoolProfile) IsManagedDisks() bool {
return a.StorageProfile == ManagedDisks

62
pkg/api/types_test.go
Просмотреть файл

@ -1359,7 +1359,23 @@ func TestAvailabilityProfile(t *testing.T) {
expectedISVMSS bool
expectedIsAS bool
expectedLowPri bool
expectedSpot bool
}{
{
p: Properties{
AgentPoolProfiles: []*AgentPoolProfile{
{
AvailabilityProfile: VirtualMachineScaleSets,
ScaleSetPriority: ScaleSetPrioritySpot,
},
},
},
expectedHasVMSS: true,
expectedISVMSS: true,
expectedIsAS: false,
expectedLowPri: false,
expectedSpot: true,
},
{
p: Properties{
AgentPoolProfiles: []*AgentPoolProfile{
@ -1373,6 +1389,7 @@ func TestAvailabilityProfile(t *testing.T) {
expectedISVMSS: true,
expectedIsAS: false,
expectedLowPri: true,
expectedSpot: false,
},
{
p: Properties{
@ -1390,6 +1407,7 @@ func TestAvailabilityProfile(t *testing.T) {
expectedISVMSS: true,
expectedIsAS: false,
expectedLowPri: false,
expectedSpot: false,
},
{
p: Properties{
@ -1403,6 +1421,7 @@ func TestAvailabilityProfile(t *testing.T) {
expectedISVMSS: false,
expectedIsAS: true,
expectedLowPri: false,
expectedSpot: false,
},
}
@ -1419,6 +1438,9 @@ func TestAvailabilityProfile(t *testing.T) {
if c.p.AgentPoolProfiles[0].IsLowPriorityScaleSet() != c.expectedLowPri {
t.Fatalf("expected IsLowPriorityScaleSet() to return %t but instead returned %t", c.expectedLowPri, c.p.AgentPoolProfiles[0].IsLowPriorityScaleSet())
}
if c.p.AgentPoolProfiles[0].IsSpotScaleSet() != c.expectedSpot {
t.Fatalf("expected IsSpotScaleSet() to return %t but instead returned %t", c.expectedSpot, c.p.AgentPoolProfiles[0].IsSpotScaleSet())
}
}
}
@ -1617,7 +1639,7 @@ func TestHasAvailabilityZones(t *testing.T) {
}
}
func TestHasLowPriorityScaleset(t *testing.T) {
func TestHasNonRegularPriorityScaleset(t *testing.T) {
cases := []struct {
p Properties
expected bool
@ -1670,6 +1692,40 @@ func TestHasLowPriorityScaleset(t *testing.T) {
},
expected: true,
},
{
p: Properties{
MasterProfile: &MasterProfile{
Count: 1,
},
AgentPoolProfiles: []*AgentPoolProfile{
{
AvailabilityProfile: VirtualMachineScaleSets,
ScaleSetPriority: ScaleSetPrioritySpot,
},
{
AvailabilityProfile: AvailabilitySet,
},
},
},
expected: true,
},
{
p: Properties{
MasterProfile: &MasterProfile{
Count: 1,
},
AgentPoolProfiles: []*AgentPoolProfile{
{
AvailabilityProfile: VirtualMachineScaleSets,
ScaleSetPriority: ScaleSetPriorityRegular,
},
{
AvailabilityProfile: AvailabilitySet,
},
},
},
expected: false,
},
{
p: Properties{
MasterProfile: &MasterProfile{
@ -1723,8 +1779,8 @@ func TestHasLowPriorityScaleset(t *testing.T) {
}
for _, c := range cases {
if c.p.HasLowPriorityScaleset() != c.expected {
t.Fatalf("expected HasLowPriorityScaleset() to return %t but instead returned %t", c.expected, c.p.HasLowPriorityScaleset())
if c.p.HasNonRegularPriorityScaleset() != c.expected {
t.Fatalf("expected HasNonRegularPriorityScaleset() to return %t but instead returned %t", c.expected, c.p.HasNonRegularPriorityScaleset())
}
}
}

3
pkg/api/vlabs/types.go
Просмотреть файл

@ -473,8 +473,9 @@ type AgentPoolProfile struct {
OSType OSType `json:"osType,omitempty"`
Ports []int `json:"ports,omitempty" validate:"dive,min=1,max=65535"`
AvailabilityProfile string `json:"availabilityProfile"`
ScaleSetPriority string `json:"scaleSetPriority,omitempty" validate:"eq=Regular|eq=Low|len=0"`
ScaleSetPriority string `json:"scaleSetPriority,omitempty" validate:"eq=Regular|eq=Low|eq=Spot|len=0"`
ScaleSetEvictionPolicy string `json:"scaleSetEvictionPolicy,omitempty" validate:"eq=Delete|eq=Deallocate|len=0"`
SpotMaxPrice *float64 `json:"spotMaxPrice,omitempty"`
StorageProfile string `json:"storageProfile" validate:"eq=StorageAccount|eq=ManagedDisks|eq=Ephemeral|len=0"`
DiskSizesGB []int `json:"diskSizesGB,omitempty" validate:"max=4,dive,min=1,max=1023"`
VnetSubnetID string `json:"vnetSubnetID,omitempty"`

21
pkg/api/vlabs/types_test.go
Просмотреть файл

@ -111,6 +111,7 @@ func TestMasterProfile(t *testing.T) {
t.Fatalf("unexpectedly detected MasterProfile.AvailabilityZones, HasAvailabilityZones returned false after unmarshal")
}
}
func TestAgentPoolProfile(t *testing.T) {
// With osType not specified
AgentPoolProfileText := `{"count" : 0, "storageProfile" : "StorageAccount", "vnetSubnetID" : "1234"}`
@ -219,6 +220,26 @@ func TestAgentPoolProfile(t *testing.T) {
t.Fatalf("unexpectedly detected AgentPoolProfile.DiskEncryptionSetID is empty after unmarshal")
}
// With VMSS and Spot VMs
AgentPoolProfileText = `{"name":"linuxpool1","osType":"Linux","distro":"rhel","count":1,"vmSize":"Standard_D2_v2",
"availabilityProfile":"VirtualMachineScaleSets","scaleSetPriority":"Spot","ScaleSetEvictionPolicy":"Delete","SpotMaxPrice":88}`
ap = &AgentPoolProfile{}
if e := json.Unmarshal([]byte(AgentPoolProfileText), ap); e != nil {
t.Fatalf("unexpectedly detected unmarshal failure for AgentPoolProfile, %+v", e)
}
if ap.ScaleSetPriority != "Spot" {
t.Fatalf("unexpectedly detected AgentPoolProfile.ScaleSetPriority != ScaleSetPrioritySpot after unmarshal")
}
if ap.ScaleSetEvictionPolicy != "Delete" {
t.Fatalf("unexpectedly detected AgentPoolProfile.ScaleSetEvictionPolicy != ScaleSetEvictionPolicyDelete after unmarshal")
}
if *ap.SpotMaxPrice != float64(88) {
t.Fatalf("unexpectedly detected *AgentPoolProfile.SpotMaxPrice != float64(88) after unmarshal")
}
// With osType Linux and coreos distro
AgentPoolProfileText = `{ "name": "linuxpool1", "osType" : "Linux", "distro" : "coreos", "count": 1, "vmSize": "Standard_D2_v2",
"availabilityProfile": "VirtualMachineScaleSets", "storageProfile" : "ManagedDisks", "diskSizesGB" : [750, 250, 600, 1000], "diskEncryptionSetID": "diskEncryptionSetID" }`

2
pkg/engine/armvariables.go
Просмотреть файл

@ -608,7 +608,7 @@ func getK8sAgentVars(cs *api.ContainerService, profile *api.AgentPoolProfile) ma
if profile.IsAvailabilitySets() {
agentVars[agentOffset] = fmt.Sprintf("[parameters('%s')]", agentOffset)
agentVars[agentAvailabilitySet] = fmt.Sprintf("[concat('%s-availabilitySet-', parameters('nameSuffix'))]", agentName)
} else if profile.IsLowPriorityScaleSet() {
} else if profile.IsLowPriorityScaleSet() || profile.IsSpotScaleSet() {
agentVars[agentScaleSetPriority] = fmt.Sprintf("[parameters('%s')]", agentScaleSetPriority)
agentVars[agentScaleSetEvictionPolicy] = fmt.Sprintf("[parameters('%s')]", agentScaleSetEvictionPolicy)
}

18
pkg/engine/armvariables_test.go
Просмотреть файл

@ -817,6 +817,24 @@ func TestK8sVars(t *testing.T) {
if diff != "" {
t.Errorf("unexpected diff while expecting equal structs: %s", diff)
}
// Test with Spot VMs
cs.Properties.AgentPoolProfiles[0].AvailabilityProfile = api.VirtualMachineScaleSets
cs.Properties.AgentPoolProfiles[0].ScaleSetPriority = api.ScaleSetPrioritySpot
varMap, err = GetKubernetesVariables(cs)
if err != nil {
t.Fatal(err)
}
agentPoolName := cs.Properties.AgentPoolProfiles[0].Name
expectedMap[fmt.Sprintf("%sScaleSetPriority", agentPoolName)] = fmt.Sprintf("[parameters('%sScaleSetPriority')]", agentPoolName)
expectedMap[fmt.Sprintf("%sScaleSetEvictionPolicy", agentPoolName)] = fmt.Sprintf("[parameters('%sScaleSetEvictionPolicy')]", agentPoolName)
diff = cmp.Diff(varMap, expectedMap)
if diff != "" {
t.Errorf("unexpected diff while expecting equal structs: %s", diff)
}
}
func TestK8sVarsMastersOnly(t *testing.T) {

5
pkg/engine/params.go
Просмотреть файл

@ -196,6 +196,11 @@ func getParameters(cs *api.ContainerService, generatorCode string, aksEngineVers
addValue(parametersMap, fmt.Sprintf("%sEndpointDNSNamePrefix", agentProfile.Name), agentProfile.DNSPrefix)
}
if !agentProfile.IsAvailabilitySets() && agentProfile.IsSpotScaleSet() {
addValue(parametersMap, fmt.Sprintf("%sScaleSetPriority", agentProfile.Name), agentProfile.ScaleSetPriority)
addValue(parametersMap, fmt.Sprintf("%sScaleSetEvictionPolicy", agentProfile.Name), agentProfile.ScaleSetEvictionPolicy)
}
// Unless distro is defined, default distro is configured by defaults#setAgentProfileDefaults
// Ignores Windows OS
if !(agentProfile.OSType == api.Windows) {

26
pkg/engine/templates_generated.go
Просмотреть файл

@ -371,6 +371,32 @@ var _agentparamsT = []byte(` "{{.Name}}Count": {
"type": "string"
},
{{end}}
{{if .IsSpotScaleSet}}
"{{.Name}}ScaleSetPriority": {
"allowedValues":[
"Spot",
"Regular",
""
],
"defaultValue": "{{.ScaleSetPriority}}",
"metadata": {
"description": "The priority for the VM Scale Set. This value can be Spot or Regular."
},
"type": "string"
},
"{{.Name}}ScaleSetEvictionPolicy": {
"allowedValues":[
"Delete",
"Deallocate",
""
],
"defaultValue": "{{.ScaleSetEvictionPolicy}}",
"metadata": {
"description": "The Eviction Policy for a Spot VM Scale Set."
},
"type": "string"
},
{{end}}
"{{.Name}}VMSize": {
{{GetKubernetesAllowedVMSKUs}}
"defaultValue": "{{.VMSize}}",

8
pkg/engine/virtualmachinescalesets.go
Просмотреть файл

@ -448,11 +448,17 @@ func CreateAgentVMSS(cs *api.ContainerService, profile *api.AgentPoolProfile) Vi
vmssVMProfile := compute.VirtualMachineScaleSetVMProfile{}
if profile.IsLowPriorityScaleSet() {
if profile.IsLowPriorityScaleSet() || profile.IsSpotScaleSet() {
vmssVMProfile.Priority = compute.VirtualMachinePriorityTypes(fmt.Sprintf("[variables('%sScaleSetPriority')]", profile.Name))
vmssVMProfile.EvictionPolicy = compute.VirtualMachineEvictionPolicyTypes(fmt.Sprintf("[variables('%sScaleSetEvictionPolicy')]", profile.Name))
}
if profile.IsSpotScaleSet() {
vmssVMProfile.BillingProfile = &compute.BillingProfile{
MaxPrice: profile.SpotMaxPrice,
}
}
vmssNICConfig := compute.VirtualMachineScaleSetNetworkConfiguration{
Name: to.StringPtr(fmt.Sprintf("[variables('%sVMNamePrefix')]", profile.Name)),
VirtualMachineScaleSetNetworkConfigurationProperties: &compute.VirtualMachineScaleSetNetworkConfigurationProperties{

28
pkg/engine/virtualmachinescalesets_test.go
Просмотреть файл

@ -556,6 +556,34 @@ func TestCreateAgentVMSS(t *testing.T) {
if diff != "" {
t.Errorf("unexpected diff while expecting equal structs: %s", diff)
}
// Test with Spot Scale Set
cs.Properties.AgentPoolProfiles[0].ScaleSetPriority = api.ScaleSetPrioritySpot
cs.Properties.AgentPoolProfiles[0].SpotMaxPrice = to.Float64Ptr(float64(22))
actual = CreateAgentVMSS(cs, cs.Properties.AgentPoolProfiles[0])
// Test VirtualMachineProfile.BillingProfile
expected.VirtualMachineProfile.BillingProfile = &compute.BillingProfile{
MaxPrice: to.Float64Ptr(float64(22)),
}
diff = cmp.Diff(actual.VirtualMachineProfile.BillingProfile, expected.VirtualMachineProfile.BillingProfile)
if diff != "" {
t.Errorf("unexpected diff while expecting equal structs: %s", diff)
}
// Test VirtualMachineProfile.Priority
expected.VirtualMachineProfile.Priority = compute.VirtualMachinePriorityTypes(fmt.Sprintf("[variables('%sScaleSetPriority')]", cs.Properties.AgentPoolProfiles[0].Name))
diff = cmp.Diff(actual.VirtualMachineProfile.BillingProfile, expected.VirtualMachineProfile.BillingProfile)
if diff != "" {
t.Errorf("unexpected diff while expecting equal structs: %s", diff)
}
// Test VirtualMachineProfile.EvictionPolicy
expected.VirtualMachineProfile.EvictionPolicy = compute.VirtualMachineEvictionPolicyTypes(fmt.Sprintf("[variables('%sScaleSetEvictionPolicy')]", cs.Properties.AgentPoolProfiles[0].Name))
diff = cmp.Diff(actual.VirtualMachineProfile.BillingProfile, expected.VirtualMachineProfile.BillingProfile)
if diff != "" {
t.Errorf("unexpected diff while expecting equal structs: %s", diff)
}
}
func TestCreateAgentVMSSHostedMasterProfile(t *testing.T) {

36
test/e2e/kubernetes/kubernetes_test.go
Просмотреть файл

@ -167,7 +167,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
} else {
var nodes []node.Node
var err error
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
nodes, err = node.GetWithRetry(1*time.Second, cfg.Timeout)
} else {
nodes = masterNodes
@ -210,7 +210,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate cloudprovider config", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
cloudproviderConfigValidateScript := "cloudprovider-config-validate.sh"
@ -264,7 +264,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should display the installed docker runtime on all nodes", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.OrchestratorProfile.KubernetesConfig.RequiresDocker() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
@ -290,7 +290,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate that every linux node has a root password", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.IsVHDDistroForAllNodes() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
@ -312,7 +312,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate Ubuntu host OS network configuration on all nodes", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.IsVHDDistroForAllNodes() {
var largeSKUPrefixes []string
if eng.ExpandedDefinition.Properties.MasterProfile != nil {
@ -354,7 +354,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate all CIS VHD-paved files", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.IsVHDDistroForAllNodes() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
@ -384,7 +384,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate kernel module configuration", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.IsVHDDistroForAllNodes() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
@ -413,7 +413,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate installed software packages", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
installedPackagesValidateScript := "ubuntu-installed-packages-validate.sh"
@ -438,7 +438,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate that every linux node has the right sshd config", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.IsVHDDistroForAllNodes() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
@ -467,7 +467,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should validate password enforcement configuration", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.IsVHDDistroForAllNodes() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
@ -638,7 +638,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should report all nodes in a Ready state", func() {
var expectedReadyNodes int
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() && !clusterAutoscalerEngaged {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() && !clusterAutoscalerEngaged {
expectedReadyNodes = eng.NodeCount()
log.Printf("Checking for %d Ready nodes\n", expectedReadyNodes)
} else {
@ -652,7 +652,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
})
It("should have node labels and annotations", func() {
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
totalNodeCount := eng.NodeCount()
nodes := totalNodeCount - len(masterNodes)
nodeList, err := node.GetByLabel("foo")
@ -669,13 +669,13 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should have node labels specific to masters or agents", func() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
Expect(len(nodes)).To(Equal(eng.NodeCount()))
}
for _, node := range nodes {
role := "master"
if !strings.HasPrefix(node.Metadata.Name, "k8s-master-") {
if eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
continue
} else {
role = "agent"
@ -1043,7 +1043,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
Describe("with a windows agent pool", func() {
It("kubelet service should be able to recover when the docker service is stopped", func() {
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.HasWindowsAgents() {
if eng.ExpandedDefinition.Properties.WindowsProfile != nil && eng.ExpandedDefinition.Properties.WindowsProfile.SSHEnabled {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
@ -1075,7 +1075,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
}
var expectedReadyNodes int
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() && !clusterAutoscalerEngaged {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() && !clusterAutoscalerEngaged {
expectedReadyNodes = len(nodes)
log.Printf("Checking for %d Ready nodes\n", expectedReadyNodes)
} else {
@ -1291,7 +1291,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
})
It("should create pv with zone labels and node affinity", func() {
if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
if eng.ExpandedDefinition.Properties.HasZonesForAllAgentPools() {
if !(common.IsKubernetesVersionGe(eng.ExpandedDefinition.Properties.OrchestratorProfile.OrchestratorVersion, "1.16.0") &&
to.Bool(eng.ExpandedDefinition.Properties.OrchestratorProfile.KubernetesConfig.UseCloudControllerManager)) {
@ -1881,7 +1881,7 @@ var _ = Describe("Azure Container Cluster using the Kubernetes Orchestrator", fu
It("should have healthy time synchronization", func() {
if cfg.BlockSSHPort {
Skip("SSH port is blocked")
} else if !eng.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
} else if !eng.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
nodes, err := node.GetWithRetry(1*time.Second, cfg.Timeout)
Expect(err).NotTo(HaveOccurred())
timeSyncValidateScript := "time-sync-validate.sh"

4
test/e2e/runner/cli_provisioner.go
Просмотреть файл

@ -303,7 +303,7 @@ func (cli *CLIProvisioner) waitForNodes() error {
if !cli.IsPrivate() {
log.Println("Waiting on nodes to go into ready state...")
var expectedReadyNodes int
if !cli.Engine.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !cli.Engine.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
expectedReadyNodes = cli.Engine.NodeCount()
log.Printf("Checking for %d Ready nodes\n", expectedReadyNodes)
} else {
@ -320,7 +320,7 @@ func (cli *CLIProvisioner) waitForNodes() error {
if err != nil {
return errors.Wrap(err, "Unable to get the list of nodes")
}
if !cli.Engine.ExpandedDefinition.Properties.HasLowPriorityScaleset() {
if !cli.Engine.ExpandedDefinition.Properties.HasNonRegularPriorityScaleset() {
for _, n := range nodes {
exp, err := regexp.Compile("k8s-master")
if err != nil {

2
test/e2e/test_cluster_configs/base.json
Просмотреть файл

@ -17,7 +17,7 @@
"count": 2,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/container_monitoring.json
Просмотреть файл

@ -31,7 +31,7 @@
"count": 3,
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/everything.json
Просмотреть файл

@ -68,7 +68,7 @@
"1",
"2"
],
"scaleSetPriority": "Low",
"scalesetPriority": "Spot",
"auditDEnabled": true
}
],

2
test/e2e/test_cluster_configs/flannel/containerd.json
Просмотреть файл

@ -27,7 +27,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/flannel/docker.json
Просмотреть файл

@ -31,7 +31,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Linux",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/gpu.json
Просмотреть файл

@ -33,7 +33,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/kube-dns.json
Просмотреть файл

@ -25,7 +25,7 @@
"count": 2,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/kubenet.json
Просмотреть файл

@ -22,7 +22,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Linux",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/large-vm-sku.json
Просмотреть файл

@ -19,7 +19,7 @@
"count": 2,
"vmSize": "Standard_D16_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/network_policy/antrea.json
Просмотреть файл

@ -24,7 +24,7 @@
"count": 1,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/network_policy/azure.json
Просмотреть файл

@ -25,7 +25,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Linux",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/network_policy/calico.json
Просмотреть файл

@ -61,7 +61,7 @@
],
"availabilityProfile": "VirtualMachineScaleSets",
"vnetSubnetId": "/subscriptions/SUB_ID/resourceGroups/RG_NAME/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/SUBNET_NAME",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/network_policy/calico_azure.json
Просмотреть файл

@ -24,7 +24,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/network_policy/cilium.json
Просмотреть файл

@ -25,7 +25,7 @@
"count": 1,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/no_outbound.json
Просмотреть файл

@ -20,7 +20,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/not_in_vhd.json
Просмотреть файл

@ -21,7 +21,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/rbac_disabled.json
Просмотреть файл

@ -25,7 +25,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Linux",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/ubuntu_distro.json
Просмотреть файл

@ -18,7 +18,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low",
"scalesetPriority": "Spot",
"distro": "ubuntu"
}
],

2
test/e2e/test_cluster_configs/vmss_master.json
Просмотреть файл

@ -18,7 +18,7 @@
"count": 3,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"linuxProfile": {

2
test/e2e/test_cluster_configs/windows/custom_image.json
Просмотреть файл

@ -26,7 +26,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Windows",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"windowsProfile": {

2
test/e2e/test_cluster_configs/windows/custom_vnet.json
Просмотреть файл

@ -35,7 +35,7 @@
128
],
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low",
"scalesetPriority": "Spot",
"vnetSubnetId": "/subscriptions/SUB_ID/resourceGroups/RG_NAME/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/SUBNET_NAME"
}
],

2
test/e2e/test_cluster_configs/windows/image_reference.json
Просмотреть файл

@ -31,7 +31,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Windows",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"windowsProfile": {

2
test/e2e/test_cluster_configs/windows/shared_image_gallery.json
Просмотреть файл

@ -34,7 +34,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Windows",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"windowsProfile": {

2
test/e2e/test_cluster_configs/windows/vhd_url.json
Просмотреть файл

@ -30,7 +30,7 @@
"vmSize": "Standard_D2_v3",
"osType": "Windows",
"availabilityProfile": "VirtualMachineScaleSets",
"scalesetPriority": "Low"
"scalesetPriority": "Spot"
}
],
"windowsProfile": {