зеркало из https://github.com/golang/build.git
env/windows-arm64: add Ubuntu/Windows ARM64 VM configuration
This change introduces a Windows ARM64 builder image for Go, running on Ubuntu on an a1.metal instance on AWS. The AMI image successfully boots, and automatically starts a Windows ARM64 VM, exposing port 443. The buildlet still needs to be updated for the appropriate mingw path. Additionally, the qemu script does not yet automatically halt the instance on termination, which makes debugging easier. - Ubuntu was chosen over Debian, as the official Debian AMI would not boot on a1.metal instances. - a1.metal is the most affordable AWS instance to support KVM on ARM64. - A prepared qemu image exists in the Go AWS account, under s3://go-builder-data. - Only 4 processors are currently supported in order to avoid a CLOCK_WATCHDOG_TIMEOUT. We're working on investigating how to increase this. - The EC2 metadata service is proxied to the guest OS via qemu. For golang/go#42604 Change-Id: I2a505a8218c2a818f0e52d179c02e6d264d2e422 Reviewed-on: https://go-review.googlesource.com/c/build/+/321959 Trust: Alexander Rakoczy <alex@golang.org> Run-TryBot: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org>
This commit is contained in:
Alexander Rakoczy
Родитель
d0819edf59
Коммит
b5eec30959
|
|
@ -0,0 +1,64 @@
|
|||
# Windows buildlet images
|
||||
|
||||
Windows images are built by creating and configuring VMs hosted on AWS
|
||||
a1.metal instances then saving the image manually.
|
||||
|
||||
## Build and test the Windows builder image
|
||||
|
||||
- Prepare the linux QEMU host image by following the instructions in
|
||||
`env/windows-arm64/aws`.
|
||||
- Create an a1.metal instance (or other instance that supports KVM)
|
||||
in AWS.
|
||||
- Download a Windows 10 ARM64 image.
|
||||
- Convert vhdx images to qcow2 via the following command:
|
||||
|
||||
```shell
|
||||
qemu-image convert -O qcow2 win.vhdx win.qcow2
|
||||
```
|
||||
|
||||
- SSH to your instance tunneling port 5901, and run `win10-arm64.sh`
|
||||
script to boot the Windows VM.
|
||||
- You may need to stop the current VM: `sudo systemctl stop qemu`
|
||||
- VNC to the tunneled port 5901.
|
||||
- Open the device control panel, and use the "Search for Drivers"
|
||||
button to search the virtio drive `D:` for drivers.
|
||||
- Matching drivers will be automatically installed.
|
||||
- This is necessary for networking to work on Windows in qemu.
|
||||
- Download the `startup.ps1` script to the Windows instance, and run
|
||||
in PowerShell. Check thoroughly for errors.
|
||||
- Alternatively, you can modify `win10-arm64.sh` to forward ssh
|
||||
access to the VM, and run PowerShell in the CLI, which is a bit
|
||||
easier than through VNC.
|
||||
- Once the image is complete, download the image to your workstation
|
||||
and upload to `s3://go-builder-data`.
|
||||
- You can find the appropriate the S3 path referenced in
|
||||
`env/windows-arm64/aws/prepare_image.sh`.
|
||||
- Re-run packer to build an AMI with your updated Windows image.
|
||||
|
||||
### Notes
|
||||
|
||||
- `QEMU_EFI.fd` is from the `qemu-efi-aarch64` Debian package, found
|
||||
at `/usr/share/qemu-efi-aarch64/QEMU_EFI.fd`. It can be regenerated
|
||||
with the following command:
|
||||
|
||||
```shell
|
||||
dd if=/dev/zero of=QEMU_EFI.fd bs=1M count=64
|
||||
dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=QEMU_EFI.fd bs=1M count=64 conv=notrunc
|
||||
```
|
||||
|
||||
- `QEMU_VARS.fd` stores saved EFI state when booting a VM. It's
|
||||
generated via the following command:
|
||||
|
||||
```shell
|
||||
dd if=/dev/zero of=QEMU_VARS.fd bs=1M count=64
|
||||
```
|
||||
|
||||
- The latest virtio driver image can be fetched from:
|
||||
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso
|
||||
|
||||
- `win10-arm64.sh` is hard-coded to run with 4 processors instead of
|
||||
the 16 available on an a1.metal instance. Higher numbers of
|
||||
processors are causing a fatal CLOCK_WATCHDOG_TIMEOUT error from
|
||||
interrupt requests not arriving in time. qemu-system-x86_64 has a
|
||||
workaround for this. We're still investigating how to increase this
|
||||
on aarch64.
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
# Copyright 2021 The Go Authors. All rights reserved.
|
||||
# Use of this source code is governed by a BSD-style
|
||||
# license that can be found in the LICENSE file.
|
||||
|
||||
env-var-check:
|
||||
ifndef AWS_ACCESS_KEY_ID
|
||||
$(error AWS_ACCESS_KEY_ID env var is not set)
|
||||
endif
|
||||
|
||||
ifndef AWS_SECRET_ACCESS_KEY
|
||||
$(error AWS_SECRET_ACCESS_KEY env var is not set)
|
||||
endif
|
||||
|
||||
create-aws-image: env-var-check
|
||||
export AWS_MAX_ATTEMPTS=600
|
||||
export AWS_POLL_DELAY_SECONDS=10
|
||||
export PACKER_LOG=1
|
||||
packer build -timestamp-ui packer_image_aws_arm64.json
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# AWS Windows ARM64 Builders
|
||||
|
||||
## Machines
|
||||
|
||||
The AWS builders use the a1 instance types which are arm64 machines.
|
||||
The base type used will be a1.metal, which are the cheapest which
|
||||
expose KVM support.
|
||||
|
||||
## Machine image
|
||||
|
||||
Machine images are stored on AWS EBS service as a snapshot. New VMs
|
||||
can use the snapshot as an image by providing the AMI ID as the base
|
||||
image when a new VM is created.
|
||||
|
||||
### Creating a new Ubuntu host image
|
||||
|
||||
Requirements:
|
||||
|
||||
Two environment variables are required to be set before initiating
|
||||
the command: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` should be
|
||||
set with the appropriate values.
|
||||
|
||||
The [packer](https://www.packer.io) binary should be in `PATH`.
|
||||
|
||||
Command:
|
||||
|
||||
`make create-aws-image`
|
||||
|
||||
or
|
||||
|
||||
`AWS_ACCESS_KEY_ID=<id> AWS_SECRET_ACCESS_KEY=<secret> make create-aws-image`
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
{
|
||||
"variables": {
|
||||
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
|
||||
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
|
||||
"region": "us-east-2"
|
||||
},
|
||||
"builders": [
|
||||
{
|
||||
"type": "amazon-ebs",
|
||||
"iam_instance_profile": "buildetS3ReadOnly",
|
||||
"access_key": "{{user `aws_access_key`}}",
|
||||
"ami_name": "go-linux-arm64-host-{{timestamp}}",
|
||||
"ami_description": "Image for linux-arm64 Go builder hosting windows-arm64",
|
||||
"instance_type": "a1.metal",
|
||||
"region": "{{user `region`}}",
|
||||
"secret_key": "{{user `aws_secret_key`}}",
|
||||
"source_ami": "ami-0b0c8ae527978b689",
|
||||
"decode_authorization_messages": true,
|
||||
"ssh_username": "ubuntu",
|
||||
"tags": {
|
||||
"Name": "Ubuntu",
|
||||
"Created": "{{isotime \"2006-01-02\"}}",
|
||||
"OS": "Ubuntu 20.04 Focal (ARM64)",
|
||||
"Release": "Latest",
|
||||
"Base_AMI_Name": "{{ .SourceAMIName }}",
|
||||
"Extra": "{{ .SourceAMITags.TagName }}",
|
||||
"Description": "{{user `description`}}"
|
||||
},
|
||||
"launch_block_device_mappings": [
|
||||
{
|
||||
"device_name": "/dev/sda1",
|
||||
"volume_size": 50,
|
||||
"volume_type": "gp2",
|
||||
"delete_on_termination": true
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"provisioners": [
|
||||
{
|
||||
"type": "file",
|
||||
"source": "./win10-arm64.sh",
|
||||
"destination": "/home/ubuntu/win10-arm64.sh"
|
||||
},
|
||||
{
|
||||
"type": "file",
|
||||
"source": "./qemu.service",
|
||||
"destination": "/tmp/qemu.service"
|
||||
},
|
||||
{
|
||||
"type": "shell",
|
||||
"script": "./prepare_image.sh"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Copyright 2021 The Go Authors. All rights reserved.
|
||||
# Use of this source code is governed by a BSD-style
|
||||
# license that can be found in the LICENSE file.
|
||||
|
||||
#
|
||||
# Installs all dependencies for an Ubuntu Linux ARM64 qemu host.
|
||||
#
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
|
||||
# Retry apt commands until we succeed.
|
||||
#
|
||||
# Metal instances are still being provisioned when we are first able
|
||||
# to connect over SSH. Some parts of apt are locked or not yet
|
||||
# present. Retrying a few times seems to do the trick.
|
||||
for i in $(seq 1 10); do
|
||||
# Give it a chance to finish before our first try,
|
||||
# and take a break between loops.
|
||||
sleep 1
|
||||
sudo apt-add-repository universe || continue
|
||||
|
||||
sudo apt-get update || continue
|
||||
sudo apt-get upgrade -y || continue
|
||||
|
||||
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg-agent gnupg jq software-properties-common \
|
||||
build-essential ninja-build && break
|
||||
done
|
||||
|
||||
# QEMU Dependencies
|
||||
sudo apt-get install -y git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev
|
||||
|
||||
# QEMU Extras
|
||||
sudo apt-get install -y git-email libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev \
|
||||
libcurl4-gnutls-dev libgtk-3-dev libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev \
|
||||
libsasl2-dev libsdl1.2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libvde-dev libvdeplug-dev libvte-2.91-dev \
|
||||
libxen-dev liblzo2-dev valgrind xfslibs-dev libnfs-dev libiscsi-dev
|
||||
|
||||
# QEMU download & build
|
||||
wget https://download.qemu.org/qemu-6.0.0.tar.xz
|
||||
tar xJf qemu-6.0.0.tar.xz
|
||||
cd qemu-6.0.0
|
||||
./configure --target-list=arm-softmmu,aarch64-softmmu
|
||||
make -j16
|
||||
cd "$HOME"
|
||||
|
||||
# S3 CLI
|
||||
sudo apt-get install -y aws-shell
|
||||
|
||||
# Copy pre-prepared Windows 10 image
|
||||
aws s3 sync s3://go-builder-data/win10 "$HOME"/win10
|
||||
|
||||
chmod u+x "$HOME/win10-arm64.sh"
|
||||
sudo cp /tmp/qemu.service /etc/systemd/user/qemu.service
|
||||
sudo systemctl enable /etc/systemd/user/qemu.service
|
||||
sudo systemctl start qemu
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[Unit]
|
||||
Description=QEMU
|
||||
|
||||
[Service]
|
||||
ExecStart=/home/ubuntu/win10-arm64.sh
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Copyright 2021 The Go Authors. All rights reserved.
|
||||
# Use of this source code is governed by a BSD-style
|
||||
# license that can be found in the LICENSE file.
|
||||
|
||||
/home/ubuntu/qemu-6.0.0/build/aarch64-softmmu/qemu-system-aarch64 \
|
||||
-name "Windows 10 ARM64" \
|
||||
-machine virt \
|
||||
-cpu host \
|
||||
--accel kvm \
|
||||
-smp 4 \
|
||||
-m 8G \
|
||||
-drive file=/home/ubuntu/win10/QEMU_EFI.fd,format=raw,if=pflash,readonly=on \
|
||||
-drive file=/home/ubuntu/win10/QEMU_VARS.fd,format=raw,if=pflash \
|
||||
-device nec-usb-xhci \
|
||||
-device usb-kbd,id=kbd0 \
|
||||
-device usb-mouse,id=tab0 \
|
||||
-device virtio-net,disable-legacy=on,netdev=net0,mac=54:91:05:C5:73:29,addr=08 \
|
||||
-netdev 'user,id=net0,hostfwd=tcp::443-:443,guestfwd=tcp:10.0.2.100:8173-cmd:netcat 169.254.169.254 80' \
|
||||
-device nvme,drive=hdd0,serial=hdd0 \
|
||||
-vnc :3 \
|
||||
-drive file=/home/ubuntu/win10/win10.qcow2,if=none,id=hdd0,cache=writethrough \
|
||||
-drive file=/home/ubuntu/win10/virtio.iso,media=cdrom,if=none,id=drivers,readonly=on \
|
||||
-device usb-storage,drive=drivers \
|
||||
-chardev file,path=/var/log/qemu-serial.log,id=char0 \
|
||||
-serial chardev:char0 \
|
||||
-device ramfb
|
||||
|
|
@ -0,0 +1,163 @@
|
|||
# Copyright 2021 The Go Authors. All rights reserved.
|
||||
# Use of this source code is governed by a BSD-style
|
||||
# license that can be found in the LICENSE file.
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
|
||||
# Helpers
|
||||
function Test-RegistryKeyExists($path, $name)
|
||||
{
|
||||
$key = Get-Item -LiteralPath $path -ErrorAction SilentlyContinue
|
||||
($key -and $null -ne $key.GetValue($name, $null)) -ne $false
|
||||
}
|
||||
|
||||
function Get-FileFromUrl(
|
||||
[string] $URL,
|
||||
[string] $Output)
|
||||
{
|
||||
Add-Type -AssemblyName "System.Net.Http"
|
||||
|
||||
$client = New-Object System.Net.Http.HttpClient
|
||||
$request = New-Object System.Net.Http.HttpRequestMessage -ArgumentList @([System.Net.Http.HttpMethod]::Get, $URL)
|
||||
$responseMsg = $client.SendAsync($request)
|
||||
$responseMsg.Wait()
|
||||
|
||||
if (!$responseMsg.IsCanceled)
|
||||
{
|
||||
$response = $responseMsg.Result
|
||||
if ($response.IsSuccessStatusCode)
|
||||
{
|
||||
$downloadedFileStream = [System.IO.File]::Create($Output)
|
||||
$copyStreamOp = $response.Content.CopyToAsync($downloadedFileStream)
|
||||
$copyStreamOp.Wait()
|
||||
$downloadedFileStream.Close()
|
||||
if ($copyStreamOp.Exception -ne $null)
|
||||
{
|
||||
throw $copyStreamOp.Exception
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# https://social.technet.microsoft.com/Forums/ie/en-US/29508e4e-a2b5-42eb-9729-6eca473716ae/disabling-password-complexity-via-command?forum=ITCG
|
||||
function Disable-PasswordComplexity
|
||||
{
|
||||
param()
|
||||
|
||||
$secEditPath = [System.Environment]::ExpandEnvironmentVariables("%SystemRoot%\system32\secedit.exe")
|
||||
$tempFile = [System.IO.Path]::GetTempFileName()
|
||||
|
||||
$exportArguments = '/export /cfg "{0}" /quiet' -f $tempFile
|
||||
$importArguments = '/configure /db secedit.sdb /cfg "{0}" /quiet' -f $tempFile
|
||||
|
||||
Start-Process -FilePath $secEditPath -ArgumentList $exportArguments -Wait
|
||||
|
||||
$currentConfig = Get-Content -Path $tempFile
|
||||
|
||||
$currentConfig = $currentConfig -replace 'PasswordComplexity = .', 'PasswordComplexity = 0'
|
||||
$currentConfig = $currentConfig -replace 'MinimumPasswordLength = .', 'MinimumPasswordLength = 0'
|
||||
$currentConfig | Out-File -FilePath $tempFile
|
||||
|
||||
Start-Process -FilePath $secEditPath -ArgumentList $importArguments -Wait
|
||||
|
||||
Remove-Item -Path .\secedit.sdb
|
||||
Remove-Item -Path $tempFile
|
||||
}
|
||||
|
||||
# Wait till network comes up
|
||||
while(-Not (Test-NetConnection 8.8.8.8 -Port 53 | ? { $_.TcpTestSucceeded })) {
|
||||
Write-Host "waiting for network (external network) to come up"
|
||||
sleep 3
|
||||
}
|
||||
|
||||
# Disable password complexity, automatic updates, windows defender, windows firewall, error reporting, and UAC
|
||||
#
|
||||
# - Update can interrupt the builds
|
||||
# - We don't care about security since this isn't going to be Internet-facing
|
||||
# - No ports will ever be accessible externally
|
||||
# - We can be trusted to run as a real Administrator
|
||||
Write-Host "disabling security features"
|
||||
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate"
|
||||
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU"
|
||||
New-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU" -Name NoAutoUpdate -Value 1 -Force | Out-Null
|
||||
New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting" -Name Disabled -Value 1 -Force | Out-Null
|
||||
New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting" -Name DontShowUI -Value 1 -Force | Out-Null
|
||||
New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\policies\system" -Name EnableLUA -PropertyType DWord -Value 0 -Force | Out-Null
|
||||
netsh advfirewall set allprofiles state off
|
||||
netsh firewall set opmode mode=disable profile=ALL
|
||||
Set-MpPreference -DisableRealtimeMonitoring $true
|
||||
|
||||
# Disable unwanted services
|
||||
Write-Host "disabling unused services"
|
||||
Set-Service -Name 'NlaSvc' -StartupType 'Disabled'
|
||||
Set-Service -Name 'LanmanServer' -StartupType 'Disabled'
|
||||
Set-Service -Name 'BITS' -StartupType 'Disabled'
|
||||
Set-Service -Name 'DPS' -StartupType 'Disabled'
|
||||
Set-Service -Name 'MSDTC' -StartupType 'Disabled'
|
||||
Set-Service -Name 'IKEEXT' -StartupType 'Disabled'
|
||||
Set-Service -Name 'RemoteRegistry' -StartupType 'Disabled'
|
||||
Set-Service -Name 'lmhosts' -StartupType 'Disabled'
|
||||
|
||||
# Download buildlet
|
||||
Write-Host "downloading stage0"
|
||||
$builder_dir = "C:\golang"
|
||||
$bootstrap_exe_path = "$builder_dir\bootstrap.exe"
|
||||
mkdir $builder_dir
|
||||
Get-FileFromUrl -URL 'https://storage.googleapis.com/go-builder-data/buildlet-stage0.windows-arm64' -Output $bootstrap_exe_path
|
||||
|
||||
# Install the OpenSSH Client
|
||||
Add-WindowsCapability -Online -Name OpenSSH.Client
|
||||
# Install the OpenSSH Server
|
||||
Add-WindowsCapability -Online -Name OpenSSH.Server
|
||||
|
||||
Start-Service sshd
|
||||
# OPTIONAL but recommended:
|
||||
Set-Service -Name sshd -StartupType 'Automatic'
|
||||
|
||||
# Download and unpack LLVM
|
||||
Write-Host "downloading LLVM"
|
||||
$dep_dir = "C:\godep"
|
||||
$llvm64_tar = "$dep_dir\llvm64.tar.gz"
|
||||
mkdir $dep_dir
|
||||
Get-FileFromUrl -URL "https://storage.googleapis.com/go-builder-data/llvm-mingw-20210423-ucrt-aarch64.tar.gz" -Output "$llvm64_tar"
|
||||
|
||||
Write-Host "extracting LLVM"
|
||||
$extract64_args=@("--untar-file=$llvm64_tar", "--untar-dest-dir=$dep_dir")
|
||||
& $bootstrap_exe_path $extract64_args
|
||||
|
||||
$builder_dir = "C:\golang"
|
||||
$bootstrap_exe_path = "$builder_dir\bootstrap.exe"
|
||||
|
||||
# Download and install Visual Studio Build Tools (MSVC)
|
||||
# https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container
|
||||
Write-Host "downloading Visual Studio Build Tools"
|
||||
$vs_buildtools = "$builder_dir\vs_buildtools.exe"
|
||||
Get-FileFromUrl -URL "https://aka.ms/vs/16/release/vs_buildtools.exe" -Output "$vs_buildtools"
|
||||
|
||||
Write-Host "installing Visual Studio Build Tools"
|
||||
& $vs_buildtools --quiet --wait --norestart --nocache --installPath "$dep_dir\vs" --all --add Microsoft.VisualStudio.Component.VC.Tools.ARM64
|
||||
|
||||
# Create a buildlet user
|
||||
Write-Host "creating buildlet user"
|
||||
$buildlet_user = "gopher"
|
||||
$buildlet_password = "gopher"
|
||||
net user $buildlet_user $buildlet_password /ADD
|
||||
net localgroup administrators $buildlet_user /ADD
|
||||
|
||||
# Run the bootstrap program on login
|
||||
Write-Host "setting stage0 to run on start"
|
||||
$bootstrap_cmd = "cmd /k ""cd $builder_dir && $bootstrap_exe_path"""
|
||||
New-ItemProperty -Path "HKLM:\Software\Microsoft\Windows\CurrentVersion\Run" -Name "Buildlet" -PropertyType ExpandString -Value $bootstrap_cmd -Force
|
||||
|
||||
# Setup autologon and reboot
|
||||
$RegPath = "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon"
|
||||
if ((Test-RegistryKeyExists $RegPath "DefaultUsername") -eq $false) {
|
||||
Write-Host "configuring auto login"
|
||||
Remove-ItemProperty -Path 'HKLM:SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon' -Name 'AutoLogonCount' -Force | Out-Null
|
||||
Set-ItemProperty $RegPath "AutoAdminLogon" -Value "1" -type String
|
||||
Set-ItemProperty $RegPath "DefaultUsername" -Value "$buildlet_user" -type String
|
||||
Set-ItemProperty $RegPath "DefaultPassword" -Value "$buildlet_password" -type String
|
||||
Set-ItemProperty $RegPath "LogonCount" -Value "99999999" -type String
|
||||
Write-Host "rebooting"
|
||||
shutdown /r /t 0
|
||||
}
|
||||
Загрузка…
Ссылка в новой задаче