This is a log from our bots, note the 14 minute gap just before printing the timing results:
```
[...]
2024-09-27T07:34:00.3958920Z Making install in dotnet
2024-09-27T07:34:01.7633820Z Validated file permissions for Xamarin.Mac.
2024-09-27T07:34:01.7800150Z Validated file permissions for Xamarin.iOS.
2024-09-27T07:34:01.7825300Z
2024-09-27T07:34:01.7872490Z Xamarin.iOS has not been installed into your system by 'make install'
2024-09-27T07:34:01.7918570Z In order to set the currently built Xamarin.iOS as your system version,
2024-09-27T07:34:01.7965090Z execute 'make install-system'.
2024-09-27T07:34:01.7987920Z
2024-09-27T07:34:01.8034290Z Xamarin.Mac has not been installed into your system by 'make install'
2024-09-27T07:34:01.8080260Z In order to set the currently built Xamarin.Mac as your system version,
2024-09-27T07:34:01.8126200Z execute 'make install-system'.
2024-09-27T07:34:01.8148530Z
2024-09-27T07:48:22.3100850Z
2024-09-27T07:48:22.3102130Z real 15m26.160s
2024-09-27T07:48:22.3102800Z user 1m4.044s
2024-09-27T07:48:22.3103270Z sys 0m18.379s
```
What happens is this:
* We're using parallel make, and parallel make will start a jobserver, managed by file descriptors, where these file descriptors must be closed in all subprocesses for make to realize it's done.
* Any 'dotnet build' might start a build server
* The build server does not close any file descriptors it may have inherited when daemonizing itself.
* Thus the build server (which will still be alive after we're done building here) might have a file descriptor open which make is waiting for.
* The proper fix is to fix the build server to close its file descriptors.
* The intermediate working is to shut down the build server instead.
This will save 10-15 minutes at the end of every build in the bots.
* Don't load maccore/mk/versions.mk anymore, which means we're not checking
any subsequent dependencies listed in maccore. Note that we still need the
maccore dependency itself for a little while longer.
* Remove some outdated testing code that called into maccore.
* Don't recurse into the maccore directory during make.
* Remove some code checking for ENABLE_XAMARIN that's not used anymore, in
particular in xharness.
Make our local .NET the default .NET (in the root's global.json), and then if
a directory wants to use the system .NET, then that directory would have to
opt-in (using its own global.json).
This way we don't have to copy global.json/NuGet.config files around to run
tests with the correct .NET setup.
Also rename DOTNET_VERSION to SYSTEM_DOTNET_VERSION to make it clear what it's
referring to (and to not clash with DOTNET6_VERSION which has now been renamed
to DOTNET_VERSION).
.NET 7 is right around the corner.
* Having .NET enabled with the windows-specific parts disabled is not a
scenario tested on CI (where we've always enabled both). This means it's
prone to bitrotting.
* It's another configuration to keep track of and make work locally.
* It doesn't work right now anyway.
So just always enable the windows-specific parts of the .NET build, if the
.NET build is enabled.
* [build] Use arcade dependency management tooling
* Apply feedback
* Apply second round of feedback
* Always make dotnet.config before trying to read it
* Debugging
* Update dependencies, trim tabs and spaces
* [dotnet] Remove the existing workload shipped with .NET and install our locally built ones.
The new version of .NET ships with our workloads, but those aren't
the workloads we want to use, so replace them with our own.
* Update .gitignores.
* Bump to 6.0.100-preview.3.21181.5
That required renaming simulator runtime packs...
* More rename for simulator packages
* moar (hopefully all)
* Bump to 6.0.100-preview.3.21201.11
This fix the issue with `Wait` that failed several tests in monotouch-tests
However it does not include the fix for AppConext.GetData on device (AOT)
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Sebastien Pouliot <sebastien@xamarin.com>
These changes add support for executing iOS and MacDev tasks remotely (on a Mac) when running a build from Windows, and creates a specific .NET6 pack for Windows that's only included in the MSI.
For now this only enables builds for the iOS Simulator, physical devices are not yet supported.
- Each task decides if it should run locally or remotely depending on the SessionId property, which will only have a value on Windows.
- The XMA Build agent is now part of this repo and will be included in the iOS .NET6 Windows pack.
- On this first version we're including some Windows specific tasks and references into the Xamarin.iOS.Tasks project for simplicity, but those will be moved to the Windows specific project.
------------
* [msbuild] Adds support for executing Xamarin.iOS tasks from Windows
* [msbuild] Adds support for executing Xamarin.MacDev tasks from Windows
* Added XMA Build Agent to Xamarin.MacDev.Tasks.sln
* Fixes some MSBuild versioning problems
* Makes the XMA Build agent load Xamarin.iOS tasks
We need to load a type from the iOS tasks assembly so we can run the tasks requested by MSBuild from Windows. We only need to load Xamarin.iOS.Tasks.dll since MacDev.tasks is already embedded in that one.
There's a little trick on the csproj, we can't directly use the Xamarin.iOS.Tasks project ref assemblies because that includes both Xamarin.iOS.Tasks.dll and Xamarin.MacDev.Tasks.dll, so the MacDev tasks will collide. We use the project ref only for build dependency purposes but we add an assembly reference to Xamarin.iOS.Tasks.dll.
* Added Xamarin.iOS.Tasks.Windows project
* Removed unnecessary references on Xamarin.iOS.Tasks.Windows.csproj
* Adds Messaging assemblies when ILRepacking Xamarin Tasks
The Xamarin Task assemblies now depend on Messaging, so we need the Messaging assemblies to be packed into Xamarin.Mac.Tasks and Xamarin.iOS.Tasks. Also had to remove the direct Messaging dependencies from the build agent since those are already contained in Xamarin.iOS.Tasks
* Adds a reference to Messaging.Core targets to the Agent's project
* [msbuild] Adds Xamarin iOS Windows targets
* [msbuild] Adds missing dependencies to Xamarin.iOS.Tasks
This should fix build errors because of missing dependencies. Had to move System.Net.Mqtt.Server from the Build agent project to the tasks one to avoid conflicts with System.Diagnostics.Tracer.
* [dotnet] Creates iOS Windows pack
Creates a new pack for Windows specific (targets, build agent, etc.) files that shouldn't be installed on the Mac. We have a separate package for this to avoid increasing the core pack size with things that are not needed when using it from macOS.
* Fixes type in dotnet makefile
* [dotnet] Fixes the iOS Windows pack generation
- The windows pack should not include the Sdk and Targets folders
- For now we'll just create an iOS pack
- Fixes the path to the files to include on the Windows Sdk pack
* Added reference to the Windows iOS SDK from the Xamarin.iOS.Common.targets
Added a property to navigate to the Windows iOS SDK folder, based on a naming convention that assumes that both packs will always have the same version
* Added reference to the core iOS SDK from the Windows iOS SDK
Added a property to navigate to the core iOS SDK folder, based on a naming convention that assumes that both packs will always have the same version
* Updated Messaging version
* Override MessagingBuildClientAssemblyFile property and correctly imported props from targets
* [dotnet] Make Windows pack using target files from the output dir
We need to take the target files from the output dir to include targets that are part of nuget packages, otherwise we will only include targets from our source
* [dotnet] Adds the Windows Sdk pack to the workload manifest
* [msbuild] Fixes the Windows Sdk pack name
* [dotnet] Merge Mqtt instead of Mqtt.Server
We only need System.Net.Mqtt to be merged into Xamarin.iOS.Tasks
* Updated Messaging version
* [dotnet] Several fixes for the Windows Sdk
- Adds missing task CollectMonotouchReferences
- Merges more dependencies into Xamarin.iOS.Tasks.dll needed by XMA
- Updates the msbuild/Makefile to include files from both the output dir and the source dir
- Overrides the agents directory to look for them on the Windows pack
* [dotnet] Fixes the XMA Build agent
- The build agent is an app so it cannot target ns2.0
- The MSBuild dependencies should be copied into the agent zip file
- Avoids copying all the Xamarin iOS SDK core targets into the build agent, since those are not needed
- Ensures the broker zip file is copied into the Xamarin.iOS.Windows.Tasks output dir so its included in the Windows pack
* Bumps Xamarin.Messaging to 1.2.102
* Adds net6-win branch to trigger builds
* Adds Messaging.Client missing dependency to Xamarin.Mac.Tasks
* Added Xamarin.Messaging.Apple.Tasks project and VerifyXcodeVersion Task
* Fix unloaded Xamarin.Messaging.Build project
* Added Build contracts project and unified Xamarin.Messaigng.Apple.Tasks in Xamarin.iOS.Tasks.Windows
Also added missing tasks and changes .After.targets
* Updated Xamarin.Messaging version
* Build agent - reference MSBuild assemblies from the framework
Since the assemblies will be included in the build agent we need those to be the ones that come from the framework to be compatible with macOS
* [msbuild] Fixes _UpdateDynamicLibraryId target
The tasks con this target need to be executed remotely (when building from Windows).
* Updates resources
* Bump Xamarin.Messaging
Fixes problems when executing Exec task remotely
* [dotnet] Overrides Publish targets to execute them remotely from Windows
The `_CopyResolvedFilesToPublishPreserveNewest` and `_CopyResolvedFilesToPublishAlways` targets essentially copy files into the app bundle. Since those are part of the .NET SDK we need to override those so we can pass to the Copy task the SessionId parameter and then it will be executed remotely when building from Windows.
This is done in a Windows.After.targets file so it won't affect builds on macOS.
* Added ILMerge to Xamarin.iOS.Tasks.Windows
Also modified ILMerge.targets to not include System assemblies because we don't need them on the Windows package
* Bumps Messaging
This new version of messaging fixes a problem when copying task inputs from Windows to the Mac
* [dotnet] Fixes copying files to the Mac when building from Windows
When building from Windows there are .NET SDK targets that copy dynamic libraries from the SDK to the intermediate output directory or other files to the publish directory, since we can't control those we can't run them remotely so we need to copy those files to the Mac to ensure other targets will find those.
* [dotnet] Fixes how files are copied to the output dir
- Before executing `_CopyResolvedFilesToPublishPreserveNewest` and `_CopyResolvedFilesToPublishAlways` we copy the input files for those targets to the Mac
- Then we override the original targets to execute the same copy task as the original ones but on the Mac, so the output files are placed in the right location for the following targets to pick them up.
* Fixes typo on Xamarin.iOS.Common.After.targets
* Bumps Xamarin.Messaging
* [msbuild] Fixes VerifyXcodeVersion and ResolveUTIs tasks
Both tasks were not being able to connect to the Mac mostly because of ILRepack, there were kind of 2 versions of Xamarin.Messaging, one merged into Xamarin.iOS.Tasks and another one merged into Xamarin.iOS.Windows.Tasks. Because of this the build connection object registered on the task could not be casted to the build connection type.
This essentially moves both tasks into the Xamarin.iOS.Tasks assembly to avoid this issue, and as part of that also includes the Messaging contracts into that same project.
* [msbuild] Fixes warnings when building from Windows
* [dotnet] Adds missing assemblies to merge into Xamarin.iOS.Tasks
Those 2 new assemblies will only be used from Windows and we need their implementation instead of the ref assemblies. In the future we will need to find a way of doing this on the Windows only pack insted of doing it on the core Xamarin.iOS.Tasks assembly.
* [dotnet] Compute PublishTrimmed on a target
We need to do this so the property is evaluated after VS on Windows connects to the Mac, otherwise by default IsMacEnabled is false from Windows.
* Bumps Messaging to 1.2.111
* [dotnet] Execute ILLink remotely when building from Windows
- Overrides the ILLink task and _RunILLink target to add the hability to execute it remotely, adding input and output properties so files are copied to the server and output files are created on Windows.
- This "custom" ILLink task will only be executed from the Windows targets so when building from a Mac it will execute the core SDK task.
* [dotnet] Fixes intput/output files creation for linker tasks
- Custom Linker options file should be created on the Mac so we need to execute WriteLinesToFile remotely
- All the *.items files from the linker are created on the Mac so we need to execute ReadItemsFromFile remotely
- CompileNativeCode: fixes the OutputFile metadata path, otherwise the execution fails; also copies all the files in the declared "IncludeDirectories" to the Mac
- Avoids copying input files from Windows to the Mac when running LinkNativeCode since the real input files already exist on the Mac, and Windows contains only empty files just to make MSBuild inputs/outputs check work. If we copy those empty files to the Mac we brake the build.
* [msbuild] Minor fixes after merging from main
* [dotnet] Adds missing output files to the Xamarin.iOS.Tasks.Windows project
The output of this project was missing Messaging build targets and the build agent zip file that are needed to create the dotnet Windows specific pack
* [dotnet] Fixes dotnet Windows specific pack generation
Ensures the Windows projects are built and the files are copied to the dotnet pack directory before creating the package.
It also adds a variable to enable building this pack.
* [dotnet] Adds iOS Windows specific pack to iOS only MSI
There's only a Windows specific pack for iOS available for now, so we should only add it to the iOS SDK MSI
* [dotnet] Create a separate bundle for the iOS Windows MSI
We need to do this to avoid including the Windows specific pack in the pkg. Also for now we'll only create an MSI for iOS since it's the only supported platform from Windows.
* Fixes spacing issues in Xamarin.iOS.Tasks.csproj
* Bumps Touch.Unit back to 05db76
* Fixes formatting problems
* [msbuild] Replaces error E0176 by E0186
Because there's a warning W0176 that will overlap with the error
* [msbuild] Fixes CompileEntitlements task
There were 2 problems:
1- The if statement on the DefaultEntitlementsPath was wrong, because we should return the base value if there's no SessionId (which means the task is running on a Mac)
2- We should copy to the Mac the default entitlements file if no custom file was specified
* Several fixes to cleanup the code to support iOS from Windows
* Apply suggestions from code review
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Formatting fixes in Xamarin.Messaging.Build
* Reverted formatting changes in CompileEntitlements.cs
* More formatting fixes
* Update msbuild/Messaging/Xamarin.Messaging.Build/Handlers/ExecuteTaskMessageHandler.cs
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Fixes order of MSBuild errors in the resource file
* Add newly added localizable strings to canary test of translated strings.
* Delete tests that ensure theres code only on the abstract tasks
These were needed to ensure all the code was in the base tasks so we could have tasks implementations on Windows to remote those. Now that code is part of this repo (and that is why these tests are failing now) so we do not need them anymore.
* [dotnet] Don't build the Windows SDK pack if not configured to do so.
Co-authored-by: mag <mauro.agnoletti@gmail.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [src] Build the .NET version of our product assemblies using a .NET 5 BCL.
We're not shipping the .NET product assemblies in any stable release, so we
can use a preview version of the .NET 5 BCL.
Also:
* Add all the nuget feeds we need to the top-level NuGet.config, even for .NET
5/6, there shouldn't be any conflicts with stable feeds since we use exact
version numbers.
* Generate a top-level global5.json which is copied to every directory that
needs a .NET 5 global.json (overriding the .NET 3.1 global.json in the root
directory).
* Use the expected dotnet binary during our local build.
* [tests] Fix the bgen tests to use .NET 5.
* [xharness] Set the current directory to the project directory when running .NET tests.
This way we end up using the dotnet version that's configured in global.json for the tests.
* There's no need to check the system when running 'make install', we already
checked it during 'make all'.
* Remove the 'download-mono' dependency for the .stamp-download-mono target,
since that makes the .stamp-download-mono target always out of date (since
there will never be a 'download-mono' file). Instead execute the
'download-mono' target manually. This fixes an issue where running 'make
all' repeatedly would continue doing the same stuff over and over again.
Create the various NuGet packages to support .NET 5+. The packages are
currently empty (and not very useful), but the actual content will come later.
The current set of NuGet packages are (this list is duplicated for each
platform: iOS, tvOS, watchOS and macOS):
* Microsoft.iOS.Sdk: currently contains the basic MSBuild targets files for an
MSBuild Project SDK. Will eventually contain all the build logic. Might also
eventually contain other tools (mlaunch, bgen, etc.), but these might also
end up in a different package.
* Microsoft.iOS.Ref: will contain the Xamarin.iOS.dll reference assembly.
* Microsoft.iOS.Runtime.[RID]: will contain architecture-specific files
(libxamarin*.dylib, the Xamarin.iOS.dll implementation assembly, etc.):
The NuGets built on CI are automatically published to a NuGet feed.
The versioning for the NuGet packages required a few changes: OS bumps are now
changed in Make.versions instead of Make.config (this is explained in the
files themselves as well).
* [configure] Add --[enable|disable]-dotnet.
And make it enabled by default on CI and disabled by default elsewhere for now
(because it makes the build significantly slower).
* [system-dependencies] Add support for provisioning .NET.
Also write a global.json in the root directory which is how we select which
.NET version to use.
Add support for building on internal Jenkins.
Jenkins has been configured to build every branch on xamarin/xamarin-macios that contains a `jenkins/Jenkinsfile`, which means it will start working as soon as this PR is merged.
Results will be posted as statuses on each commit, which can be viewed using the url `https://github.com/xamarin/xamarin-macios/commits/<branch>`:
* The `continuous-integration/jenkins/branch` status links to the jenkins job.
* The other two are XI and XM packages (the `Jenkins-` prefix will be removed once we officially switch from Wrench to Jenkins).
More detailed information will be added as a comment to each commit, which can be seen by clicking on the commit and scrolling to the bottom (url of the format `https://github.com/xamarin/xamarin-macios/commit/<sha1>`)
Unfortunately GitHub does not display the commit statuses when viewing a single commit, so to view those statuses you'll have to view the list of commits (the `/commits/` url). Tip: it's possible to use `<sha1>` instead of `<branch>` (and vice versa for that matter) if you're interested in the statuses of a particular commit.
Pull requests will also be built (only from contributors with write access), but by default nothing will be done (the job will exit immediately, although a green check mark will still show up). Jenkins will **not** add a comment in the pull request in this case.
However, if the label `build-package` [1] is set for a pull request, the internal jenkins job will run (it will do everything except the local xharness test run: this includes creating and publishing packages, creating various diffs, run tests on older macOS versions, test docs, etc). A detailed comment will also be added to the pull request (see below for multiple examples), which means that there will be two Jenkins comments: one for the public Jenkins which builds every PR, and one for the internal Jenkins [2].
[1] I don't quite like the name of the label, because it doesn't get even close to explain all that will actually happen, but `run-on-internal-jenkins-and-create-package` is a bit too long IMHO... Also it's non-obvious that this is the label to apply if the reason for executing on the internal jenkins is some other reason (for instance to test a maccore bump). Other ideas:
* `run-internal-jenkins`: doesn't make it obvious that a package will be created (which is probably the most common reason to want to run on internal jenkins)
* We could have multiple labels that mean the same thing: `build-package`, `internal-build`, `run-internal-jenkins`, etc, but it's redundant and I don't quite like it either.
* Any other ideas?
[2] I'm noticing now that these two look quite similar and this might end up confusing (the main difference is that the comment from the public jenkins will say **Build success/failure** and **Build comment file:** at the top. If something goes wrong the failure will also show up differently). Should this be made clearer?
* Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes#3316.
If the Xamarin build is enabled, then build mlaunch from source, otherwise get it from macios-binaries.
Fixes#3316.
* Bump macios-binaries.
Commit list for xamarin/macios-binaries:
* xamarin/macios-binaries@69a9088 Bump mlaunch to xamarin/maccore@4505cd6f02 (#8)
Diff: e1e8bdf7a8...69a90882a0
* make msbuild/ after tools/.
Build msbuild/ after tools/, so that mlaunch builds when building from source,
since its build needs the MSBuild logic installed by msbuild/.
* Bump macios-binaries to get Xcode 9.3-capable mlaunch.
[VSTS] Adding --no-sudo option in system-dependencies.sh
This PR also brings in the package step from the release-scripts makefile. This allows all the build tasks and scripts to be self-contained in one repo.
* [src] Rebuild Constants.cs when Make.config.inc changes.
Also unify the makefile targets/dependencies for the various Constants.cs recipes.
* [build] Fix dependencies for various installed files.
Some auto-generated installed files with version numbers must depend on
Make.config.inc, since that's where the revision number is stored.
Make.config.inc depends on Makefile, so there's no need to depend on Makefile
as well.
* [build] Fix build error.
Rolf Bjarne Kvinge
Haritha Mohan
Peter Collins
Manuel de la Pena
Alex Soto
Emanuel Fernandez Dell'Oca
Sebastien Pouliot
Alexander Köplinger
Vincent Dondain
Joseph Stroman
Joseph Stroman
Chris Hamons
