This has been bothering me for a while... the symptom is that the build just
hangs at the end. Curiously it's never happend on the bots, only locally.
1. It only happens when using parallel make. When using parallel make, make is
in a jobserver mode, where sub-makes are controlled using a pair of file
descriptors inherited by the sub-makes. A consequence of this algorithm is
that the controlling make process will wait until all inherited file
descriptors have been closed before it will realize that all its sub-makes
have finished.
2. 'dotnet pack' will build the corresponding project, and that might start a
background compiler server.
3. This background compiler server does not seem to close any file descriptors
it inherits.
4. The background compiler server does not necessarily exit by the time `make`
is done.
5. The result is that `make` things there are still sub-makes doing stuff,
because there are inherited file descriptors still open.
6. Killing the compiler server (in another terminal for instance) will make
make realize it's done (and the hang is resolved).
So I'm applying the last point: shutting down the compiler server after
packing all the .NET NuGets.
Fixes https://github.com/xamarin/xamarin-macios/issues/13355.
Xcode14 broke actool. We have created two diff bot images:
- Stable: Contains the latests stable xcode (xcode13.4)
- Beta: Contains the latests beta xcode OR will allow it to be
installed.
By doing this we mitigate the possible issues that the new Xcode might
create. Some of the bots have already been migrated.
This commit SHOULD NOT be backported to xcode14.
When the label is not present we set the value to false. We also needed
to make sure that the project is checked out when we try to write the
comment.
This should be backported to xcode14.
Allow to pass the xcode channel we are going to be using. This is to
ensure that we do not try to build the xcode14 branch in beta bots. At
the moment all bots are using the Beta channel, but once the lab has
been updated we should be able to move main to Stable.
Get the bot information to improve the debugging of build failures. The
following is an example of the output:
Software:
System Software Overview:
System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Computer Name: Mandels Home iMac Pro
User Name: Manuel de la Pena Saenz (mandel)
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
Time since boot: 3 days 23:43
Hardware:
Hardware Overview:
Model Name: iMac Pro
Model Identifier: iMacPro1,1
Processor Name: 8-Core Intel Xeon W
Processor Speed: 3,2 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per Core): 1 MB
L3 Cache: 11 MB
Hyper-Threading Technology: Enabled
Memory: 64 GB
System Firmware Version: 1731.120.10.0.0 (iBridge: 19.16.15071.0.0,0)
OS Loader Version: 540.120.3~6
Serial Number (system): C02WD0V7HX8F
Hardware UUID: 85EE3276-4E8F-592A-A47B-599DFAB6DF1C
Provisioning UDID: 85EE3276-4E8F-592A-A47B-599DFAB6DF1C
Activation Lock Status: Disabled
Developer:
Developer Tools:
Version: 13.3.1 (13E500a)
Location: /Applications/Xcode_13.3.0.app
Applications:
Xcode: 13.3.1 (20103)
Instruments: 13.3.1 (64552.71)
SDKs:
DriverKit:
21,4:
iOS:
15,4: (19E239)
iOS Simulator:
15,4: (19E239)
macOS:
12,3: (21E226)
tvOS:
15,4: (19L439)
tvOS Simulator:
15,4: (19L439)
watchOS:
8,5: (19T241)
watchOS Simulator:
8,5: (19T241)
Other teams in xamarin/microsoft do no use forks but use dev branhces.
The correct thing in our repo is to use forks, yet other developers
insist in not following our developement practices. The fact that this
branches are created results in 2 builds:
- One for CI
- On for the PR
It is harder to educate other developers than it is to ignore their
branches, therefore we have added the pattern dev/* to the exclude
list for branches in the CI build.
* Add these methods to shared code so they're available on all platforms
(they're already available on mobile platforms), since there's no reason to
exclude them on macOS:
* NSObject.Init
* NSObject.Alloc
* NSObject.InvokeInBackground
* Remove unused usings.
* Move identical code in platform-specific files to shared code.
There are certain bots misbehaving and taking longer. We believ ethat it
might be due to some throttiling depending on their positio in the lab.
We are increasing the timeout for those cases.
Attempts to push build asset information to Maestro started failing
recently:
D:\a\1\s\xamarin-macios\packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21212.6\tools\SdkTasks\PublishBuildAssets.proj(43,5): error MSB4062: The "PushMetadataToBuildAssetRegistry" task could not be loaded from the assembly C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.maestro.tasks\1.1.0-beta.20570.1\tools\netcoreapp3.1\Microsoft.DotNet.Maestro.Tasks.dll. Could not load file or assembly 'C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.maestro.tasks\1.1.0-beta.20570.1\tools\netcoreapp3.1\Microsoft.DotNet.Maestro.Tasks.dll'. The system cannot find the path specified.
Commit a1d0b6eb looks like it may have broken this, as it changed the
`globalPackagesFolder` used for NuGet packages across the repo.
Looking at [PublishBuildAssets.proj][0] we should be able to set the
`$(NuGetPackageRoot)` property to the new `globalPackagesFolder` value,
fixing attempts to load `Microsoft.DotNet.Maestro.Tasks.dll`.
[0]: b8007eed82/src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/PublishBuildAssets.proj (L32)
Fixed the following two issues:
1. Update tests to use the correct constructor.
2. Renamed $context because it is a known pester variable name and would
get overriden.
We are getting errors with the following:
> Error creating archive at '/Users/builder/azdo/_work/2/s/diagnostic-sim-output/output.tar.gz'.
> Files are still in /Users/builder/azdo/_work/2/s/diagnostic-sim-output/output
> An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=17):
> Unable to write or file already exists
> File exists
we remove the path in case it is present and continue even if there was
an error since it does not imply a test failure.
This allows the CI to run ALL the tests that the project has in
parallel. This is divided in two main changes:
1. Xharness - We move away from using boolenas to use a flag that states
the tests to run.
2. yaml - We have move the code to use a template per label. This new
jobs all run in parallel and the results are later collected by a
funel job
3. pwsh - Added a new class that understands that we have several mark
downs with the tests results. The classes parses them and them writes
a single comment (and example can be found here: https://github.com/xamarin/xamarin-macios/pull/15201#issuecomment-1162366240
The changes gives the following advantages vs how we used to run tests:
1. The CI run for all tests moves from taking 13 hours to 3/4 hours
(depending on the number of bots in the pool).
2. The download needed to verify the results on a case of failure is
smaller. Rather than downloading several GBs we now just download
that part of the html that we are interested in.
3. Better bot utlization. Bots are just used to a max of 2 hours, this
means that we can use the bots better since they are fragmented.
4. Less VMs. VSDrops has added support for macOS and Linux, we take
advanges of that here.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Collect diagnostic logs from the simulator to try to get better diagnostic information for https://github.com/xamarin/maccore/issues/2558.
Interestingly it seems like just collecting diagnostic information makes the problem much less likely to occur...
Tests work because other tests do use the nugets and gets picked up by
the runner. Yet, if we execute this projects witjout others, the tests
will fail when trying to use the nunit2 format.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
The job that hides the previous bot comments tries to find comments with
[OR\CI build] to decide if a comment should be hidden. This is added
when a title is present, so we just add a titles so that we can hide the
comments.
Note that the call to native code still _always_ happen (not cached) since the application could use `class_addProtocol` to add conformance to a protocol at runtime.
So the cache is limited to the .net specific reflection code that is present (only) when the dynamic registrar is included inside applications. This is the default for macOS apps, but not iOS / tvOS or MacCatalyst apps.
The linker/trimmer will remove the caching code when the dynamic registrar is removed. IOW this PR should not have any impact, performance or size, for most iOS apps (where the dynamic registrar is removed by default).
Fix https://github.com/xamarin/xamarin-macios/issues/14065
Running Dope on macOS, a 2 minutes benchmark, shows the following times (in seconds and percentage) spent calling this API:
## Before
<img width="1051" alt="Screen Shot 2022-06-15 at 9 21 22 PM" src="https://user-images.githubusercontent.com/260465/174201703-a91860a5-ec29-4e19-9de0-5158fd7aafa7.png">
* `RemoveFromSuperview` 7.99s (6.4%)
* `NSObject.ConformsToProtocol` 3.26s (2.6%)
## After
<img width="1228" alt="Screen Shot 2022-06-15 at 9 24 42 PM" src="https://user-images.githubusercontent.com/260465/174201708-92193e77-ea8e-41bc-9672-bddaaa18a4f6.png">
* `RemoveFromSuperview` 4.67s (3.8%)
* `NSObject.ConformsToProtocol` 0.32s (.26%)
So a 10x improvements on `ConformsToProtocol` which helps a lot the code path calling `RemoveFromSuperview`.
This makes it not necessary to check for the currently selected Xcode in our
system dependency check. It also means it'll become much easier to work with
multiple branches simultaneously where each branch needs its own Xcode.
When we changed SCNMatrix4 to be column-major instead of row-major in .NET, there
were several other related changes we should have done but didn't do. In particular
we should have made transformation operations based on column-vectors instead of
row-vectors.
In legacy Xamarin, a vector would be transformed by a transformation matrix by doing
matrix multiplication like this:
[ x y z w] * [ 11 21 31 41 ]
| 12 22 32 42 |
| 13 23 33 43 |
[ 14 24 34 41 ]
In this case the vector is a row-vector, and it's the left operand in the multiplication.
When using column-major matrices, we want to use column-vectors, where the vector
is the right operand, like this:
[ 11 21 31 41 ] * [ x ]
| 12 22 32 42 | | y |
| 13 23 33 43 | | z |
[ 14 24 34 41 ] [ w ]
This affects numerous APIs in SCNMatrix4, SCNVector3 and SCNVector4:
* The M## fields have been changed to make the first number the column and the
second number the row, to reflect that it's a column-major matrix (this is
also how it's defined in the native SCNMatrix4 type).
* Functions that return a transformation matrix have been modified to return column-vector
transformers. Technically this means that these matrices are transposed compared
to legacy Xamarin. The functions involved are:
* CreateFromAxisAngle
* CreateRotation[X|Y|Z]
* CreateTranslation
* CreatePerspectiveFieldOfView
* CreatePerspectiveOffCenter
* Rotate
* LookAt
* Combining two column-vector transforming transformation matrices is done by multiplying
them in the reverse order, so the Mult function (and the multiplication operator)
have been modified to multiply the given matrices in the opposite order (this matches
how the SCNMatrix4Mult function does it). To make things clearer I've changed the
parameter names for XAMCORE_5_0.
* Functions that transform a vector using a transformation matrix have been modified
to do a column-vector transformation instead of a row-vector transformation. This
involves the following functions:
* SCNVector3.TransformVector
* SCNVector3.TransformNormal
* SCNVector3.TransformNormalInverse
* SCNVector3.TransformPosition
* SCNVector4.Transform
* Numerous new tests.
Fixes https://github.com/xamarin/xamarin-macios/issues/15094.
* Enable Nullability
* throw better exceptions
* use is null
* Style fixes
* address Rolf suggestions
* use better syntax
* make properties private and use ActualOpenGLContext
Co-authored-by: tj_devel709 <antlambe@microsoft.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
This was mostly a clean merge, with a few minor differences:
* We no longer compute whether we're running in the simulator or not when building for Mac Catalyst.
* The task now supports building remotely for macOS (due to code sharing).
Will be useful if we ever support building macOS apps remotely.
* We now call AppleSdkSettings.Init () on macOS. No idea why we weren't
before, but it seems logical for macOS to behave like our other platforms.
There shouldn't be any other functional differences.
Rolf Bjarne Kvinge
Manuel de la Pena
Steve Hawley
dependabot[bot]
Naveen
Peter Collins
nathannaveen
Sebastien Pouliot
github-actions[bot]
TJ Lambert
