This reverts commit 9a27951a99.
The purpose of the original commit was to take out 4 tasks that were only executed from Windows to the Windows specific pack (and enabling more tasks that need to be executed remotely on that pack).
To execute tasks remotely the build will initially run `SayHello` which connects to the Mac and shares the connection with all the tasks through IBuildEngine4[1]. When that connection object is retrieved on the tasks we need to cast it as a Messaging connection type, and this works from Visual Studio and msbuild because there's just one messaging assembly loaded.
But it doesn't work on dotnet/msbuild, because there's a feature called ALC (AssemblyLoadContext) which will load each assembly task into it's own context[2], loading all its dependencies in that context. In this scenario sharing custom objects between tasks won't work because the original object type could be from a different context than the one retrieving it. This is described here: https://github.com/dotnet/msbuild/issues/5084.
[1] https://docs.microsoft.com/en-us/dotnet/api/microsoft.build.framework.ibuildengine4?view=msbuild-16-netcore
[2] https://github.com/dotnet/msbuild/blob/master/documentation/specs/task-isolation-and-dependencies.md
* [dotnet] Stops merging Messaging into Xamarin.iOS.Tasks
- Creates a separate Xamarin.Messaging project that will produce a single assembly after merging all the Xamarin Messaging dependencies.
- Stops merging Messaging into Xamarin.iOS.Tasks to be able to reference it from the Windows specific pack.
- Refactors the ILMerge.targets to reuse common code from ILMerge.Messaging.targets.
* [dotnet] Moves Windows specific tasks to the Windows tasks project
* [dotnet] Adds Xamarin.Messaging.dll to the different packages
* [msbuild] Add logic to build & install Xamarin.Messaging.dll.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
These changes add support for executing iOS and MacDev tasks remotely (on a Mac) when running a build from Windows, and creates a specific .NET6 pack for Windows that's only included in the MSI.
For now this only enables builds for the iOS Simulator, physical devices are not yet supported.
- Each task decides if it should run locally or remotely depending on the SessionId property, which will only have a value on Windows.
- The XMA Build agent is now part of this repo and will be included in the iOS .NET6 Windows pack.
- On this first version we're including some Windows specific tasks and references into the Xamarin.iOS.Tasks project for simplicity, but those will be moved to the Windows specific project.
------------
* [msbuild] Adds support for executing Xamarin.iOS tasks from Windows
* [msbuild] Adds support for executing Xamarin.MacDev tasks from Windows
* Added XMA Build Agent to Xamarin.MacDev.Tasks.sln
* Fixes some MSBuild versioning problems
* Makes the XMA Build agent load Xamarin.iOS tasks
We need to load a type from the iOS tasks assembly so we can run the tasks requested by MSBuild from Windows. We only need to load Xamarin.iOS.Tasks.dll since MacDev.tasks is already embedded in that one.
There's a little trick on the csproj, we can't directly use the Xamarin.iOS.Tasks project ref assemblies because that includes both Xamarin.iOS.Tasks.dll and Xamarin.MacDev.Tasks.dll, so the MacDev tasks will collide. We use the project ref only for build dependency purposes but we add an assembly reference to Xamarin.iOS.Tasks.dll.
* Added Xamarin.iOS.Tasks.Windows project
* Removed unnecessary references on Xamarin.iOS.Tasks.Windows.csproj
* Adds Messaging assemblies when ILRepacking Xamarin Tasks
The Xamarin Task assemblies now depend on Messaging, so we need the Messaging assemblies to be packed into Xamarin.Mac.Tasks and Xamarin.iOS.Tasks. Also had to remove the direct Messaging dependencies from the build agent since those are already contained in Xamarin.iOS.Tasks
* Adds a reference to Messaging.Core targets to the Agent's project
* [msbuild] Adds Xamarin iOS Windows targets
* [msbuild] Adds missing dependencies to Xamarin.iOS.Tasks
This should fix build errors because of missing dependencies. Had to move System.Net.Mqtt.Server from the Build agent project to the tasks one to avoid conflicts with System.Diagnostics.Tracer.
* [dotnet] Creates iOS Windows pack
Creates a new pack for Windows specific (targets, build agent, etc.) files that shouldn't be installed on the Mac. We have a separate package for this to avoid increasing the core pack size with things that are not needed when using it from macOS.
* Fixes type in dotnet makefile
* [dotnet] Fixes the iOS Windows pack generation
- The windows pack should not include the Sdk and Targets folders
- For now we'll just create an iOS pack
- Fixes the path to the files to include on the Windows Sdk pack
* Added reference to the Windows iOS SDK from the Xamarin.iOS.Common.targets
Added a property to navigate to the Windows iOS SDK folder, based on a naming convention that assumes that both packs will always have the same version
* Added reference to the core iOS SDK from the Windows iOS SDK
Added a property to navigate to the core iOS SDK folder, based on a naming convention that assumes that both packs will always have the same version
* Updated Messaging version
* Override MessagingBuildClientAssemblyFile property and correctly imported props from targets
* [dotnet] Make Windows pack using target files from the output dir
We need to take the target files from the output dir to include targets that are part of nuget packages, otherwise we will only include targets from our source
* [dotnet] Adds the Windows Sdk pack to the workload manifest
* [msbuild] Fixes the Windows Sdk pack name
* [dotnet] Merge Mqtt instead of Mqtt.Server
We only need System.Net.Mqtt to be merged into Xamarin.iOS.Tasks
* Updated Messaging version
* [dotnet] Several fixes for the Windows Sdk
- Adds missing task CollectMonotouchReferences
- Merges more dependencies into Xamarin.iOS.Tasks.dll needed by XMA
- Updates the msbuild/Makefile to include files from both the output dir and the source dir
- Overrides the agents directory to look for them on the Windows pack
* [dotnet] Fixes the XMA Build agent
- The build agent is an app so it cannot target ns2.0
- The MSBuild dependencies should be copied into the agent zip file
- Avoids copying all the Xamarin iOS SDK core targets into the build agent, since those are not needed
- Ensures the broker zip file is copied into the Xamarin.iOS.Windows.Tasks output dir so its included in the Windows pack
* Bumps Xamarin.Messaging to 1.2.102
* Adds net6-win branch to trigger builds
* Adds Messaging.Client missing dependency to Xamarin.Mac.Tasks
* Added Xamarin.Messaging.Apple.Tasks project and VerifyXcodeVersion Task
* Fix unloaded Xamarin.Messaging.Build project
* Added Build contracts project and unified Xamarin.Messaigng.Apple.Tasks in Xamarin.iOS.Tasks.Windows
Also added missing tasks and changes .After.targets
* Updated Xamarin.Messaging version
* Build agent - reference MSBuild assemblies from the framework
Since the assemblies will be included in the build agent we need those to be the ones that come from the framework to be compatible with macOS
* [msbuild] Fixes _UpdateDynamicLibraryId target
The tasks con this target need to be executed remotely (when building from Windows).
* Updates resources
* Bump Xamarin.Messaging
Fixes problems when executing Exec task remotely
* [dotnet] Overrides Publish targets to execute them remotely from Windows
The `_CopyResolvedFilesToPublishPreserveNewest` and `_CopyResolvedFilesToPublishAlways` targets essentially copy files into the app bundle. Since those are part of the .NET SDK we need to override those so we can pass to the Copy task the SessionId parameter and then it will be executed remotely when building from Windows.
This is done in a Windows.After.targets file so it won't affect builds on macOS.
* Added ILMerge to Xamarin.iOS.Tasks.Windows
Also modified ILMerge.targets to not include System assemblies because we don't need them on the Windows package
* Bumps Messaging
This new version of messaging fixes a problem when copying task inputs from Windows to the Mac
* [dotnet] Fixes copying files to the Mac when building from Windows
When building from Windows there are .NET SDK targets that copy dynamic libraries from the SDK to the intermediate output directory or other files to the publish directory, since we can't control those we can't run them remotely so we need to copy those files to the Mac to ensure other targets will find those.
* [dotnet] Fixes how files are copied to the output dir
- Before executing `_CopyResolvedFilesToPublishPreserveNewest` and `_CopyResolvedFilesToPublishAlways` we copy the input files for those targets to the Mac
- Then we override the original targets to execute the same copy task as the original ones but on the Mac, so the output files are placed in the right location for the following targets to pick them up.
* Fixes typo on Xamarin.iOS.Common.After.targets
* Bumps Xamarin.Messaging
* [msbuild] Fixes VerifyXcodeVersion and ResolveUTIs tasks
Both tasks were not being able to connect to the Mac mostly because of ILRepack, there were kind of 2 versions of Xamarin.Messaging, one merged into Xamarin.iOS.Tasks and another one merged into Xamarin.iOS.Windows.Tasks. Because of this the build connection object registered on the task could not be casted to the build connection type.
This essentially moves both tasks into the Xamarin.iOS.Tasks assembly to avoid this issue, and as part of that also includes the Messaging contracts into that same project.
* [msbuild] Fixes warnings when building from Windows
* [dotnet] Adds missing assemblies to merge into Xamarin.iOS.Tasks
Those 2 new assemblies will only be used from Windows and we need their implementation instead of the ref assemblies. In the future we will need to find a way of doing this on the Windows only pack insted of doing it on the core Xamarin.iOS.Tasks assembly.
* [dotnet] Compute PublishTrimmed on a target
We need to do this so the property is evaluated after VS on Windows connects to the Mac, otherwise by default IsMacEnabled is false from Windows.
* Bumps Messaging to 1.2.111
* [dotnet] Execute ILLink remotely when building from Windows
- Overrides the ILLink task and _RunILLink target to add the hability to execute it remotely, adding input and output properties so files are copied to the server and output files are created on Windows.
- This "custom" ILLink task will only be executed from the Windows targets so when building from a Mac it will execute the core SDK task.
* [dotnet] Fixes intput/output files creation for linker tasks
- Custom Linker options file should be created on the Mac so we need to execute WriteLinesToFile remotely
- All the *.items files from the linker are created on the Mac so we need to execute ReadItemsFromFile remotely
- CompileNativeCode: fixes the OutputFile metadata path, otherwise the execution fails; also copies all the files in the declared "IncludeDirectories" to the Mac
- Avoids copying input files from Windows to the Mac when running LinkNativeCode since the real input files already exist on the Mac, and Windows contains only empty files just to make MSBuild inputs/outputs check work. If we copy those empty files to the Mac we brake the build.
* [msbuild] Minor fixes after merging from main
* [dotnet] Adds missing output files to the Xamarin.iOS.Tasks.Windows project
The output of this project was missing Messaging build targets and the build agent zip file that are needed to create the dotnet Windows specific pack
* [dotnet] Fixes dotnet Windows specific pack generation
Ensures the Windows projects are built and the files are copied to the dotnet pack directory before creating the package.
It also adds a variable to enable building this pack.
* [dotnet] Adds iOS Windows specific pack to iOS only MSI
There's only a Windows specific pack for iOS available for now, so we should only add it to the iOS SDK MSI
* [dotnet] Create a separate bundle for the iOS Windows MSI
We need to do this to avoid including the Windows specific pack in the pkg. Also for now we'll only create an MSI for iOS since it's the only supported platform from Windows.
* Fixes spacing issues in Xamarin.iOS.Tasks.csproj
* Bumps Touch.Unit back to 05db76
* Fixes formatting problems
* [msbuild] Replaces error E0176 by E0186
Because there's a warning W0176 that will overlap with the error
* [msbuild] Fixes CompileEntitlements task
There were 2 problems:
1- The if statement on the DefaultEntitlementsPath was wrong, because we should return the base value if there's no SessionId (which means the task is running on a Mac)
2- We should copy to the Mac the default entitlements file if no custom file was specified
* Several fixes to cleanup the code to support iOS from Windows
* Apply suggestions from code review
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Formatting fixes in Xamarin.Messaging.Build
* Reverted formatting changes in CompileEntitlements.cs
* More formatting fixes
* Update msbuild/Messaging/Xamarin.Messaging.Build/Handlers/ExecuteTaskMessageHandler.cs
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Fixes order of MSBuild errors in the resource file
* Add newly added localizable strings to canary test of translated strings.
* Delete tests that ensure theres code only on the abstract tasks
These were needed to ensure all the code was in the base tasks so we could have tasks implementations on Windows to remote those. Now that code is part of this repo (and that is why these tests are failing now) so we do not need them anymore.
* [dotnet] Don't build the Windows SDK pack if not configured to do so.
Co-authored-by: mag <mauro.agnoletti@gmail.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
String Localization Capability and Unit Test
Co-authored-by: TJ Lambert <tjlambert@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* The ILRepack package uses a 4-year old version of Mono.Cecil, which does not
support portable pdbs. Work is in progress to use a newer version of
Mono.Cecil (https://github.com/gluck/il-repack/pull/236), but this is taking
some time (the PR is over a year old). In the meantime, switch to the
ILRepack.MSBuild.Task package, which ships a forked and updated ILRepack.exe
executable. This means the assembly merging process will produce a working
pdb for the merged assembly, which in turn means we'll get stack traces with
source code location.
* Update the makefile to ship the pdbs we produce. Also don't copy files into
a temporary build/ directory, since it's not needed. This avoids an
intermediate file copy.
Instead of having multiple symlinks in the /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac directory
pointing to files in /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/msbuild, have the
/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Mac directory point to the
/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/lib/msbuild directory. This way we
don't have to update the symlinks whenever there are new files somewhere.
* Bump Xamarin.MacDev.
New commits in xamarin/Xamarin.MacDev:
* xamarin/Xamarin.MacDev@210c664 Adds net451 to Xamarin.MacDev.csproj
* xamarin/Xamarin.MacDev@64db365 [winios] Changes provisioning profiles default path
* xamarin/Xamarin.MacDev@d34430a Switch to short-form projects and build for both net461 and netstandard2.0. (#68)
Diff: 0f578f51e6..210c664e56
* [msbuild] Update to latest Mono.Cecil.
The older version doesn't support netstandard2.0.
No code changes were required.
* [msbuild] Remove unused usings.
* [msbuild] Make ILMerge work when building for netstandard2.0.
Also unify/deduplicate the ILMerge logic between Xamarin.iOS and Xamarin.Mac.
* [msbuild] Build for netstandard2.0 in addition to net461.
* [msbuild] Use custom project configurations to support running the tests for both netstandard2.0 and net461.
Use custom project configurations to support running the tests for when the
tasks assembly is built for netstandard2.0 and net461.
* [tests] Make command-line based 'make test-ios-tasks' run tests for both netstandard2.0 and net461.
* [xharness] Add test configuration to run iOS MSBuild tests using either netstandard2.0 or net461.
* [msbuild] Make the netstandard2.0-buils task assemblies the default.
* [msbuild] ILRepack lib assemblies, not ref assemblies.
Ask MSBuild to copy lib assemblies to the output folder when building for
netstandard2.0, this way we can easily find the actual implementation
libraries to pass to ILRepack.
* [msbuild] Merge System.Text.Encodings.Web.dll as well.
* [xharness] Fix build of MSBuild tests for iOS.
* [msbuild] Convert to short-form csproj.
* [msbuild] Make asserts more useful.
* [msbuild] Make tests ignore the actual location of the test assembly.
* [msbuild] Short-style projects default to deterministic builds, which is not compatible with wildcard versions.
* [msbuild] Adjust test.
* Update .gitignore.
* Bump NUnit.ConsoleRunner version.
* [msbuild] Fix indentation.
* [msbuild] Simplify csproj.
* Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/801867
System.Memory.dll not being copied anywhere when Roslyn package is referenced
The issue was that the XM FrameworkList was shared for the 2 profiles and there are differences.
The Full profile for instance doesn't have `System.Memory` and that causes some issues with Nuget package references.
* Updated tests to report issues against the XM Full and Mobile Framework lists
* Now using 2 separate XM FrameworkList files (updated makefile) since the list of assemblies is different and that's expected by the mono team.
- Existing binding projects embed the native libraries within the assembly as managed resource
- This does not scale well and has performance implications
- This PR creates a new property, NoBindingEmbedding which when true processes the building and consumption of binding projects differently.
- Existing binding projects are not affected, they will continue as is
- I've written a full XM test suite and ported a subset to iOS. Since iOS only supports checked in projects, and I didn't want to make the existing situation worse by adding more, I only wrote tests that could use the existing test projects.
-When we complete some form of msbuild testing reform, we'll revisit these tests.
- Remove two files in MyiOSFrameworkBinding that are not used (we use copies elsewhere)
- Remove unnecessary sleep and fix broken touch command
- Output failing test log to console instead of test output
- VSfM does not handle thousands of lines of test failure message well
- Add ability to generate binding projects with LinkWith
Mono.Options doesn't (yet) support escaped quotes when parsing response files,
which becomes a problem because escaped quotes are necessary when passing
paths with spaces as gcc/linker flags.
So don't write gcc/linker flags in the response file, and instead pass them as
normal command line arguments, and to be on the safe side, do the same thing
for all extra arguments passed to mmp/mtouch.
Also add tests.
Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/649776.
Fixes this warning:
> Makefile:317: target `/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/msbuild/iOS' given more than once in the same rule.
because the same directory was listed twice.
For some strange reason the following sometimes work with make v3.8.1:
X/:
mkdir X
X/Y: X
mkdir X/Y
Note how the second target specifies `X` as a dependency, but the actual
target to be executed is `X/` (additional trailing slash).
It does not work with remake (v4.2.1), nor in a simple test case like the one
above with make (v3.8.1), but it works in our own makefiles (which are
admittedly slightly more complicated).
Since it's trivial to fix, and I don't understand how it works in make in the
first place, I'm just changing it to what makes sense (and works everywhere):
remove trailing slashes from all directories that are used as targets.
* [msbuild] Pack all iOS MSBuild Task assemblies into a single assembly
* Fixed the build
* Renamed ProcessArgumentBuilder to CommandLineArgumentBuilder
This is needed to prevent symbol conflicts with Xamarin.MacDev's
ProcessArgumentBuilder (which is functionally different from
Xamarin.MacDev.Tasks.Core's class of the same name).
* Fixed ILRepack logic for filtering dll's to repack
* Fixed building of Xamarin.iOS.Tasks.Tests now that X.iOS.Tasks.dll contains all symbols
* Updated Makefile now that only 1 iOS Task assembly needs to be distributed
* ILRepack Xamarin.Mac.Tasks as well
* Fixed up *.targets to specify The One Assembly To Rule Them All
* [xharness] Build MSBuild tests with MSBuild.
* Touch the ilrepack stamp file *after* invoking ILRepack, not before.
* Same for Xamarin.Mac.Tasks
* [builds] Improve mono/llvm dependencies.
* Create a list of all the files in the mono and llvm repositories, and save
these lists as a Make variable (in a generated Makefile - .deps.*.mk). We
don't list _all_ the files in each repository, because there are quite a few
(55k for mono), and Make measurably takes a while to check all of them, so
try to limit it to a sane subset, without risking missing changes to files
that actually matters.
* Always create stamp files when we're done with mono builds.
* Modify the mono/llvm builds to depend on all the files in their
repositories.
* Explicitly list the corresponding .stamp-build-* files as dependencies for
various files that are produced by the mono builds, so that make knows how
to build these files.
* Rewrite the *-facade-check targets to depend on the corresponding
*_BCL_TARGETS, so that we can avoid running a submake to the same Makefile
to execute the facade checks.
It now takes a little while (less than a second on my machine, which is
fine) for make to list all dependencies and get their timestamps, but if
executing multiple submakes this adds up to a multi-second timewaste.
So avoid the timewaste by not doing submakes, but instead use dependencies
to enforce the required target execution ordering.
* Don't depend on nicely named intermediate targets, since won't prevent
rebuilds:
build-cross64: setup-cross64
Since the `setup-cross64` file doesn't exist, `build-cross64` will always
execute. Instead depend on the stamp file:
build-cross64: .stamp-configure-cross64
And now `build-cross64` will only rebuild if needed.
* Don't try to list all intermediate files as .SECONDARY dependencies, instead
list none at all, which works as if all files were listed as dependencies.
* Some targets had to move later in the file, since variables used in dependencies:
foo: $(VARIABLE)
must be defined before that point in the file, as opposed to variables used in recipes:
foo:
$(MAKE) $(VARIABLE)
can be defined anywhere in the Makefile.
* Simplify the targets that sign assemblies significantly.
There are a few end results:
* It's now possible to do `make install`, without doing `make all` first. This
might seem weird, but that also ensures the more common `make all install`
works properly.
* Remakes (without any mono/llvm changes) in build/ are much faster, because
we now won't recurse into every mono build:
$ time make all -C builds/ -j8
[...]
real 0m1.873s
This even means that we might be able to make it a habit to remake in the
root directory, which doesn't take forever now:
$ time make all -j8
[...]
real 0m4.521s
Unfortunately adding `make install` to the mix still does some useless
stuff, and it ends up taking ~30 seconds to complete a full build:
$ time make all install -j8
[...]
real 0m32.542s
* [msbuild] Don't verify the xml syntax of targets files unless the files change.
* [build] Don't depend on installed files.
Don't depend on installed files, because that causes a rebuild when installing
to a different directory (i.e. package creation).
* Bump maccore to get build improvements.
Rebuilds are now very fast:
$ make all install -j8
$ time make all install -j8
real 0m5.735s
Less than 6s to figure out that nothing needs to be done.
And strangely flushing the disk cache doesn't make it much slower:
$ sudo purge
$ time make all install -j8
real 0m7.309s
Which probably means that Make mostly reads file metadata, and not actual file
contents (which is good).
* [msbuild] Remove unused FrameworkList.xmls
* [msbuild] Make files in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/msbuild/iOS the real deal, not a symlink.
* [msbuild] Make /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS a symlink, instead of each file inside.
* [msbuild] Don't put anything in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/mono/2.1 anymore.
* [msbuild] Remove support for XI/Classic binding projects.
* Improve 'install-system' to clean up old files.
* [msbuild] Simplify XI/Classic targets files a bit.
* [msbuild] Remove dead XI/Classic code.
* Bump maccore to get fix for xamarin-analysis.
commit xamarin/maccore@34c04c2bf1
Author: Rolf Bjarne Kvinge <rolf@xamarin.com>
Date: Mon Oct 10 16:46:18 2016 +0200
[analysis] Update to put files in /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS.
XI/Classic is being removed now, which means files should go into
/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/msbuild/iOS/ instead of into
/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/mono/2.1.
* Enables CoreCompile target for WatchOS App projects
The iOS Designer depends on Roslyn Workspace APIs to inspect and get notified of project changes, which needs CoreCompile target to work.
Fixes Bug #41766 (https://bugzilla.xamarin.com/show_bug.cgi?id=41766)
* [msbuild] Adds empty cs file to avoid errors and warnings when building watchOS apps
Xbuild fails to build projects with no @(Compile). This change workaround it for watchOS apps.
* [msbuild] Add nuget diagnostics.
* [msbuild] Make restore less verbose.
To see if that's what's causing an infinite newline print loop on jenkins bots.
* [msbuild] nuget can't list sources on jenkins for some reason, so just do it manually.
* [msbuild] Make sure diagnostics doesn't fail the build.
Emanuel Fernandez Dell'Oca
Rolf Bjarne Kvinge
TJ Lambert
Waleed Chaudhry
Vincent Dondain
Chris Hamons
Jeffrey Stedfast
joj