This requires passing the root directory around in multiple places, since ResolveAllPaths
doesn't have access to the static class where we define the root directory.
Also call ResolveAllPaths in a few more places to ensure paths everywhere are resolved.
In .NET projects there's a default value for most properties, which means that
there won't necessarily be an AssemblyName property in a csproj. We need to know the
AssemblyName, so calculate it from the csproj filename (which is how .NET does it).
This turned out slighly complicated, because we're pass an XmlDocument around,
and the XmlDocument doesn't know the file from where it was loaded, so we need
to keep that information separately.
This also means that setting 'run-all-tests' enables the .NET tests, which
means they'll now start running on internal Jenkins (as was the intention from
the beginning).
* [Harness] Add support to create tunnels.
Add support to create tunnels in case the devices cannot connect to
the host. This option is false by default, which means that unless told
otherwise xharness will try to se a tcp connection over the WiFi.
We are not setting it as default because the devices in DDFun will have
access to an unrestricted network. Nevertheless after this PR we will
create a new one with the following logic:
1. Try to use the tcp connection using the network.
2. If devices cannot connect to the host via the network, fall back to
the tcp tunnel.
This change executes a tunnel process per test application, most of the
cases out of the 150 test application we execute, most of them (maybe 98%
but most % are made up) will pass, and just a few of them will fail. The
reason is that there is a daemon in the OS that gets underwater.
Rather than tryingt o find a hacky way to re-use the tunnel, lets KISS
and if we need to hack that, do it as an enhancement.
Moved all the code that can be shared with the CLI to the common
library. We neede some small changes, but mainly due to namespaces and a
forgotten interface implemenation (was already implemented, was missing
in the class).
This was fun :)
Refactor TestProject so that we can move all the tasks to the common
assembly. We had to remove all the references from Harness, that
included the MonoNativeInfo.
Move all the logic outside and use it as a Composition pattern, later
this class can be used in the CLI so that we share the logic of building
and tested.
In order to de-couple the RunTestTask from Jenkins, create an interface
to be implemented, which is pass to use as a member (Composition
pattern). In order to do that, do not expose Jenkins as a property of
the interface because it is required just by the base constructor.
Moving the the use an interface meant a lot of small changes that
should have no real effect (the compiler should have caught any possible
issues).
The initial idea of the refactoring looked nice but as soon as we want
to get the RunTestTask out of jenkins, we have a number of naming
issues. Move the tools to not use the *Task postfix so that it is
cleaner and we can later extra the RunTestTask better.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Move all the logic outside of the Jenkins namespace. Rework a little the
inheritance to make it nicer in the constructors.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Move the task and use composition so that we can reuse the code. This
will later allow other projects to use the class without the need of
Jenkins or Harness and just implement the base class.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Some of the Jenkins test tasks are very useful and do A LOT of stuff. So
we try to generalize the base class, to later be able to share the most
usebul ones in the shared lib.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
We weren't properly reporting all failures when there were multiple failures
in parameterized tests, because we'd incorrectly skip too many xml nodes when
we were looking for the next test failure.
In particular NUnit uses reflection to get a private method, and the linker removes
the corresponding private method:
1c680b4dc8/src/NUnitFramework/framework/Internal/TestExecutionContext.cs (L552)
So add an xml definition to keep this private method, and modify project files to
pass the xml definition to mtouch and mmp.
Some care needs to be taken to make sure xharness is still able to clone these project
files.
* [xharness] Mark the .NET tests project as a .NET project so that we don't try to call nuget restore on it.
* [xharness] Mark the .NET generator project as a .NET project as well.
Packages are listed in the csproj when we're using package references.
Also only list files in git, otherwise we pick up all the generated project files as well.
It turns out some Xamarin.Mac projects are library projects (for unit tests),
and we still need to generate makefile targets for those (because we use them
to build Xamarin.Mac tests to run on older macOS bots).
It's a valid scenario to set a null environment variable.
This happens when changing InCI to always return true (to test locally what's
done on the bots), in which case we try to forward BUILD_REVISION to mlaunch,
but BUILD_REVISION isn't necessarily set. Not forwarding it to mlaunch should
not cause problems in most tests, so just allow this scenario.
According to the existing code, we're already not supposed to create makefile
targets for BCL tests. However, the code to detect BCL tests wasn't quite
reliable, so we ended up creating makefile targets for some BCL tests.
So always use the same logic to detect BCL tests.
Also remove a lot of dead code to generate makefile code for BCL tests.
Split the BCL project generation in two: one part that figures out which
projects to generate, and which is done synchronously (because we need it to
compute the list of tests), then split out the actual generation and run it on
a background thread (since that doesn't have to happen until we want to
execute those tests).
This speeds up launching xharness in server mode significantly (from ~2s to
~0.2s).
* [xharness] Don't generate into the same directory/files for macOS Modern and macOS Full.
This meant that we were overwriting some generated files, which meant that we
were executing the Modern set of tests instead of the Full set of tests for at
least some configurations.
* Add a few ignored tests to the System.Configuration test now that we actually run it.
When a process times out, we try to print a stack trace for all threads. This
involves executing "lldb --source <script>", where the script contains:
process attach --pid 1234
thread list
thread backtrace all
detach
quit
Basically we attach to the project, and ask lldb to print stack traces.
The problem:
16:09:02.9522580 Printing backtrace for pid=25276
16:09:02.9528060 /usr/bin/lldb --source /var/folders/q7/mkzwrzcn7bzf3g2v38f3c1cw0000gn/T/tmp58e75d85.tmp
16:09:04.6127570 (lldb) command source -s 0 '/var/folders/q7/mkzwrzcn7bzf3g2v38f3c1cw0000gn/T/tmp58e75d85.tmp'
16:09:04.6130020 Executing commands in '/var/folders/q7/mkzwrzcn7bzf3g2v38f3c1cw0000gn/T/tmp58e75d85.tmp'.
16:09:04.6130200 (lldb) process attach --pid 25276
16:09:05.6458750 error: attach failed: Error 1
16:09:05.7529100 25276 Execution timed out after 1200 seconds and the process was killed.
16:09:05.7588770 Execution timed out after 1200 seconds.
If any of those commands fail, the subsequent commands aren't executed. This
includes the final "quit" command, which means we end up waiting forever for
lldb to do its thing, when lldb doesn't think it needs to do anything at all.
The fix: pass two different scripts. It turns out subsequent scripts are
executed even if previous scripts fail, so we do the equivalent of:
lldb --source <attach script> --source <quit script>
And now lldb will quit no matter what the attach script does (it still works
even if the attach script succeeds, in which case we'll ask lldb to quit
twice).
This avoids:
* Getting the resource stream for every template expansion
* Reading the resource stream
* A lot of async code.
Instead just read the resource stream once, into a string, and return that.
This speeds up the startup by ~10% (from ~0.96s to ~0.83s on my machine).
Grouping simulators means we have to wait until we know which simulators are
available, because we group by the simulator's UDID.
When running in server mode, we don't need to group simulator test runs,
because we run the as the user chooses, not in any particular order.
This way the test list loads faster at startup, since we don't have to wait
until we've loaded the simulators before we can show the list.
This is slightly faster - ~0.95s vs ~1.4s - (probably because reflection tries
to load a lot of other referenced assemblies, which may or may not exist,
causing exceptions (if they don't exist) or spend time loading them (which
Cecil won't)).
It also avoids a lot of exception details showing up when tracing xharness
execution.
* [xharness] Add support for generating a tvOS version of .NET iOS projects.
And use it to run the tvOS version of introspection for .NET.
* [xharness] Change according to reviews.
They end up with the same make targets as non-.NET targets, which prints
warnings in the terminal when running make.
We're not using the makefile targets much anymore, so postpone implementing
them for .NET until we need them for some reason.
.NET projects are vastly simplified, which means that the OutputPath can't be
determined by reading the project file itself, it has to be calculated by
MSBuild.
So that's exactly what we do: we run MSBuild on the project file and get it to
print the property we're interested in.
.NET projects will include files from the current directory by default, which
means that if we clone the project file and write the cloned project file in a
different directory, we'll have to add an automatically included files into
the cloned file manually.
Add crash reports to a logs collection we care about, instead of to a logs
collection that's promptly forgotten.
This makes sure crash reports actually show up in the html report.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This fixes this:
Unexpected exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (ApplicationName='[...]/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/bin/mlaunch', CommandLine='--listsim=/var/folders/43/h027tm1n101cdrq2_b6n9n2m0000gn/T/tmp12c3d415.tmp --output-format=XML', CurrentDirectory='', Native error= Cannot find the specified file)) ---> System.AggregateException: One or more errors occurred. (ApplicationName='[...]/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/bin/mlaunch', CommandLine='--listsim=/var/folders/43/h027tm1n101cdrq2_b6n9n2m0000gn/T/tmp12c3d415.tmp --output-format=XML', CurrentDirectory='', Native error= Cannot find the specified file) ---> System.ComponentModel.Win32Exception: ApplicationName='[...]/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/bin/mlaunch', CommandLine='--listsim=/var/folders/43/h027tm1n101cdrq2_b6n9n2m0000gn/T/tmp12c3d415.tmp --output-format=XML', CurrentDirectory='', Native error= Cannot find the specified file
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0029f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/mcs/class/System/System.Diagnostics/Process.cs:778
at System.Diagnostics.Process.Start () [0x0003a] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/mcs/class/referencesource/System/services/monitoring/system/diagnosticts/Process.cs:2006
at (wrapper remoting-invoke-with-check) System.Diagnostics.Process.Start()
at Microsoft.DotNet.XHarness.iOS.Shared.Execution.ProcessManager.RunAsyncInternal (System.Diagnostics.Process process, Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog log, Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog stdout, Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog stderr, System.Nullable`1[T] timeout, System.Collections.Generic.Dictionary`2[TKey,TValue] environmentVariables, System.Nullable`1[T] cancellationToken, System.Nullable`1[T] diagnostics) [0x00469] in [...]/xamarin-macios/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Execution/ProcessManager.cs:185
at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.SimulatorLoader+<>c__DisplayClass16_0.<LoadDevices>b__0 () [0x00107] in [...]/xamarin-macios/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/SimulatorLoader.cs:65
at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.SimulatorLoader.LoadDevices (Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog log, System.Boolean includeLocked, System.Boolean forceRefresh, System.Boolean listExtraData) [0x00155] in [...]/xamarin-macios/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/SimulatorLoader.cs:54
at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.SimulatorLoader.FindOrCreateDevicesAsync (Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog log, System.String runtime, System.String devicetype, System.Boolean force) [0x0023c] in [...]/xamarin-macios/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/SimulatorLoader.cs:156
at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.SimulatorLoader.FindSimulators (Microsoft.DotNet.XHarness.iOS.Shared.TestTarget target, Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog log, System.Boolean create_if_needed, System.Boolean min_version) [0x00267] in [...]/xamarin-macios/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/SimulatorLoader.cs:277
at Xharness.Jenkins.TestTasks.RunSimulatorTask.FindSimulatorAsync () [0x0005a] in [...]/xamarin-macios/tests/xharness/Jenkins/TestTasks/RunSimulatorTask.cs:53
at Xharness.Jenkins.Jenkins.CreateRunSimulatorTasksAsync () [0x004ee] in [...]/xamarin-macios/tests/xharness/Jenkins/Jenkins.cs:558
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2027
at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:496
at System.Threading.Tasks.Task`1[TResult].get_Result () [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:466
at Xharness.Jenkins.Jenkins+<>c__DisplayClass72_0.<PopulateTasksAsync>b__0 (System.Threading.Tasks.Task`1[TResult] v) [0x0001e] in [...]/xamarin-macios/tests/xharness/Jenkins/Jenkins.cs:941
at System.Threading.Tasks.ContinuationTaskFromResultTask`1[TAntecedentResult].InnerInvoke () [0x00024] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:155
at System.Threading.Tasks.Task.Execute () [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2319
--- End of stack trace from previous location where exception was thrown ---
Rolf Bjarne Kvinge
Manuel de la Pena
Sebastien Pouliot
monojenkins