techcasestudies/2017-03-23-Almaviva.md в master

24 KiB

Исходник Постоянная ссылка Ответственный История

layout

title

author

author-link

date

Key technologies

Core team

Gianluca Bertelli – Technical Evangelist, Microsoft Italy
Paola Presutto – Senior Technical Evangelist, Microsoft Italy
Gabriele Mazzocca – Developer, AlmavivA
Antonio Parisi – Head of ALM and DevOps, AlmavivA

Customer profile

AlmavivA is the leading Italian group in information and communications technology with over 45,000 employees and 59 offices worldwide. The fundamentals of this company are proven experience, unique skills, ongoing research, and in-depth knowledge of a range of public and private market sectors.

The mission of the group is to create exceptional technological solutions that can develop the systems and operating processes of private companies and government organizations, and improve service levels in a constantly changing market in terms of operational continuity, privacy, and data security.

Their product and service offerings include cloud computing and ICT services, CRM BPO and CX services, and people-centered technology.

The company has developed several software solutions. The scope of this project is the AlmaToolBox, a solution for DevOps-oriented integration that contains a mixture of technologies, IT tools, and change management activities for organizations so they can develop software products and services quickly and efficiently.

Problem statement

Visual Studio Team Services (formerly Visual Studio Online), and its on-premises version, allows you to organize and easily manage your tests, create collections, and group, delete, and execute tests. With a test plan, you can keep track of all the tests that are run across your builds. This is extremely useful when you have automated tests that are executed as part of a build process. Within a test plan, you can associate a single test with a unit test method to have the test result automatically.

Unfortunately, this automated test association is available only for .NET projects. If you have a Java project, you cannot use such integration. The request from AlmavivA is to use the test plan feature for Java projects in the very same way they are using it for .NET projects. To that end, we are proposing a solution that uses a dynamically created C# unit test assembly to act as a bridge, linking Java unit tests with test plan work items in Visual Studio Team Services.

With such a solution, AlmavivA will be able to execute the automated tests of a test plan for a pure Java project as part of their build process, achieving continuous integration and continuous deployment. AlmavivA is willing to use this solution in AlmaToolBox and several other Java projects developed internally.

Solution, steps, and delivery

The development process for AlmaToolBox is well-structured, and includes separate teams with different roles such as developers, testers, and quality. The entire chain was already using different DevOps practices and is heavily based on test-driven development (TDD). For example, they have different environments built by using Infrastructure as Code (IaC) techniques; dev and test environments are hosted on Microsoft Azure, while the production environment is on-premises.

Value stream mapping

Figure 1. Value stream mapping.

![Current value stream mapping]({{ site.baseurl }}/images/JavaTestBridge/vsm.png)

With a value stream mapping (VSM) exercise, we identified some points where the process wastes time and has some roadblocks to becoming a fully automated DevOps process:

No real continuous integration and no continuous deployment
Partially automated testing that needs an interactive process to run
Manual steps required to validate test results

Continuous integration and continuous deployment are not achieved mostly because of the lack of fully automated tests and the need for human intervention for the validation of the already executed tests. Based on the VSM, this task wastes a lot of time simply because it cannot be automated. This is most evident in case of test failures because the developers receive feedback only after the manual review from the quality team is submitted days after the code commit.

As part of their process, AlmavivA needs to execute test plans to confirm the quality of each build before the release. However, they cannot use the test plan feature as is because the test results are not reported and consolidated automatically. Instead, their process requires an operator to manually check test work items and map the Java test results for each build; this manual and error-prone step creates a huge delay in the entire process.

When we create a test work item inside a test plan, we have the option to associate a single unit test method to this item, which transforms the test into an automated test that can be executed by a machine. Currently this association can be performed only by using Visual Studio Team Services and can be done only for .NET unit test assemblies. This cannot be done for a Java-based project, and this impossibility of association is a clear design limitation.

As described, the customer's DevOps process is heavily based on the use of test plans to coordinate the test strategy. The request is to also use the test plan feature for Java projects, the very same way they are using it for .NET projects. More specifically, AlmavivA wants to run Java Selenium tests (browser-based Java JUnit tests) as part of the build process and get the results in the test plan view to achieve continuous integration and continuous deployment.

Java test bridge for test plan solution

As the lack of fully automated tests has been identified as a cause of multiple roadblocks for DevOps practices, the goal was to have a Visual Studio Team Services Marketplace extension to use as part of the build process so that they could effortlessly associate Java tests to test plan items. This would be done without changing any line of code and without any human intervention.

The solution behind this extension is called the Java test bridge for test plan.

Having such an extension allows us to achieve the intended ideal flow:

The developer prepares the tests and commits the code.
The tester/reviewer creates a test plan, and then creates a JSON file that maps 1:1 each Java test with a test plan work item (by using the ID).
Apache Maven is used to compile and test the Java solution.
This extension is added as a step in the build workflow.
The test plan is executed.
The Java test results are automatically consolidated within the test plan.

The idea of this solution is quite simple and is based on these facts:

Java tests are executed as part of the compilation (Maven) and the results are saved as XML files with a consistent naming convention.
In Visual Studio Team Services, each test work item has a unique ID.
The automated test association can be done only for .NET assemblies.
Team Services exposes management API that allows us to create the automated test association by code.

The main point is to dynamically create a .NET assembly, being the only way to create the automation association. Fortunately, the .NET Compiler Platform (Roslyn) allows such a task with few lines of code. Basically, by using the mapping specified via the JSON definition, this extension converts Java tests to .NET unit tests, creating on the fly a brand new .NET assembly. This .NET assembly is then used to associate each .NET unit test method with the corresponding test work item. Copying this new assembly to our repository allows us to execute the test plan and get the results effortlessly.

To achieve this result, we created several different projects:

Microsoft.DX.JavaTestBridge.UnitTestGenerator, a .NET project written in C# that, starting from the JSON definition of the test mapping, creates a .NET assembly on the fly by using Roslyn. Each method has the same body that basically searches for the corresponding test result file by using a standard naming convention.
Microsoft.DX.JavaTestBridge.VSTS, a .NET project that throughout reflection is able to discover .NET unit tests and to associate them to corresponding Team Services work items by using the Visual Studio Team Services API.
JavaTestBridge extension, a Visual Studio Team Services Marketplace extension (build task). Basically a PowerShell script that leverages the previous projects to properly generate the automated tests' dynamic-link library (DLL) and performs the Visual Studio Team Services association.

Creating and configuring the JSON mapping file

For the end customer, the usage is really simple. You just need to create the JSON mapping file, a step that can be easily automated:

[
  {
    "className": "HelloWorldJUnitTest",
    "methodName": "TestOne",
    "workItemID": 210
  },
  {
    "className": "HelloWorldJUnitTest",
    "methodName": "TestTwo",
    "workItemID": 211
  }
]

After they have this JSON, where each array item contains the link between a single Java test and a test plan work item, you have all the required information for the association. The ID of each test plan item can be easily retrieved from Team Services.

This JSON is one of the required inputs, and it can be passed as a string argument or saved in a file. In the case of a string argument, you just need to escape the special characters:

[{\"className\": \"HelloWorldJava.Demo.HelloWorldJunitTest\",\"methodName\": \"testTrue\",\"workItemID\": 216}]

Having the JSON, you can add this extension to the build workflow and configure the required parameters. After adding this extension, you must execute a functional test task that enables the test plan to run.

Figure 2. Build process with the extension.

![Build process with the extension]({{ site.baseurl }}/images/JavaTestBridge/build.png)

The results will be reported automatically and integrated in the test plan view.

Implementation details

The JSON mapping input is processed by Microsoft.DX.JavaTestBridge.UnitTestGenerator.exe. The result is a .NET DLL that contains one single class and one test method for each entry in the JSON. The assembly is dynamically created by using Roslyn.

(JSON) --> JavaTestBridge.UnitTestGenerator.exe --> (.NET DLL) -->  JavaTestBridge.VSTS.exe --> VSTS API

For each created method, the body logic is the same; we changed only a few parameters to identify the right method name and result file.

    [AutomatedTestID(216), TestMethod]
    public void testTrue()
    {
        try
        {
            string str = ".\\target\\surefire-reports";
            string str2 = "TEST-HelloWorldJava.Demo.HelloWorldJunitTest";
            string testMethodName = "testOne";
            XDocument xDocument = XDocument.Load(new StreamReader(str + Path.DirectorySeparatorChar.ToString() + str2 + ".xml"));
            IEnumerable<XElement> source =
                from t in xDocument.Element("testsuite").Elements("testcase")
                where t.Attribute("name").Value == testMethodName
                select t;
            XElement xElement = source.FirstOrDefault<XElement>();
            bool flag = xElement == null;
            if (flag)
            {
                throw new Exception("Java Method not found");
            }
            IEnumerable<XElement> source2 = xElement.Descendants("failure");
            bool flag2 = source2.Count<XElement>() > 0;
            if (flag2)
            {
                Assert.Fail(source2.First<XElement>().Attribute("message").Value);
            }
        }
        catch (Exception ex)
        {
            Assert.Fail(ex.Message);
        }
    }

As you can see, the method's body simply searches for the XML file that contains the result of the test execution by using a well-known naming convention. The test fails the assertion in case of any issues or in case of negative results (from the Java execution).

After this step, we have the bridge between Java and .NET, and therefore we have all the necessary structure to execute our tests and understand the results.

The last part of this solution is to associate the .NET unit tests with Team Services. This association can fortunately be done in code through the use of the Visual Studio Team Services SDK (NuGet package). The association itself is a mere collection of three pieces of information:

The test type (fixed: "unit test")
The test assembly name
The test method name

Using .NET Reflection to discover the test methods (methods marked with the attribute [AutomatedTestID(...)), the Microsoft.DX.JavaTestBridge.VSTS project performs the following association.

Microsoft.DX.JavaTestBridge.VSTS.exe https://xxxxxxx.visualstudio.com ProjectName AutomatedTestAssembly.dll Username Password

After its completion, we successfully associated the Java tests to Team Services test items (that is, to a test plan).

The customer can now use the test plans in its build process. The created DLL assembly must be copied to a repository folder because it must be found by the test agent.

Because this solution requires more steps to be run in a sequence, we have automated this error-prone workflow by creating a PowerShell script that is executed as part of the Marketplace extension.

Source code

The full source code of this solution is available in GitHub at Java Test Bridge. The AlmaToolBox extension download is available on the Visual Studio Marketplace.

Following are some code samples of the key parts.

How Roslyn is used to create a dynamic .NET assembly

  public override bool CreateTestAssembly(FileInfo assemblyFile)
        {
            try
            {
                //load the class -  C# code
                SyntaxTree syntaxTree = CSharpSyntaxTree.ParseText(CreateTestClass());

                //add all the library references
                MetadataReference[] references = new MetadataReference[]
                {
                MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
                MetadataReference.CreateFromFile(typeof(Enumerable).Assembly.Location),
                MetadataReference.CreateFromFile(typeof(JsonConvert).Assembly.Location),
                MetadataReference.CreateFromFile(typeof(Microsoft.VisualStudio.TestTools.UnitTesting.Assert).Assembly.Location),
                MetadataReference.CreateFromFile(typeof(Microsoft.DX.JavaTestBridge.Common.JavaTestItem).Assembly.Location),
                MetadataReference.CreateFromFile(typeof(Process).Assembly.Location)
                };

                //set compilation options
                CSharpCompilation compilation = CSharpCompilation.Create(
                    assemblyFile.Name,
                    syntaxTrees: new[] { syntaxTree },
                    references: references,
                    options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));

                using (var ms = new MemoryStream())
                {
                    //compile it
                    EmitResult result = compilation.Emit(ms);
                    if (!result.Success)
                    {
                        IEnumerable<Diagnostic> failures = result.Diagnostics.Where(diagnostic =>
                            diagnostic.IsWarningAsError ||
                            diagnostic.Severity == DiagnosticSeverity.Error);
                        foreach (Diagnostic diagnostic in failures)
                        {
                            Trace.TraceError($"{diagnostic.Id}: {diagnostic.GetMessage()}");
                        }
                        return false;
                    }
                    else
                    {
                        //correct, flush to disk
                        ms.Seek(0, SeekOrigin.Begin);
                        File.WriteAllBytes(assemblyFile.FullName, ms.GetBuffer());
                        return true;
                    }
                }
            }
            catch (Exception ex)
            {
              ...
            }
        }

How .NET Reflection is used to discover the marked tests

 public static List<AutomatedTestMethod> DiscoverAutomatedTests(FileInfo assemblyFile)
        {
            if (!assemblyFile.Exists || assemblyFile == null)
                throw new FileNotFoundException($"assembly file not found");

            List<AutomatedTestMethod> foundTests = new List<AutomatedTestMethod>();

            Assembly testAssembly = Assembly.LoadFrom(assemblyFile.FullName);

            foreach (Type type in testAssembly.GetTypes())
            {
                if (type.IsClass)
                {
                    foreach (MethodInfo methodInfo in type.GetMethods())
                    {
                        var automatedTestAttribute = Attribute.GetCustomAttribute(methodInfo, typeof(AutomatedTestIDAttribute));
                        if (automatedTestAttribute != null)
                        {
                            AutomatedTestMethod t = new AutomatedTestMethod(methodInfo.Name, methodInfo.DeclaringType.FullName + "." + methodInfo.Name, assemblyFile.Name);
                            t.TestID = ((AutomatedTestIDAttribute)automatedTestAttribute).WorkItemID;
                            foundTests.Add(t);
                            Trace.TraceInformation($"Found TEST method: {t}" );
                        }
                    }
                }
            }

            return foundTests;

        }

How the PowerShell script is used in the extension to run all the required processes


Invoke-VstsTool -FileName ".\Microsoft.DX.JavaTestBridge.UnitTestGenerator.exe" -Arguments "AutomatedTestAssembly `"$jsonMapping`" $testResultPath" -RequireExitCodeZero

    if($LASTEXITCODE -eq 0){

        Write-host ".NET unit test assembly created (AutomatedTestAssembly.dll)"
        Write-host "Association of tests with VSTS..."
        Write-host "-------------------------------------------------"

        Invoke-VstsTool -FileName ".\Microsoft.DX.JavaTestBridge.VSTS.exe" -Arguments "$VSTSAccountName $projectName AutomatedTestAssembly.dll $username $password" -RequireExitCodeZero
        if($LASTEXITCODE -eq 0)
        {
          Write-host "Association completed successfully"
          #required DLL to run
          Move-Item .\AutomatedTestAssembly.dll $outputFolder -Force
          Copy-Item .\Newtonsoft.Json.dll $outputFolder -Force
          Copy-Item .\Microsoft.DX.JavaTestBridge.Common.dll $outputFolder -Force

          Write-host "Files copied to $outputFolder"
        }
        else
        {
            Write-Error "Association of tests failed"
        }
    }
    else
    {
      Write-Error "Creation of .NET dynamic test DLL failed";
    }

Continuous integration and continuous deployment

By leveraging Visual Studio Team Services and this created extension, we defined a basic workflow to implement a real continuous integration and continuous deployment for the customer environments:

Testers create test plans and test work items.
- A mapping JSON file is created and added to the source code repo.
Developers commit the code into Visual Studio Team Services.
A build is triggered.
- A build/test agent is deployed to a remote machine.
- Selenium tests are automatically executed (interactive process).
- Test results are automatically reported back to Team Services.
This extension task (Java test bridge) is executed.
If the build and test plan succeed, the solution is automatically deployed.

This process avoids any human interaction, allows the developers to not take part in the testing process, and achieves a real code integration allowing each commit to be deployed automatically if it does not fail any tests. This also permits the stakeholders that use the test plan feature to have an always updated view of the overall quality of the solution.

Conclusion

The use of automated tests and test plans during the build process is really important for the customer. The current design that limits this to a .NET assembly was a clear roadblock for a complete automated DevOps solution.

The lack of fully automated tests has been identified as a cause of wasted time in the value stream mapping. This was also the contributory cause of the missing continuous integration and continuous deployment practices because several manual steps and human intervention were required.

With such a solution, AlmavivA is going to have a fully integrated and automated DevOps process. AlmavivA plans to use this solution in AlmaToolBox and several other Java projects developed internally.

The DevOps practices we introduced and enhanced will speed up the customer process. Based on the future value stream mapping, the time from commit to deployment will be reduced to ~15 hours (previously was 3 days).

Figure 3. Future value stream mapping.

![Future value stream mapping]({{ site.baseurl }}/images/JavaTestBridge/vsm-after.png)

Instead of this being a technically complex project, the end-user experience is really simple: using a Visual Studio Team Services add-in that can be reused by just adding it to any build or release pipeline as a task.

This solution can be the starting point for a bigger solution. Because it's open source, it can be extended or customized to fit other scenarios. In future developments, it can be extended to physically run Java tests, or to read and understand several other test result formats.

We see a huge opportunity here because this can be reused by other customers, and by solving a design limitation of the current version, it can bring more Java developers into the Visual Studio Team Services ecosystem.

24 KiB Исходник Постоянная ссылка Ответственный История