267 строки
22 KiB
Markdown
267 строки
22 KiB
Markdown
---
|
||
layout: post
|
||
title: "Using DevOps practices enables continuous delivery at Amadeus"
|
||
author: "David Tesar"
|
||
author-link: "https://twitter.com/dtzar"
|
||
#author-image: "{{ site.baseurl }}/images/authors/photo.jpg"
|
||
date: 2016-04-05
|
||
categories: [DevOps]
|
||
color: "blue"
|
||
image: "images/feat_Amadeus-VSM-starting.jpg"
|
||
excerpt: Microsoft teamed up with Amadeus to help them begin a DevOps journey and assess where they could make improvements with the most impact.
|
||
verticals: [Retail & Consumer Goods]
|
||
language: [English]
|
||
geolocation: [North America]
|
||
#permalink: /<page-title>.html
|
||
---
|
||
|
||
|
||
|
||
Microsoft teamed up with Amadeus in November 2015 to help them begin a DevOps journey, assess where they could make improvements with the most impact, and implement recommendations to better deliver value to their customers. The DevOps practices that were fully or partially implemented during this hack were:
|
||
|
||
- Continuous integration (CI)
|
||
- Automated testing
|
||
- Configuration as code
|
||
|
||
Since the hackfest, the customer has also implemented these two DevOps practices:
|
||
|
||
- Continuous deployment (CD)
|
||
- Release management
|
||
|
||
All of these practices working together have enabled a foundation for continuous delivery.
|
||
|
||
In addition, this hackfest sparked Amadeus to start a permanent DevOps transformation involving the non-technical aspects such as culture, roles, and shared metrics changes.
|
||
|
||
**Key technologies used:**
|
||
|
||
- [Visual Studio Team Services](https://www.visualstudio.com/team-services/)
|
||
- [Microsoft Team Foundation Server 2015](https://www.visualstudio.com/tfs/)
|
||
- [Windows PowerShell Desired State Configuration](https://docs.microsoft.com/en-us/powershell/dsc/overview)
|
||
|
||
**The core hack team:**
|
||
|
||
- Jay Hilyard – Team Development Lead, Amadeus
|
||
- Jason Dusseault – Release Engineer, Amadeus
|
||
- Dave Flanders – QA Lead, Amadeus
|
||
- Johnathan Govednik – Cloud Automation Engineer, Amadeus
|
||
- [David Tesar](https://twitter.com/dtzar) – Senior Technical Evangelist, Microsoft
|
||
- Gil Isaacs – Senior Software Development Engineer, Microsoft
|
||
|
||
## Customer profile
|
||
|
||
<img alt="Figure 0: Amadeus logo" src="{{ site.baseurl }}/images/Amadeus-Logo.png" width="400">
|
||
|
||
|
||
[Amadeus](https://amadeus-hospitality.com/) is an enterprise that delivers business solutions for hospitality organizations worldwide. The company has more than 40,000 installations and 150,000 users in 154 countries. Amadeus offers solutions for sales and marketing automation, catering and events, workflow management, and other areas for informing hospitality professionals.
|
||
|
||
The focus of the hackfest was on the eRest API, which was one service in Amadeus' web services platform that encompasses multiple products. Amadeus releases new features for the platform every three months.
|
||
|
||
## Problem statement
|
||
|
||
Amadeus used multiple manual hand-offs to build and test code for the app. In the development phase, developers built code locally and then checked in changes to Team Foundation Version Control (TFVC). Then they manually interacted with release engineers who built the code, depending on their availability. Automated and manual tests were run at a two-week regression cycle.
|
||
|
||
Deployments for the app were also manual, causing increased lead times. A developer communicated with a release engineer or IT to manually deploy a feature to a development environment, typically taking around four days per request to deploy to development or quality assurance. Deployments to staging and production were also manual, taking up to a week each, with patches falling outside of the process.
|
||
|
||
Amadeus used InstallShield installers in the staging environment, but this added complexity to the deployment. When manual deployments to environments occurred, configuration-related issues often caused problems. In total, the lead time for deployment, from development to production, took more than six weeks—if no problems arose and only a single feature was released.
|
||
|
||
## Solution, steps, and delivery
|
||
|
||
In order to know where they were headed and where the biggest improvements could be made, Amadeus needed a realistic assessment of where they were at. It was essential to understand the part that each team played in the process, from conception of a feature to final production and release to customers. This is where [value stream mapping](https://en.wikipedia.org/wiki/Value_stream_mapping) (VSM) came in. The exercise helped identify the following, from development to production:
|
||
|
||
- Each step involved in each environment and the roles of specific teams.
|
||
- Tools or products used for each step.
|
||
- The estimated lead times for each step, as well as per environment.
|
||
|
||
This is the completed VSM diagram of the state Amadeus was in before we began to make any improvements:
|
||
|
||
<br/>
|
||
|
||
|
||
|
||
<br/>
|
||
|
||
Value stream mapping helped the team identify numerous opportunities for improvement. Of those, they decided the hackfest would focus on the following areas:
|
||
|
||
- Automated build and testing with [**Team Foundation Server 2015**](https://www.visualstudio.com/tfs/) could save at least two hours of lead time per build, as well as be a foundation to enable other DevOps practices such as automated testing and continuous deployment.
|
||
- A two-week test regression cycle was hard-set into the release process. About 80 percent of the tests run during this cycle had the potential to be run automatically with the new build system. Automatically running the tests nightly could provide justification to eliminate the hard-coded two-week regression cycle and significantly reduce the overall lead time and provide faster feedback to developers.
|
||
- The manual InstallShield process for deploying builds and configuring the virtual machine took at least an hour for each new build, so [**PowerShell Desired State Configuration**](https://docs.microsoft.com/en-us/powershell/dsc/overview) (DSC) would provide an excellent way to automate the install and enforce configuration of the virtual machines where the app needed to run.
|
||
|
||
### CI and CD with Team Foundation Server 2015 and Visual Studio Team Services release management
|
||
|
||
Amadeus had recently upgraded from Team Foundation Server 2010 to Team Foundation Server 2015, but had no plans to use the build system. The first part of the hackfest implemented a new build definition with a continuous integration trigger. The build would run every time a developer checked in code and it would compile the code with MSBuild.
|
||
|
||
The continuous integration and continuous deployment build process included the following steps for the web API application (eRest):
|
||
|
||
<br/>
|
||
|
||
|
||
|
||
<br/>
|
||
|
||
1. PowerShell – Update versions for the build itself.
|
||
2. Visual Studio build – Build the solution.
|
||
3. Visual Studio test – Execute unit tests against the solution using the Test Filter criteria of "TestCategory=DevUnitTest".
|
||
|
||
**Note:** At this point in the build definition, it is sufficient for CI. If all unit tests pass, then the steps below continue on to implement the steps for Team Foundation Server 2015 to create a new Visual Studio Team Services release that does CD to their development environment.
|
||
4. PowerShell – Add eRest NuGet sources.
|
||
5. PowerShell – Transform web.config to have values required for the development environment where it will first be deployed.
|
||
6. Command – Delete the local NuGet packages to avoid potential version conflicts. Arguments for NuGet are as follows:
|
||
|
||
`/C del /F /Q "$(agent.builddirectory)\*.nupkg"`
|
||
|
||
<br/>
|
||
|
||
7. Command – Create a local eRest API NuGet package. Arguments for NuGet are as follows:
|
||
|
||
`pack "$(Build.SourcesDirectory)\Cloud\Web Services\Newmarket.Web.Services\Newmarket.Web.Services.csproj" -OutputDirectory $(agent.builddirectory) -Version $(Build.BuildNumber) -NoPackageAnalysis`
|
||
|
||
<br/>
|
||
|
||
8. Command – Publish the eRest API NuGet package to Team Services package management service. Arguments for NuGet are as follows:
|
||
|
||
`$(agent.builddirectory)\*.nupkg -Source https://theiraccountname.pkgs.visualstudio.com/DefaultCollection/_packaging/NWS-eRest/nuget/v3/index.json -ApiKey VSTS`
|
||
|
||
<br/>
|
||
|
||
9. PowerShell – Copies the NuGet package to an internal file share.
|
||
10. Command – Install the eRest API configuration NuGet package. Arguments for NuGet are as follows:
|
||
|
||
`install Newmarket.Web.Services.Configuration -Source https://theiraccountname.pkgs.visualstudio.com/DefaultCollection/_packaging/NWS-eRest/nuget/v3/index.json`
|
||
|
||
<br/>
|
||
|
||
11. Command – Install configuration updater package. Arguments for NuGet are as follows:
|
||
|
||
`install EnvironmentConfigurationUpdater -Source https://theiraccountname.pkgs.visualstudio.com/DefaultCollection/_packaging/DevOpsUtilities/nuget/v3/index.json`
|
||
|
||
<br/>
|
||
|
||
12. Command that references PowerShell.exe – Executes the Release utility that initiates a new release in Team Services and a Team Services task in the release definition does CD to the dev environment. Arguments for this task are as follows:
|
||
|
||
`-command iex """$((dir .\EnvironmentConfigurationUpdater*).fullname)\content\EnvironmentConfigurationUpdater.exe config $((dir .\Newmarket.Web.Services.Configuration*).fullname)\content\EnvironmentConfig.json releaseid 1 buildversion $(build.buildnumber)"""`
|
||
|
||
<br/>
|
||
|
||
13. Command that references PowerShell.exe – Deletes the NuGet install directory on the build agent. Arguments for this task are as follows:
|
||
|
||
`-command "$thispath=(get-location).path;Get-ChildItem -Path .\ -Recurse|Select -ExpandProperty FullName |?{($_ -notlike """$($thispath)\*.Nupkg""") -and ($_ -notlike """$($thispath)\s\*""") -and ($_ -notlike """$($thispath)\TestResults\*""")}|sort length -Descending|Remove-Item -recurse -force -confirm:$false -verbose"`
|
||
|
||
<br/>
|
||
|
||
Enabling this build definition eliminated the bottleneck of waiting for manual builds and deployments.
|
||
|
||
They removed the custom-built website running on a virtual machine for developers to request a new build and replaced that with simply being able to access the Team Foundation Server build service website.
|
||
|
||
Developers also received immediate feedback from the automated builds if their changes broke the build and were able to address the issues quickly, without becoming familiar with the context a week later.
|
||
|
||
Furthermore, successful builds are able to be tested against a shared development environment quickly because they are automatically deployed. This was important for Amadeus to better embrace DevOps practices with agility.
|
||
|
||
### Automated testing with Team Foundation Server 2015
|
||
|
||
Next, the automated tests that were typically run by QA later in the process were introduced as Visual Studio Test tasks in automated builds using the standard Visual Studio test build task. The shorter tests (such as unit tests and a smaller list of feature tests) were added into the continuous integration build, and the longer tests were added into a scheduled nightly build. Previously, a manual process by QA was involved to categorize tests, but Amadeus could categorize in test classes and test methods and include that as part of the build definition and quality of the build.
|
||
|
||
|
||
|
||
<br/>
|
||
|
||
|
||
|
||
<br/>
|
||
|
||
You can see how they run separate functional test tasks for various test categories. Each test task specifies the test filter criteria corresponding to the TestCategory they specified in the test code. The last build task publishes the test results from all test tasks to the test center and links them to the build.
|
||
|
||
By running tests earlier in the development process with the builds, Amadeus was able to identify defects and bugs sooner, remove the two-week regression cycle, and ultimately provide a better product to customers.
|
||
|
||
> "Implementing TFS 2015 and the automated testing and delivery cycles that we’ve put together, we can get issues in front of our developers faster to cut down some of the lead time in the cycle. You get more immediate feedback as a developer when you check something in, it runs through a bunch of automated tests, and you’re immediately getting feedback as to whether or not you may have broken something, or everything worked so that you can address it when you’re thinking about it. You don’t have to reacquire that context in the next week after a whole series of other things have run—you can go right to it at that point and fix it."
|
||
>
|
||
>— Jay Hilyard, Team Development Lead, Amadeus
|
||
|
||
### Configuration as code with PowerShell DSC
|
||
|
||
Amadeus created a PowerShell Desired State Configuration (DSC) script to configure the virtual machine dependencies and install the build. This helped to eliminate the wait for a release engineer to manually click through a GUI InstallShield wizard to do the same steps.
|
||
|
||
[Deploy_eRest](https://github.com/Newmarket/Deployment-DSC-Scripts/) is the master PowerShell DSC script that does the following:
|
||
|
||
- Pull down NuGet packages for the eRest application build, secrets, and environment configuration updates required for the application.
|
||
- Install and configure all dependent services on the Windows Server 2012 R2 server including:
|
||
- Internet Information Services (IIS) features (DSC\_1_IISFeature.ps1)
|
||
- .NET Framework 4.5 (DSC\_2_NetFramework45.ps1)
|
||
- .NET Framework 4.6 (DSC\_3_NetFramework46.ps1)
|
||
- Web Administration management tools (DSC\_4_InstallxWebAdministration.ps1)
|
||
- Custom performance counters related to the application
|
||
- Custom event log sources related to the application
|
||
- IIS website application pool (DSC\_5\_Create_naasWebsite.ps1)
|
||
- Transform web.config values for the environment.
|
||
- Create the eRest API application (DSC\_6\_Create_eRestWebApp.ps1).
|
||
|
||
The process of using PowerShell DSC and PowerShell to deploy the build and configure the virtual machines began the conversation of a repeatable procedure across the development, staging, and production environments. Reliable deployments removed manual delays and configuration-related issues. By running deployment scripts with PowerShell DSC, InstallShield installers were no longer needed.
|
||
|
||
This process also opened the possibility of allowing developers and testers to create their own development and QA environments hosted in Azure rather than break the shared development environment.
|
||
|
||
> "If we could quickly spin up new environments required to develop against eRest API, this would be beneficial to over 75% of those developers."
|
||
>
|
||
>— Jay Hilyard, Team Development Lead, Amadeus
|
||
|
||
### Release management with Team Foundation Server 2015 and Team Services
|
||
|
||
Amadeus has a number of on-premises environments they go through before pushing updates to production. They wanted an automatic and traceable way to move a known good build of the software and configuration of the servers. To accomplish this goal, they used Team Foundation Server 2015 build steps (see above) with the Visual Studio Team Services package management service, two custom-built utilities called VSTS Release Creator and ConfigurationSecrets, and the Team Services release management service.
|
||
|
||
The VSTS Release Creator utility pushes the key/value entries in the Configuration JSON file to Team Services as environment variables via the VSTS RM API. It also takes the ReleaseDefinitionID and the Configuration JSON file, creates a new release draft, updates the configuration variables in each environment that matches environments defined within the Configuration JSON file, and starts the release.
|
||
|
||
In the build step 12 above, you can see the creation of the release using this utility. The VSTS Release Creator utility source code can be found at [https://github.com/Newmarket/Release-Creator-Utility](https://github.com/Newmarket/Release-Creator-Utility).
|
||
|
||
The ConfigurationSecrets utility allows pointers to secrets within the Configuration JSON to retrieve protected/private values from their enterprise secret/key management system at deploy time. This prevents “protected” values such as passwords, SAS keys, Storage keys, and so on from being committed into source control as required with configuration as code.
|
||
|
||
The steps that the release task does to deploy the bits into the environment are as follows:
|
||
|
||
1. Downloads the infrastructure as code (IaC) package – a set of PowerShell DSC scripts and Azure Resource Manager templates used across projects to create and update Resource Manager resources as defined in the configuration package (see the PowerShell section above for more details).
|
||
2. Executes the IaC Wrapper script, determines which IaC scripts are to be executed by which configuration variables.
|
||
3. Deploys Node1 (run from an agent on Node1, “Local deployment”).
|
||
4. Downloads the eRest API Application NuGet package and the ConfigurationSecret NuGet package.
|
||
5. DSC Scripts executed in order – Any “secret pointer” variables used in the DSC scripts are resolved using the ConfigurationSecret utility.
|
||
6. Untokenized web.config is transformed using XDT into a tokenized web.config.
|
||
7. Tokenized web.config: tokens are replaced with values from the Team Services RM environment “variables” – “secret pointer” variables are resolved using the ConfigurationSecret utility.
|
||
|
||
Steps 3-7 get executed on additional Windows Server 2012 R2 nodes where the application needs to be deployed.
|
||
|
||
Implementing release management has eliminated wasted time doing manual deployments to every environment, reduced configuration-related bugs/errors, and allowed visibility into what code is running on what environment and calculating the lead time.
|
||
|
||
> "Pretty much since the day I started (3 years ago), I've wanted to see things be fully automated from the developers' check-in right up through our production environment ... It wasn't until this hackfest we actually started to knock it out and make it a reality."
|
||
>
|
||
>— Jason Dusseault, Release Engineer, Amadeus
|
||
|
||
## Conclusion
|
||
|
||
By automating continuous integration builds with Team Foundation Server 2015, ensuring consistent configuration of the virtual machines with PowerShell DSC, and removing the two-week regression test cycle and automating functional tests, the lead time from development to production is reduced by 25 percent, allowing for a repeatable build/test/deploy process with opportunities for improvement in other areas of development.
|
||
|
||
The original lead time from development to deployment: **6 weeks, 0 days, and 7 hours**. The new lead time after a week of improvement with adding continuous integration builds, nightly tests, and automated deployments: **4 weeks, 1 day, 2.5 hours**. This reduced the best case lead time by **13 days and 4.5 hours**.
|
||
|
||
Furthermore, feedback was much faster with tests running automatically each night versus every 2 weeks. Automatically building and deploying saved about 8 hours per bug discovered.
|
||
|
||
Amadeus now has buy-in from upper management to spend real time in the next program increments with DevOps backlog items in the sprints following for development, quality assurance, operations, and release engineering.
|
||
|
||
> "We read about the changes that are required and I think at first we were questioning, do we really have to co-locate people? And I think we’ve come to the realization that yes we do, because when you put people from different disciplines together, there is no wait time anymore and they can answer each other’s questions immediately.
|
||
>
|
||
> "I think more importantly you start to cross-train people on the different disciplines. For example, the week here with Microsoft, we had a developer of mine that was actually writing scripts that was deploying actual infrastructure. On the flip side, we had operations people that were looking at the release engineering cycle and how to automate InstallShield installers that we have. So there was a lot of ... shifting around even in just one week of time; that was an eye-opener for us that is making us realize how much more effective we can be if we put people together."
|
||
>
|
||
>— Alex Shore, Amadeus
|
||
|
||
## General lessons
|
||
|
||
The hackfest team came away with the following insights:
|
||
|
||
- When teams work together toward a larger common goal, amazing opportunities for collaboration happen, blurring the lines of roles and responsibilities. For instance, members of different Amadeus teams made contributions in non-typical areas of responsibility. A developer wrote code for the PowerShell DSC script that helped eliminate the InstallShield installers, and operations and test engineers helped with creating build definitions and build agents.
|
||
|
||
- It is very uncommon for companies to wipe out their current software development lifecycle process, so it is pertinent to start where you are.
|
||
|
||
- Begin the conversation by running the value stream exercise with team members across different disciplines and show the current process to reveal pain points and bottlenecks.
|
||
|
||
- Although the PowerShell DSC details may not align perfectly to every situation, the principles and tasks are generally the same and may be applied in various environments using Windows Server regardless of whether running on bare metal, on-premises, or in Azure.
|
||
|
||
## Additional resources
|
||
|
||
- [Overview of Team Foundation Build 2015](https://msdn.microsoft.com/en-us/library/vs/alm/build/feature-overview)
|
||
- [Implementing Continuous Delivery with Build vNext and Release Management](http://blogs.msdn.com/b/charles_sterling/archive/2015/06/22/ci-and-cd-hands-on-lab.aspx)
|
||
- [Testing in Continuous Integration and Continuous Deployment Workflows](https://blogs.msdn.microsoft.com/visualstudioalm/2015/05/29/testing-in-continuous-integration-and-continuous-deployment-workflows/)
|
||
- [Windows PowerShell Desired State Configuration Overview](https://msdn.microsoft.com/en-us/powershell/dsc/overview)
|