Adding a definition of a pipeline to produce NuGet packages for the MSIX SDK.
Moved the definitions of each platform-specific job in the build pipelines to a template which is used by the respective pipeline, and added a new pipeline that uses all of these jobs and then creates the packages in a new stage. The new stage is based on an existing internal release pipeline, with minor changes (replaced build artifacts by pipeline artifacts, enabled generation for Linux, added a step to produce the SBoM manifest)
This includes several changes related to the pipelines for MSIX SDK.
* The pipelines for iOS, aosp and Linux builds were not running as we were using an outdated VM image that is no longer available. This updates all pipelines to a more recent image.
+ The update of the image used for iOS involved an update to the version of XCode used, from 11 to 12. On XCode 12 there is now support for the simulator running on arm64, and by default running the simulator will try to use that. This would cause an error when trying to run the tests for x86_64 as we were not building the msix library for arm64. To prevent it, I'm explicitly excluding arm64 on the test project when building for the simulator.
* The Windows pipeline was using the `windows-latest` image, which will soon change from `windows-2019` to `windows-2022`. I'm changing the image we use to `-2022` so we are sure that the change in `-latest` will not affect us.
* Some tests were failing on Windows builds; seemingly due to an OS update. I'm disabling those tests while I investigate.
Changes to the build pipelines for the AzDO extension:
* Added CredScan and Semmle code security analysis tasks.
* Factored out the common steps between the PR and release build pipelines.
* Moved to use a custom agent pool.
Updating dependencies per npm audit.
Also updated MSBuildHelpers copied from azure-pipelines-task-lib to latest, which includes detection for MSBuild 17, and fixed project URL on package.json files.
1. opensslconf.h.cmake - moved to sub-directory openssl
2. bn_conf.h.cmake & dso_conf.h.cmake - moved to sub-directory openssl/crypto to align with openssl include directory structure from which these cmake are generated.
When we examine defination of async_posix.h it appears to exclude android platform from its definitaion by defining !defined(__ANDROID__) at the begining.
As a result async_posix.c definition will not be applicable for AOSP platform in that case it use async_null.c as default definition. So added conditional statement such that for AOSP it should consider async_null.c , for windows async_win.c and linux and ios async_posix.c source code respectively
ios issue - CMakeFiles/crypto.dir/openssl/crypto/dso/dso_dl.c.o has no symbols
Adjusted async code order such that arch/async_posix.c compiled first and then
async.c as some of the code definitation such as async_local_cleanup exists as part of platform specific code and is referred in async.c as a possible fix to error: undefined reference to 'async_local_cleanup on adroid specific builds.
as MSIX SDK code doesn't set explicitly any DSO macro and hence default DSO_METHOD was used as per old dso_openssl.c source code.
With the latest OpenSSL 1.1.1j - there is no default DSO as such inorder to use DSO_METHOD_null we have to set DSO_NONE macro.
- So, Crypto.cmake has been updated to set DSO_NONE to retain old behavior
- Removed dso_dl.c from compilation list as we DSO_NONE as default this is not going to consumed. Compiling this file without defining DSO_DL macro resulting failures on iOS x86 platform. It appears iOS x86 platform is too sensitive compiling a *.c file which endsup not having any content (due to entire content is wrapped on the existing of DSO_DL definition in the recent OpenSSL version)
* makemsix: Remove wrong help output
The help output for the bundle command referred to options the p, ep,
kt and kf options, that do not actually exist.
* makemsix: Minor fixes in bundle option descriptions
Some spaces were missing and / was used for some option names.
* makemsix: Split very long lines
The commit includes:
1. Updates to crypto_sources.cmake
a) to include all the neccessary sources/header file from the new OpenSSL 1.1.1j version
b) Removed sources files from crypto_sources.cmake that don't exist/deleted with the latest OpenSSL code
2. Updated OpenSSL version to 1.1.1 specificied in CMakeList.txt
3. Updates to SignatureValidator.cpp
a) added a local macro definition for CRYPTO_NUM_LOCKS - as this macro support has been taken out from OpenSSL but MSIX SDK code consumes it with std::lock constructs. It appears We should be fine using this constant, as code doesn't depend on OpenSSL Locking constructs which are out support.
b) Comment M_ASN1_OCTET_STRING_print code as this function has been removed in the latest OpenSSL version. Will have to discuss with experts for the alternatives.
c) Also update STACK_OF(X509_EXTENSION) *exts = cert->cert_info.extensions; "cert_info.extensions" instead "cert_info->extensions" as it appears this type has been updated else it would cause compile error
c) CRYPTO_THREADID_get_callback() == NULL) - comparison has been updated to use NULL else it would cause compile time error
[How Validated:]
- Compiled MSIX SDK code with the helps of
./makelinux.sh
- executed .vs/msixtest/msixtest and ensure all the tests passes
[Test Result:]
All tests passed (6268 assertions in 72 test cases)
The changes includes addition of:
1. bn_conf.h.cmake - generated with the help of running Makefile on "\msix-packaging\lib\openssl\include\crypto\bn_conf.h.in"
2. dso_conf.h.cmake - generated with the help of running Makefile on "\msix-packaging\lib\openssl\include\crypto\dso_conf.h.in"
3. opensslconf.h.cmake - generated with the help of running Makefile on "\msix-packaging\lib\openssl\include\openssl\opensslconf.h.in"
Above cmake file has neccessary cmake macro expansion flag which get resolved inside crypto.cmake when configure_file called on each of these
files
Additionally, crypto.cmake has some additional changes
- Copying the all the includes as per new OpenSSL source includes layout
- Additional flag settings to resolve certain compiler time definitions
* Invalid Memory Access in SignatureValidator
Problem:
* SignatureValidator::IsStoreOrigin() tries to read X.509 extensions
to determine if the origin of the signature matches the Windows Store OID.
* Extension data is converted from a raw buffer to an std::string for
comparision.
* The raw buffer is not null-terminated, and therefore, running
std::strlen() on it causes invalid memory access.
* This invalid access is caught by ASAN on macOS.
Solution:
* Null-terminate the raw buffer before trying to build an std::string
from it.
Tests:
* Ran app test suite that uses libmsix.dylib with ASAN on. No crashes
were reported.
* Invalid Memory Access in SignatureValidator
Problem:
* As @JohnMcPMS pointed out, writing "" with 1 byte size is null termination.
* A better solution would be to use bptr->length and avoid writing the null
byte altogether.
Tests:
* Ran app test suite that uses libmsix.dylib with ASAN on. No crashes
were reported.
Co-authored-by: Sayan Chaliha <sachalih@microsoft.com>
Adds new property on AppxBundleMetadata: ChildAppxPackages. This replaces the now [Obsolete] InternalAppxPackagesRelativePaths, as it has the file paths but also the metadata unique to each child package.
* Added option to AppxFactory to create package writer with file hash enabled in block map xml
* Changed PCSTR to LPCSTR as PCSTR is not defined in Linux and MacOS. Also updated test data about expected block map xml size
* Added CoCreateAppxFactoryWithOptions and CoCreateAppxFactoryWithHeapAndOptions to export list
Madhusudhan Gumbalapura Sudarshan
Chacón
dependabot[bot]
Ruben Guerrero
Joakim
Madhusudhan-MSFT
ePirat
stephenk-msft
Sayan Chaliha
Union Palenshus
rhuang-msft