## Description:
+ Adds ML-KEM API surface
+ Implements the API with initial C implementation, with sprinkling of SSE/NEON for (I)NTT
+ Adds low level ML-KEM polynomial arithmetic testing which tests self-consistency and exercises internal assertion in debug builds
+ Adds multi-implementation functionality testing to enable comparative functionality and performance testing on E2E functionality
+ For now, comparison testing is just between SymCrypt static and dynamic, with 3rd party comparison with libcrux disabled until they publish a final ML-KEM implementation
+ Adds ML-KEM KATs from NIST
Related work items: #50913735
## Description:
Adds support for stateful hash-based signatures XMSS and XMSS^MT per RFC 8391 and NIST SP800-208.
## Admin Checklist:
- [ ] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [ ] You have updated CHANGELOG.md to reflect any changes in behavior
- [ ] You have updated symcryptunittest to exercise any new functionality
- [ ] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [ ] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [ ] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [ ] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
Related work items: #52453903
## Description:
SSKDF implementation and unit tests.
## Admin Checklist:
- [x] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [x] You have updated CHANGELOG.md to reflect any changes in behavior
- [x] You have updated symcryptunittest to exercise any new functionality
- [x] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [x] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [X] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [X] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
Related work items: #51795170
Since the KATs for some of the PQC algorithms have very long data values, it's convenient to be able to split the data across multiple lines to avoid having excessively long lines. This change adds support for that in the KAT parser (via escaping the newline with `\`), and adds a Python script that automatically reformats existing KATs with lines that are too long.
Tested: unit tests
- Clean up files that were added accidentally
- MacOS build working again
- Unit test fixes
- Cleanup, address comments, add Mac pipeline
- Windows and Linux build fixes
- Publish pipeline artifacts
- More Windows build fixes
- Add macOS build to official pipeline
- More pipeline fiddling
- Generate symbols for unit test
- Workaround for speculative load hardening bug on Clang 12+. Updated documentation
- Test availability of macOS 14 ARM64 agent
- Revert change to pipeline; macos-14-arm64 doesn't work in ADO. Add skeleton CMakePresets.json file for work to be continued on newer Mac
- ARM64 build fixes
- Pipeline changes
- Fix PublishPipelineArtifact name
- Address comments
Get rid of misleading comment.
First step towards turning the linux build path into a generic unix build path that supports Linux and macOS.
Make a build check that is already effectively for Linux only explicitly for Linux.
Remove unnecessary/breaking Apple includes.
test_lib.h: Generally use gnuc definitions on macOS but define GENRANDOM separately for macOS and Linux.
Define GET_PERF_CLOCK on macOS/ARM
Define body of getTimeInMs on macOS
We need C++17, and in order for CMake to set that properly on macOS we need to explicitly set the compilers to clang/clang++.
Use arc4random in unittest code on macOS.
build changes
+ Enable Arm64 Windows build with CMake (not production build as it is
missing support for Arm64X)
+ Avoid naming collision in macros between test code and product code,
enabling inclusion of sc_lib.h in test_lib.h
+ Remove sc_lib-testhooks.h
## Description:
Add OpenSSL implementation for SHA2 and SHA3 algorithms
## Admin Checklist:
- [ ] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [ ] You have updated CHANGELOG.md to reflect any changes in behavior
- [ ] You have updated symcryptunittest to exercise any new functionality
- [ ] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [ ] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [ ] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [ ] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
## Description:
## Admin Checklist:
- [ ] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [ ] You have updated CHANGELOG.md to reflect any changes in behavior
- [ ] You have updated symcryptunittest to exercise any new functionality
- [ ] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [ ] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [ ] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [ ] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
This change fixes a build break in the CMake pipeline by removing `SymCryptWipeKnownSize` from the exports for both the kernel and user mode modules. Since `SymCryptWipeKnownSize` is marked `FORCEINLINE`, it should be inlined everywhere and thus cannot be exported. (It's not clear why this didn't break the Windows Undocked Pipeline build--probably some difference in flags.)
In the future we'll remove this function from the Linux module exports as well. We have already documented that as an upcoming breaking change in `doc/breaking_changes.md`. Since the Windows modules haven't shipped yet, we don't have to consider this a breaking change for those modules.
Tested: unit tests with user mode + kernel mode dynamic modules
## Description:
## Admin Checklist:
- [ ] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [ ] You have updated CHANGELOG.md to reflect any changes in behavior
- [ ] You have updated symcryptunittest to exercise any new functionality
- [ ] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [ ] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [ ] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [ ] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
Add OpenSSL implementation for AesGcm test
This change updates the MSBuild projects to make the inclusion of msbignum and rsa32 optional. This will allow external users to more easily use MSBuild, and also unblock testing potential future changes to undocked pipelines.
## Description:
We add OpenSSL as submodule to 3rdparty and link symcryptunittest to it so we can compare the SymCrypt implementation. We add perf and functional test for XtsAes as well.
## Admin Checklist:
- [ ] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [ ] You have updated CHANGELOG.md to reflect any changes in behavior
- [ ] You have updated symcryptunittest to exercise any new functionality
- [ ] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [ ] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [ ] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [ ] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
Related work items: #49347468
## Description:
- Updates version.py to optionally accept git commit timestamp, commit ID, and branch name from the environment. SymCrypt is built from a source tarball in Mariner and cannot get this information from from git.
- Fixed build warnings encountered in Mariner build system
## Admin Checklist:
- [x] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [x] You have updated CHANGELOG.md to reflect any changes in behavior
- [x] You have updated symcryptunittest to exercise any new functionality
- [x] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [x] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [x] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
- [x] If you have added new algorithms/modes, you have updated the status indicator text for the associated modules if necessary
## Description:
This PR addresses the FIPS 140-3 requirement for crypto modules to support an approved services status indicator function.
## Admin Checklist:
- [X] You have updated documentation in symcrypt.h to reflect any changes in behavior
- [X] You have updated CHANGELOG.md to reflect any changes in behavior
- [X] You have updated symcryptunittest to exercise any new functionality
- [X] If you have introduced any symbols in symcrypt.h you have updated production and test dynamic export symbols (exports.ver / exports.def / symcrypt.src) and tested the updated dynamic modules with symcryptunittest
- [ ] If you have introduced functionality that varies based on CPU features, you have manually tested with and without relevant features
- [ ] If you have made significant changes to a particular algorithm, you have checked that performance numbers reported by symcryptunittest are in line with expectations
Related work items: #47548894
Adding support for salt length detection in RSA-PSS verification
+ Add SYMCRYPT_FLAG_RSA_PSS_VERIFY_WITH_MINIMUM_SALT flag to
SymCryptRsaPssVerify and SymCryptRsaPssVerifySignaturePadding
+ When verifying a PSS signature with this flag specified, allow salt
length >= the caller specified cbSalt
+ In symcryptunittest, test that verification succeeds and fails w.r.t. the cbSalt
and flag in the correct way
+ Some small unrelated changes
Related work items: #33692439
+ Add SymCryptXtsAes(En|De)cryptEx to induce XTS-AES with 128b tweak
+ Add SymCryptXtsAesExpandKeyEx to do FIPS IG check for non-equal
halves of XTS key
+ Refactor Multi-DataUnit logic to reduce code duplication using
new xtsaes_pattern.c file to instantiate the logic with various
specializations
+ Add support for ciphertext stealing in both old and new API surface,
allowing data unit sizes which are not a multiple of 16 bytes
+ Add IEEE test vectors for odd data unit sizes
+ Update randomized tests to exercise both 64-bit and 128-bit tweak
interfaces, and new supported data unit sizes
+ Update XTS tests to allow CNG to fail with unsupported parameters
+ Update XTS FIPS self-test to use FIPS-approved key expansion function
+ Fix a couple of typos found while making changes
Related work items: #43854713
+ Separate out SYMCRYPT_INTERNAL_ECURVE_TYPE representing the types that
SymCrypt can use for an Ecurve object at runtime from
SYMCRYPT_ECURVE_TYPE representing the types that callers can specify a
curve to be in curve parameters
+ Create a new SYMCRYPT_INTERNAL_ECURVE_TYPE which represents a Short-Weierstrass
curve with A == -3 (the form of all NIST prime curves)
+ Recognise A == -3 in Ecurve creation, and set the Ecurve type
appropriately
+ For NIST prime curves, dispatch to specialized EcPointDouble function
which uses fewer Modular operations
+ Add W25519 and W448 as test cases to exercise non-NIST prime Short
Weierstrass point doubling function. Just include these in test
binaries, rather than in production modules, we do not want to
encourage external callers to use these curves (they can specify the
ECC parameters themselves if they really need them, but callers almost
certainly should be using Montgomery or Twisted Edwards forms of these
curves)
Related work items: #45077527, #45539705
This PR adds the necessary YAML pipeline files for building SymCrypt via the Windows undocked pipeline. It also includes minor changes to existing files to support the tooling used by the pipeline. Currently, the pipeline only exists for pull requests. The next step is to create an official pipeline, including the option to package the binaries and ingest them into Windows. This will come in a subsequent PR.
This pull request adds MSBuild solution and project files so that SymCrypt can be built using the undocked OneBranch pipeline, including the kernel mode components. See the SymCrypt EO Compliance document for more information on why this is being done, and the high-level overview of how it will be accomplished.
In addition to adding the MSBuild files, I removed a bunch of files that were no longer being used, such as the iOS workspace and project files, old kernel test drivers that are not used in the RI-TP, etc.
Related work items: #42154697
Add HMAC-SHA3-256, HMAC-SHA3-384, HMAC-SHA3-512 implementations.
Other changes:
- Add generic HMAC API
- Introduce a unique state for each of the Keccak based algorithms
- Convert tabs to spaces in symcryptasm files
Related work items: #41559779
+ In Windows unit test environment only fatal when SymCrypt reports a vector extension as available when Windows does not report the register set is available
+ Previously would fatal when SymCrypt detected AVX2 not available when Windows reports AVX registers available, or AVX512 not available when Windows reports AVX512 registers available, causing a crash on IvyBridge or Knights Landing processors respectively
+ Also dump CPUID info on fatal error in unit tests to help debug this kind of issue in future
Related work items: #38706387
This change rewrites our Azure DevOps pipelines to be compatible with OneBranch pipelines. It also adds new scripts to help with building, testing and packaging SymCrypt. These scripts replicate some of the functionality of `scbuild` but are also compatible with Linux builds. They can be used directly on the command line by developers, but the OneBranch pipeline also uses them to move as much as possible of the "business logic" of building SymCrypt out of the YAML templates and into Python scripts.
Also includes various reorganization and small fixes.
+ New dynamic module SymCryptKernelTestModule_UM.dll and kernel driver
SymCryptKernelTestModule.sys which enable unit tests to call into
driver using the existing SymCrypt dynamic module flow
+ Update a few places in tests which assume objects created by module
under test can have their internals inspected (which is not the case
when the object lives in kernel and the unit tests are running in user
mode)
+ SymCryptKernelTestModule.sys tracks all allocations it makes and ensures it
frees everything when it is unloaded
Related work items: #38706387
+ Enable measurements of Linux RNG system
+ Make various performance improvements to defer costly calls into JitterEntropy until they are strictly required,
and reduce cost of calls when they are made.
Related work items: #42441472, #42441492
+ Do not use git status in createBuildString.cmd, it writes
all of the git lock files, which breaks Razzle build (ABT3101)
+ Do not include selftests that need allocations in the list
used by kernel test; there is no SymCryptCallbackAlloc/Free
defined there, so we have a link error.
`SymCryptDsaSelftest` was passing the `SYMCRYPT_DLGROUP_FIPS_LATEST` flag when calling `SymCryptDlgroupSetValue`, which causes that function to regenerate the primes P and Q and perform Rabin-Miller primality tests on them. This is very computationally expensive, to the point that running the DSA selftest caused significant performance problems in some scenarios.
The fix is to instead use `SYMCRYPT_DLGROUP_FIPS_NONE`; since we're using a hardcoded, known-good key, we do not need to perform the additional validation on it.
Also added error injection to selftests that were missing it, and added basic performance measurement to the selftests when they are run as part of the unit test, so that we can catch selftest performance issues more easily.
- Add SRTP-KDF and SSH-KDF implementations
- Update `SYMCRYPT_HASH` structure to contain hash state copying function member
Related work items: #38101963, #38102026
+ Just targeting Linux modules for now as we do not yet have a Windows SymCrypt module
+ At the command line when running symcryptunittest add option to load and test a dynamic SymCrypt module from a path
+ Enable a shim layer in our test code calling SymCrypt functions, and based on the template specialization or a global variable (depending upon the context of the function being called), the shim directs the function call to different places
+ Rework the multi-implementation part of the code handling SymCrypt's implementation (sc_imp*) to enable multiple SymCrypt implementations to coexist
+ The pre-existing ImpSc's shim is augmented to always add vector register save/restore testing around every call into the statically linked SymCrypt function. All previous test calls for vector register testing are removed. We may add other tests here in future (i.e. checking the contents of the stack are clean on SymCrypt function return?).
+ The new ImpScStatic implementation directly calls the statically linked SymCrypt function. This is used in performance tests
+ The new ImpScDynamic implementation uses a static variable in a lambda function to store dynamic symbol pointers that are looked up (once per lambda function) at runtime using dlsym. These pointers are then used to test the dynamic module directly (i.e. without any unit-test specific environment). This enables both comparative functional testing with static and dynamic SymCrypt, and realistic performance testing of the dynamic SymCrypt module
+ Also tests directly calling the SymCrypt APIs (not all tests calling low-level SymCrypt APIs) to shim to static or dynamic versions
Related work items: #38706387
+ Make VAES CPU feature depend only on VAES and VPCLMULQDQ
+ Make features for VAES_256 code depend on AVX2 feature and VAES (so
disabling AVX2 will disable VAES_256)
+ Similarly make features for VAES_512 code depend on AVX512 feature
+ Check xgetbv correctly to enable/disable AVX512 appropriately based on
OS support
+ Remove GetEnabledXStateFeatures logic from linux env file. For Windows
unittest env, check if xgetbv result is different to GetEnabledXStateFeatures, as
that indicates an OS bug
+ This enables all AVX2 on Linux which can support it, rather than
only supporting VAES_256 on Linux which supports AVX2 (the prior
situation)
+ Reintroduce reduced Xmm save/restore testing on Windows AMD64 user
mode, to check that Xmm6-Xmm15 are correctly saved/restored in SymCrypt code
+ Introduce optional Ymm save/restore testing which can be run on Linux
successfully today using runtime options for telling glibc to not use AVX. This allows
us to test the SymCrypt[Save|Restore]Ymm logic (relevant to Windows kernel mode)
accurately in the SymCrypt ADO pipeline.
Related work items: #32997124
Add AES-GCM session implementation
+ The GCM encryption session implementation enables FIPS certification of AES-GCM as the nonce generation is within the FIPS boundary
+ The GCM decryption session enables replay protection for callers. It is designed to be useful for fresh a higher level protocol like QCC
Related work items: #38643032
+ Re-enable Ymm save/restore tests in Windows unit tests
+ Fix Ymm save/restore tests in all unit tests (previously would never
trigger)
+ Add flag to skip Ymm verification if unit tests artificially fallback
to non-Ymm SymCrypt code which can still lead to (volatile) Ymm
registers being wiped with CRT functions
+ Opt-in to checking Ymm registers in unit tests for AES-GCM and AES-XTS
+ Add Save/Restore Ymm logic to AES-GCM and AES-XTS codepaths
+ Move GCM check for sufficient buffer length to use Ymm up a level to
avoid saving/restoring Ymm state for small buffers
Related work items: #37812709
+ KMAN uses SP800-108 with HmacSha512 - we will need to certify this in the SymCrypt module for Overlake / Mariner, so add SelfTests for this combination.
+ Also tidy up some comments which are clearly copy-paste errors
+ Ignore build artifacts that are in the jitterentropy submodule
Related work items: #37166368
+ Don't use x86/AMD64 specific timing functions in generic build
unit tests
+ Only include AMD64-specific files in AMD64 builds
Related work items: #35287257
+ Resolves all issues flagged by runoacr in symcrypt\lib
+ Leaves some oacr issues in test code
+ Also includes some unrelated fixes to typos etc.
Related work items: #35052770
This change adds a new SymCrypt shared object module for Linux. The shared object module implements integrity verification for FIPS compliance by reading its own memory at runtime, reversing any relocations, calculating the HMAC-SHA256 digest of the module contents in memory, and comparing the digest to a known-good value which is injected into the module (outside the FIPS boundary) post compilation by a Python script.
Related work items: #30397153, #30397542, #30397643, #30397707, #30397781, #32407416
+ Enable compiler optimizations (important for comparable perf with
intrinsics)
+ On my devbox this has the added advantage of the runtime of the
unit tests being "only" ~10 mins, so I suspect we can reenable running
tests in the pipeline
+ Enable SYMCRYPT_NOINLINE macro for GCC/clang (I was worried
measurements could be skewed if this macro isn't respected)
+ Also fix use of print with std::string rather than c_string in unit
test
Related work items: #32141800
+ Add validation flags for DlKey and EcKey Import and Generate
+ Enables callers to perform the correct amount of validation to be compliant with SP800-56arev3
+ Add support for named safe-prime groups in DH, as shortest path to supporting compliance using FFC
+ This will enable BCrypt callers to explicitly or implicitly use named safe-prime groups groups and have the requisite validation, rather than having to create a new in-memory representation of DH keys to expose the prime Q
+ Add tests to for validity of the named safe-prime group constants, so the unit tests should pick up on inadvertent changes, or errors in definitions of new safe-prime groups if/when they may be added
+ Enable the strictest validation that makes sense in all existing tests to exercise validation code.
+ Also randomly exercise the use of all the different valid flag combinations in tests
Related work items: #26526301, #26527152, #31528841, #31528936, #31529044, #31782556
Mitch Lindgren 🦎
Samuel Lee
Cagdas Calik
Maxwell Moyer-McKee
Josh Aas
Changyu Li
Yuval Harpaz