* Fix Utf8Decoder operator== handling of the last code point in the input
While working on diagnostics for https://github.com/microsoft/vcpkg-tool/pull/1210 I observed that we were printing the caret ^ in the wrong place when it goes after the input.
The way this works is we form the line of text to print, then decode the unicode encoding units, and when we hit the target, we stop and print ^:
5b8f9c40dd/src/vcpkg/base/parse.cpp (L51-L68)
however, if the intended location for the ^ is the "end" of the line, we hit this bug:
5b8f9c40dd/src/vcpkg/base/unicode.cpp (L273)
The iterator only compares the last_ pointers, but both the "points at the last code point in the input" and "points to the end of the input" state set `next_ == last_`. See:
5b8f9c40dd/src/vcpkg/base/unicode.cpp (L222-L226)
This means that the points to the end and points one past the end iterator compare equal, so the loop in parse.cpp stops one position too early.
Also adds a bunch of testing for this specific case, for other parts of Utf8Decoder, adds a way to parse the first code point without failing, makes all the operators 'hidden friends', and removes localized strings for bugs-in-vcpkg-itself.
* Add noexcepts as requested by @Thomas1664
A customer reported a problem where sometimes `vcpkg-cmake` can't detect that Ninja is safe to use. There, despite being on an amd64 system, this block didn't detect that things were OK:
(#1297)08c4e71048/ports/vcpkg-cmake/vcpkg_cmake_configure.cmake (L55-L61)
We are unable to replicate the problem, but suspect that they may be in an environment where PROCESSOR_ARCHITECTURE isn't set properly, similar to what we experienced in https://github.com/microsoft/vcpkg-tool/pull/769 .
Try to fix this problem by explicitly setting it to what it should be.
Before:
```console
PS D:\vcpkg> rm env:PROCESSOR_ARCHITECTURE
Remove-Item: Cannot find path 'Env:\PROCESSOR_ARCHITECTURE' because it does not exist.
PS D:\vcpkg> rm env:PROCESSOR_ARCHITEW6432
Remove-Item: Cannot find path 'Env:\PROCESSOR_ARCHITEW6432' because it does not exist.
PS D:\vcpkg> echo "PROCESSOR_ARCHITEW6432: $env:PROCESSOR_ARCHITEW6432 PROCESSOR_ARCHITECTURE: $env:PROCESSOR_ARCHITECTURE"
PROCESSOR_ARCHITEW6432: PROCESSOR_ARCHITECTURE:
PS D:\vcpkg> .\vcpkg.exe env
Microsoft Windows [Version 10.0.22631.2715]
(c) Microsoft Corporation. All rights reserved.
D:\vcpkg>echo PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: %PROCESSOR_ARCHITECTURE%
PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: %PROCESSOR_ARCHITECTURE%
D:\vcpkg>exit
PS D:\vcpkg> echo "PROCESSOR_ARCHITEW6432: $env:PROCESSOR_ARCHITEW6432 PROCESSOR_ARCHITECTURE: $env:PROCESSOR_ARCHITECTURE"
PROCESSOR_ARCHITEW6432: PROCESSOR_ARCHITECTURE:
PS D:\vcpkg> .\vcpkg.exe env C:\Windows\SysWOW64\cmd.exe
Microsoft Windows [Version 10.0.22631.2715]
(c) Microsoft Corporation. All rights reserved.
D:\vcpkg>echo PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: %PROCESSOR_ARCHITECTURE%
PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: %PROCESSOR_ARCHITECTURE%
```
After:
```console
PS D:\vcpkg> rm env:PROCESSOR_ARCHITECTURE
PS D:\vcpkg> rm env:PROCESSOR_ARCHITEW6432
Remove-Item: Cannot find path 'Env:\PROCESSOR_ARCHITEW6432' because it does not exist.
PS D:\vcpkg> echo "PROCESSOR_ARCHITEW6432: $env:PROCESSOR_ARCHITEW6432 PROCESSOR_ARCHITECTURE: $env:PROCESSOR_ARCHITECTURE"
PROCESSOR_ARCHITEW6432: PROCESSOR_ARCHITECTURE:
PS D:\vcpkg> .\vcpkg.exe env
Microsoft Windows [Version 10.0.22631.2715]
(c) Microsoft Corporation. All rights reserved.
D:\vcpkg>echo PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: %PROCESSOR_ARCHITECTURE%
PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: AMD64
D:\vcpkg>exit
PS D:\vcpkg> echo "PROCESSOR_ARCHITEW6432: $env:PROCESSOR_ARCHITEW6432 PROCESSOR_ARCHITECTURE: $env:PROCESSOR_ARCHITECTURE"
PROCESSOR_ARCHITEW6432: PROCESSOR_ARCHITECTURE:
PS D:\vcpkg> .\vcpkg.exe env C:\Windows\SysWOW64\cmd.exe
Microsoft Windows [Version 10.0.22631.2715]
(c) Microsoft Corporation. All rights reserved.
D:\vcpkg>echo PROCESSOR_ARCHITEW6432: %PROCESSOR_ARCHITEW6432% PROCESSOR_ARCHITECTURE: %PROCESSOR_ARCHITECTURE%
PROCESSOR_ARCHITEW6432: AMD64 PROCESSOR_ARCHITECTURE: x86
```
In https://github.com/microsoft/vcpkg/pull/35164 there was some confusion where a customer thought `vcpkg ci` was building an old version of openvino because we only printed the version to build for the `--dry-run` prepass and didn't print the version to build for the actual build. The customer thought we were building the old version rather than the new version as a result.
This change ensures we print the version we are discussing after "Installing" or "Building" and consistently uses the "packagespec@version" form in all the places. For consistency the "packagespec -> version" form we used in a few places is replaced with the @ form.
* Fix VS static analyzer warnings.
1. Optional and expected were not properly forwarding noexcept for their special member functions, so is_nothrow_move_constructible_v<Expected<T>> was false for all T. (This has bad implications if the expecteds are in a vector)
2. system.process.cpp uses a 32k stack buffer but it's the "top" of the stack so we don't care.
3. dependencies.cpp was calling std::move() on a constant variable `info_ptr->default_features`; I changed the declarations to better show that this was const.
4. (Drive by) All users of make_optional were actually doing emplacement.
* Use the git repo of --x-builtin-ports-root and --x-builtin-registry-versions-dir when talking about port commit shas and version tree shas, respectively.
I believe this is necessary to make the instructions in https://learn.microsoft.com/vcpkg/produce/publish-to-a-git-registry#4---update-the-versions-database actually work.
This also reduces the number of things that assume VCPKG_ROOT is a git repo. (Since it is not when we are shipped in VS or acquired with the one liner)
Alternate resolution of https://github.com/microsoft/vcpkg/pull/34441
The URL is supposed to contain query parameters and similar, so we can't go all the way to full urlencode(). But spaces should do the same thing with curl as they do for WinHTTP and get encoded correctly.
The registries machinery like version database parsing is annoying to reuse because it has this VersionDbEntry type which can represent version records in both filesystem and git registries, forcing all callers that interact with everything to special case both or otherwise tolerate 'silent UB' all over the place. I found that all callers always pass constant values, meaning this all can be totally separate functions and structs, thus eliminating that entire class of error.
This also means that the interface most clients *want*, 'get_builtin_versions', now maps closely to what the actual implementation is doing.
This also means that reading git registries no longer requires clients to cough up a meaningless 'root' variable.
This also means that callers like add-version and ci-verify-versions can use the same version database record type that all the parsing machinery uses rather than repackaging into pair.
* Add name, synopsis, and example information to CommandMetadata.
* Clean up needing to make thunks for all help text in command metadata machinery.
* Power autocomplete and help with the same tables that power command lookup. Resolves this line item first discovered in https://github.com/microsoft/vcpkg-tool/pull/1164
* Improve help by printing synopsis and grouped examples powered by CommandMetadata.
* Update the main usage to make sure every relevant command is included.
* Add website link to all help commands.
* split rarely used and artifacts into separate `vcpkg help commands` from @ras0219-msft
* make tense for all synopsis and argument help text consistent from @ras0219-msft
* burninate 'instead of' and 'writes out'
* Deduplicate the all commands help topic and remove 'vcpkg' from 'vcpkg help blah'.
This removes the 'in September 2023' message and does what the message said we would do. For the next 6 months we'll warn that the behavior changed.
The specific change to triplet.cpp to change the triplet setting is from https://github.com/microsoft/vcpkg-tool/pull/640
Co-authored-by: Alexander Neumann <30894796+Neumann-A@users.noreply.github.com>
* Delete 'vcpkg cache'.
'vcpkg cache' was a helper for folks to attempt to implement caching systems. It prints the contents of the "packages" directory. It was a kind of helper for before we had real binary caching support built into the tool proper.
@vicroms indicates that telemetry shows fewer than 100 invocations of `vcpkg cache` in the last 90 days, so this should be OK to outright remove without a deprecation period.
* Also get rid of unique messages.
Also uses the new infrastructure to avoid needing a temporary file in:
[dependencygraph] Fix "command line too long" when sending to GitHub's dependency-graph API #1144
from @klalumiere .
Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Kevin Lalumiere <kevin.lalumiere@gmail.com>
* initial implementation
* avoid value or exit and reuse MissinOption message
* improve messaging
* reduce redundancy
* add end2end test file
* add test cases for raw based exports in manifest mode
* get_all_port_names => append_all_port_names
* Extract get_all_reachable_port_names.
* Fix redundant equal compare in package_match_prefix when the prefix is a wildcard.
* Fix handling of wildcards when getting all reachable names.
* Teach autocomplete to do its work without hitting the network.
* Add unit test.
* Make commands physical design consistent.
In the vcpkg technical review meeting today we said:
* We want commands.Xxx.cpp for every "vcpkg Xxx" subcommand, rather than Xxx.cpp.
* We want x- commands to not have the x in their code names or x- in their file names, and z- commands have the z in their code names and z- in their file names.
* Command cpps have dashes.
Drive by fixes as part of this audit:
* FooCommand structs are gone, all users were stateless and are better served by plain function pointers. This also allows the command tables to be constexpr.
* commands.interface.h is dead, as it only existed to declare those structs that are dead.
Future changes:
* I really want all the Commands::Xxx and the commands namespace itself to go away. But ran out of time to sell this to the team.
* Xxx::COMMAND_STRUCTURE => XxxCommandMetadata
* Xxx::perform_and_exit => xxx_command_and_exit
Notes: build, install, and export probably deserve to have functionality split out, but that would probably be intricate so I have not attempted to do it here.
export.*.cpp seem like they *really* shouldn't be separate .cpps.
* Missed some _command s.
* wip
* wip
* format
* minimal impl
* add .xz
* Add help text for command
* vcpkg-artifacts uses x-extract
* format
* oops
* x-extract -> z-extract
* rename extract_archive_to_empty -> extract_archive and rename the old extract_archive to set_directory_to_archive_content
* fix comment
* wip
* add strip setting
* add strip option localized message
* format
* wip
* Add testing for strip mapping
* add --extract-type option to bypass file extension detection
* format
* wip
* test
* wip
* wip
* wip
* forward slash delim for non-windows
* wip
* try again
* maybe fix paths for non-windows
* use generic_Xxx
* use + instead of /
* convert back to path
* use appropriate path separators in test
* set VCPKG_COMMAND env variable
* set VCPKG_COMMAND
* set VCPKG_COMMAND in yaml
* wip
* use matrix.preset
* remove archive unit test from artifacts
* debug
* const everything
* create destination if it does not exist
* use fsPath instead
* bug fixes and remove installers
* fix end-to-end check
* wip
* wip
* debug messages
* fix strip for macos and linux
* print output
* wip
* cleanup
* format
* rename to command_extract_and_exit as per convention
* response to feedback
* remove the extract-type option and just fall back to cmake
* fix strip_map to proper behavior, better temp dir handling, etc
* fix unit tests
* for real this time
* wip
* wip
* reuse logic for is_slash functionality
* oops
* testing
* test
* testing
* verify problem
* try this
* fix
* use Strings::strto<int> instead of std::stoi
* add AUTO option + unit test
* add strip=AUTO
* switch artifacts to use strip=AUTO
* format
* cleanup
* fix compiler warnings
* fix conversion
* format
* remove magic -1, respond to feedback, format
* update localized messages
* add e2e
* test executable bits on non-windows
* oops
* better naming
* try this
* update end2end
* for real this time
* verify file exists
* print UnixMode
* oops
* fix indentation
* look for the right unixMode
* Simplify tests.
* Delete unreferenced test assets, add archive start events back.
* remove = from #define
* format
* minor fix
* Fix casing for enum class
* rename strip_map and get_common_prefix_count and add minimum documentation
* wip
* missed a couple renames
* rename test folder from "archive" to "folder0"
* remove then from artifact caller
* some structural changes
* enforce relationship between stripMode and stripCount
* use files is_slash
* check strip option is >=0
* format
* minor refactoring
* remove empty entries from get_archive_deploy_operations
* minor refactor
* adds testing for guess_extraction_type and get_strip_setting
* add testing for get_strip_setting
* minor feedback
---------
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* Remove const Optional<>& from the constructor. The caller unpacks them prior to calling constructor
* rename put_file() parameter 'request' to 'method'
* Added `get_cache_entry` to downloads and `lookup_cache_entry` uses it.
* wip
* reserve_cache_entry and push success now use get_entry
* print output for debugging
* fix reference to temporary bug
* wip
* wip
* wip..
* try again
* rename get_entry to invoke_http_request
* cleanup
* wip
* change `invoke_http_request` to return an expected
* change key to be package name + abi
* debug print statements
* reverse method and url order on command
* more print statements.
* try again
* checking value of invoke_http_request
* one more time
* print exit code
* print exit code
* try again
* wip
* try to print cachId
* wip
* wip
* write url has an s at the end...
* cleanup
* [working]removing debug print messages
* add option -L
* use -X method instead of -d
* ensure query_params is not empty before joining
* wip
* wip
* restructure curl command
* if GET, queries delimited by ?
* testing without flattening just letting -d do its thing
* print command
* wip
* remove -L
* move the method option
* don't pass -d to GET requests
* confirming that -X GET is not needed
* merge main
* wip
* encoding data manually rather than using -d
* print error if there is one.
* for real this time
* do not encode data since content-type = application/json
* use -d instead of --data-raw
* const string& everything
* oops
* try --data-raw
* try my own implementation of filesize
* rename filesize to file_size
* do ec.assign since I don't need to assign it to anything custom
* set ec in a consistent manner
* format
* const the parameters
* format again
* rename url_encode to percent_encode because it's no longer needed
* remove unused variable
* wrap linux code in #else to avoid the 'unreachable code' warning
* apply diff
* add User-Agent header
* change to base version
* use vcpkg_version_base-vcpkg_version
* wip
* update cache key/value
* missed headers
* fix headers
* wip
* wip
* try again
* try again
* debug print
* remove debug messages
* some error handling
* remove unused variable
* format
* format
* response to feedback
---------
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
* Add an e2e test for removed-from-baseline ports.
* Rename patterns.ps1 to e2e-registry.ps1, to reflect that there are
tests for more than just patterns now.
* Add removed.json.in and a call to it.
See also https://github.com/microsoft/vcpkg-tool/pull/1108 which
fixes all the e2e infrastructure to consistently use -s.
* Actually fix the bug: anything accepting both an ActionPlan and a PortFileProvider is suspect. In this case it was:
```
void load_tag_vars(const ActionPlan& action_plan,
const PortFileProvider& port_provider,
Triplet host_triplet) const;
```
Fix that by just pushing the ActionPlan part down to the underlying virtual load_tag_vars.
Also:
* Make all the things const so that the overall variable can be constexpr.
* Implement the readonlyness with a drived interface type rather than cv qualifiers, as requested by @ras0219-msft.
* Add --format to depend-info.
Add error handling and testing to depend-info switch handling.
Documentation update: https://github.com/microsoft/vcpkg-docs/pull/89
* xtree => x-tree, can not => cannot
* Add / use map utilities.
While reviewing https://github.com/microsoft/vcpkg-tool/pull/989/ I nerd sniped myself into looking for other instances of this pattern. Ultimately I proved to myself that I should *not* make a comment, but proving that required doing this work anyway...
* CR feedback
* [telemetry] track process tree using hashes
* send metrics
* error handling
* fix tests
* Fix process handle leak
* Fix non-Windows build
* Use CreateToolhelp32Snapshot API
* Track process tree on Linux
* PR comments
* Handle parenthesis in stat file + add tests
* Apply suggestions from code review
Co-authored-by: Billy O'Neal <bion@microsoft.com>
---------
Co-authored-by: Billy O'Neal <bion@microsoft.com>
Billy O'Neal
Thomas1664
autoantwort
data-queue
Victor Romero
Capric
Robert Schumacher
Alexander Neumann
Guillaume Racicot
Julian Zimmermann
Javier Matos Denizac
Haowu Ge
Kevin Lalumiere
comex
dan-shaw
Wim Leflere
Pierre Wendling
dubejf