- Add timestamp arguments to build_mariner_toolchain.sh
- Split tools.mk into tools.mk and chroot.mk, fix the include order in the main Makefile to use go tools in toolchain.mk
* add PACKAGE_REBUILD_LIST for testing
* Add sodiff to 2.0
* Add documentation of SODIFF_OPTIONAL_SOURCES
* Add sodiff changes to makefile
* fix sodiff for 2.0
* comment updates to various makefiles and sodiff script
* add sodiff pipeline files
* fix spacking in PackageBuild.yml
* add logging to sodiff
* change logging for sodiff
* send dnf errors to dev/null
* bump json-glb version for testing
* bump jq and protobuf for testing
* Add failure condition for having packages present in summary
* add missing semicolon
* Revert "bump json-glb version for testing"
This reverts commit 30926dcac21826ee3ae487259a0bc2cc12bcdfe3.
* change return to exit
* move SodiffCheck template, add sodiff to PR check, change release ver to grep from mariner-release.spec
* correct parameteres to parameters
* update SodiffCheck.yml to strip whitepsace from mariner version
* fix configuration typos
* add srpm pack list for testing
* remove buildRepoRoot param from sodiff call
* remove testing changes with srpm pack list
* Revert "bump jq and protobuf for testing"
This reverts commit eff92ca649.
* remove out of date doc update
---------
Co-authored-by: Sam Meluch <sam.meluch@microsoft.com>
Co-authored-by: MateuszMalisz <mateusz.malisz@microsoft.com>
* Add the repoquerywrapper tool.
The repoquerywrapper tools creates a chroot, installs dnf packages, and
runs specific repoquery commands within that environment.
This tools allows us to run repoquery on systems that may not have repoquery
installed/available.
It also allows us to run queries that are not directly supported by repoquery -
for example, in this check-in, it allows us to check which packages in a list
are available on the target repros.
Such queries are very specific - and can be added incrementally to the tool as
the need arises.
* Fix formatting (make go-tidy-all).
* Fix typo in help message.
* Add support for per-package ccache.
* Add support for removing ccache archives that are not the latest.
* Protect against failed ccache manager initialization.
* Update ccache configuration parameter dump.
* Use runtime build architecture instead of rpm node architecture.
* Fix error wrapping + minor changes.
* Fix formatting (make go-tidy-all).
* Fix typo in error message.
* Introduces a basic pattern for ##help comments that allow expressing the help content for well-known variables and targets at their respective definition sites.
* Adds help Makefile target that extracts the ##help comments via sed/grep/awk and does a simple rendering of it to stdout.
* Adds help.mk, which documents the format of these comments and also includes the toplevel help preamble.
* Populates a set of initial ##help comments for commonly used targets and variables. The intention is for this to be easy to iterate on in-tree over time.
Set `CONFIG_FILE=""` as the new default instead of `CONFIG_FILE="./imageconfigs/core-efi.json"`.
For teams that use the Mariner toolkit to build custom packages or images using the Mariner toolkit:
- Previously the toolkit would default to the core-efi.json image definition for all invocations.
- This would result it building all packages required for that image definition when calling `make build-packages` in the core repo
- This would mean that `make image` would succeed without explicitly setting a desired image config
- In future versions of the toolkit, CONFIG_FILE will default to “”
- Package builds will no longer include extraneous packages that may be unneeded
- Image builds will return an error if a config is not explicitly selected
Considerations:
- Packages like the kernel will no longer build during normal package builds unless requested. Set desired packages via PACKAGE_BUILD_LIST=”pkg1 pkg2 …”
- `make image` will no longer succeed without CONFIG_FILE=”/path/to/config.json” being set explicitly
* add flag for grapher to resolve cycles by downloading RPM from repos in repolist
add flag to clear version. replace run node with remote node in lookupTable for graphpkgfetcher
send cloner object instead of multiple parameters
* add info to show cycle resolution and use pre-allocated array instead of counted size
* preallocate nodes instead of getting actual count and add info stmt
* Avoid clearing version in original node
* use a different directory for saving chroot
* revert new NodeTypes
* Revert "Avoid clearing version in original node"
This reverts commit e1f17452dbd0df2c5e17109fb3c5b8085cee57d3.
* graphpkgfetcher satisfy with fetched packages
* clear version only if package cannot be found with version
* add Info stmt to show how cycle is fixed
* use rpmsprovidedbySRPM
* Revert "use rpmsprovidedbySRPM"
This reverts commit 571e5be0d7ddb6819ecf998f5397dc46eabf805c.
* Revert "graphpkgfetcher satisfy with fetched packages"
This reverts commit 162882f0d41691011ac9bec4c82fc30aac37097f.
* Revert "Revert "Avoid clearing version in original node""
This reverts commit e36664cf5214aa7b180a369f42a0a9e05e41c13d.
* introduce clonerErr to avoid it tamper the return error
* sort is not deferred when adding remote node to lookup table
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
* do not defer sort when adding remote node to lookup table
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
* remove packageURLlist from grapher
* add disableDefaultrepos flag to grapher
* split MakeDAG
* continue on clonerErr
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* continue early if not buildToRunEdge
* function and arg name changes in replaceCurrentRunNodeWithNewNode
* return error from addRemoteToLookup if tried to add a nonRemote Node
* use proper tmp dir for grapher worker_chroot
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* drop extra debug print
* Try to break cycle with prebuilt RPMs before ignoring version
* Panic if AddRemoteToLookup is called with a non-remote node
* Remove debug function - printLookupTable
* Break inconsistancy in scheduler
Call AddRunToLookup in scheduler.go to avoid inconsistency of the
LookupTable
* Update toolkit/tools/internal/pkggraph/pkggraph.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/grapher/grapher.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Address review comments to
* Functionalize the breaking cycle using upstream at a particular node
* Update function namereplaceCurrentRunNodeWithNewNode to meaningful
replaceSRPMBuildDependency
* Update proper name in function call
* Update toolkit/tools/internal/pkggraph/pkggraph.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Return proper error message from fixCyclesWithExistingRPMS
* Remove Add*ToLookup APIs
* Replace AllRunNodes with AllPreferredRunNodes and make AllRunNodes
travers the entire graph
* Elaborate comments for AllPreferredRunNodes
* Adding suggestions.
* Remove unnecessary assignment in pkggraph.go
---------
Co-authored-by: Sindhu Karri <lakarri@microsoft.com>
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
Co-authored-by: Bala <balakumaran.kannan@microsoft.com>
Co-authored-by: Bala <kumaran.4353@gmail.com>
This PR introduces 3 additional flags, by default all the three flags are disabled.
- ENABLE_TRACE: Setting this flag to y, instruments the various tools in the
toolkit to record trace data. Tracing is a way to instrument code to analyze
latency throughout the lifecycle of a call or user request. Traces provide an
overview of how much latency each component contributes to the overall latency
in a system. Traces can span multiple Go processes.
- ENABLE_MEM_PROFILE: Setting this flag to y instruments the toolkit to record
the memory profile.
- ENABLE_CPU_PROFILE: Setting this flag to y instruments the toolkit to record
the cpu profile.
Profiling tools analyze the complexity and costs of a Go program such as its
memory usage and frequently called functions to identify the expensive sections
of a Go program.
It is advisable to only turn one of the flags for a run to get data that does
not have noise due to other profiling/tracing flags.
The trace/profile data is created in the build/profile directory.
To use the profile data: `go tool pprof -http=:8080 file.profile`
To use the trace data: `go tool trace -http=:8080 file.trace`
Reference: https://go.dev/doc/diagnostics
Signed-off-by: Muhammad Falak R Wani <falakreyaz@gmail.com>
* added timestamp feature for image-building process
* Update toolkit/docs/how_it_works/1_initial_prep.md
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/pkggen/worker/create_worker_chroot.sh
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Removed test with direct filepath to home directory
* Removed unused imports in timestamp testing file
* Update toolkit/tools/roast/roast.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Pluralize timestamp flag description message
* add timestamp in create_worker_chroot; fix format in roast.go; removed unnecessary tests
* fix boilerplate template; relocate 'bldtracker' in go_tool_list to be alphabetical
* fixed formatting
* changed alphabetical order of go tools
* removed timestamp.TrackToFile (not used)
* Update toolkit/tools/bldtracker/bldtracker.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/bldtracker/bldtracker.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/pkggen/worker/create_worker_chroot.sh
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/pkggen/worker/create_worker_chroot.sh
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* fixed formatting & naming; changed log file output for bldtracker
* fixed minor mistakes
* replace raw strings with well-named constants
* csvparser now dynamically gets the list of CSV files to parse
* Address feedback
- Fix magic numbers in csvparser.go.
- Delete timestamp_test.go for this PR.
- Change initialize mode in bldtracker.go to "i".
* Fix formatting
* Fix formatting again
* Rename csvparser to timestampcsvparser
timestampcsvparser is a more accurate name than csvparser because it
indicates that the parser is not generic to all CSVs.
* Address syntax errors
- moved "$(go-bldtracker)" into "$(worker_chroot_deps)" in tools.mk
- wrapped “$timestamp_dir” with ""s when
calling bldtracker.go in create_worker_chroot.sh
- added missing space after "record" in the description for the "mode"
variable in bldtracker.go
* Fix calls to bldtracker tool
- add bldtracker back as an input for create_worker_chroot
- create the variable "script_name" to avoid duplicated operations
* Fix boilerplate
- Remove unnecessary CSV file generated as an output from
calling boilerplate.go
- Removed unnecessary comments
* Refined bldtracker variables
- made the modes constants
- changed the scope of "completePath" from global to local (in main())
* Use join for filepath concatenation.
* Move init filepath to above the err conditional.
* dashboard
* path finding adapted to user directories
* Added error handling; now checks every file in a list
* Change output from number of bytes to number of lines
* Rename dashboard function
* combined two slices into a map
* Done with the MVP
* changed output location to build/timestamp
* added progress bar visualization
* update bar
* progress bar in chronological order
* added timestamp at the beginning of any bash script; fixed progress bar to display the correct %
* a
* multiple progress bar
* added goroutine calls
* Recalculated the image build start time and duration using the init file modTime
* Modify timestamp descriptions and add dashboard labels
* Optimize dashboard label
* Remove comment
* fix formatting issues
* Change padleft and modify step names
* Modify step name in imager
* fixed dashboard length for smaller screens
* initialized new branch
* explained tests; moved /demo into /internal
* moved demo folder back to tools
* demo now adds timestamps in intervals
* done with generating json for demo
* fixed json name
* create wrapper around uiprogress to enable removing bars
* Nested progress bar on updating JSON file
* delayed progress bar removal
* new files
* fixed target dir and code structure
* fixed naming format
* cleaned up code
* Fixup after rebase
* wip
* wip 2
* wip 3
* bit of cleanup
* why is this a separate set of changes?
* WIP
* remove binary output file
* update manifests to stable
* use up to date toolchain versions
* wip
* fix build
* migrate from timestamp_v2 and fix logic
* resolve deadlock
* add comments
* formatting
* revert timestamp change
* remove unused scripts
* sync with main
* sync with main
* remove weighted stuff
* make sure CompleteTiming can only be run once
* Fix null ptr and add some test cases
* add test cases
* Add timestamp code to existing tooling (#5465)
* Revert "revert timestamp change"
This reverts commit 6f07ef16a3.
* Revert "revert timestamp change"
This reverts commit 6f07ef16a3.
* stop collecting timestamp on sigterm
* Address comments
* remove toolchain_download_timestamp.sh
* more test coverage
---------
Co-authored-by: (Apple) Xuanchen Li <apple20010209@163.com>
Co-authored-by: Applelxc <59279209+Applelxc@users.noreply.github.com>
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
Co-authored-by: james-zhou1 <james.zhou@berkeley.edu>
Co-authored-by: James Zhou <91182187+james-zhou1@users.noreply.github.com>
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
Co-authored-by: james-zhou1 <james@james-Virtual-Machine>
Co-authored-by: Michelle <t-michelin@microsoft.com>
- Created a new variable ALLOW_TOOLCHAIN_REBUILDS to use this flag for the pipelines to not report and fail the pipeline on the toolchain error.
- Set up the flag to be "false" during all the builds except testing pipelines. Included changes in CBL-Mariner-pipelines code.
- Added allowToolchainRebuilds into RecordBuildResult function conditions and changed the PrintBuildSummary function to not print the error log but print info log if the flag is true
---------
Co-authored-by: Betty Lakes <bettylakes@microsoft.com>
This change allows you to build a package for either arm64 for x86_64 by appending TARGET_ARCH to the build-packages make command. This change combined with the cross-toolchain allows developers to cross-build RPM packages.
* Restore Production Repos. Create Release Candidate Version
* replaced the enabled status for base repo
* Update update_manifests.sh to remove preview and reflect recent debuginfo changes
* Update tools to build packages with dependencies hydrated
Hydrated build is a new approach where all the dependencies are hydrated from pre-built RPMs. This provides two advantages.
1. Faster build. As no package need to wait for its dependency's build to complete, all packages will be fired to build in parallel. (Practically with slight delay as firing of pre-built nodes takes place before).
2. Failure of any dependency build will not cause failure to the current package, as the dependency is fetched from cached RPMs.
Introduced a new flag HYDRATED_BUILD to enable this feature. Also it is pre-requisite to run `make hydrate-toolchain` and `make hydrate-rpms` with appropriate tar files before using this option.
This is achieved by replacing the BUILD to RUN node edge with BUILD to PREBUILT node edge (clone of the RUN node) nodes (if there is a corresponding RPM hydrated already).
* Fix go fmt check failures
* Fix typos
* Introduce graphscrubber
graphscrubber is for preprocessing the build graph to scheduler.
Currently it is only doing hydrated build related work. Ideally all the
graph optimization related jobs to be moved from scheduler to
graphscrubber.
* go fmt tidy
* Remove unnecessary functions from graphscrubber
* Fix Make rule to build graphscrubber before using it
* Use inline hydrated-build flag
* Update documentaion for HYDRATED_BUILD flag
* Update changes recommended in PR discussion
* go tidy
* Remove unnecessary arguments from graphPreprocessor
* Update toolkit/tools/graphPreprocessor/graphPreprocessor.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/graphPreprocessor/graphPreprocessor.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* Update toolkit/tools/graphPreprocessor/graphPreprocessor.go
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
* retrugger checks
* Update toolkit/scripts/pkggen.mk
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
* Update toolkit/scripts/pkggen.mk
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
* Update toolkit/scripts/pkggen.mk
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
* Update toolkit/scripts/pkggen.mk
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
* retrugger checks
* retrugger checks
Co-authored-by: Pawel Winogrodzki <pawelwi@microsoft.com>
Co-authored-by: Daniel McIlvaney <damcilva@microsoft.com>
Daniel McIlvaney
Trung
Sam Meluch
George Mileka
reuben olinsky
sindhu-karri
Pawel Winogrodzki
Neha Agarwal
Muhammad Falak R Wani
Andrew Phelps
Betty
osamaesmailmsft
Dallas Delaney
Olivia Crain
jslobodzian
Thomas Crain
Bala