Граф коммитов

2770 Коммитов

Автор SHA1 Сообщение Дата
Chi Song 30e27715bc Improve ssh shell to support reconnect
It needs to call close() explictly. In future, it may need reconnect on
exception automatically.
2020-10-03 07:11:02 +08:00
Chi Song c4bbf0500e Improve os requirements support super types
for example, Unbuntu satisifies Linux

Co-authored-by: Leandro A. F. Pereira <leandro@hardinfo.org>
2020-10-03 07:11:02 +08:00
Chi Song 3c299f6e85 Improve SetSpace for serialization
The original collection of SetSpace cannot be serialized, so that azure
location cache cannot save features to disk. So add items field for
serialization.

Note, it's a minimum patch to support our functionality, not target to
create a search_space works any place else.

1. add __post_init__ to load data from deserialized data.
2. unfortunately, #1 doesn't work automatically like document. So add
decode_set_space to make it can be called explictly.
3. override add, update methods, because they are used in code.
4. call decoders in schema.
2020-10-03 07:11:02 +08:00
Chi Song 998ee75011 add cat command to output sys files 2020-10-03 07:11:02 +08:00
Chi Song 997a932c74 Process: remove unused code. 2020-10-03 07:11:02 +08:00
Chi Song c3fe545caf node: remove Dict base to avoid confusing
The nodes is a list, which supports queried by name. It doesn't
implement some dict method. So remove dict to get it clearer.
2020-10-03 07:11:02 +08:00
Chi Song 36cf117a6a improve log of search_space 2020-10-03 07:11:02 +08:00
Chi Song 8213e09569 update comments in executable.py 2020-10-03 07:11:02 +08:00
Chi Song b39c1bad2b change feature in schema to str, not a class.
The feature class will be parsed in platform level, so it doesn't need
to know feature class in schema. the excluded feature set won't be
merged into included, after this change.
2020-10-03 07:11:02 +08:00
Chi Song ce3d0682bd remove unused from_value methods
As final implementation, no need to create capability from value. The
capability should be defined, if it's needed. If a capability is not
defined, a node uses default capability.
2020-10-03 07:11:02 +08:00
Andrew Schwartzmeyer 929fbc0326 Update to Poetry version 1.1.0b4 2020-09-23 17:36:41 -07:00
Andrew Schwartzmeyer 29019c4432 Enable cyclomatic complexity checking
And disable existing violations with noted TODOs.
2020-09-23 17:36:41 -07:00
Andrew Schwartzmeyer 57dc0339d7 Reformat due to updated Black package 2020-09-23 17:36:41 -07:00
Andrew Schwartzmeyer 300121735a Explicitly update developer tool packages
Biggest change is the move to Black 20.8b1. W503 is now disabled by
default, and since we control the versions of the tools we use, we can
safely update our configuration.
2020-09-23 17:36:41 -07:00
Andrew Schwartzmeyer eb98969695 Run `poetry update` 2020-09-23 17:36:41 -07:00
Andrew Schwartzmeyer 0ca0798cc5 Use Makefile to centralize build/test commands
This Makefile is optional because the commands can be run independently,
it was added just to simplify and centralize all our commands. On Linux,
GNU Make is expected to almost universally be installed, but on Windows
it may need to be installed manually to use it. Fortunately the GitHub
Actions Windows image has it.
2020-09-23 17:36:41 -07:00
Chi Song cd037c286f Add UT for azure platform
It doesn't need to access azure, but it covers limited scope.
2020-09-22 09:34:15 +08:00
Chi Song 3879356cd1 remove deprecated assert 2020-09-22 09:34:15 +08:00
Chi Song 0ad9f4f730 fix a bug found by UT
Schema should be copied, when NodeSpace generates min_value.
So that, if first batches doesn't match, the data won't override original
requirement.
2020-09-22 09:34:15 +08:00
Chi Song f5cd25d327 AzurePlatform: don't fail prepare env on empty
Platform should support empy requirement, like ready.
2020-09-22 09:34:15 +08:00
Chi Song b9444f3a39 fix a typo eligable to eligible 2020-09-22 09:34:15 +08:00
Chi Song 34d82580a8 Implement to deploy multiple nics
Since arm doesn't support nested loop, so use nested template to
workaround. Also add field and logic to support it.

other changes,
1. adjust priority of demo test cases, so easy to select by priority.
2. Remove AzureArmParameterNode, reuse AzureNodeSchema as their schema
    is almost the same.
3. some performance tuning like adjust loop priority, reuse caches.
4. remove some unused code.
5. improve log messages.
6. other minor improvements.
2020-09-21 12:46:00 +08:00
Chi Song 13d9be733f
Add unittest for lisarunner, and bug fixes (#1019)
* Add ut for lisarunner, and bug fixes

1. add ut for lisa runner
2. fix bugs found by ut.
3. remove some unused code.
4. improve testability.
5. move EnvironmentSpace from shema to env, because it's confused in
  schema folder. As it's not a schema.
6. other minor fixes.
2020-09-18 11:31:02 +08:00
Chi Song 9319ce0a3d Add platform test cases, and refactor
1. Add platform test cases
2. remove global platforms cache for testability.
3. remove unused platform_schema and node_schema on platform.
4. skip test files when importing to exclude mock classes.
5. other minor improvements.
2020-09-17 07:32:15 +08:00
Chi Song 57189562f8 add SkipTestCaseException to unblock test suite
It will be handled later.
2020-09-17 07:32:15 +08:00
Chi Song 694de48b0f add test cases for env, and testsuite
refine test cases to reuse test case util functions
minor update on comments
2020-09-17 07:32:15 +08:00
Chi Song 13a41d20af add capability schema for local and remote node. 2020-09-17 07:32:15 +08:00
Chi Song 265c9ae72d add min_nic_count in simple requirement 2020-09-17 07:32:15 +08:00
Chi Song 9d2b44b42c fix bugs and improve log
1. fix bug that not break when find first matched location
2. fix bug that cost caculated multiple times
3. improve logs
2020-09-17 07:32:15 +08:00
Chi Song c9f23c1271 update example runbooks
add azure runbook
remove confused local_dev, local dev can use hello_world
2020-09-17 07:32:15 +08:00
Chi Song 64fb01afc9 Azure support prepare environment
Other changes,
1. remove yum update on centos, as it's not necessary
2. fix lscpu under windows
3. fix logger to show debug in file always
4. show console as configured
5. update test case behavior, it's still acceptable
6. tuning some log
7. minor improvements
2020-09-17 07:32:15 +08:00
Chi Song 6bd5f6c2a3 fix conflict on import same name folders
add a prefix module name, and change the approach to import package.

other updates,
1. simplify hello_world runbook
2. fix words
2020-09-17 07:32:15 +08:00
Chi Song 6186dfb76c add a timer to measure deployment time 2020-09-17 07:32:15 +08:00
Chi Song a698de989e support OS detection and add some tools
1. update OS to support Ubuntu and CentOS
2. add logic to detect OS.
3. support package installation
4. add gcc, make tools
5. support git, ntttcp install
6. other minor improvements
2020-09-17 07:32:15 +08:00
Chi Song 75ce9d66d2 support test case requirement flow
1. add work flow to create env from cases.
2. add capability on environment and node.
3. add EnvironmentSpace for requirement and capability.
4. add prepare_environment method to match capability and requirements
5. use delay node schema nodes replace ugly package instrument.
6. move some test only code to test file.
7. add create factory to support windows tools.
8. support test case settings like retry, times, and so on.
9. other minor improvements.
2020-09-17 07:32:15 +08:00
Chi Song 0a3bb363be organize code for better logic
1. Move runbook related init code into runbook.py
2. cleanup commands.py, make it like a command wrapper.
3. move test case selector to lisarunner to centralize flow.
4. other minor improvements.
2020-09-17 07:32:15 +08:00
Chi Song 6385e90ddc add operating_system and initialable mixin
1. add operating system package for os information
2. add initialable mixin for reusing code
3. improve test suite message
4. other minor improvements
2020-09-17 07:32:15 +08:00
Chi Song bd9c233e9f add multiple nodes demo case 2020-09-17 07:32:15 +08:00
Chi Song f5e855881b implement requirement and capability comparison
Terms
1. Search Space. A couple of values, partial match is ok. For example,
   test case need at leat 8 cores, but also ok for 16 cores. But if a
   case needs exact 4 cores, so 6 or 2 doesn't meet requirement.
2. Requirement: it's from test case, it's requirement on test
   environment, platform and so on.
3. Capability: it's from platform, it describes what kind of environment
   can be provided by platform.

Two major methods,
1. check to check if a requirement can be met by a capability.
2. generate_min_capaiblity, if meet requirement, get the minimum meet object.

Support count, set (unordered) are enougth to current
design.

Replace original template with NodeSpace.
2020-09-17 07:32:15 +08:00
Chi Song 0c86e5a4de add support for code coverage 2020-09-17 07:32:15 +08:00
Chi Song 9cce7a727a bug fixes and refactoring
1. rename test file for consistent
2. fix log path regression bug
3. use peotry 1.1.0b2 for linux
4. other fixes and improvements
2020-09-17 07:32:15 +08:00
Chi Song e94e02d635 azure support deploy flag to reuse env 2020-09-17 07:32:15 +08:00
Chi Song 6dc0808bcd redirect stdout/stderr to improve secret 2020-09-17 07:32:15 +08:00
Chi Song 8c3615c333 update connect char 2020-09-17 07:32:15 +08:00
Chi Song 87e0f85bb0 fix incorrect name in Linux 2020-09-17 07:32:15 +08:00
Chi Song e0426ad966 improve stability on different windows behavior
add more log and fix bugs.
2020-09-17 07:32:15 +08:00
Chi Song 70fb7fe5d1 Support Windows SSH with spur 2020-09-17 07:32:15 +08:00
Andy Schwartzmeyer 622da16e20
Merge pull request #1005 from LIS/andschwa/getting-started
Update getting started instructions
2020-09-03 11:08:43 -07:00
Andrew Schwartzmeyer 3576467dc0 Update getting started instructions
This were tested with fresh installations of Ubuntu 18.04 and 20.04 via
WSL in order to capture all errant package dependencies.
2020-09-02 18:03:13 -07:00
Chi Song 0d6db713a6 remove unused folder 2020-09-01 10:02:58 +08:00