ml-agents/utils/validate_versions.py

199 строки
6.6 KiB
Python
Исходник Постоянная ссылка Обычный вид История

2020-03-03 00:58:08 +03:00
#!/usr/bin/env python3
import os
import json
import sys
from typing import Dict, Optional
import argparse
VERSION_LINE_START = "__version__ = "
Develop python api ga (#6) * Dropped support for python 3.6 * Pinning python 3.9.9 for tests due to typing issues with 3.9.10 * Testing new bokken image. * Testing new bokken image. * Updated yamato standalone build test. * Updated yamato standalone build test. * Updated standalone build test. * Updated yamato configs to use mla bokken vm. * Bug fixes for yamato yml files. * Fixed com.unity.ml-agents-test.yml * Bumped min python version to 3.7.2 * pettingzoo api prototype * add example * update file names * support multiple behavior names * fix multi behavior action index * add install in colab * add setup * update colab * fix __init__ * clone single branch * import tags only * import in init * catch import error * update colab * move colab and add readme * handle agent dying * add tests * update doc * add info * add action mask * fix action mask * update action masks in colab * change default env * set version * fix hybrid action * fix colab for hybrid actions * add note on auto reset * Updated colab name. * Update README.md * Following petting_zoo registry API (#5557) * init petting_zoo registry * cherrypick Custom trainer editor analytics (#5511) * cherrypick "Update dotnet-format to address breaking changes introduced by upstream changes (#5528)" * Update colab to match pettingZoo import api * ToRevert: pull exp-petting-registry branch * Add init file to tests * Install pettingzoo-unity requirements for pytest * update pytest command * Add docstrings and comments * update coverage to pettingzoo folder * unset log level * update env string * Two small bugfixes (#5589) 1. Add the missing `_cumulative_rewards` property 2. Update `agent_selection` to not error out when an agent finishes an episode. * Updated gym to 0.21.0 and petting zoo to 1.13.1, fixed bugs with AEC wrapper for gym and PZ updates. API tests are passing. * Some refactoring. * Finished inital implementation of parallel. Tests not passing. * Finished parallel API implementation and refactor. All PZ tests passing. * Cleanup. * Refactoring. * Pinning numpy version. * add metadata and behavior_specs initialization * addressing behaviour_spec issues * Bumped PZ version to 1.14.0. Fixed failing tests. * Refactored gym-unity and petting-zoo into ml-agents-envs * Added TODO to pydoc-config.yaml * Refactored gym and pz to be under a subpackage in mlagents_env package * Refactored ml-agents-envs docs. * Minor update to PZ API doc. * Updated mlagents_envs docs and colab. * Updated pytest gh workflow to remove ref to gym and pz. * Refactored to remove some test coupling between trainers and envs. * Updated installation doc. * Update ml-agents-envs/README.md Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com> * Updated failing yamato jobs. * pettingzoo api prototype * add example * update file names * support multiple behavior names * fix multi behavior action index * add install in colab * add setup * update colab * fix __init__ * clone single branch * import tags only * import in init * catch import error * update colab * move colab and add readme * handle agent dying * add tests * update doc * add info * add action mask * fix action mask * update action masks in colab * change default env * set version * fix hybrid action * fix colab for hybrid actions * add note on auto reset * Updated colab name. * Update README.md * Following petting_zoo registry API (#5557) * init petting_zoo registry * cherrypick Custom trainer editor analytics (#5511) * cherrypick "Update dotnet-format to address breaking changes introduced by upstream changes (#5528)" * Update colab to match pettingZoo import api * ToRevert: pull exp-petting-registry branch * Add init file to tests * Install pettingzoo-unity requirements for pytest * update pytest command * Add docstrings and comments * update coverage to pettingzoo folder * unset log level * update env string * Two small bugfixes (#5589) 1. Add the missing `_cumulative_rewards` property 2. Update `agent_selection` to not error out when an agent finishes an episode. * Updated gym to 0.21.0 and petting zoo to 1.13.1, fixed bugs with AEC wrapper for gym and PZ updates. API tests are passing. * Some refactoring. * Finished inital implementation of parallel. Tests not passing. * Finished parallel API implementation and refactor. All PZ tests passing. * Cleanup. * Refactoring. * Pinning numpy version. * add metadata and behavior_specs initialization * addressing behaviour_spec issues * Bumped PZ version to 1.14.0. Fixed failing tests. * Refactored gym-unity and petting-zoo into ml-agents-envs * Added TODO to pydoc-config.yaml * Refactored gym and pz to be under a subpackage in mlagents_env package * Refactored ml-agents-envs docs. * Minor update to PZ API doc. * Updated mlagents_envs docs and colab. * Updated pytest gh workflow to remove ref to gym and pz. * Refactored to remove some test coupling between trainers and envs. * Updated installation doc. * Update ml-agents-envs/README.md Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com> * Updated CHANGELOG. * Updated Migration guide. * Doc updates based on CR. * Updated github workflow for colab tests. * Updated github workflow for colab tests. * Updated github workflow for colab tests. * Fixed yamato import error. Co-authored-by: Ruo-Ping Dong <ruoping.dong@unity3d.com> Co-authored-by: Miguel Alonso Jr <miguelalonsojr> Co-authored-by: jmercado1985 <75792879+jmercado1985@users.noreply.github.com> Co-authored-by: Maryam Honari <honari.m94@gmail.com> Co-authored-by: Henry Peteet <henry.peteet@unity3d.com> Co-authored-by: mahon94 <maryam.honari@unity3d.com> Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com>
2022-02-03 03:32:23 +03:00
DIRECTORIES = ["ml-agents/mlagents/trainers", "ml-agents-envs/mlagents_envs"]
MLAGENTS_PACKAGE_JSON_PATH = "com.unity.ml-agents/package.json"
MLAGENTS_EXTENSIONS_PACKAGE_JSON_PATH = "com.unity.ml-agents.extensions/package.json"
ACADEMY_PATH = "com.unity.ml-agents/Runtime/Academy.cs"
PYTHON_VERSION_FILE_TEMPLATE = """# Version of the library that will be used to upload to pypi
__version__ = {version}
# Git tag that will be checked to determine whether to trigger upload to pypi
__release_tag__ = {release_tag}
"""
def _escape_non_none(s: Optional[str]) -> str:
"""
Returns s escaped in quotes if it is non-None, els e "None"
:param s:
:return:
"""
if s is not None:
return f'"{s}"'
else:
return "None"
def extract_version_string(filename):
with open(filename) as f:
for line in f.readlines():
if line.startswith(VERSION_LINE_START):
return line.replace(VERSION_LINE_START, "").strip()
return None
def check_versions() -> bool:
version_by_dir: Dict[str, str] = {}
for directory in DIRECTORIES:
path = os.path.join(directory, "__init__.py")
version = extract_version_string(path)
print(f"Found version {version} for {directory}")
version_by_dir[directory] = version
# Make sure we have exactly one version, and it's not none
versions = set(version_by_dir.values())
if len(versions) != 1 or None in versions:
print("Each setup.py must have the same VERSION string.")
return False
return True
def set_version(
python_version: str,
csharp_version: str,
csharp_extensions_version: str,
release_tag: Optional[str],
) -> None:
# Sanity check - make sure test tags have a test or dev version
if release_tag and "test" in release_tag:
if not ("dev" in python_version or "test" in python_version):
raise RuntimeError('Test tags must use a "test" or "dev" version.')
new_contents = PYTHON_VERSION_FILE_TEMPLATE.format(
version=_escape_non_none(python_version),
release_tag=_escape_non_none(release_tag),
)
for directory in DIRECTORIES:
path = os.path.join(directory, "__init__.py")
print(f"Setting {path} to version {python_version}")
with open(path, "w") as f:
f.write(new_contents)
if csharp_version is not None:
# removed exp.1 tag from version strings since MLA package is a supported package again
# package_version = f"{csharp_version}-exp.1"
package_version = f"{csharp_version}"
if csharp_extensions_version is not None:
# since this has never been promoted we need to keep
# it in preview forever or CI will fail
extension_version = f"{csharp_extensions_version}-preview"
print(
f"Setting package version to {package_version} in {MLAGENTS_PACKAGE_JSON_PATH}"
f" and {MLAGENTS_EXTENSIONS_PACKAGE_JSON_PATH}"
)
set_package_version(package_version)
set_extension_package_version(package_version, extension_version)
print(f"Setting package version to {package_version} in {ACADEMY_PATH}")
set_academy_version_string(package_version)
def set_package_version(new_version: str) -> None:
with open(MLAGENTS_PACKAGE_JSON_PATH) as f:
package_json = json.load(f)
if "version" in package_json:
package_json["version"] = new_version
with open(MLAGENTS_PACKAGE_JSON_PATH, "w") as f:
json.dump(package_json, f, indent=2)
f.write("\n")
def set_extension_package_version(
new_dependency_version: str, new_extension_version
) -> None:
with open(MLAGENTS_EXTENSIONS_PACKAGE_JSON_PATH) as f:
package_json = json.load(f)
package_json["dependencies"]["com.unity.ml-agents"] = new_dependency_version
if new_extension_version is not None:
package_json["version"] = new_extension_version
with open(MLAGENTS_EXTENSIONS_PACKAGE_JSON_PATH, "w") as f:
json.dump(package_json, f, indent=2)
f.write("\n")
def set_academy_version_string(new_version):
needle = "internal const string k_PackageVersion"
found = 0
with open(ACADEMY_PATH) as f:
lines = f.readlines()
for i, l in enumerate(lines):
if needle in l:
left, right = l.split(" = ")
right = f' = "{new_version}";\n'
lines[i] = left + right
found += 1
if found != 1:
raise RuntimeError(
f'Expected to find search string "{needle}" exactly once, but found it {found} times'
)
with open(ACADEMY_PATH, "w") as f:
f.writelines(lines)
Merge release 2 to master (#4000) * update versions for patch release (#3970) * update versions for patch releae * Update precommit flake8 (#3961) * fix changelog * Release 2 cherry pick (#3971) * [bug-fix] Fix issue with initialize not resetting step count (#3962) * Develop better error message for #3953 (#3963) * Making the error for wrong number of agents raise consistently * Better error message for inputs of wrong dimensions * Fix #3932, stop the editor from going into a loop when a prefab is selected. (#3949) * Minor doc updates to release * add unit tests and fix exceptions (#3930) Co-authored-by: Ervin T <ervin@unity3d.com> Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com> Co-authored-by: Chris Goy <christopherg@unity3d.com> * update changelog (#3975) * [docs] Add memory_size hyperparameter (#3973) * Release 2 docs (#3976) * Add v1.0 blog post and update reference paper. (#3947) * Develop mm fix readme releases (#3966) * Fix broken link and clean-up Releases section. * Updated link to be consistent with the table. * Update one of the bullets for consistency. * update table, add Versioning doc * release_2_docs Co-authored-by: Marwan Mattar <marwan@unity3d.com> * update barracuda to 0.7.1 (#3977) * Wrong variable naming in code example (#3983) (#3988) Co-authored-by: Sebastian Schuchmann <schuchmannsebastian@gmail.com> * Fix barracuda version in changelog * [docs] Add missing config and make sure to use floats in example (#3989) * Add missing config and make sure to use floats in example * Moved init_path * fix typo in log message (#3987) * Fix Barracuda assembly reference. (#3994) * fix missing metafile (#3999) * add missing metafile, change package to 1.0.2 * changelog * undo DevProject * Update make_readme_table.py * reapply markdown config file changes * fix release_1 references (#4001) Co-authored-by: Ervin T <ervin@unity3d.com> Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com> Co-authored-by: Chris Goy <christopherg@unity3d.com> Co-authored-by: Marwan Mattar <marwan@unity3d.com> Co-authored-by: Sebastian Schuchmann <schuchmannsebastian@gmail.com>
2020-05-21 04:08:29 +03:00
def print_release_tag_commands(
python_version: str, csharp_version: str, release_tag: str
):
python_tag = f"python-packages_{python_version}"
csharp_tag = f"com.unity.ml-agents_{csharp_version}"
docs_tag = f"{release_tag}_docs"
print(
f"""
###
Use these commands to create the tags after the release:
###
git checkout {release_tag}
git tag -f latest_release
git push -f origin latest_release
git tag -f {docs_tag}
git push -f origin {docs_tag}
git tag {python_tag}
git push -f origin {python_tag}
git tag {csharp_tag}
git push -f origin {csharp_tag}
"""
)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--python-version", default=None)
parser.add_argument("--csharp-version", default=None)
parser.add_argument("--csharp-extensions-version", default=None)
parser.add_argument("--release-tag", default=None)
# unused, but allows precommit to pass filenames
parser.add_argument("files", nargs="*")
args = parser.parse_args()
if args.python_version:
print(f"Updating python library to version {args.python_version}")
if args.csharp_version:
print(f"Updating C# package to version {args.csharp_version}")
if args.csharp_extensions_version:
print(
f"Updating C# extensions package to version {args.csharp_extensions_version}-preview"
)
set_version(
args.python_version,
args.csharp_version,
args.csharp_extensions_version,
args.release_tag,
)
Merge release 2 to master (#4000) * update versions for patch release (#3970) * update versions for patch releae * Update precommit flake8 (#3961) * fix changelog * Release 2 cherry pick (#3971) * [bug-fix] Fix issue with initialize not resetting step count (#3962) * Develop better error message for #3953 (#3963) * Making the error for wrong number of agents raise consistently * Better error message for inputs of wrong dimensions * Fix #3932, stop the editor from going into a loop when a prefab is selected. (#3949) * Minor doc updates to release * add unit tests and fix exceptions (#3930) Co-authored-by: Ervin T <ervin@unity3d.com> Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com> Co-authored-by: Chris Goy <christopherg@unity3d.com> * update changelog (#3975) * [docs] Add memory_size hyperparameter (#3973) * Release 2 docs (#3976) * Add v1.0 blog post and update reference paper. (#3947) * Develop mm fix readme releases (#3966) * Fix broken link and clean-up Releases section. * Updated link to be consistent with the table. * Update one of the bullets for consistency. * update table, add Versioning doc * release_2_docs Co-authored-by: Marwan Mattar <marwan@unity3d.com> * update barracuda to 0.7.1 (#3977) * Wrong variable naming in code example (#3983) (#3988) Co-authored-by: Sebastian Schuchmann <schuchmannsebastian@gmail.com> * Fix barracuda version in changelog * [docs] Add missing config and make sure to use floats in example (#3989) * Add missing config and make sure to use floats in example * Moved init_path * fix typo in log message (#3987) * Fix Barracuda assembly reference. (#3994) * fix missing metafile (#3999) * add missing metafile, change package to 1.0.2 * changelog * undo DevProject * Update make_readme_table.py * reapply markdown config file changes * fix release_1 references (#4001) Co-authored-by: Ervin T <ervin@unity3d.com> Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com> Co-authored-by: Chris Goy <christopherg@unity3d.com> Co-authored-by: Marwan Mattar <marwan@unity3d.com> Co-authored-by: Sebastian Schuchmann <schuchmannsebastian@gmail.com>
2020-05-21 04:08:29 +03:00
if args.release_tag is not None:
print_release_tag_commands(
args.python_version, args.csharp_version, args.release_tag
)
else:
ok = check_versions()
return_code = 0 if ok else 1
sys.exit(return_code)