Support --lang-pair in remote_settings script (#134)
* Represent new records internally as a list * Add --lang-pair flag * Write test for --lang-pair * Add README.md to remote_settings * Add help message for no files found * Add top-level example to README * Add test for empty directory given language pair
This commit is contained in:
Erik Nordin
GitHub
Родитель
c8c896d640
Коммит
e03ba4ea06
|
|
@ -132,5 +132,6 @@ dmypy.json
|
|||
._.DS_Store
|
||||
*.bin
|
||||
*.spm
|
||||
!tests/remote_settings/attachments/*.bin
|
||||
!tests/remote_settings/attachments/*.spm
|
||||
!tests/remote_settings/attachments/**/*.bin
|
||||
!tests/remote_settings/attachments/**/*.spm
|
||||
!tests/remote_settings/attachments/**/*.gz
|
||||
|
|
|
|||
|
|
@ -69,10 +69,7 @@ Models are deployed to Remote Settings to be delivered to Firefox.
|
|||
Records and attachments are uploaded via a CLI tool which lives in the
|
||||
`remote_settings` directory in this repository.
|
||||
|
||||
At present, records are uploaded by invoking the script manually, but we would
|
||||
like to automate this process whenever we merge a PR that adds new language models.
|
||||
|
||||
Run `poetry run python -m remote_settings --help` to see more information.
|
||||
View the `remote_settings` [README](https://github.com/mozilla/firefox-translations-models/blob/main/remote_settings/README.md) for more details on publishing models.
|
||||
|
||||
# Currently supported Languages
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,185 @@
|
|||
# Firefox Translations Models: remote_settings
|
||||
|
||||
A CLI tool to upload language models to [Remote Settings](https://remote-settings.readthedocs.io/).
|
||||
|
||||
## When to use remote_settings
|
||||
|
||||
The `remote_settings` CLI tool should be used when language models are added or
|
||||
updated within the repository to publish them to Remote Settings.
|
||||
|
||||
### Example: Publish a language pair to Remote Settings
|
||||
|
||||
**1) Export your authentication token (see [Authentication](#authentication))**
|
||||
> ```
|
||||
> export REMOTE_SETTINGS_BEARER_TOKEN="Bearer ..."`
|
||||
> ```
|
||||
|
||||
**2) Ensure your files are present locally, and not stored remotely via [git-lfs](https://git-lfs.com/)**
|
||||
> ```
|
||||
> git lfs fetch --all
|
||||
> git lfs pull {path_to_your_files}/*
|
||||
> ```
|
||||
|
||||
> [!NOTE]
|
||||
> Requires [installing](https://github.com/git-lfs/git-lfs#installing) git-lfs.
|
||||
|
||||
**3) Unzip the model files you wish to publish**
|
||||
> ```
|
||||
> gzip -d {path_to_your_files}/*
|
||||
> ```
|
||||
|
||||
> [!NOTE]
|
||||
> Requires [installing](https://www.gnu.org/software/gzip/) gzip.
|
||||
|
||||
**4) Inspect the metadata of your to-be-published records by using the --dry-run flag**
|
||||
> ```
|
||||
> poetry run python -m remote_settings create --dry-run --lang-pair {lang_pair} --server {dev,stage,prod} --version {version}
|
||||
> ```
|
||||
|
||||
**5) Publish the model records for a language pair by removing the --dry-run flag**
|
||||
> ```
|
||||
> poetry run python -m remote_settings create --lang-pair {lang_pair} --server {dev,stage,prod} --version {version}
|
||||
> ```
|
||||
|
||||
> [!NOTE]
|
||||
> At this time [versions](#arg---version) are manually entered by the CLI user.
|
||||
>
|
||||
> For example, if you are publishing records for a version `1.0a2`, and `1.0a1` already exists,
|
||||
> you will need to input the version `1.0a2` manually.
|
||||
>
|
||||
> In the future, we would like to support one or many of the following:
|
||||
>
|
||||
> 1) Bumping versions relatively via the CLI, e.g. `--version {bump-alpha,bump-beta,bump-minor,bump-major}`
|
||||
>
|
||||
> 2) Retrieving version data elsewhere, possibly from the model names, paths, or JSON data generated alongside the models.
|
||||
|
||||
## Creating model records and attachments
|
||||
|
||||
The `create` subcommand is used to upload a new model to the Remote Settings.
|
||||
|
||||
## Examples
|
||||
|
||||
Create a record and attachment for esen/vocab.esen.spm on the dev server with a version of 1.0,
|
||||
but mock the server connection to inspect the record data and configuration without uploading.
|
||||
```
|
||||
poetry run python -m remote_settings create --path models/prod/esen/vocab.esen.spm --server dev --version 1.0 --mock-connection
|
||||
```
|
||||
|
||||
Create a record and attachment for esen/vocab.esen.spm on the prod server with a version of 1.0a1,
|
||||
but only do a dry run to inspect the record data, configuration, and server authentication without uploading.
|
||||
```
|
||||
poetry run python -m remote_settings create --path models/prod/esen/vocab.esen.spm --server prod --version 1.0a1 --dry-run
|
||||
```
|
||||
|
||||
Create a record and attachment for esen/vocab.esen.spm on the stage server with a version of 2.1b2.
|
||||
```
|
||||
poetry run python -m remote_settings create --path models/prod/esen/vocab.esen.spm --server stage --version 2.1b2
|
||||
```
|
||||
|
||||
Create records and attachments for every model file in the esen directory with a version of 1.0 and upload them
|
||||
to the prod server
|
||||
```
|
||||
poetry run python -m remote_settings create --lang-pair esen --server prod --version 1.0
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
To authenticate your user with the server, you must export you token as an environment variable called `REMOTE_SETTINGS_BEARER_TOKEN`.
|
||||
|
||||
You can retrieve a bearer token from the Remote Settings admin dashboards.
|
||||
|
||||
* Dev: https://settings.dev.mozaws.net/v1/admin
|
||||
* Stage: https://remote-settings.allizom.org/v1/admin
|
||||
* Prod: https://remote-settings.mozilla.org/v1/admin
|
||||
|
||||
On the top right corner, use the 📋 icon to copy the authentication string
|
||||
|
||||
## Required arguments
|
||||
|
||||
There are three required arguments for the script to run:
|
||||
|
||||
* `--server` - The server to which the record and attachment will be uploaded.
|
||||
* `--version` - The semantic version of the record and the attachment.
|
||||
|
||||
The final argument is either of the following:
|
||||
|
||||
* `--path` - The path to a single file attachment to upload.
|
||||
* `--lang-pair` - The language pair for which to upload all file attachments.
|
||||
|
||||
### Arg: --server
|
||||
|
||||
Determines which Remote Settings Server will receive the uploaded record.
|
||||
|
||||
Models can be uploaded to one of three servers using the `--server` flag:
|
||||
|
||||
* `--server dev`
|
||||
* `--server stage`
|
||||
* `--server prod`
|
||||
|
||||
> [!NOTE]
|
||||
> Uploading to the `prod` or `stage` servers require VPN access, which can be acquired
|
||||
> by following the steps [here](https://mozilla-hub.atlassian.net/wiki/spaces/IT/pages/15761733/Mozilla+Corporate+VPN).
|
||||
|
||||
### Arg: --version
|
||||
|
||||
Applies a semantic version to the record.
|
||||
|
||||
Record versions follow typical semantic versioning used throughout Firefox,
|
||||
and records will be made available only in certain channels based on their version.
|
||||
|
||||
* Alpha-version records, e.g. `--version 1.0a1` will be available only in local builds and nightly builds.
|
||||
* Beta-version records, e.g. `--version 1.0b1` will be available in all builds except release builds.
|
||||
* Release-version records, e.g. `--version 1.0` will be available in all builds.
|
||||
|
||||
### Arg: --path
|
||||
|
||||
Uploads a single record and attachment located at the provided path.
|
||||
|
||||
Model attachment files are stored in the `models` directory at the root of the repository.
|
||||
|
||||
The `remote_settings` script derives metadata from the name of the file itself.
|
||||
|
||||
For example, the file named `trgvocab.esen.spm` will by of type `trgvocab` with a from-language of `es` and a to-language of `en`.
|
||||
|
||||
> [!NOTE]
|
||||
> Files are stored in compressed gzip archives. They must be decompressed before uploading.
|
||||
|
||||
### Arg: --lang-pair
|
||||
|
||||
Uploads all file attachments in the directory associated with the given language pair.
|
||||
|
||||
This argument will take a language pair, e.g. `"enes"` and upload all files in the relevant path.
|
||||
|
||||
The path itself is based on the provided `--version` argument.
|
||||
|
||||
* If the version is a release version, e.g. `1.0`, the script will search for the language pair in the `models/prod` directory.
|
||||
* If the version is a pre-release version, e.g. `1.0a1`, the script will search for the language pair in the `models/dev` directory.
|
||||
|
||||
> [!NOTE]
|
||||
> Files are stored in compressed gzip archives. They must be decompressed before uploading.
|
||||
|
||||
### Args: --mock-connection and --dry-run
|
||||
|
||||
Before uploading a model, it can be useful to test your configuration before committing.
|
||||
|
||||
The `remote_settings` script offers two levels of preparation:
|
||||
|
||||
* `-m | --mock-connection`
|
||||
* `-d | --dry-run`
|
||||
|
||||
The `--mock-connection` flag mocks the user authentication and connection to the server, allowing you to inspect
|
||||
the derived metadata of your record and attachment.
|
||||
|
||||
The `--dry-run` flag will attempt to connect and authenticate with the server as well as output the metadata,
|
||||
but it will _not_ upload the record and attachment.
|
||||
|
||||
Using these flags is recommended to ensure that your setup is correct before you upload attachments to Remote Settings.
|
||||
|
||||
|
||||
### Testing
|
||||
|
||||
Tests can be run via the following command:
|
||||
|
||||
```
|
||||
poetry run python -m pytest tests
|
||||
```
|
||||
|
|
@ -59,13 +59,13 @@ class RemoteSettingsClient:
|
|||
collection=COLLECTION,
|
||||
auth=BearerTokenAuth(self._auth_token),
|
||||
)
|
||||
self._new_record_info = None
|
||||
self._new_records = None
|
||||
|
||||
@classmethod
|
||||
def init_for_create(cls, args):
|
||||
"""Initializes the RemoteSettingsClient for the create subcommand
|
||||
This expects the CLI args to have information regarding creating a
|
||||
new record, which populates the _new_record_info data member.
|
||||
new record, which populates the _new_records data member.
|
||||
|
||||
Args:
|
||||
args (argparse.Namespace): The arguments passed through the CLI
|
||||
|
|
@ -74,27 +74,73 @@ class RemoteSettingsClient:
|
|||
RemoteSettingsClient: A RemoteSettingsClient that can create new records
|
||||
"""
|
||||
this = cls(args)
|
||||
name = os.path.basename(args.path)
|
||||
if args.path is not None:
|
||||
new_record_info = RemoteSettingsClient._create_record_info(args.path, args.version)
|
||||
this._new_records = [new_record_info]
|
||||
else:
|
||||
paths = this._paths_for_lang_pair(args)
|
||||
this._new_records = [
|
||||
RemoteSettingsClient._create_record_info(path, args.version) for path in paths
|
||||
]
|
||||
|
||||
return this
|
||||
|
||||
@staticmethod
|
||||
def _paths_for_lang_pair(args):
|
||||
"""Retrieves all of the file paths for the given language pair and version in args.
|
||||
|
||||
Args:
|
||||
args (argparse.Namespace): The arguments passed through the CLI
|
||||
|
||||
Returns:
|
||||
List[str]: A list of file paths in the specified language-pair directory.
|
||||
"""
|
||||
parsed_version = version.parse(args.version)
|
||||
|
||||
if parsed_version.is_prerelease:
|
||||
directory = os.path.join(RemoteSettingsClient._base_dir(args), "dev")
|
||||
else:
|
||||
directory = os.path.join(RemoteSettingsClient._base_dir(args), "prod")
|
||||
|
||||
full_path = os.path.join(directory, args.lang_pair)
|
||||
|
||||
if not os.path.exists(full_path):
|
||||
print_error(f"Path does not exist: {full_path}")
|
||||
exit(1)
|
||||
|
||||
return [os.path.join(full_path, f) for f in os.listdir(full_path) if not f.endswith(".gz")]
|
||||
|
||||
@staticmethod
|
||||
def _create_record_info(path, version):
|
||||
"""Creates a record-info dictionary for a file at the given path.
|
||||
|
||||
Args:
|
||||
path (str): The path to the file
|
||||
version (str): The version of the record attachment
|
||||
|
||||
Returns:
|
||||
dict: A dictionary containing the record metadata
|
||||
"""
|
||||
name = os.path.basename(path)
|
||||
file_type = RemoteSettingsClient._determine_file_type(name)
|
||||
from_lang, to_lang = RemoteSettingsClient._determine_language_pair(name)
|
||||
filter_expression = RemoteSettingsClient._determine_filter_expression(args.version)
|
||||
mimetype, _ = mimetypes.guess_type(args.path)
|
||||
this._new_record_info = {
|
||||
filter_expression = RemoteSettingsClient._determine_filter_expression(version)
|
||||
mimetype, _ = mimetypes.guess_type(path)
|
||||
return {
|
||||
"id": str(uuid.uuid4()),
|
||||
"data": {
|
||||
"name": os.path.basename(args.path),
|
||||
"name": os.path.basename(path),
|
||||
"fromLang": from_lang,
|
||||
"toLang": to_lang,
|
||||
"version": args.version,
|
||||
"version": version,
|
||||
"fileType": file_type,
|
||||
"filter_expression": filter_expression,
|
||||
},
|
||||
"attachment": {
|
||||
"path": args.path,
|
||||
"path": path,
|
||||
"mimeType": mimetype,
|
||||
},
|
||||
}
|
||||
return this
|
||||
|
||||
@staticmethod
|
||||
def _retrieve_remote_settings_bearer_token():
|
||||
|
|
@ -192,6 +238,21 @@ class RemoteSettingsClient:
|
|||
file_type_segment = segments[0]
|
||||
return file_type_segment
|
||||
|
||||
@staticmethod
|
||||
def _base_dir(args):
|
||||
"""Get the base directory in which to search for record attachments.
|
||||
|
||||
Args:
|
||||
args (argparse.Namespace): The arguments passed through the CLI
|
||||
|
||||
Returns:
|
||||
str: The base directory for record attachments.
|
||||
"""
|
||||
if args.test:
|
||||
return os.path.join("tests", "remote_settings", "attachments")
|
||||
else:
|
||||
return "models"
|
||||
|
||||
def server_url(self):
|
||||
"""Retrieves the url of the server that this client is connected to.
|
||||
|
||||
|
|
@ -208,65 +269,91 @@ class RemoteSettingsClient:
|
|||
"""
|
||||
return self._client.server_info()["user"]["id"]
|
||||
|
||||
def attachment_path(self):
|
||||
def attachment_path(self, index):
|
||||
"""Retrieves the path of the attachment that will be attached to a newly created record.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
|
||||
Returns:
|
||||
str: The attachment path
|
||||
"""
|
||||
return self._new_record_info["attachment"]["path"]
|
||||
return self._new_records[index]["attachment"]["path"]
|
||||
|
||||
def attachment_name(self):
|
||||
def attachment_name(self, index):
|
||||
"""Retrieves the name of the attachment that will be attached to a newly created record.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
|
||||
Returns:
|
||||
str: The attachment name
|
||||
"""
|
||||
return os.path.basename(self.attachment_path())
|
||||
return os.path.basename(self.attachment_path(index))
|
||||
|
||||
def attachment_mimetype(self):
|
||||
def attachment_mimetype(self, index):
|
||||
"""Retrieves the determined mimetype of the attachment that will be attached to a newly created record.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
|
||||
Returns:
|
||||
Union[None | str]: The determined mimetype
|
||||
"""
|
||||
return self._new_record_info["attachment"]["mimeType"]
|
||||
return self._new_records[index]["attachment"]["mimeType"]
|
||||
|
||||
def attachment_content(self):
|
||||
def attachment_content(self, index):
|
||||
"""Retrieves the file content of the attachment that will be attached to a newly created record.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
|
||||
Returns:
|
||||
bytes: The content of the attachment
|
||||
"""
|
||||
with open(self.attachment_path(), "rb") as f:
|
||||
with open(self.attachment_path(index), "rb") as f:
|
||||
attachment_content = f.read()
|
||||
return attachment_content
|
||||
|
||||
def record_info_json(self):
|
||||
def record_count(self):
|
||||
"""Returns the count of new records to be created"""
|
||||
return len(self._new_records)
|
||||
|
||||
def record_info_json(self, index):
|
||||
"""Returns the information of the record to be created as JSON data.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
|
||||
Returns:
|
||||
str: The JSON-formatted string containing the record info
|
||||
"""
|
||||
return json.dumps(self._new_record_info, indent=2)
|
||||
return json.dumps(self._new_records[index], indent=2)
|
||||
|
||||
def create_new_record(self):
|
||||
"""Creates a new record in the Remote Settings server along with its file attachment."""
|
||||
id = self._new_record_info["id"]
|
||||
data = self._new_record_info["data"]
|
||||
def create_new_record(self, index):
|
||||
"""Creates a new record in the Remote Settings server along with its file attachment.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
"""
|
||||
id = self._new_records[index]["id"]
|
||||
data = self._new_records[index]["data"]
|
||||
self._client.create_record(id=id, data=data)
|
||||
self.attach_file_to_record()
|
||||
self.attach_file_to_record(index)
|
||||
|
||||
def attach_file_to_record(self):
|
||||
def attach_file_to_record(self, index):
|
||||
"""Attaches the file attachment to the record of the matching id.
|
||||
|
||||
Args:
|
||||
index (int): The index of the record.
|
||||
|
||||
Raises:
|
||||
KintoException: An exception if the record was not able to be uploaded.
|
||||
"""
|
||||
headers = {"Authorization": f"Bearer {self._auth_token}"}
|
||||
|
||||
attachment_endpoint = "buckets/{}/collections/{}/records/{}/attachment".format(
|
||||
BUCKET, COLLECTION, self._new_record_info["id"]
|
||||
BUCKET, COLLECTION, self._new_records[index]["id"]
|
||||
)
|
||||
|
||||
response = requests.post(
|
||||
|
|
@ -275,9 +362,9 @@ class RemoteSettingsClient:
|
|||
(
|
||||
"attachment",
|
||||
(
|
||||
self.attachment_name(),
|
||||
self.attachment_content(),
|
||||
self.attachment_mimetype(),
|
||||
self.attachment_name(index),
|
||||
self.attachment_content(index),
|
||||
self.attachment_mimetype(index),
|
||||
),
|
||||
)
|
||||
],
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import argparse, os
|
|||
from packaging import version
|
||||
|
||||
from remote_settings.client import RemoteSettingsClient
|
||||
from remote_settings.format import print_info
|
||||
from remote_settings.format import print_info, print_error, print_help
|
||||
|
||||
|
||||
def attach_create_subcommand(subparsers):
|
||||
|
|
@ -44,18 +44,32 @@ def attach_create_subcommand(subparsers):
|
|||
help="the server where records will be created",
|
||||
required=True,
|
||||
)
|
||||
create_parser.add_argument(
|
||||
"--path",
|
||||
type=validate_path,
|
||||
help="the path to the file attachment to upload",
|
||||
required=True,
|
||||
)
|
||||
create_parser.add_argument(
|
||||
"--version",
|
||||
metavar="VERSION",
|
||||
type=validate_version,
|
||||
help="the semantic version of the record",
|
||||
required=True,
|
||||
)
|
||||
create_parser.add_argument(
|
||||
"--test",
|
||||
action="store_true",
|
||||
help=argparse.SUPPRESS,
|
||||
)
|
||||
|
||||
group = create_parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument(
|
||||
"--path",
|
||||
metavar="PATH",
|
||||
type=validate_path,
|
||||
help="the path to the file attachment to upload",
|
||||
)
|
||||
group.add_argument(
|
||||
"--lang-pair",
|
||||
metavar="PAIR",
|
||||
type=validate_lang_pair,
|
||||
help="the language pair for which to publish all associated files, e.g. 'enes'",
|
||||
)
|
||||
|
||||
|
||||
def validate_path(value):
|
||||
|
|
@ -64,6 +78,14 @@ def validate_path(value):
|
|||
return value
|
||||
|
||||
|
||||
def validate_lang_pair(value):
|
||||
if len(value) != 4:
|
||||
raise argparse.ArgumentTypeError(
|
||||
f"invalid language pair '{value}', expected only four letters, e.g. 'enes'"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_version(value):
|
||||
try:
|
||||
version.parse(value)
|
||||
|
|
@ -86,14 +108,22 @@ def do_create(args):
|
|||
print_info(args)
|
||||
print_info(args, f"User: {client.authenticated_user()}")
|
||||
print_info(args, f"Server: {client.server_url()}")
|
||||
print_info(args, f"Record: {client.record_info_json()}")
|
||||
print_info(args)
|
||||
|
||||
if args.dry_run or args.mock_connection:
|
||||
return
|
||||
|
||||
print_info(args, f"Creating record...")
|
||||
client.create_new_record()
|
||||
print_info(args, f"{client.attachment_name()} created")
|
||||
if client.record_count() == 0:
|
||||
print_error("No records found.")
|
||||
print_help("You may need to unzip the archives in the desired directory.")
|
||||
exit(1)
|
||||
|
||||
print_info(args)
|
||||
|
||||
for i in range(client.record_count()):
|
||||
print_info(args, f"Record: {client.record_info_json(i)}")
|
||||
|
||||
if not (args.dry_run or args.mock_connection):
|
||||
print_info(args, "Creating record...")
|
||||
client.create_new_record(i)
|
||||
print_info(args, f"{client.attachment_name(i)} created")
|
||||
|
||||
print_info(args)
|
||||
|
||||
print_info(args)
|
||||
|
|
|
|||
|
|
@ -2,8 +2,12 @@ import pytest
|
|||
import subprocess
|
||||
|
||||
SUCCESS = 0
|
||||
ERROR = 1
|
||||
INVALID_USE = 2
|
||||
|
||||
PROD_LANG_PAIR = "esen"
|
||||
DEV_LANG_PAIR = "enes"
|
||||
|
||||
LEX_TYPE = "lex"
|
||||
MODEL_TYPE = "model"
|
||||
QUALITY_MODEL_TYPE = "qualityModel"
|
||||
|
|
@ -19,14 +23,15 @@ SRCVOCAB_NAME = "srcvocab.esen.spm"
|
|||
TRGVOCAB_NAME = "trgvocab.esen.spm"
|
||||
VOCAB_NAME = "vocab.esen.spm"
|
||||
|
||||
ATTACHMENTS_PATH = "tests/remote_settings/attachments"
|
||||
LEX_PATH = f"{ATTACHMENTS_PATH}/{LEX_NAME}"
|
||||
LEX_5050_PATH = f"{ATTACHMENTS_PATH}/{LEX_5050_NAME}"
|
||||
MODEL_PATH = f"{ATTACHMENTS_PATH}/{MODEL_NAME}"
|
||||
QUALITY_MODEL_PATH = f"{ATTACHMENTS_PATH}/{QUALITY_MODEL_NAME}"
|
||||
SRCVOCAB_PATH = f"{ATTACHMENTS_PATH}/{SRCVOCAB_NAME}"
|
||||
TRGVOCAB_PATH = f"{ATTACHMENTS_PATH}/{TRGVOCAB_NAME}"
|
||||
VOCAB_PATH = f"{ATTACHMENTS_PATH}/{VOCAB_NAME}"
|
||||
DEV_ATTACHMENTS_PATH = "tests/remote_settings/attachments/dev/enes"
|
||||
PROD_ATTACHMENTS_PATH = "tests/remote_settings/attachments/prod/esen"
|
||||
LEX_PATH = f"{PROD_ATTACHMENTS_PATH}/{LEX_NAME}"
|
||||
LEX_5050_PATH = f"{PROD_ATTACHMENTS_PATH}/{LEX_5050_NAME}"
|
||||
MODEL_PATH = f"{PROD_ATTACHMENTS_PATH}/{MODEL_NAME}"
|
||||
QUALITY_MODEL_PATH = f"{PROD_ATTACHMENTS_PATH}/{QUALITY_MODEL_NAME}"
|
||||
SRCVOCAB_PATH = f"{PROD_ATTACHMENTS_PATH}/{SRCVOCAB_NAME}"
|
||||
TRGVOCAB_PATH = f"{PROD_ATTACHMENTS_PATH}/{TRGVOCAB_NAME}"
|
||||
VOCAB_PATH = f"{PROD_ATTACHMENTS_PATH}/{VOCAB_NAME}"
|
||||
|
||||
DEV_SERVER_URL = "https://remote-settings-dev.allizom.org/v1/"
|
||||
PROD_SERVER_URL = "https://remote-settings.mozilla.org/v1"
|
||||
|
|
@ -43,6 +48,7 @@ class CreateCommand:
|
|||
def __init__(self):
|
||||
self._server = None
|
||||
self._version = None
|
||||
self._lang_pair = None
|
||||
self._path = None
|
||||
self._quiet = None
|
||||
|
||||
|
|
@ -54,6 +60,10 @@ class CreateCommand:
|
|||
self._version = version
|
||||
return self
|
||||
|
||||
def with_lang_pair(self, lang_pair):
|
||||
self._lang_pair = lang_pair
|
||||
return self
|
||||
|
||||
def with_path(self, path):
|
||||
self._path = path
|
||||
return self
|
||||
|
|
@ -70,10 +80,12 @@ class CreateCommand:
|
|||
"-m",
|
||||
"remote_settings",
|
||||
"create",
|
||||
"--test",
|
||||
"--mock-connection",
|
||||
]
|
||||
command.extend(["--server", self._server] if self._server else [])
|
||||
command.extend(["--version", self._version] if self._version else [])
|
||||
command.extend(["--lang-pair", self._lang_pair] if self._lang_pair else [])
|
||||
command.extend(["--path", self._path] if self._path else [])
|
||||
command.extend(["--quiet"] if self._quiet else [])
|
||||
return subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
|
|
@ -102,11 +114,26 @@ def test_create_command_missing_version():
|
|||
assert "the following arguments are required: --version" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_missing_path():
|
||||
def test_create_command_missing_path_or_lang_pair():
|
||||
result = CreateCommand().with_server("dev").with_version("1.0").quiet().run()
|
||||
assert result.returncode == INVALID_USE, f"The return code should be {INVALID_USE}"
|
||||
assert "" == result.stdout, "The standard output stream should be empty"
|
||||
assert "the following arguments are required: --path" in result.stderr
|
||||
assert "one of the arguments --path --lang-pair is required" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_with_path_and_lang_pair():
|
||||
result = (
|
||||
CreateCommand()
|
||||
.with_server("dev")
|
||||
.with_path(MODEL_PATH)
|
||||
.with_lang_pair(PROD_LANG_PAIR)
|
||||
.with_version("1.0")
|
||||
.quiet()
|
||||
.run()
|
||||
)
|
||||
assert result.returncode == INVALID_USE, f"The return code should be {INVALID_USE}"
|
||||
assert "" == result.stdout, "The standard output stream should be empty"
|
||||
assert "argument --path: not allowed with argument --lang-pair" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_invalid_server():
|
||||
|
|
@ -157,6 +184,52 @@ def test_create_command_invalid_path():
|
|||
assert "argument --path: invalid value 'invalid_path' (path does not exist)" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_lang_pair_too_short():
|
||||
result = (
|
||||
CreateCommand().with_server("dev").with_version("1.0").with_lang_pair("ese").quiet().run()
|
||||
)
|
||||
assert result.returncode == INVALID_USE, f"The return code should be {INVALID_USE}"
|
||||
assert "" == result.stdout, "The standard output stream should be empty"
|
||||
assert "argument --lang-pair: invalid language pair 'ese'" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_lang_pair_too_long():
|
||||
result = (
|
||||
CreateCommand()
|
||||
.with_server("dev")
|
||||
.with_version("1.0")
|
||||
.with_lang_pair("esene")
|
||||
.quiet()
|
||||
.run()
|
||||
)
|
||||
assert result.returncode == INVALID_USE, f"The return code should be {INVALID_USE}"
|
||||
assert "" == result.stdout, "The standard output stream should be empty"
|
||||
assert "argument --lang-pair: invalid language pair 'esene'" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_lang_pair_does_not_exist_in_dev():
|
||||
result = (
|
||||
CreateCommand()
|
||||
.with_server("dev")
|
||||
.with_version("1.0a1")
|
||||
.with_lang_pair("esen")
|
||||
.quiet()
|
||||
.run()
|
||||
)
|
||||
assert result.returncode == ERROR, f"The return code should be {ERROR}"
|
||||
assert "" == result.stdout, "The standard output stream should be empty"
|
||||
assert "Path does not exist: tests/remote_settings/attachments/dev/esen" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_lang_pair_does_not_exist_in_prod():
|
||||
result = (
|
||||
CreateCommand().with_server("dev").with_version("1.0").with_lang_pair("enes").quiet().run()
|
||||
)
|
||||
assert result.returncode == ERROR, f"The return code should be {ERROR}"
|
||||
assert "" == result.stdout, "The standard output stream should be empty"
|
||||
assert "Path does not exist: tests/remote_settings/attachments/prod/enes" in result.stderr
|
||||
|
||||
|
||||
def test_create_command_display_authenticated_user():
|
||||
result = CreateCommand().with_server("dev").with_version("1.0").with_path(MODEL_PATH).run()
|
||||
assert result.returncode == SUCCESS, f"The return code should be {SUCCESS}"
|
||||
|
|
@ -302,15 +375,101 @@ def test_create_command_trgvocab_esen():
|
|||
assert f'"mimeType": null' in result.stdout
|
||||
|
||||
|
||||
def test_create_command_vocab_esen():
|
||||
result = CreateCommand().with_server("stage").with_version("1.0").with_path(VOCAB_PATH).run()
|
||||
LEX_PATH = f"{PROD_ATTACHMENTS_PATH}/{LEX_NAME}"
|
||||
LEX_5050_PATH = f"{PROD_ATTACHMENTS_PATH}/{LEX_5050_NAME}"
|
||||
MODEL_PATH = f"{PROD_ATTACHMENTS_PATH}/{MODEL_NAME}"
|
||||
QUALITY_MODEL_PATH = f"{PROD_ATTACHMENTS_PATH}/{QUALITY_MODEL_NAME}"
|
||||
SRCVOCAB_PATH = f"{PROD_ATTACHMENTS_PATH}/{SRCVOCAB_NAME}"
|
||||
TRGVOCAB_PATH = f"{PROD_ATTACHMENTS_PATH}/{TRGVOCAB_NAME}"
|
||||
VOCAB_PATH = f"{PROD_ATTACHMENTS_PATH}/{VOCAB_NAME}"
|
||||
|
||||
|
||||
def test_create_command_lang_pair_esen():
|
||||
result = CreateCommand().with_server("stage").with_version("1.0").with_lang_pair("esen").run()
|
||||
assert result.returncode == SUCCESS, f"The return code should be {SUCCESS}"
|
||||
assert "" == result.stderr, "The standard error stream should be empty"
|
||||
|
||||
assert f"{PROD_ATTACHMENTS_PATH}" in result.stdout
|
||||
assert f"{DEV_ATTACHMENTS_PATH}" not in result.stdout
|
||||
|
||||
assert f'"name": "{LEX_NAME}"' in result.stdout
|
||||
assert f'"name": "{LEX_5050_NAME}"' in result.stdout
|
||||
assert f'"name": "{MODEL_NAME}"' in result.stdout
|
||||
assert f'"name": "{QUALITY_MODEL_NAME}"' in result.stdout
|
||||
assert f'"name": "{SRCVOCAB_NAME}"' in result.stdout
|
||||
assert f'"name": "{TRGVOCAB_NAME}"' in result.stdout
|
||||
assert f'"name": "{VOCAB_NAME}"' in result.stdout
|
||||
|
||||
assert f'"fromLang": "es"' in result.stdout
|
||||
assert f'"fromLang": "en"' not in result.stdout
|
||||
|
||||
assert f'"toLang": "en"' in result.stdout
|
||||
assert f'"toLang": "es"' not in result.stdout
|
||||
|
||||
assert f'"version": "1.0"' in result.stdout
|
||||
assert f'"version": "1.0a1"' not in result.stdout
|
||||
|
||||
assert f'"fileType": "{LEX_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{MODEL_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{QUALITY_MODEL_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{SRCVOCAB_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{TRGVOCAB_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{VOCAB_TYPE}"' in result.stdout
|
||||
|
||||
assert f'"filter_expression": "{RELEASE_FILTER_EXPRESSION}"' in result.stdout
|
||||
assert f'"filter_expression": "{ALPHA_FILTER_EXPRESSION}"' not in result.stdout
|
||||
|
||||
assert f'"path": "{LEX_PATH}"' in result.stdout
|
||||
assert f'"path": "{LEX_5050_PATH}"' in result.stdout
|
||||
assert f'"path": "{MODEL_PATH}"' in result.stdout
|
||||
assert f'"path": "{QUALITY_MODEL_PATH}"' in result.stdout
|
||||
assert f'"path": "{SRCVOCAB_PATH}"' in result.stdout
|
||||
assert f'"path": "{TRGVOCAB_PATH}"' in result.stdout
|
||||
assert f'"path": "{VOCAB_PATH}"' in result.stdout
|
||||
assert f'"mimeType": null' in result.stdout
|
||||
|
||||
|
||||
def test_create_command_lang_pair_enes():
|
||||
result = (
|
||||
CreateCommand().with_server("stage").with_version("1.0a1").with_lang_pair("enes").run()
|
||||
)
|
||||
assert result.returncode == SUCCESS, f"The return code should be {SUCCESS}"
|
||||
assert "" == result.stderr, "The standard error stream should be empty"
|
||||
|
||||
assert f"{DEV_ATTACHMENTS_PATH}" in result.stdout
|
||||
assert f"{PROD_ATTACHMENTS_PATH}" not in result.stdout
|
||||
|
||||
assert f'"name": "{LEX_NAME}"' not in result.stdout
|
||||
assert f'"name": "{LEX_5050_NAME}"' not in result.stdout
|
||||
assert f'"name": "{MODEL_NAME}"' not in result.stdout
|
||||
assert f'"name": "{QUALITY_MODEL_NAME}"' not in result.stdout
|
||||
assert f'"name": "{SRCVOCAB_NAME}"' not in result.stdout
|
||||
assert f'"name": "{TRGVOCAB_NAME}"' not in result.stdout
|
||||
assert f'"name": "{VOCAB_NAME}"' not in result.stdout
|
||||
|
||||
assert f'"fromLang": "en"' in result.stdout
|
||||
assert f'"fromLang": "es"' not in result.stdout
|
||||
|
||||
assert f'"toLang": "es"' in result.stdout
|
||||
assert f'"toLang": "en"' not in result.stdout
|
||||
|
||||
assert f'"version": "1.0a1"' in result.stdout
|
||||
assert f'"version": "1.0"' not in result.stdout
|
||||
|
||||
assert f'"fileType": "{LEX_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{MODEL_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{QUALITY_MODEL_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{SRCVOCAB_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{TRGVOCAB_TYPE}"' in result.stdout
|
||||
assert f'"fileType": "{VOCAB_TYPE}"' in result.stdout
|
||||
|
||||
assert f'"filter_expression": "{ALPHA_FILTER_EXPRESSION}"' in result.stdout
|
||||
assert f'"filter_expression": "{RELEASE_FILTER_EXPRESSION}"' not in result.stdout
|
||||
|
||||
|
||||
def test_create_command_no_files_in_directory():
|
||||
result = (
|
||||
CreateCommand().with_server("stage").with_version("1.0a1").with_lang_pair("emty").run()
|
||||
)
|
||||
assert result.returncode == ERROR, f"The return code should be {ERROR}"
|
||||
assert "No records found" in result.stderr
|
||||
assert "You may need to unzip" in result.stdout
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче