From eb4a7cd1c64432c59d5b7d8bffdf346b3ba34f9d Mon Sep 17 00:00:00 2001 From: jeburke Date: Tue, 13 Apr 2021 18:19:05 -0700 Subject: [PATCH 1/6] add page for connected registry error code reference --- .../connected-registry-error-codes.md | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 docs/preview/connected-registry/connected-registry-error-codes.md diff --git a/docs/preview/connected-registry/connected-registry-error-codes.md b/docs/preview/connected-registry/connected-registry-error-codes.md new file mode 100644 index 0000000..9ed212a --- /dev/null +++ b/docs/preview/connected-registry/connected-registry-error-codes.md @@ -0,0 +1,36 @@ +--- +title: Connected registry error code reference +description: Details about error codes shown in the statusDetails property of a connected registry resource. For each error, possible solutions are listed. +ms.topic: article +ms.date: 04/13/2021 +ms.author: jeburke +author: jaysterp +--- + +# Connected registry error code reference + +This article helps you troubleshoot error codes you might encounter in the `StatusDetails` property of a connected registry. + +## Status Details Format + +When a connected registry has a connection state of `Unhealthy`, this indicates there is a critical error on the instance running on-premises. You may reference the `StatusDetails` property of the connected registry resource to view the corresponding error. + +Run the `az acr connected-registry show` command to view the statusDetails property for your connected registry. + +`StatusDetails` provides a list of errors, each with the following format: + +```json +{ + "code": "Error code", + "correlationId": "CorrelationId of the error on the on-premises connected registry instance", + "description": "Description corresponding to this error", + "timestamp": "Timestamp corresponding to this error", + "type": "Component of the connected registry instance corresponding to the error" +} +``` + +Every time the connected registry instance syncs with the cloud, these status details are updated. When the connected registry no longer has status details listed, it is considered healthy and its connection state is transitioned from `Unhealthy` to `Online`. + +## Error Code Reference + +This section lists the possible codes you may see in the `StatusDetails` property of a connected registry, which indicate critical errors. For each error, possible solutions are listed. From 04d27b41ce4802cbd629dc990cd98a76a6cb7ebf Mon Sep 17 00:00:00 2001 From: jeburke Date: Wed, 14 Apr 2021 13:24:12 -0700 Subject: [PATCH 2/6] add error codes --- .../connected-registry-error-codes.md | 58 +++++++++++++++++-- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/docs/preview/connected-registry/connected-registry-error-codes.md b/docs/preview/connected-registry/connected-registry-error-codes.md index 9ed212a..6deb59d 100644 --- a/docs/preview/connected-registry/connected-registry-error-codes.md +++ b/docs/preview/connected-registry/connected-registry-error-codes.md @@ -11,11 +11,13 @@ author: jaysterp This article helps you troubleshoot error codes you might encounter in the `StatusDetails` property of a connected registry. +## Connection State + +The connection state of a connected registry indicates the current overall health status of the on-premises instance. The connection state is `Online` when the instance is healthy, `Offline` when the instance is not connected to internet, and `Unhealthy` when there is a critical error on the instance while it is online. When the connected registry resource has a connection state of `Unhealthy`, you may reference the `StatusDetails` property to view the corresponding error. + ## Status Details Format -When a connected registry has a connection state of `Unhealthy`, this indicates there is a critical error on the instance running on-premises. You may reference the `StatusDetails` property of the connected registry resource to view the corresponding error. - -Run the `az acr connected-registry show` command to view the statusDetails property for your connected registry. +When your connected registry has a connection state of `Unhealthy` you can run the `az acr connected-registry show` command to view the statusDetails. `StatusDetails` provides a list of errors, each with the following format: @@ -29,8 +31,56 @@ Run the `az acr connected-registry show` command to view the statusDetails prope } ``` -Every time the connected registry instance syncs with the cloud, these status details are updated. When the connected registry no longer has status details listed, it is considered healthy and its connection state is transitioned from `Unhealthy` to `Online`. +Every time the connected registry instance syncs with the cloud, these status details are updated. When the connected registry no longer has status details listed, it is considered healthy and its connection state is transitioned from `Unhealthy` to `Online`. Once the connected registry is no longer connected to internet, its connection state will transition to `Offline`. ## Error Code Reference This section lists the possible codes you may see in the `StatusDetails` property of a connected registry, which indicate critical errors. For each error, possible solutions are listed. + +### DISK_PERMISSION_DENIED + +The connected registry was unable to write any file to the disk because it did not have sufficient permissions. + +*Potential solution:* Ensure that the host storage path used to run the connected registry container gives sufficient access to the container user. Update the permissions of the host system directory so that the user profile for your container has read, write, and execute access. By default, docker containers run as root. If the container is run as a non-root user, please ensure that user has the above permissions. + +### DISK_STORAGE_FULL + +The connected registry was unable to write files to the local host because there was no storage available on disk. + +*Potential solution:* Connected registry container logs are integrated with docker. By default, docker does not set container log size limits. Over time, these logs can take up much of your host's storage capacity. If your disk is out of space then you can place limits on the container logs retained by docker. + +#### Option 1: Place global log limits for all containers on the host + +Create or update the docker daemon file `/etc/docker/daemon.json` to add logging limits accross all containers on this host. The following example sets the log driver to `json-file` and sets `max-size` and `max-file` properties in order to enable automatic log rotation. If the configured threshold is reached, docker will remove the oldest log file first in order to make space for new logs. + +```json +{ + "log-driver": "json-file", + "log-opts": { + "max-size": "10m", + "max-file": "3" + } +} +``` + +Restart the container engine in order for the configuration to take effect. + +#### Option 2: Place log limits only for the connected registry container only + +You can also update the log level of the connected registry container only. Add the following flags to your `docker run` command: + +`--log-driver json-file --log-opt max-size=10m --log-opt max-file=3` + +Please reference how to set module-level log restrictions when running your connected registry on [IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge/production-checklist?view=iotedge-2020-11#option-adjust-log-settings-for-each-container-module). + +#### Update log verbosity on your connected registry + +After making the above docker configuration changes to free up disk space, you can also update the connected registry resource in order to limit logs sent to docker. By default, connected registries are created with `Information` log level. To minimize the verbosity of the logs stored, set the log level to `Warning`, `Error`, or `None`. Using the Az CLI, run + +`az acr connected-registry update -r MyRegistry -n MyConnectedRegistry --log-level Error` + +The configuration will take effect on-premises during the next scheduled sync with the cloud. + +### DISK_ERROR + +This is the default error code when the connected registry is unable create, write, or delete a file on the local disk. From 2400e30de3fe2733d0a7546f8ba11cf7746a214e Mon Sep 17 00:00:00 2001 From: jeburke Date: Mon, 19 Apr 2021 15:08:16 -0700 Subject: [PATCH 3/6] respond to PR comments --- .../connected-registry/connected-registry-error-codes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/preview/connected-registry/connected-registry-error-codes.md b/docs/preview/connected-registry/connected-registry-error-codes.md index 6deb59d..d92c42e 100644 --- a/docs/preview/connected-registry/connected-registry-error-codes.md +++ b/docs/preview/connected-registry/connected-registry-error-codes.md @@ -1,7 +1,7 @@ --- title: Connected registry error code reference description: Details about error codes shown in the statusDetails property of a connected registry resource. For each error, possible solutions are listed. -ms.topic: article +ms.topic: troubleshooting ms.date: 04/13/2021 ms.author: jeburke author: jaysterp @@ -65,7 +65,7 @@ Create or update the docker daemon file `/etc/docker/daemon.json` to add logging Restart the container engine in order for the configuration to take effect. -#### Option 2: Place log limits only for the connected registry container only +#### Option 2: Place log limits only for the connected registry container You can also update the log level of the connected registry container only. Add the following flags to your `docker run` command: @@ -83,4 +83,4 @@ The configuration will take effect on-premises during the next scheduled sync wi ### DISK_ERROR -This is the default error code when the connected registry is unable create, write, or delete a file on the local disk. +This is the default error code when the connected registry is unable to create, write, or delete a file on the local disk. From ffa4d2db98e5979444b42b88d1fef0334a6b3eb0 Mon Sep 17 00:00:00 2001 From: jeburke Date: Wed, 29 Sep 2021 11:18:41 -0700 Subject: [PATCH 4/6] update with current Disk error code --- .../connected-registry-error-codes.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/preview/connected-registry/connected-registry-error-codes.md b/docs/preview/connected-registry/connected-registry-error-codes.md index d92c42e..518b759 100644 --- a/docs/preview/connected-registry/connected-registry-error-codes.md +++ b/docs/preview/connected-registry/connected-registry-error-codes.md @@ -2,7 +2,7 @@ title: Connected registry error code reference description: Details about error codes shown in the statusDetails property of a connected registry resource. For each error, possible solutions are listed. ms.topic: troubleshooting -ms.date: 04/13/2021 +ms.date: 09/29/2021 ms.author: jeburke author: jaysterp --- @@ -37,21 +37,25 @@ Every time the connected registry instance syncs with the cloud, these status de This section lists the possible codes you may see in the `StatusDetails` property of a connected registry, which indicate critical errors. For each error, possible solutions are listed. -### DISK_PERMISSION_DENIED +### DiskError -The connected registry was unable to write any file to the disk because it did not have sufficient permissions. +This is the default error code when the connected registry is unable to create, write, or delete a file on the local disk. There are a few scenarios that may cause a `DiskError` code. Please reference below for possible scenarios and remediations. -*Potential solution:* Ensure that the host storage path used to run the connected registry container gives sufficient access to the container user. Update the permissions of the host system directory so that the user profile for your container has read, write, and execute access. By default, docker containers run as root. If the container is run as a non-root user, please ensure that user has the above permissions. +1. The connected registry was unable to write any file to the disk because it did not have sufficient permissions. -### DISK_STORAGE_FULL +Sample status description: "Access to the path '/var/acr/data/registry/dummy.txt' is denied." -The connected registry was unable to write files to the local host because there was no storage available on disk. +*Potential solution:* Ensure that the host storage path used to run the connected registry container gives sufficient access to the container user. In the sample above, this path is `/var/acr/data/registry`. Update the permissions of the host system directory so that the user profile for your container has read, write, and execute access. By default, docker containers run as root. If the container is run as a non-root user, please ensure that user has the above permissions. + +2. The connected registry was unable to write files to the local host because there was no storage available on disk. + +Sample status description: "There is not enough space on the disk." *Potential solution:* Connected registry container logs are integrated with docker. By default, docker does not set container log size limits. Over time, these logs can take up much of your host's storage capacity. If your disk is out of space then you can place limits on the container logs retained by docker. #### Option 1: Place global log limits for all containers on the host -Create or update the docker daemon file `/etc/docker/daemon.json` to add logging limits accross all containers on this host. The following example sets the log driver to `json-file` and sets `max-size` and `max-file` properties in order to enable automatic log rotation. If the configured threshold is reached, docker will remove the oldest log file first in order to make space for new logs. +If running the connected registry on docker, create or update the docker daemon file `/etc/docker/daemon.json` to add logging limits accross all containers on this host. The following example sets the log driver to `json-file` and sets `max-size` and `max-file` properties in order to enable automatic log rotation. If the configured threshold is reached, docker will remove the oldest log file first in order to make space for new logs. ```json { @@ -73,14 +77,10 @@ You can also update the log level of the connected registry container only. Add Please reference how to set module-level log restrictions when running your connected registry on [IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge/production-checklist?view=iotedge-2020-11#option-adjust-log-settings-for-each-container-module). -#### Update log verbosity on your connected registry +#### Option 3: Update log verbosity on your connected registry After making the above docker configuration changes to free up disk space, you can also update the connected registry resource in order to limit logs sent to docker. By default, connected registries are created with `Information` log level. To minimize the verbosity of the logs stored, set the log level to `Warning`, `Error`, or `None`. Using the Az CLI, run `az acr connected-registry update -r MyRegistry -n MyConnectedRegistry --log-level Error` The configuration will take effect on-premises during the next scheduled sync with the cloud. - -### DISK_ERROR - -This is the default error code when the connected registry is unable to create, write, or delete a file on the local disk. From 7b5315590fff06ec3d1c2e7f30727e5b9b58970e Mon Sep 17 00:00:00 2001 From: jeburke Date: Wed, 29 Sep 2021 14:25:24 -0700 Subject: [PATCH 5/6] respond to PR comments' --- .../connected-registry-error-codes.md | 94 ++++++++++++++++--- 1 file changed, 82 insertions(+), 12 deletions(-) diff --git a/docs/preview/connected-registry/connected-registry-error-codes.md b/docs/preview/connected-registry/connected-registry-error-codes.md index 518b759..0bcb8af 100644 --- a/docs/preview/connected-registry/connected-registry-error-codes.md +++ b/docs/preview/connected-registry/connected-registry-error-codes.md @@ -7,7 +7,7 @@ ms.author: jeburke author: jaysterp --- -# Connected registry error code reference +# Connected Registry Error Code Reference This article helps you troubleshoot error codes you might encounter in the `StatusDetails` property of a connected registry. @@ -15,11 +15,35 @@ This article helps you troubleshoot error codes you might encounter in the `Stat The connection state of a connected registry indicates the current overall health status of the on-premises instance. The connection state is `Online` when the instance is healthy, `Offline` when the instance is not connected to internet, and `Unhealthy` when there is a critical error on the instance while it is online. When the connected registry resource has a connection state of `Unhealthy`, you may reference the `StatusDetails` property to view the corresponding error. +Use the [az acr connected-registry show][az-acr-connected-registry-show] command to view the current connection state of your connected registry. + +```azurecli +az acr connected-registry show \ + --registry MyRegistry \ + --name MyConnectedRegistry \ + --output table +``` + +You should see a response as follows. Note that this connected registry has a connection state of `Unhealthy`. + +``` +NAME MODE CONNECTION STATE PARENT LOGIN SERVER LAST SYNC (UTC) SYNC SCHEDULE SYNC WINDOW +--------------------- -------- ------------------ -------- -------------- ------------------------- --------------- ------------- +MyConnectedRegistry ReadOnly Unhealthy 2021-09-29T12:59:00+00:00 * * * * * +``` + ## Status Details Format -When your connected registry has a connection state of `Unhealthy` you can run the `az acr connected-registry show` command to view the statusDetails. +When your connected registry has a connection state of `Unhealthy` you can run the [az acr connected-registry show][az-acr-connected-registry-show] command to view the list of status details. -`StatusDetails` provides a list of errors, each with the following format: +```azurecli +az acr connected-registry show \ + --registry MyRegistry \ + --name MyConnectedRegistry + --query statusDetails +``` + +The `StatusDetails` property provides a list of error objects, each with the following format: ```json { @@ -33,25 +57,60 @@ When your connected registry has a connection state of `Unhealthy` you can run t Every time the connected registry instance syncs with the cloud, these status details are updated. When the connected registry no longer has status details listed, it is considered healthy and its connection state is transitioned from `Unhealthy` to `Online`. Once the connected registry is no longer connected to internet, its connection state will transition to `Offline`. -## Error Code Reference +# Error Codes -This section lists the possible codes you may see in the `StatusDetails` property of a connected registry, which indicate critical errors. For each error, possible solutions are listed. +This section lists the possible codes you may see in the `StatusDetails` property of a connected registry, which indicate critical errors. For each error, possible solutions are listed. You can view the status details of your connected registry by running the [az acr connected-registry show][az-acr-connected-registry-show] command. -### DiskError +```azurecli +az acr connected-registry show \ + --registry MyRegistry \ + --name MyConnectedRegistry + --query statusDetails +``` + +## DiskError This is the default error code when the connected registry is unable to create, write, or delete a file on the local disk. There are a few scenarios that may cause a `DiskError` code. Please reference below for possible scenarios and remediations. -1. The connected registry was unable to write any file to the disk because it did not have sufficient permissions. +### Insufficient Permissions -Sample status description: "Access to the path '/var/acr/data/registry/dummy.txt' is denied." +Sample status detail: + +```json +[ + { + "code": "DiskError", + "correlationId": "73a46395-b89b-49c7-5621-d54e8b1574b5", + "description": "Access to the path '/var/acr/data/registry/dummy.txt' is denied.", + "timestamp": "2021-09-16T01:17:45.394512+00:00", + "type": "Disk" + } +] +``` + +This status `description` indicates that the connected registry was unable to write any file to the disk because it did not have sufficient permissions. *Potential solution:* Ensure that the host storage path used to run the connected registry container gives sufficient access to the container user. In the sample above, this path is `/var/acr/data/registry`. Update the permissions of the host system directory so that the user profile for your container has read, write, and execute access. By default, docker containers run as root. If the container is run as a non-root user, please ensure that user has the above permissions. -2. The connected registry was unable to write files to the local host because there was no storage available on disk. +### No Storage Available On Disk -Sample status description: "There is not enough space on the disk." +Sample status detail: -*Potential solution:* Connected registry container logs are integrated with docker. By default, docker does not set container log size limits. Over time, these logs can take up much of your host's storage capacity. If your disk is out of space then you can place limits on the container logs retained by docker. +```json +[ + { + "code": "DiskError", + "correlationId": "73a46395-b89b-49c7-5621-d54e8b1574b5", + "description": "There is not enough space on the disk.", + "timestamp": "2021-09-16T01:17:45.394512+00:00", + "type": "Disk" + } +] +``` + +This status `description` indicates that the connected registry was unable to write any file to the disk because it did not have sufficient permissions. + +*Potential solution:* Connected registry container logs are integrated with docker. By default, docker does not set container log size limits. Over time, these logs can take up much of your host's storage capacity. If your disk is out of space then you can place limits on the container logs retained by docker. See the following options for limiting storage space used by connected registry logs. #### Option 1: Place global log limits for all containers on the host @@ -79,8 +138,19 @@ Please reference how to set module-level log restrictions when running your conn #### Option 3: Update log verbosity on your connected registry -After making the above docker configuration changes to free up disk space, you can also update the connected registry resource in order to limit logs sent to docker. By default, connected registries are created with `Information` log level. To minimize the verbosity of the logs stored, set the log level to `Warning`, `Error`, or `None`. Using the Az CLI, run +After making the above docker configuration changes to free up disk space, you can also update the connected registry resource in order to limit logs sent to docker. By default, connected registries are created with `Information` log level. To minimize the verbosity of the logs stored, set the log level to `Warning`, `Error`, or `None`. Use the connected registry [az acr connected-registry update][az-acr-connected-registry-update] command to update the log level. `az acr connected-registry update -r MyRegistry -n MyConnectedRegistry --log-level Error` +```azurecli +az acr connected-registry update \ + --registry MyRegistry \ + --name MyConnectedRegistry \ + --log-level Error +``` + The configuration will take effect on-premises during the next scheduled sync with the cloud. + + +[az-acr-connected-registry-show]: https://docs.microsoft.com/cli/azure/acr/connected-registry?view=azure-cli-latest#az_acr_connected_registry_show +[az-acr-connected-registry-update]: https://docs.microsoft.com/cli/azure/acr/connected-registry?view=azure-cli-latest#az_acr_connected_registry_update From 6b729d20923e2cc26f95c1e9f0f73f77a692d7a9 Mon Sep 17 00:00:00 2001 From: jeburke Date: Mon, 4 Oct 2021 11:27:33 -0700 Subject: [PATCH 6/6] define connection states in a table --- .../connected-registry/connected-registry-error-codes.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/preview/connected-registry/connected-registry-error-codes.md b/docs/preview/connected-registry/connected-registry-error-codes.md index 0bcb8af..fe5225d 100644 --- a/docs/preview/connected-registry/connected-registry-error-codes.md +++ b/docs/preview/connected-registry/connected-registry-error-codes.md @@ -13,7 +13,13 @@ This article helps you troubleshoot error codes you might encounter in the `Stat ## Connection State -The connection state of a connected registry indicates the current overall health status of the on-premises instance. The connection state is `Online` when the instance is healthy, `Offline` when the instance is not connected to internet, and `Unhealthy` when there is a critical error on the instance while it is online. When the connected registry resource has a connection state of `Unhealthy`, you may reference the `StatusDetails` property to view the corresponding error. +The connection state of a connected registry indicates the current overall health status of the deployed connected registry instance. The possible connection states are defined as follows: + +| Connection State | Description | +|--------------|-----------| +| Online | The connected registry instance is currently connected with the cloud and in a healthy state. | +| Offline | The connected registry instance is currently disconnected from the cloud. | +| Unhealthy | The connected registry instance is currently connected with the cloud, but it is reporting critical errors. Reference the `StatusDetails` property to view the corresponding errors. | Use the [az acr connected-registry show][az-acr-connected-registry-show] command to view the current connection state of your connected registry.