GitHub Enterprise Server 3.9 general availability (#38599)

Co-authored-by: Laura Coursen <lecoursen@github.com>
Co-authored-by: Siara <108543037+SiaraMist@users.noreply.github.com>

content/admin/configuration/configuring-your-enterprise/configuring-backups-on-your-instance.md

@@ -1,5 +1,5 @@
 ---
-title: Configuring backups on your appliance
+title: Configuring backups on your instance
 shortTitle: Configuring backups
 redirect_from:
   - /enterprise/admin/categories/backups-and-restores
@@ -14,6 +14,7 @@ redirect_from:
   - /enterprise/admin/installation/configuring-backups-on-your-appliance
   - /enterprise/admin/configuration/configuring-backups-on-your-appliance
   - /admin/configuration/configuring-backups-on-your-appliance
+  - /admin/configuration/configuring-your-enterprise/configuring-backups-on-your-appliance
 intro: 'As part of a disaster recovery plan, you can protect production data on {% data variables.location.product_location %} by configuring automated backups.'
 versions:
   ghes: '*'
@@ -94,7 +95,7 @@ Backup snapshots are written to the disk path set by the `GHE_DATA_DIR` data dir
 
      **Note:** If {% data variables.location.product_location %} is deployed as a cluster or in a high availability configuration using a load balancer, the `GHE_HOSTNAME` can be the load balancer hostname, as long as it allows SSH access (on port 122) to {% data variables.location.product_location %}.
 
-     To ensure a recovered appliance is immediately available, perform backups targeting the primary instance even in a geo-replication configuration.
+     To ensure a recovered instance is immediately available, perform backups targeting the primary instance even in a geo-replication configuration.
 
      {% endnote %}
    1. Set the `GHE_DATA_DIR` value to the filesystem location where you want to store backup snapshots. We recommend choosing a location on the same filesystem as your backup host, but outside of where you cloned the Git repository in step 1.
@@ -203,34 +204,50 @@ To use Git instead of a compressed archive for upgrades, you must back up your e
 
 ## Scheduling a backup
 
+{% ifversion backup-utilities-encryption-bug %}
+{% warning %}
+
+**Warning**: {% data reusables.enterprise_backup_utilities.enterprise-backup-utils-encryption-keys %}
+
+{% endwarning %}
+{% endif %}
+
 You can schedule regular backups on the backup host using the `cron(8)` command or a similar command scheduling service. The configured backup frequency will dictate the worst case recovery point objective (RPO) in your recovery plan. For example, if you have scheduled the backup to run every day at midnight, you could lose up to 24 hours of data in a disaster scenario. We recommend starting with an hourly backup schedule, guaranteeing a worst case maximum of one hour of data loss if the primary site data is destroyed.
 
 If backup attempts overlap, the `ghe-backup` command will abort with an error message, indicating the existence of a simultaneous backup. If this occurs, we recommended decreasing the frequency of your scheduled backups. For more information, see the "Scheduling backups" section of the [{% data variables.product.prodname_enterprise_backup_utilities %} README](https://github.com/github/backup-utils#scheduling-backups) in the {% data variables.product.prodname_enterprise_backup_utilities %} project documentation.
 
 ## Restoring a backup
 
-In the event of prolonged outage or catastrophic event at the primary site, you can restore {% data variables.location.product_location %} by provisioning another {% data variables.product.prodname_enterprise %} appliance and performing a restore from the backup host. You must add the backup host's SSH key to the target {% data variables.product.prodname_enterprise %} appliance as an authorized SSH key before restoring an appliance.
+{% ifversion backup-utilities-encryption-bug %}
 
-{% note %}
+{% warning %}
 
-**Note:** When performing backup restores to {% data variables.location.product_location %}, the same version supportability rules apply. You can only restore data from at most two feature releases behind.
+**Warning**: {% data reusables.enterprise_backup_utilities.enterprise-backup-utils-encryption-keys %}
 
-For example, if you take a backup from {% data variables.product.product_name %} 3.0.x, you can restore the backup to a {% data variables.product.product_name %} 3.2.x instance. You cannot restore data from a backup of {% data variables.product.product_name %} 2.22.x to an instance running 3.2.x, because that would be three jumps between versions (2.22 to 3.0 to 3.1 to 3.2). You would first need to restore to an instance running 3.1.x, and then upgrade to 3.2.x.
+{% endwarning %}
 
-{% endnote %}
+{% endif %}
 
-To restore {% data variables.location.product_location %} from the last successful snapshot, use the `ghe-restore` command.
+In the event of prolonged outage or catastrophic event at the primary site, you can restore {% data variables.location.product_location %} by provisioning another instance and performing a restore from the backup host. You must add the backup host's SSH key to the target {% data variables.product.prodname_enterprise %} instance as an authorized SSH key before restoring an instance.
 
-{% note %}
+When performing backup restores to {% data variables.location.product_location %}, you can only restore data from at most two feature releases behind. For example, if you take a backup from {% data variables.product.product_name %} 3.0.x, you can restore the backup to an instance running {% data variables.product.product_name %} 3.2.x. You cannot restore data from a backup of {% data variables.product.product_name %} 2.22.x to an instance running 3.2.x, because that would be three jumps between versions (2.22 to 3.0 to 3.1 to 3.2). You would first need to restore to an instance running 3.1.x, and then upgrade to 3.2.x.
 
-**Note:** Prior to restoring a backup, ensure:
-- Maintenance mode is enabled on the primary instance and all active processes have completed. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/enabling-and-scheduling-maintenance-mode)."
-- Replication is stopped on all replicas in high availability configurations. For more information, see the `ghe-repl-stop` command in "[AUTOTITLE](/admin/enterprise-management/configuring-high-availability/about-high-availability-configuration#ghe-repl-stop)."
-- If {% data variables.location.product_location %} has {% data variables.product.prodname_actions %} enabled, you must first configure the {% data variables.product.prodname_actions %} external storage provider on the replacement appliance. For more information, see "[AUTOTITLE](/admin/github-actions/advanced-configuration-and-troubleshooting/backing-up-and-restoring-github-enterprise-server-with-github-actions-enabled)."
+Network settings are excluded from the backup snapshot. After restoration, you must manually configure networking on the target {% data variables.product.prodname_ghe_server %} instance.
 
-{% endnote %}
+### Prerequisites
 
-When running the `ghe-restore` command, you should see output similar to this:
+1. Ensure maintenance mode is enabled on the primary instance and all active processes have completed. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/enabling-and-scheduling-maintenance-mode)."
+1. Stop replication on all replica nodes in a high-availability configuration. For more information, see "[AUTOTITLE](/admin/enterprise-management/configuring-high-availability/about-high-availability-configuration#ghe-repl-stop)."
+1. If {% data variables.location.product_location %} has {% data variables.product.prodname_actions %} enabled, you must configure the external storage provider for {% data variables.product.prodname_actions %} on the replacement instance. For more information, see "[AUTOTITLE](/admin/github-actions/advanced-configuration-and-troubleshooting/backing-up-and-restoring-github-enterprise-server-with-github-actions-enabled)."
+
+### Starting the restore operation
+
+To restore {% data variables.location.product_location %} from your backup host using the last successful snapshot, use the `ghe-restore` command. You can use the following additional options with `ghe-restore`.
+
+- The `-c` flag overwrites the settings, certificate, and license data on the target host even if it is already configured. Omit this flag if you are setting up a staging instance for testing purposes and you wish to retain the existing configuration on the target. For more information, see the "Using backup and restore commands" section of the [{% data variables.product.prodname_enterprise_backup_utilities %} README](https://github.com/github/backup-utils#using-the-backup-and-restore-commands) in the github/backup-utils repository.
+- The `-s` flag allows you to select a different backup snapshot.
+
+After you run `ghe-restore`, the command confirms the restoration, then outputs details and status during the operation.
 
 ```shell
 $ ghe-restore -c 169.154.1.1
@@ -253,16 +270,27 @@ $ ghe-restore -c 169.154.1.1
 Optionally, to validate the restore, configure an IP exception list to allow access to a specified list of IP addresses. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/enabling-and-scheduling-maintenance-mode#validating-changes-in-maintenance-mode-using-the-ip-exception-list)."
 {% endif %}
 
-{% note %}
+On an instance in a high-availability configuration, after you restore to new disks on an existing or empty instance, `ghe-repl-status` may report that Git or Alambic replication is out of sync due to stale server UUIDs. These stale UUIDs can be the result of a retired node in a high-availability configuration still being present in the application database, but not in the restored replication configuration. 
 
-**Note:**
+To remediate after the restoration completes and before starting replication, you can tear down stale UUIDs using `ghe-repl-teardown`. If you need further assistance, contact {% data variables.contact.contact_ent_support %}.
 
-- The network settings are excluded from the backup snapshot. You must manually configure the network on the target {% data variables.product.prodname_ghe_server %} appliance as required for your environment.
+{% ifversion backup-utilities-progress %}
+## Monitoring backup or restoration progress
 
-- When restoring to new disks on an existing or empty {% data variables.product.prodname_ghe_server %} instance, stale UUIDs may be present, resulting in Git and/or Alambic replication reporting as out of sync. Stale server entry IDs can be the result of a retired node in a high availability configuration still being present in the application database, but not in the restored replication configuration. To remediate, stale UUIDs can be torn down using `ghe-repl-teardown` once the restore has completed and prior to starting replication. In this scenario, contact {% data variables.contact.contact_ent_support %} for further assistance.
+During a backup or restoration operation, you can use the `ghe-backup-progress` utility on your backup host to monitor the operation's progress. The utility prints the progress of each job sequentially.
 
-{% endnote %}
+To monitor progress on the backup host, from the directory containing {% data variables.product.prodname_enterprise_backup_utilities %}, run the following command.
 
-You can use these additional options with `ghe-restore` command:
-- The `-c` flag overwrites the settings, certificate, and license data on the target host even if it is already configured. Omit this flag if you are setting up a staging instance for testing purposes and you wish to retain the existing configuration on the target. For more information, see the "Using backup and restore commands" section of the [{% data variables.product.prodname_enterprise_backup_utilities %} README](https://github.com/github/backup-utils#using-the-backup-and-restore-commands) in the {% data variables.product.prodname_enterprise_backup_utilities %} project documentation.
-- The `-s` flag allows you to select a different backup snapshot.
+```shell copy
+bin/ghe-backup-progress
+```
+
+By default, the utility prints progress continuously until the operation is complete. You can press any key to return to the prompt.
+
+Optionally, you can run the following command to print the current progress, the last completed job, and then immediately exit.
+
+```shell copy
+bin/ghe-backup-progress --once
+```
+
+{% endif %}

content/admin/configuration/configuring-your-enterprise/index.md

@@ -20,7 +20,8 @@ children:
   - /initializing-github-ae
   - /accessing-the-administrative-shell-ssh
   - /enabling-and-scheduling-maintenance-mode
-  - /configuring-backups-on-your-appliance
+  - /configuring-backups-on-your-instance
+  - /known-issues-with-backups-for-your-instance
   - /site-admin-dashboard
   - /enabling-private-mode
   - /managing-github-mobile-for-your-enterprise

content/admin/configuration/configuring-your-enterprise/known-issues-with-backups-for-your-instance.md

@@ -0,0 +1,122 @@
+---
+title: Known issues with backups for your instance
+intro: 'See an overview of workarounds for issues that impact the backup or restoration process for {% data variables.product.prodname_ghe_server %}.'
+versions:
+  feature: backup-utilities-encryption-bug
+type: overview
+topics:
+  - Enterprise
+  - Troubleshooting
+  - Backups
+shortTitle: Known issues with backups
+---
+
+## About known issues with {% data variables.product.prodname_ghe_server %} backups
+
+{% data variables.product.company_short %} provides workarounds for the following issues that could impact backup or restoration of data for a {% data variables.product.prodname_ghe_server %} instance. For more information, see "Known issues" in the [{% data variables.product.prodname_ghe_server %} release notes](/admin/release-notes).
+
+{% ifversion backup-utilities-encryption-bug %}
+
+## Users cannot sign in after restoration of a backup
+
+If you used {% data variables.product.prodname_enterprise_backup_utilities %} {% ifversion ghes = 3.7 %}3.7.0{% elsif ghes = 3.8 or ghes = 3.9 %}3.7.0 or 3.8.0{% endif %} to back up an instance running any release in the {% data variables.product.product_name %} 3.7{% ifversion ghes = 3.8 or ghes = 3.9 %} or 3.8{% endif %} series, after you restore the backup to a new instance, users cannot sign in. Though users cannot sign in, the backup itself is unaffected and all data is intact.
+
+To address this issue, upgrade {% data variables.product.prodname_enterprise_backup_utilities %} on your backup host, then create a new backup. Alternatively, you can modify the configuration on a new target instance to restore an existing backup affected by this issue.
+
+- [Upgrading {% data variables.product.prodname_enterprise_backup_utilities %}](#upgrading-github-enterprise-server-backup-utilities)
+- [Restoring from an existing backup](#restoring-from-an-existing-backup)
+
+### Upgrading {% data variables.product.prodname_enterprise_backup_utilities %}
+
+To create a new backup, upgrade {% data variables.product.prodname_enterprise_backup_utilities %} on your backup host to version {% ifversion ghes = 3.7 %}3.7.1{% elsif ghes = 3.8 or ghes = 3.9 %}3.7.1 or 3.8.1{% endif %}, then use the `ghe-backup` utility to back up your instance running {% data variables.product.product_name %} {% ifversion ghes = 3.7 %}3.7{% elsif ghes = 3.8 or ghes = 3.9 %}3.7 or 3.8{% endif %}. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/configuring-backups-on-your-appliance)" and [{% ifversion ghes = 3.7 %}the 3.7.1 release{% elsif ghes = 3.8 or ghes = 3.9 %}releases{% endif %}](https://github.com/github/backup-utils/releases{% ifversion ghes = 3.7 %}/tag/v3.7.1{% endif %}) in the github/backup-utils repository.
+
+{% ifversion ghes = 3.9 %}
+
+If your instance is already running {% data variables.product.product_name %} 3.9 and users can sign in, you can upgrade to {% data variables.product.prodname_enterprise_backup_utilities %} 3.9.0 on your backup host and continue backing up normally. For more information, see the [3.9.0 release](https://github.com/github/backup-utils/releases/tag/v3.9.0) in the `github/backup-utils` repository.
+
+{% endif %}
+
+### Restoring from an existing backup
+
+If you've restored an existing backup from {% data variables.product.prodname_enterprise_backup_utilities %} {% ifversion ghes = 3.7 %}3.7.0{% elsif ghes = 3.8 or ghes = 3.9 %}3.8.0{% endif %} to a new instance and users cannot sign in, you must output configuration data from the source {% data variables.product.product_name %} instance and adjust the configuration on the target instance.
+
+To ensure users can sign into the new target instance, ensure that your environment meets the following requirements.
+
+- The source {% data variables.product.product_name %} instance must be running and accessible via SSH.
+- You must have an existing backup from {% data variables.product.prodname_enterprise_backup_utilities %} {% ifversion ghes = 3.7 %}3.7.0{% elsif ghes = 3.8 or ghes = 3.9 %}3.8.0{% endif %}.
+- You must have provisioned a new target {% data variables.product.product_name %} instance and restored the backup. For more information, see "[AUTOTITLE](/admin/installation/setting-up-a-github-enterprise-server-instance)" and "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/configuring-backups-on-your-instance)."
+
+1. SSH into the source {% data variables.product.product_name %} instance that you backed up. If your instance comprises multiple nodes, for example if high availability or geo-replication are configured, SSH into the primary node. If you use a cluster, you can SSH into any node. Replace HOSTNAME with the actual hostname of your instance. For more information about SSH access, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/accessing-the-administrative-shell-ssh)."
+
+   ```shell copy
+   ssh -p 122 admin@HOSTNAME
+   ```
+{%- ifversion ghes = 3.7 %}
+1. To display a list of encryption and decryption keys, run the following command.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-keying-material
+   ```
+{%- elsif ghes = 3.8 or ghes = 3.9 %}
+1. To display a list of decryption keys, run the following command.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-keying-material
+   ```
+1. Copy the output to a safe and temporary location.
+1. To display a list of encryption keys, run the following command.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-current-encryption-key
+   ```
+1. Copy the output to a safe and temporary location.
+{%- endif %}
+1. SSH into the destination {% data variables.product.product_name %} instance where you restored the backup. Replace HOSTNAME with the actual hostname of your instance.
+
+   ```shell copy
+   ssh -p 122 admin@HOSTNAME
+   ```
+1. Enable maintenance mode. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/enabling-and-scheduling-maintenance-mode#enabling-maintenance-mode-immediately-or-scheduling-a-maintenance-window-for-a-later-time)."
+1. To verify that the destination instance is ready for configuration, run the following {% ifversion ghes = 3.7 %}command{% elsif ghes = 3.8 or ghes = 3.9 %}commands{% endif %}. There should be no output displayed.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-keying-material
+   {%- ifversion ghes = 3.8 or ghes = 3.9 %}
+   ghe-config secrets.github.encrypted-column-current-encryption-key
+   {%- endif %}
+   ```
+{%- ifversion ghes = 3.7 %}
+1. To update the list of keys on the destination instance, run the following command. Replace KEY-LIST with the output from step 1.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-keying-material "KEY-LIST"
+   ```
+{%- elsif ghes = 3.8 %}
+1. To update the decryption keys on the destination instance, run the following command. Replace DECRYPTION-KEY-LIST with the output from step 1.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-keying-material "DECRYPTION-KEY-LIST"
+   ```
+1. To update the encryption key on the destination instance, run the following command. Replace ENCRYPTION-KEY with the output from step 4.
+
+   ```shell copy
+   ghe-config secrets.github.encrypted-column-current-encryption-key "ENCRYPTION-KEY"
+   ```
+{%- endif %}
+1. To apply the configuration, run the following command.
+
+    ```shell copy
+    ghe-config-apply
+    ```
+1. Wait for the configuration run to complete.
+1. To ensure that the target instance's configuration contains the keys, run the following {% ifversion ghes = 3.7 %}command{% elsif ghes = 3.8 or ghes = 3.9 %}commands{% endif %} and verify that the output matches step 1{% ifversion ghes = 3.8 or ghes = 3.9 %} and step 4{% endif %}.
+
+   ```shell{:copy}
+   ghe-config secrets.github.encrypted-column-keying-material
+   {%- ifversion ghes = 3.8 or ghes = 3.9 %}
+   ghe-config secrets.github.encrypted-column-current-encryption-key
+   {%- endif %}
+   ```
+1. Have a user sign into the destination instance. If any issues arise, contact {% data variables.contact.enterprise_support %}. For more information, see "[AUTOTITLE](/support/contacting-github-support)."
+
+{% endif %}

content/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance.md

@@ -1,6 +1,6 @@
 ---
 title: Known issues with upgrades to your instance
-intro: '{% data variables.product.company_short %} is aware of issues that impact the upgrade process for {% data variables.product.prodname_ghe_server %}, or impact your instance after you complete an upgrade.'
+intro: 'See an overview of workarounds for issues that impact the upgrade process for {% data variables.product.prodname_ghe_server %}, or impact your instance after you complete an upgrade.'
 versions:
   ghes: '>=3.7'
 type: overview
@@ -19,7 +19,7 @@ shortTitle: Known issues with upgrades
 
 {% ifversion mysql-8-upgrade %}
 
-## Known issue: increased I/O utilization from MySQL 8 upgrade
+## Increased I/O utilization from MySQL 8 upgrade
 
 If you upgrade from {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 to 3.9 or later, an upgrade to the database software on your instance will increase I/O utilization. In some cases, this may affect your instance's performance.
 

data/features/backup-utilities-encryption-bug.yml

@@ -0,0 +1,5 @@
+# Reference: ghes#6726, ghes#6731
+# Encryption bug in GitHub Enterprise Server Backup Utilities
+
+versions:
+  ghes: '>=3.7 <=3.9'

data/features/backup-utilities-progress.yml

@@ -0,0 +1,5 @@
+# Reference: #9925
+# ghe-backup-progress utility for GitHub Enterprise Server Backup Utilities
+
+versions:
+  ghes: '>=3.9'

data/release-notes/enterprise-server/3-7/0.yml

@@ -322,6 +322,8 @@ sections:
     - 'The maximum number of self-hosted runners in a runner group is limited to 10,000. Previously, there was no limit. [Updated: 2023-05-24]'
 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/1.yml

@@ -26,6 +26,8 @@ sections:
     - If a site administrator has not yet configured GitHub Actions for the instance, the UI for setting up code scanning will prompt the user to configure GitHub Actions.
     - To avoid failing domain verification due to the 63-character limit enforced by DNS providers for DNS records, the GitHub-generated `TXT` record to verify domain ownership is now limited to 63 characters.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/10.yml

@@ -24,6 +24,8 @@ sections:
     - |
       People with administrative SSH access who generate a support bundle using the `ghe-support-bundle` or `ghe-cluster-support-bundle` utilities can specify the period of time to gather data with `-p` or `--period` without using spaces or quotes. For example, in addition to `'-p 5 days'` or `-p '4 days 10 hours'`,  `-p 5days` or `-p 4days10hours` are valid.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-7/11.yml

@@ -12,6 +12,8 @@ sections:
   changes:
     - People with administrative SSH access to an instance can configure the maximum memory usage in gigabytes for Redis using `ghe-config redis.max-memory-gb VALUE`. 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-7/2.yml

@@ -26,6 +26,8 @@ sections:
     - A user's list of recently accessed repositories no longer includes deleted repositories.
     - '{% data reusables.release-notes.scim-custom-mappings-supported-change %}'
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/3.yml

@@ -22,6 +22,8 @@ sections:
     - The performance of configuration runs started with `ghe-config-apply` has been improved.
     - When exporting account data, backing up a repository, or performing a migration, the link to a repository archive now expires after 1 hour. Previously the archive link expired after 5 minutes.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/4.yml

@@ -4,6 +4,8 @@ sections:
     - |
       {% data reusables.release-notes.2023-01-git-vulnerabilities %}
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/5.yml

@@ -12,6 +12,8 @@ sections:
     - The additional committers count for GitHub Advanced Security always showed 0.
     - In some cases, users were unable to convert existing issues to discussions. If an issue is stuck while being converted to a discussion, enterprise owners can review the "Known issues" section below for more information.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/6.yml

@@ -15,6 +15,8 @@ sections:
   changes:
     - After the Dependency submission REST API receives a submission with one or more dependencies without a version, the dependency graph will now correctly report this fact.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/7.yml

@@ -8,6 +8,8 @@ sections:
     - |
       In the rare case when primary shards for Elasticsearch were located on a replica node, the `ghe-repl-stop` command would fail with `ERROR: Running migrations`.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-7/8.yml

@@ -41,6 +41,8 @@ sections:
     - To avoid intermittent issues with the success of Git operations on an instance with multiple nodes, GitHub Enterprise Server checks the status of the MySQL container before attempting a SQL query. The timeout duration has also been reduced. 
     - The default path for output from `ghe-saml-mapping-csv -d` is `/data/user/tmp` instead of `/tmp`. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-saml-mapping-csv)."
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - |

data/release-notes/enterprise-server/3-7/9.yml

@@ -15,6 +15,8 @@ sections:
   changes:
     - If a site administrator provides an invalid configuration for blob storage for GitHub Actions or GitHub Packages on an instance, the preflight checks page displays details and troubleshooting information. 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-8/0.yml

@@ -436,6 +436,8 @@ sections:
       Before squash-merging a pull request, the web UI displays the email address of the commit's author. Previously, the commit author was only displayed when merging with a merge commit.
 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - Custom firewall rules are removed during the upgrade process.
     - When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

data/release-notes/enterprise-server/3-8/1.yml

@@ -37,6 +37,8 @@ sections:
     - The default path for output from `ghe-saml-mapping-csv -d` is `/data/user/tmp` instead of `/tmp`. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-saml-mapping-csv)."
     - On an instance with a GitHub Advanced Security license, users who author custom patterns for secret scanning can provide expressions that must or must not match that are up to 2,000 characters. This limit is an increase from 1,000 characters.
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
     - |

data/release-notes/enterprise-server/3-8/2.yml

@@ -23,6 +23,8 @@ sections:
       After a site administrator exports a migration archive using GitHub Enterprise Importer's `gh-migrator` utility, the link to the archive remains accessible for 48 hours instead of one hour. 
     - On an instance with a GitHub Advanced Security license, users who author custom patterns for secret scanning can provide expressions that must or must not match that are up to 2,000 characters. This limit is an increase from 1,000 characters. 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-8/3.yml

@@ -26,6 +26,8 @@ sections:
       People with administrative SSH access who generate a support bundle using the `ghe-support-bundle` or `ghe-cluster-support-bundle` utilities can specify the period of time to gather data with `-p` or `--period` without using spaces or quotes. For example, in addition to `'-p 5 days'` or `-p '4 days 10 hours'`,  `-p 5days` or `-p 4days10hours` are valid.
     - After a site administrator exports a migration archive using GitHub Enterprise Importers `gh-migrator` utility, the link to the archive remains accessible for 48 hours instead of one hour. 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-8/4.yml

@@ -15,6 +15,8 @@ sections:
   changes:
     - People with administrative SSH access to an instance can configure the maximum memory usage in gigabytes for Redis using `ghe-config redis.max-memory-gb VALUE`. 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-8/5.yml

@@ -19,6 +19,8 @@ sections:
   changes:
     - If a configuration runs fails due to Elasticsearch errors, `ghe-config-apply` displays a more actionable error message. 
   known_issues:
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |
@@ -39,5 +41,3 @@ sections:
       On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render.
     - |
       Organization owners cannot register a new SSH certificate authorities (CAs) due to an erroneous suggestion to start a trial. Organization SSH CAs configured before an upgrade to an affected version are still usable after the upgrade. Enterprise owners can can still register SSH CAs for all organizations.
-    - |
-      `backup-utils` < v3.9.0 does not correctly backup encryption keys, which prevents encrypted data from being decrypted during a restore.

data/release-notes/enterprise-server/3-9/0-rc1.yml

@@ -1,6 +1,6 @@
 date: '2023-06-08'
 release_candidate: true
-deprecated: false
+deprecated: true
 intro: |
   {% note %}
 
@@ -287,17 +287,7 @@ sections:
         - |
           Users of the GraphQL API can revert a merged pull request by using the revertPullRequest mutation. For more information, see "[AUTOTITLE](/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/reverting-a-pull-request)" and "[AUTOTITLE](/graphql/reference/mutations#revertpullrequest)" in the GraphQL API documentation.
 
-  changes:
-    # HOLD FOR GA:
-    #
-    # https://github.com/github/releases/issues/3050
-    #- |
-    # Before beginning a backup with GitHub Enterprise Server Backup Utilities 3.9.0 and later, the `ghe-host-check` utility will now perform a preflight check on the backup host to confirm the software version and disk space requirements. For more information, see the [3.9.0 release](https://github.com/github/backup-utils/releases/tag/v3.9.0) in the `github/backup-utils` repository on GitHub.com.
-
-    # https://github.com/github/releases/issues/3052
-    #- |
-    # GitHub Enterprise Server Backup Utilities 3.9.0 and later displays a progress indicator for backup and restoration operations. For more information, see the [3.9.0 release](https://github.com/github/backup-utils/releases/tag/v3.9.0) in the `github/backup-utils` repository on GitHub.com.
-      
+  changes: 
     # https://github.com/github/releases/issues/2909
     - |
       Field names for some service logs on GitHub Enterprise Server have changed as part of GitHub's gradual migration to internal semantic conventions for [OpenTelemetry](https://opentelemetry.io/). Additional field names will change in upcoming releases. If any tooling or processes in your environment rely on specific field names within logs, or log entries in specific files, the following changes may affect you.
@@ -380,9 +370,20 @@ sections:
     - |
       If a user archives a repository, responses from the GraphQL API that include information about the repository now include an `archivedAt` value with a timestamp representing the archival date.
 
+  backups:
+    # https://github.com/github/releases/issues/3050
+    - |
+     Before beginning a backup with GitHub Enterprise Server Backup Utilities 3.9.0 and later, the `ghe-host-check` utility will now perform a preflight check on the backup host to confirm the software version and disk space requirements. For more information, see the [3.9.0 release](https://github.com/github/backup-utils/releases/tag/v3.9.0) in the github/backup-utils repository on GitHub.com.
+
+    # https://github.com/github/releases/issues/3052
+    - |
+     GitHub Enterprise Server Backup Utilities 3.9.0 allows administrators to view the progress of backup and restoration operations on the backup host using the `ghe-backup-progress` utility. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/configuring-backups-on-your-appliance#monitoring-backup-or-restoration-progress)."
+
   known_issues:
     - |
       If you upgrade from {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 to 3.9, the database server on your instance will be upgraded from MySQL 5.7 to MySQL 8.0. I/O utilization will increase as a result, and in some cases this may affect your instance's performance. Do not upgrade to this RC in a production environment, and ensure that you take and verify a backup of the instance before upgrading to the GA release. For more information, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance)."
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |
       {% data reusables.release-notes.manage-api-unreachable %}
     - |

data/release-notes/enterprise-server/3-9/0.yml

@@ -0,0 +1,427 @@
+date: '2023-06-08'
+release_candidate: false
+deprecated: false
+intro: |
+  For upgrade instructions, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/upgrading-github-enterprise-server)."
+
+sections:
+  features:
+    - heading: Instance administration
+      notes:
+        # https://github.com/github/releases/issues/3019
+        - |
+          To improve security posture and protect data from threats, enterprise owners can see user activity from the Management Console within the enterprise audit log, including events from the UI, API, and administrative SSH access. For more information, see "[AUTOTITLE](/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/audit-log-events-for-your-enterprise#management_console-category-actions)."
+
+        # https://github.com/github/releases/issues/3053
+        - |
+          During an upgrade of an instance to a new release, people with administrative SSH access to the instance can monitor the progress of routine migrations using the `ghe-migrations` utility. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-migrations)."
+
+        # https://github.com/github/releases/issues/3054
+        - |
+          On an instance with multiple nodes, site administrators can use the Manage GitHub Enterprise Server API to monitor the health of replication. For more information, see "[AUTOTITLE](/admin/enterprise-management/configuring-high-availability/monitoring-a-high-availability-configuration)."
+
+        # https://github.com/github/releases/issues/3097
+        - |
+          On an instance in a cluster configuration, administrators can ensure a balanced distribution of jobs across nodes by using the `ghe-cluster-rebalance` utility. For more information, see "[AUTOTITLE](/admin/enterprise-management/configuring-clustering/rebalancing-cluster-workloads)."
+
+        # https://github.com/github/releases/issues/3096
+        - |
+          On an instance in a cluster configuration, administrators can proactively monitor the health of individual nodes and control the reintroduction of unhealthy nodes into the cluster using Node Eligibility Service. For more information, see "[AUTOTITLE](/admin/enterprise-management/configuring-clustering/monitoring-the-health-of-your-cluster-nodes-with-node-eligibility-service)."
+
+    - heading: Identity and access management
+      notes:
+        # https://github.com/github/releases/issues/3019
+        - |
+          On an instance configured for SAML SSO, enterprise owners can review information about the Identity Provider (IdP) configured for user authentication using the GraphQL API. The personal access token (PAT) used to authenticate requests to this API requires the `read:enterprise` scope. Previously, the PAT required the `admin:enterprise` scope. For more information, see "[AUTOTITLE](/graphql/reference/objects#enterpriseidentityprovider)" in the GraphQL API documentation.
+
+    - heading: Authentication
+      notes:
+        # https://github.com/github/releases/issues/2833
+        - |
+          For an instance or organization with 2FA enabled, users can configure a 2FA method to be a preferred method. Users can also update 2FA methods from `http(s)://HOSTNAME/settings/security`. For more information, see "[AUTOTITLE](/authentication/securing-your-account-with-two-factor-authentication-2fa/configuring-two-factor-authentication)" and "[AUTOTITLE](/authentication/securing-your-account-with-two-factor-authentication-2fa/changing-your-preferred-two-factor-authentication-method)."
+
+    - heading: REST API
+      notes:
+        # https://github.com/github/releases/issues/2022
+        - |
+          To provide API integrators a smooth migration path and time to update integrations after GitHub makes occasional breaking changes, the REST API now uses calendar-based versioning. GitHub Enterprise Server 3.9 provides version `2022-11-28` of the REST API. For more information, see "[AUTOTITLE](/rest/overview/api-versions?apiVersion=2022-11-28)" in the REST API documentation.
+
+    - heading: GitHub Connect
+      notes:
+        # https://github.com/github/releases/issues/2783
+        - |
+          Enterprise owners who configure Server Statistics on an instance with GitHub Actions enabled will transmit usage metrics related to GitHub Actions. For more information, see "[AUTOTITLE](/admin/monitoring-activity-in-your-enterprise/analyzing-how-your-team-works-with-server-statistics/about-server-statistics#server-statistics-data-collected)."
+
+    - heading: GitHub Advanced Security
+      notes:
+        # https://github.com/github/releases/issues/2452
+        - |
+          To more easily discover potential security or quality issues in code, users can configure code scanning directly through the web interface without adding a GitHub Actions workflow to the repository. This feature finds and sets up the best CodeQL configuration for the repository, detecting supported languages and enabling CodeQL analysis for every pull request and every push to the default branch and any protected branches. Analysis of JavaScript (including TypeScript), Python, and Ruby code, are currently supported. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically)."
+
+        # https://github.com/github/releases/issues/2888
+        - |
+          To simplify the configuration of code scanning, organization owners can enable code scanning for all eligible repositories in an organization using a default configuration, either via the web interface or REST API. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-at-scale)" and "[AUTOTITLE](/rest/orgs/orgs?apiVersion=2022-11-28#enable-or-disable-a-security-feature-for-an-organization)" in the REST API documentation.
+
+        # https://github.com/github/releases/issues/2845
+        - |
+          To ensure that relevant alerts remain visible and actionable, users can manually remove stale alerts from code scanning. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/managing-code-scanning-alerts-for-your-repository#removing-stale-configurations-and-alerts-from-a-branch)."
+
+        # https://github.com/github/releases/issues/2796
+        - |
+          To better understand the status of CodeQL and other code scanning tools for a repository, and to help troubleshoot, users can review the tool status page. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page)."
+
+        # https://github.com/github/releases/issues/2943
+        - |
+          To customize the behavior of code scanning on a per-repository basis, repository administrators can configure what severity levels for code scanning alerts will cause checks in a pull request to fail. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/triaging-code-scanning-alerts-in-pull-requests#code-scanning-results-check-failures)."
+
+        # https://github.com/github/releases/issues/2699
+        # https://github.com/github/releases/issues/2800
+        - |
+          To protect repositories from pushes that contain custom secret scanning patterns defined at the enterprise, organization, or repository level, users can enable push protection for those patterns. For more information, see "[AUTOTITLE](/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning)."
+
+        # https://github.com/github/releases/issues/2794
+        - |
+          Organization owners can view the enablement status of security features for the organization's repositories using the REST API. The endpoint provides details for GitHub Advanced Security, secret scanning, and push protection. For more information, see "[Repositories](/rest/repos/repos?apiVersion=2022-11-28#list-organization-repositories)" in the REST API documentation.
+
+        # https://github.com/github/releases/issues/2840
+        - |
+          Repository administrators can programmatically enable code scanning with a default CodeQL configuration using the REST API.  For more information, see the following documentation.
+
+          - "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically)"
+          - "[Get the code scanning default setup configuration](/rest/code-scanning#get-a-code-scanning-default-setup-configuration)" in the Code Scanning REST API documentation
+          - "[Update the code scanning default setup configuration](/rest/code-scanning#update-a-code-scanning-default-setup-configuration)" in the Code Scanning REST API documentation
+
+    - heading: Dependabot
+      notes:
+        # https://github.com/github/releases/issues/2976
+        - |
+          To improve the security of GitHub Actions workflows that pin references, Dependabot can update the versioning for calls to reusable workflows within workflow files. For more information, see "[AUTOTITLE](/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates)."
+
+        # https://github.com/github/releases/issues/2911
+        - |
+          On an instance with GitHub Actions and the dependency graph enabled, as well as automatic access to GitHub.com actions using GitHub Connect, the web interface will suggest submission actions within a repository with supported languages. For more information, see the following documentation.
+          
+          - "[AUTOTITLE](/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api)"
+          - "[AUTOTITLE](/admin/github-actions/getting-started-with-github-actions-for-your-enterprise/about-github-actions-for-enterprises)"
+          - "[AUTOTITLE](/admin/github-actions/managing-access-to-actions-from-githubcom/enabling-automatic-access-to-githubcom-actions-using-github-connect)"
+          
+          For repositories that use a language that has a submission action, when users with write access visit their dependency graph (this page), we will show them a prompt that directs them to the Marketplace to find an action that would help them.
+
+        # https://github.com/github/releases/issues/3007
+        - |
+          To improve the security of projects that use npm v9, the dependency graph and Dependabot can parse and update `package-lock.json` files that specify `lockfileVersion: 3`. For more information, see "[AUTOTITLE](/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph)," "[AUTOTITLE](/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates)," and [`lockfileVersion`](https://docs.npmjs.com/cli/v9/configuring-npm/package-lock-json#lockfileversion) in the npm documentation.
+
+        # https://github.com/github/releases/issues/2980
+        - |
+          To improve the security of Gradle projects, the dependency graph and Dependabot can parse and update Gradle version catalogs in `settings.gradle`.  For more information, see "[AUTOTITLE](/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates)" and [Sharing dependency versions between projects](https://docs.gradle.org/current/userguide/platforms.html) in the Gradle User Manual.
+
+        # https://github.com/github/releases/issues/2806
+        - |
+          To ensure that users receive the most relevant and actionable alerts about dependency updates, repository administrators and organization owners can enable or disable Dependabot alerts for an individual repository or organization. For more information, see "[AUTOTITLE](/code-security/getting-started/securing-your-repository#managing-dependabot-alerts)" or "[AUTOTITLE](/code-security/getting-started/securing-your-organization#managing-dependabot-alerts-and-the-dependency-graph)."
+
+        # https://github.com/github/releases/issues/2601
+        - |
+          If people with access to a repository do not interact with Dependabot security updates for over 90 days, Dependabot will pause automated pull request activity. For more information, see "[AUTOTITLE](/code-security/dependabot/dependabot-security-updates/about-dependabot-security-updates#about-automatic-deactivation-of-dependabot-updates)."
+
+        # https://github.com/github/releases/issues/3068
+        - |
+          To help users evaluate the stability risk of a dependency update, Dependabot can fetch release notes, changelogs, and commit history in pull requests that update Docker dependencies. For more information, see "[AUTOTITLE](/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#docker)."
+
+        # https://github.com/github/releases/issues/2873
+        - |
+          To assist with software security and supply chain risk management, people with read access to a repository can export a software bill of materials (SBOM) for a repository's dependency graph using the web interface or REST API. The SBOM adheres to the SPDX 2.3 specification. For more information, see "[AUTOTITLE](/code-security/supply-chain-security/understanding-your-software-supply-chain/using-the-dependency-submission-api#generating-and-submitting-a-software-bill-of-materials-sbom)," "[AUTOTITLE](/code-security/supply-chain-security/understanding-your-software-supply-chain/exporting-a-software-bill-of-materials-for-your-repository)," and [The Software Package Data Exchange® (SPDX®) Specification Version 2.3](https://spdx.github.io/spdx-spec/v2.3/) on the SPDX website.
+
+        # https://github.com/github/releases/issues/2871
+        - |
+          The dependency graph can parse Python dependencies for `pyproject.toml` files that follow the PEP 621 standard. For more information,  see "[AUTOTITLE](/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph)" and [PEP 621 – Storing project metadata in pyproject.toml](https://peps.python.org/pep-0621/) in the Index of Python Enhancement Proposals.
+
+        # https://github.com/github/releases/issues/3023
+        - |
+          Users can use the GraphQL API to review dependencies submitted using the Dependency submission API. For more information, see "[AUTOTITLE](/graphql/overview/schema-previews#access-to-a-repositorys-dependency-graph-preview)."
+
+    - heading: GitHub Actions
+      notes:
+        # https://github.com/github/releases/issues/3006
+        - |
+          On instances in a cluster configuration, GitHub Actions is available as a private beta. Beta features are subject to change. For more information, and to enroll in the beta, [contact your representative on GitHub's Sales team](https://github.com/enterprise/contact).
+
+        # https://github.com/github/releases/issues/2617
+        - |
+          Administrators of self-hosted runners for GitHub Actions can configure auto-scaling runners using Actions Runner Controller and runner scale sets. For more information, see "[AUTOTITLE](/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller)."
+
+        # https://github.com/github/releases/issues/2896
+        - |
+          Administrators can bypass all protection rules for a given environment and force the pending jobs referencing the environment to proceed. For more information, see "[ AUTOTITLE](/actions/deployment/targeting-different-environments/using-environments-for-deployment#allow-administrators-to-bypass-configured-protection-rules)."
+
+        # https://github.com/github/releases/issues/2801
+        - |
+          Users who deploy with OIDC can define more advanced access policies by including additional custom claims within a token. To help uniquely verify the source of a workflow job, include the following claims.
+
+          - `actor_id`
+          - `repository_id`
+          - `repository_owner_id`
+          - `workflow_ref`
+          - `workflow_sha`
+          - `job_workflow_sha`
+
+          For more information, see [AUTOTITLE](/actions/deployment/security-hardening-your-deployments).
+
+        # https://github.com/github/releases/issues/2905
+        - |
+          To improve security for workflows that use `GITHUB_TOKEN`, the following defaults apply to new organizations and repositories.
+
+          - New organizations that users create inherit permissions from the instance's enterprise-level configuration. For more information, see "[AUTOTITLE](/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization#configuring-the-default-github_token-permissions)."
+          - New repositories that users create within an organization inherit permissions from the organization. For more information, see "[AUTOTITLE](/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-the-default-github_token-permissions)."
+          - New user-owned repositories have a read-only `GITHUB_TOKEN`. For more information, see "[AUTOTITLE](/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-the-default-github_token-permissions)."
+
+        # https://github.com/github/releases/issues/2979
+        - |
+          To allow workflow authors to pin a required workflow file to a fully validated version, required workflows can be referenced using any branch, tag, or commit SHA from the repository containing the workflow file. For more information, see "[AUTOTITLE](/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization#configuring-a-required-workflow-for-your-organization)."
+        - |
+          To enforce required workflows throughout an organization, GitHub Enterprise Server blocks direct pushes to branches where required workflows are enforced. To allow direct pushes for a particular repository, remove the repository as a target for the required workflow. For more information, see "[AUTOTITLE](/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization#configuring-a-required-workflow-for-your-organization)."
+
+        # https://github.com/github/releases/issues/2861
+        - |
+          To improve performance for workflows that build Go, caching is enabled by default when using the `setup-go` action. For more information, see "[AUTOTITLE](/actions/automating-builds-and-tests/building-and-testing-go#caching-dependencies)."
+
+    # IN PROGRESS
+    #- heading: GitHub Packages
+    # notes:
+    #   # https://github.com/github/releases/issues/2924
+    #   - |
+    #     Users can manage packages in repositories and organizations using the Packages REST API. For more information, see "[AUTOTITLE](/rest/packages?apiVersion=2022-11-28)" in the REST API documentation.
+
+    - heading: Organizations
+      notes:
+        # https://github.com/github/releases/issues/2986
+        - |
+          Organization owners can improve security posture and protect data from threats by enabling the display of organization members' IP addresses in audit log events. This feature is in beta and is subject to change. For more information, see "[AUTOTITLE](/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/displaying-ip-addresses-in-the-audit-log-for-your-organization)."
+
+        # https://github.com/github/releases/issues/2916
+        - |
+          To allow the management of branch protection rules without granting admin access, organization owners can create a custom role with the "Edit repository rules" permission. For more information, see "[AUTOTITLE](/organizations/managing-peoples-access-to-your-organization-with-roles/managing-custom-repository-roles-for-an-organization)."
+
+        # https://github.com/github/releases/issues/2462
+        # https://github.com/github/releases/issues/2556
+        - |
+          Users of the REST API can programmatically create and update least-privilege roles for repositories using the Custom Repository Roles REST API. The API is generally available, with a breaking change to the API's endpoint paths. Previously, the API was accessible at `/orgs/{org}/custom_roles`, and is now accessible at `/orgs/{org}/custom-repository-roles`. The [List custom repository roles in an organization](/rest/orgs/custom-roles#list-custom-repository-roles-in-an-organization) will no longer be available in the next version of the REST API. For more information, see "[AUTOTITLE](/organizations/managing-peoples-access-to-your-organization-with-roles/about-custom-repository-roles)" and "[AUTOTITLE](/rest/orgs/custom-roles?apiVersion=2022-11-28)" in the REST API documentation.
+
+        # https://github.com/github/releases/issues/3067
+        - |
+          Enterprise and organization owners can delete an organization and all of the organization's repositories using the REST API. After deletion, organization names are locked for 90 days. For more information, see "[AUTOTITLE](/rest/orgs/orgs?apiVersion=2022-11-28#delete-an-organization)" in the REST API documentation.
+
+    - heading: Repositories
+      notes:
+        # https://github.com/github/releases/issues/2707
+        - |
+          Within the "Insights" tab for a repository, the sidebar's "Forks" tab provides more information about a project's forks, including a sortable and filterable list of forks and more details about each fork.
+
+        # https://github.com/github/releases/issues/2791
+        - |
+          Repository administrators can unarchive a repository using the REST API. For more information, see "[AUTOTITLE](/rest/repos/repos?apiVersion=2022-11-28#update-a-repository)" in the REST API documentation.
+
+    - heading: Projects
+      notes:
+        # https://github.com/github/releases/issues/2827
+        - |
+          To visualize a project at a high level and across a configurable timespan, users can apply a roadmap layout to any project view. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/customizing-views-in-your-project/changing-the-layout-of-a-view#about-the-roadmap-layout)."
+
+        # https://github.com/github/releases/issues/2821
+        - |
+          To get started with a new project faster, users can copy an existing project, including the source project's views, custom fields, and draft issues. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/creating-projects/copying-an-existing-project)."
+
+        # https://github.com/github/releases/issues/2820
+        - |
+          To save time when adding items to a project, users can configure a workflow to automatically add new items from a repository as people create or update items that match specific criteria. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/automating-your-project/adding-items-automatically)."
+
+        # https://github.com/github/releases/issues/2503
+        - |
+          To keep a long-lived project focused, users can define filters to automatically archive items. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/automating-your-project/archiving-items-automatically)."
+
+        # https://github.com/github/releases/issues/2826
+        - |
+          To easily organize items within a project's columns while using the board layout, users can sort the project by field values using the view configuration menu. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/customizing-views-in-your-project/customizing-the-board-layout)."
+
+        # https://github.com/github/releases/issues/2829
+        - |
+          To quickly add a new issue to a project without changing context, users can create a new issue from a project's omnibar by clicking `+`, then clicking **Create new issue**. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/managing-items-in-your-project/adding-items-to-your-project#creating-issues)."
+
+        # https://github.com/github/releases/issues/2917
+        - |
+          To help people scan a project and take action, users can add a color and a text description to each value for a project's single select fields. For more information, see "[AUTOTITLE](/issues/planning-and-tracking-with-projects/understanding-fields/about-single-select-fields#editing-a-single-select-field)."
+
+        # https://github.com/github/releases/issues/2984
+        - |
+          Users of the GitHub CLI can manage projects from the command line. For more information, see "[AUTOTITLE](/github-cli/github-cli/about-github-cli)" and the [README](https://github.com/github/gh-projects#cli-extension-for-projects) for the `github/gh-projects` repository on GitHub.com.
+
+        # https://github.com/github/releases/issues/2978
+        - |
+          For users who programmatically access projects using the GraphQL API, additional mutations are available. For more information, see "[createProjectV2Field](/graphql/reference/mutations#createprojectv2field)," "[deleteProjectV2Field](/graphql/reference/mutations#deleteprojectv2field)," and "[deleteProjectV2](/graphql/reference/mutations#deleteprojectv2)" in the "Mutations" GraphQL documentation.
+
+    - heading: GitHub Discussions
+      notes:
+        # https://github.com/github/releases/issues/2967
+        - |
+          To indicate that a discussion is resolved, outdated, or a duplicate, users can close the discussion. For more information, see "[AUTOTITLE](/discussions/managing-discussions-for-your-community/managing-discussions#closing-a-discussion)."
+
+        # https://github.com/github/releases/issues/2825
+        - |
+          To encourage other users to include specific, structured information in discussions, users can create discussion category forms. For more information, see "[AUTOTITLE](/discussions/managing-discussions-for-your-community/creating-discussion-category-forms)."
+
+        # https://github.com/github/releases/issues/2675
+        - |
+          After a user locks a discussion and disallows further comments, the user can permit emoji reactions on the discussion. For more information, see "[AUTOTITLE](/discussions/managing-discussions-for-your-community/moderating-discussions#locking-discussions)."
+
+    - heading: Pull requests
+      notes:
+        # https://github.com/github/releases/issues/3026
+        - |
+          To provide feedback on an entire file, or a file that's been deleted, users can comment on a file from a pull request's "Files changed" tab. For more information, see "[AUTOTITLE](/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/commenting-on-a-pull-request)."
+
+        # https://github.com/github/releases/issues/2857
+        - |
+          Users of the GraphQL API can revert a merged pull request by using the revertPullRequest mutation. For more information, see "[AUTOTITLE](/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/reverting-a-pull-request)" and "[AUTOTITLE](/graphql/reference/mutations#revertpullrequest)" in the GraphQL API documentation.
+
+  changes: 
+    # https://github.com/github/releases/issues/2909
+    - |
+      Field names for some service logs on GitHub Enterprise Server have changed as part of GitHub's gradual migration to internal semantic conventions for [OpenTelemetry](https://opentelemetry.io/). Additional field names will change in upcoming releases. If any tooling or processes in your environment rely on specific field names within logs, or log entries in specific files, the following changes may affect you.
+
+      - `level` is now `SeverityText`.
+      - `log_message`, `msg`, or `message` is now `Body`.
+      - `now` is now `Timestamp`.
+      - Custom field names such as `gh.repo.id` or `graphql.operation.name` use semantic names.
+      - Log statements that the instance would previously write to `auth.log`, `ldap.log`, or `ldap-sync.log` now appear in containerized logs for `github-unicorn` if the statement originated from a web request, or in logs for `github-resqued` if the statement originated from a background job.
+
+      For a full list of mappings, download the [OpenTelemetry attribute mapping CSV](/assets/ghes-3.9-opentelemetry-attribute-mappings.csv).
+
+    # https://github.com/github/ghes/issues/6342
+    - |
+      On a configured instance, the name for the HAProxy service is now `haproxy-frontend`. Previously, the name was `haproxy`. Additionally, on an unconfigured instance, there is a new service named `haproxy-pre-config`. If your instance forwards logs to an external system, update your rules to reflect these changes. For more information, see "[AUTOTITLE](/admin/monitoring-activity-in-your-enterprise/exploring-user-activity/log-forwarding)" article
+
+    # https://github.com/github/releases/issues/2757
+    - |
+      For an instance or organization with 2FA enabled, when a user sets up 2FA, GitHub Enterprise Server suggests an authenticator app (TOTP) by default.
+
+    # https://github.com/github/releases/issues/3160
+    - |
+      When a person with administrative SSH access to an instance submits a support bundle using either the `ghe-support-bundle` or `ghe-cluster-support-bundle` utility, a period for log collection specified with the `-p` or `--period` no longer requires quotes to enclose the date value. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-support-bundle)."
+
+    # https://github.com/github/releases/issues/2745
+    - |
+      To provide additional context within the web interface on an instance where Dependabot alerts are enabled, links to Dependabot alerts in an issue or pull request comment display an improved label and hovercard with alert details.
+
+    # https://github.com/github/releases/issues/2599
+    - |
+      On an instance with Dependabot alerts enabled, people with write or maintain access to a repository can view or act on Dependabot alerts by default. Custom roles, the security manager role, organization permissions, and notification settings are not affected.
+
+    # https://github.com/github/releases/issues/2946
+    - |
+      On an instance with a GitHub Advanced Security license and GitHub Connect enabled for the synchronization of actions from GitHub.com, CodeQL code scanning is up to 16% faster. For more information, see "[AUTOTITLE](/admin/code-security/managing-github-advanced-security-for-your-enterprise/configuring-code-scanning-for-your-appliance#configuring-github-connect-to-sync-github-actions)."
+
+    # https://github.com/github/releases/issues/2865
+    - |
+      On an instance with a GitHub Advanced Security license and email configured for notifications, users can receive notifications for secret scanning alerts by watching a repository and choosing "All activity" or "Security alerts". To continue receiving notifications for secret scanning alerts in GitHub Enterprise Server 3.9 and later, users must enable email notifications in the web interface at `http(s)://HOSTNAME/settings/notifications` under "Watching" by choosing "Email".
+
+    # https://github.com/github/releases/issues/2724
+    - |
+      On an instance with a GitHub Advanced Security license, secret scanning alerts display whether detected tokens from GitHub are valid.
+
+    # https://github.com/github/releases/issues/2776
+    - |
+      On an instance with a GitHub Advanced Security license, the enterprise and organization audit logs now display an event when an owner enables or disables a push protection for a custom pattern for a repository, organization, or the enterprise. For more information, see "[AUTOTITLE](/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/reviewing-the-audit-log-for-your-organization#org-category-actions)" and "[AUTOTITLE](/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/audit-log-events-for-your-enterprise#business_secret_scanning_push_protection-category-actions)."
+
+    # https://github.com/github/releases/issues/2892
+    - |
+      Users can filter the lists of alerts for Dependabot, code scanning, and secret scanning by repository topic or team in the security overview for an organization. For more information, see "[AUTOTITLE](/code-security/security-overview/filtering-alerts-in-security-overview)."
+
+    # https://github.com/github/releases/issues/3073
+    - |
+      In the security overview for an organization, the following improvements apply to the "Security coverage" view during feature enablement.
+
+      - To provide insight into the number of GitHub Advanced Security licenses used, active committers for the repository are visible. For repositories where GitHub Advanced Security is not enabled, the number indicates the number of licenses required to enable the feature.
+      - Unsaved changes are now labeled with a "Modified" tag, and the "Save security settings" button now displays the total number of changes to save.
+      - While a security feature is being enabled, the "Security coverage" view shows a status of "Updating..." to inform you of the ongoing process.
+      
+      For more information, see "[AUTOTITLE](/code-security/security-overview/about-security-overview)."
+
+    # https://github.com/github/releases/issues/2811
+    - |
+      In the security overview's "Security risk" and "Security coverage" views, when a user selects a team from the "Team" drop-down or filters by team, results appear for repositories where the team has write or administrative access or has been granted access to security alerts. Previously, users could only view results for repositories where the team had administrative access or had been granted access to security alerts.
+
+    # https://github.com/github/releases/issues/2822
+    - |
+      To provide more context within a project, users can share a deep link to a specific issue in a project to have the issue open in the project's side panel.
+
+    # https://github.com/github/releases/issues/2958
+    - |
+      Organization owners can create up to five custom repository roles. Previously, the limit was three. For more information, see "[AUTOTITLE](/organizations/managing-peoples-access-to-your-organization-with-roles/about-custom-repository-roles)."
+
+    # https://github.com/github/releases/issues/2799
+    - |
+      When transferring a repository, users can also rename the repository. For more information, see "[AUTOTITLE](/repositories/creating-and-managing-repositories/transferring-a-repository)."
+
+    # https://github.com/github/releases/issues/2961
+    - |
+      If a user archives a repository, responses from the GraphQL API that include information about the repository now include an `archivedAt` value with a timestamp representing the archival date.
+
+  backups:
+    # https://github.com/github/releases/issues/3050
+    - |
+     Before beginning a backup with GitHub Enterprise Server Backup Utilities 3.9.0 and later, the `ghe-host-check` utility will now perform a preflight check on the backup host to confirm the software version and disk space requirements. For more information, see the [3.9.0 release](https://github.com/github/backup-utils/releases/tag/v3.9.0) in the `github/backup-utils` repository on GitHub.com.
+
+    # https://github.com/github/releases/issues/3052
+    - |
+     GitHub Enterprise Server Backup Utilities 3.9.0 allows administrators to view the progress of backup and restoration operations on the backup host using the `ghe-backup-progress` utility. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/configuring-backups-on-your-appliance#monitoring-backup-or-restoration-progress)."
+
+  known_issues:
+    - |
+      If you upgrade from {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 to 3.9, the database server on your instance will be upgraded from MySQL 5.7 to MySQL 8.0. I/O utilization will increase as a result, and in some cases this may affect your instance's performance. Do not upgrade to this RC in a production environment, and ensure that you take and verify a backup of the instance before upgrading to the GA release. For more information, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance)."
+    - |
+      {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
+    - |
+      {% data reusables.release-notes.manage-api-unreachable %}
+    - |
+      On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run `ghe-config-apply`.
+
+      ```
+      Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
+      ```
+
+      You can safely ignore this message.
+    - |
+      Custom firewall rules are removed during the upgrade process.
+    - |
+      The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.
+    - |
+      On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.
+    - |
+      The `mbind: Operation not permitted` error in the `/var/log/mysql/mysql.err` file can be ignored. MySQL 8 does not gracefully handle when the `CAP_SYS_NICE` capability isn't required, and outputs an error instead of a warning.
+    - |
+      When using an outbound web proxy server, the `ghe-btop` command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".
+    - |
+      If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using `ghe-ssl-ca-certificate-install` are not respected, and connections to the server fail.
+    - |
+      When running `ghe-config-apply`, the process may stall with the message `Deployment is running pending automatic promotion`. 
+      During the validation phase of a configuration run, a `No such object` error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.
+    - |
+      If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "[AUTOTITLE](/admin/configuration/administering-your-instance-from-the-management-console/troubleshooting-access-to-the-management-console#unlocking-the-root-site-administrator-account)."
+
+  deprecations:
+    # https://github.com/github/releases/issues/2826
+    - heading: Change to command-line utility for management of replication
+      notes:
+        - |
+          On an instance with multiple nodes, people with administrative SSH access to the instance should use `ghe-spokesctl` for management of Git replication instead of `ghe-spokes`. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#ghe-spokesctl)."
+
+    # https://github.com/github/releases/issues/2773
+    - heading: Dependency graph no longer ingests go.sum files
+      notes:
+        - |
+          Because `go.sum` files are not lock files and may result in false positive Dependabot alerts, on an instance with the dependency graph enabled, the `go.sum` files are no longer ingested for users' Go repositories. If Dependabot alerts are enabled, Dependabot will no longer alert users for vulnerabilities in a `go.sum` file's dependencies. The dependency graph continues to support `go.mod` files, the recommended format for Go projects. Use Go 1.17 or higher to ensure your `go.mod` file contains a comprehensive view of all direct and transitive dependencies. For more information, see "[AUTOTITLE](/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph)."
+
+    # https://github.com/github/releases/issues/2938
+    - heading: Only GitHub Actions can publish a GitHub Pages site if source includes symbolic links
+      notes:
+        - |
+          To improve the security of an instance where users deploy sites using GitHub Pages, sites that contain symbolic links will no longer build outside of GitHub Actions. If a user's site is affected and a site administrator has configured email for the instance, the user will receive an email with instructions about how to fix the error. To continue using symbolic links in the site's source, the instance must be configured for GitHub Actions, and the user must write a GitHub Actions workflow to use as a publishing source. For more information, see "[AUTOTITLE](/pages/getting-started-with-github-pages/about-github-pages#publishing-sources-for-github-pages-sites)."

data/reusables/enterprise_backup_utilities/enterprise-backup-utils-encryption-keys.md

@@ -0,0 +1 @@
+After restoration of a backup created using {% data variables.product.prodname_enterprise_backup_utilities %} {% ifversion ghes = 3.7 %}3.7.0{% elsif ghes = 3.8 or ghes = 3.9 %}3.7.0 or 3.8.0{% endif %}, users may not be able to sign into the instance. For more information, see "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/known-issues-with-backups-for-your-instance#known-issue-users-cannot-sign-in-after-restoration-of-a-backup)."

data/reusables/release-notes/enterprise-backup-utils-encryption-keys.md

@@ -0,0 +1 @@
+{% data reusables.enterprise_backup_utilities.enterprise-backup-utils-encryption-keys %} [Updated: 2023-06-29]

data/variables/release_candidate.yml

@@ -1 +1 @@
-version: enterprise-server@3.9
+version: ''

tests/fixtures/data/variables/release_candidate.yml

@@ -1 +1 @@
-version: enterprise-server@3.9
+version: ''