github
/
automatic-contrib-prs
зеркало из
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

feat: support github enterprise api

This commit is contained in:
ricardojdsilva87 2024-10-30 14:25:28 +00:00
Родитель 8065048872
Коммит 70a357bb0b
8 изменённых файлов: 304 добавлений и 57 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

4
.coveragerc Normal file
Просмотреть файл

@ -0,0 +1,4 @@
[run]
omit =
# omit test files
test_*.py

13
.env-example
Просмотреть файл

@ -1,12 +1,13 @@
GH_ACTOR=""
GH_ACTOR = ""
GH_ENTERPRISE_URL = ""
GH_TOKEN=""
ORGANIZATION=""
PR_TITLE=""
PR_BODY=""
REPOS_JSON_LOCATION=""
GH_TOKEN = ""
ORGANIZATION = ""
PR_TITLE = ""
PR_BODY = ""
REPOS_JSON_LOCATION = ""
# GITHUB APP
GH_APP_ID = ""
GH_INSTALLATION_ID = ""
GH_PRIVATE_KEY = ""
GITHUB_APP_ENTERPRISE_ONLY = ""

58
README.md
Просмотреть файл

@ -31,7 +31,7 @@ It is desirable, for example, for all Open Source and InnerSource projects to ha
## Use as a GitHub Action
1. Create a repository to host this GitHub Action or select an existing repository.
1. Create the env values from the sample workflow below (GH_TOKEN, GH_ACTOR, PR_TITLE, PR_BODY, and ORGANIZATION) with your information as repository secrets. More info on creating secrets can be found [here](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
1. Create the env values from the sample workflow below (`GH_TOKEN`, `GH_ACTOR`, `PR_TITLE`, `PR_BODY`, and `ORGANIZATION`) with your information as repository secrets. More info on creating secrets can be found [here](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
Note: Your GitHub token will need to have read/write access to all the repositories in the `repos.json` file.
1. Copy the below example workflow to your repository and put it in the `.github/workflows/` directory with the file extension `.yml` (ie. `.github/workflows/auto-contrib-file.yml`)
@ -45,11 +45,12 @@ This action can be configured to authenticate with GitHub App Installation or Pe
##### GitHub App Installation
| field | required | default | description |
| ------------------------ | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GH_APP_ID` | True | `""` | GitHub Application ID. See [documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) for more details. |
| `GH_APP_INSTALLATION_ID` | True | `""` | GitHub Application Installation ID. See [documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) for more details. |
| `GH_APP_PRIVATE_KEY` | True | `""` | GitHub Application Private Key. See [documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) for more details. |
| field | required | default | description |
| ---------------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GH_APP_ID` | True | `""` | GitHub Application ID. See [documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) for more details. |
| `GH_APP_INSTALLATION_ID` | True | `""` | GitHub Application Installation ID. See [documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) for more details. |
| `GH_APP_PRIVATE_KEY` | True | `""` | GitHub Application Private Key. See [documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app) for more details. |
| `GITHUB_APP_ENTERPRISE_ONLY` | False | `false` | Set this input to `true` if your app is created in GHE and communicates with GHE. |
##### Personal Access Token (PAT)
@ -106,6 +107,51 @@ jobs:
PR_BODY: ${{ secrets.PR_BODY }}
```
#### Using GitHub app
```yaml
name: Find proper repos and open CONTRIBUTING.md prs
on:
workflow_dispatch:
permissions:
contents: read
jobs:
build:
name: Open CONTRIBUTING.md in OSS if it doesnt exist
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Find OSS repository in organization
uses: docker://ghcr.io/zkoppert/innersource-crawler:v1
env:
GH_TOKEN: ${{ secrets.GH_TOKEN }}
ORGANIZATION: ${{ secrets.ORGANIZATION }}
TOPIC: open-source
- name: Open pull requests in OSS repository that are missing contrib files
uses: docker://ghcr.io/github/automatic-contrib-prs:v2
env:
GH_APP_ID: ${{ secrets.GH_APP_ID }}
GH_APP_INSTALLATION_ID: ${{ secrets.GH_APP_INSTALLATION_ID }}
GH_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }}
# GITHUB_APP_ENTERPRISE_ONLY: True --> Set to true when created GHE App needs to communicate with GHE api
GH_ENTERPRISE_URL: ${{ github.server_url }}
# GH_TOKEN: ${{ secrets.GH_TOKEN }} --> the token input is not used if the github app inputs are set
ORGANIZATION: ${{ secrets.ORGANIZATION }}
GH_ACTOR: ${{ secrets.GH_ACTOR }}
PR_TITLE: ${{ secrets.PR_TITLE }}
PR_BODY: ${{ secrets.PR_BODY }}
```
## Scaling for large organizations
- GitHub Actions workflows have time limits currently set at 72 hours per run. If you are operating on more than 1400 repos or so with this action, it will take several runs to complete.

64
auth.py
Просмотреть файл

@ -1,42 +1,42 @@
"""
This is the module that contains functions related to authenticating to GitHub with
a personal access token or GitHub App, depending on the environment variables set.
"""
"""This is the module that contains functions related to authenticating to GitHub with a personal access token."""
import github3
import requests
def auth_to_github(
token: str | None,
gh_app_id: int | None,
gh_app_installation_id: int | None,
gh_app_private_key_bytes: bytes,
gh_enterprise_url: str | None,
token: str | None,
ghe: str | None,
gh_app_enterprise_only: bool,
) -> github3.GitHub:
"""
Connect to GitHub.com or GitHub Enterprise, depending on env variables.
Args:
gh_app_id (int | None): the GitHub App ID
gh_installation_id (int | None): the GitHub App Installation ID
gh_app_private_key (bytes): the GitHub App Private Key
gh_enterprise_url (str): the GitHub Enterprise URL
token (str): the GitHub personal access token
gh_app_id (int | None): the GitHub App ID
gh_app_installation_id (int | None): the GitHub App Installation ID
gh_app_private_key_bytes (bytes): the GitHub App Private Key
ghe (str): the GitHub Enterprise URL
gh_app_enterprise_only (bool): Set this to true if the GH APP is created on GHE and needs to communicate with GHE api only
Returns:
github3.GitHub: the GitHub connection object
"""
if gh_app_id and gh_app_private_key_bytes and gh_app_installation_id:
gh = github3.github.GitHub()
if ghe and gh_app_enterprise_only:
gh = github3.github.GitHubEnterprise(url=ghe)
else:
gh = github3.github.GitHub()
gh.login_as_app_installation(
gh_app_private_key_bytes, gh_app_id, gh_app_installation_id
)
github_connection = gh
elif gh_enterprise_url and token:
github_connection = github3.github.GitHubEnterprise(
gh_enterprise_url, token=token
)
elif ghe and token:
github_connection = github3.github.GitHubEnterprise(url=ghe, token=token)
elif token:
github_connection = github3.login(token=token)
else:
@ -47,3 +47,35 @@ def auth_to_github(
if not github_connection:
raise ValueError("Unable to authenticate to GitHub")
return github_connection # type: ignore
def get_github_app_installation_token(
ghe: str | None,
gh_app_id: int | None,
gh_app_private_key_bytes: bytes,
gh_app_installation_id: int | None,
) -> str | None:
"""
Get a GitHub App Installation token.
API: https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-as-a-github-app-installation
Args:
ghe (str): the GitHub Enterprise endpoint
gh_app_id (str): the GitHub App ID
gh_app_private_key_bytes (bytes): the GitHub App Private Key
gh_app_installation_id (str): the GitHub App Installation ID
Returns:
str: the GitHub App token
"""
jwt_headers = github3.apps.create_jwt_headers(gh_app_private_key_bytes, gh_app_id)
api_endpoint = f"{ghe}/api/v3" if ghe else "https://api.github.com"
url = f"{api_endpoint}/app/installations/{gh_app_installation_id}/access_tokens"
try:
response = requests.post(url, headers=jwt_headers, json=None, timeout=5)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Request failed: {e}")
return None
return response.json().get("token")

25
env.py
Просмотреть файл

@ -11,6 +11,22 @@ MAX_TITLE_LENGTH = 70
MAX_BODY_LENGTH = 65536
def get_bool_env_var(env_var_name: str, default: bool = False) -> bool:
"""Get a boolean environment variable.
Args:
env_var_name: The name of the environment variable to retrieve.
default: The default value to return if the environment variable is not set.
Returns:
The value of the environment variable as a boolean.
"""
ev = os.environ.get(env_var_name, "")
if ev == "" and default:
return default
return ev.strip().lower() == "true"
def get_int_env_var(env_var_name: str, default: int = -1) -> int | None:
"""Get an integer environment variable.
@ -39,8 +55,9 @@ class EnvVars:
gh_app_id (int | None): The GitHub App ID to use for authentication
gh_app_installation_id (int | None): The GitHub App Installation ID to use for authentication
gh_app_private_key_bytes (bytes): The GitHub App Private Key as bytes to use for authentication
gh_token (str | None): GitHub personal access token (PAT) for API authentication
gh_app_enterprise_only (bool): Set this to true if the GH APP is created on GHE and needs to communicate with GHE api only
ghe (str): The GitHub Enterprise URL to use for authentication
gh_token (str | None): GitHub personal access token (PAT) for API authentication
gh_actor (str): The GitHub actor to use for authentication
organization (str): The GitHub organization to use for the PR
pr_body (str): The PR body to use for the PR
@ -54,6 +71,7 @@ class EnvVars:
gh_app_id: int | None,
gh_app_installation_id: int | None,
gh_app_private_key_bytes: bytes,
gh_app_enterprise_only: bool,
gh_enterprise_url: str | None,
gh_token: str | None,
organization: str | None,
@ -65,6 +83,7 @@ class EnvVars:
self.gh_app_id = gh_app_id
self.gh_app_installation_id = gh_app_installation_id
self.gh_app_private_key_bytes = gh_app_private_key_bytes
self.gh_app_enterprise_only = gh_app_enterprise_only
self.gh_enterprise_url = gh_enterprise_url
self.gh_token = gh_token
self.organization = organization
@ -79,6 +98,7 @@ class EnvVars:
f"{self.gh_app_id},"
f"{self.gh_app_installation_id},"
f"{self.gh_app_private_key_bytes},"
f"{self.gh_app_enterprise_only}"
f"{self.gh_enterprise_url},"
f"{self.gh_token},"
f"{self.organization},"
@ -100,6 +120,7 @@ def get_env_vars(test: bool = False) -> EnvVars:
gh_app_id (int | None): The GitHub App ID to use for authentication
gh_app_installation_id (int | None): The GitHub App Installation ID to use for authentication
gh_app_private_key_bytes (bytes): The GitHub App Private Key as bytes to use for authentication
gh_app_enterprise_only (bool): Set this to true if the GH APP is created on GHE and needs to communicate with GHE api only
gh_enterprise_url (str): The GitHub Enterprise URL to use for authentication
gh_token (str | None): The GitHub token to use for authentication
organization (str): The GitHub organization to use for the PR
@ -116,6 +137,7 @@ def get_env_vars(test: bool = False) -> EnvVars:
gh_app_id = get_int_env_var("GH_APP_ID")
gh_app_private_key_bytes = os.environ.get("GH_APP_PRIVATE_KEY", "").encode("utf8")
gh_app_installation_id = get_int_env_var("GH_APP_INSTALLATION_ID")
gh_app_enterprise_only = get_bool_env_var("GITHUB_APP_ENTERPRISE_ONLY")
if gh_app_id and (not gh_app_private_key_bytes or not gh_app_installation_id):
raise ValueError(
@ -157,6 +179,7 @@ def get_env_vars(test: bool = False) -> EnvVars:
gh_app_id,
gh_app_installation_id,
gh_app_private_key_bytes,
gh_app_enterprise_only,
gh_enterprise_url,
gh_token,
organization,

27
open_contrib_pr.py
Просмотреть файл

@ -17,21 +17,34 @@ if __name__ == "__main__":
pr_title = env_vars.pr_title
repos_json_location = env_vars.repos_json_location
token = env_vars.gh_token
gh_app_id = env_vars.gh_app_id
gh_app_installation_id = env_vars.gh_app_installation_id
gh_app_private_key_bytes = env_vars.gh_app_private_key_bytes
ghe = env_vars.gh_enterprise_url
gh_app_enterprise_only = env_vars.gh_app_enterprise_only
# Auth to GitHub.com
github_connection = auth.auth_to_github(
env_vars.gh_app_id,
env_vars.gh_app_installation_id,
env_vars.gh_app_private_key_bytes,
env_vars.gh_enterprise_url,
token,
gh_app_id,
gh_app_installation_id,
gh_app_private_key_bytes,
ghe,
gh_app_enterprise_only,
)
if not token and gh_app_id and gh_app_installation_id and gh_app_private_key_bytes:
token = auth.get_github_app_installation_token(
ghe, gh_app_id, gh_app_private_key_bytes, gh_app_installation_id
)
endpoint = ghe if ghe else "github.com"
os.system("git config --global user.name 'GitHub Actions'")
os.system("git config --global user.email 'no-reply@github.com'")
os.system(f"git config --global user.email 'no-reply@{endpoint}'")
# Get innersource repos from organization
os.system(f"git clone https://{gh_actor}:{token}@github.com/{repos_json_location}")
os.system(f"git clone https://{gh_actor}:{token}@{endpoint}/{repos_json_location}")
with open(str(repos_json_location), "r", encoding="utf-8") as repos_file:
innersource_repos = json.loads(repos_file.read())
@ -46,7 +59,7 @@ if __name__ == "__main__":
repo_full_name = repo["full_name"]
repo_name = repo["name"]
os.system(
f"git clone https://{gh_actor}:{token}@github.com/{repo_full_name}"
f"git clone https://{gh_actor}:{token}@{endpoint}/{repo_full_name}"
)
# checkout a branch called contributing-doc
BRANCH_NAME = "contributing-doc"

78
test_auth.py
Просмотреть файл

@ -4,33 +4,23 @@ import unittest
from unittest.mock import MagicMock, patch
import auth
import github3
class TestAuthToGitHub(unittest.TestCase):
class TestAuth(unittest.TestCase):
"""
Test case for the auth module.
"""
def test_auth_to_github_with_token(self):
@patch("github3.login")
def test_auth_to_github_with_token(self, mock_login):
"""
Test the auth_to_github function when the token is provided.
"""
mock_login.return_value = "Authenticated to GitHub.com"
result = auth.auth_to_github("", "", b"", "", "token")
result = auth.auth_to_github("token", "", "", b"", "", False)
self.assertIsInstance(result, github3.github.GitHub)
@patch("github3.github.GitHub.login_as_app_installation")
def test_auth_to_github_with_github_app(self, mock_login):
"""
Test the auth_to_github function when the token is provided.
"""
mock_login.return_value = MagicMock()
result = auth.auth_to_github("12345", "678910", b"hello", "", "")
self.assertIsInstance(result, github3.github.GitHub)
self.assertEqual(result, "Authenticated to GitHub.com")
def test_auth_to_github_without_token(self):
"""
@ -38,20 +28,68 @@ class TestAuthToGitHub(unittest.TestCase):
Expect a ValueError to be raised.
"""
with self.assertRaises(ValueError) as context_manager:
auth.auth_to_github("", "", b"", "", "")
auth.auth_to_github("", "", "", b"", "", False)
the_exception = context_manager.exception
self.assertEqual(
str(the_exception),
"GH_TOKEN or the set of [GH_APP_ID, GH_APP_INSTALLATION_ID, GH_APP_PRIVATE_KEY] environment variables are not set",
)
def test_auth_to_github_with_github_enterprise_url(self):
@patch("github3.github.GitHubEnterprise")
def test_auth_to_github_with_ghe(self, mock_ghe):
"""
Test the auth_to_github function when the GitHub Enterprise URL is provided.
"""
result = auth.auth_to_github("", "", b"", "https://github.example.com", "token")
mock_ghe.return_value = "Authenticated to GitHub Enterprise"
result = auth.auth_to_github(
"token", "", "", b"", "https://github.example.com", False
)
self.assertIsInstance(result, github3.github.GitHubEnterprise)
self.assertEqual(result, "Authenticated to GitHub Enterprise")
@patch("github3.github.GitHubEnterprise")
def test_auth_to_github_with_ghe_and_ghe_app(self, mock_ghe):
"""
Test the auth_to_github function when the GitHub Enterprise URL is provided and the app was created in GitHub Enterprise URL.
"""
mock = mock_ghe.return_value
mock.login_as_app_installation = MagicMock(return_value=True)
result = auth.auth_to_github(
"", "123", "123", b"123", "https://github.example.com", True
)
mock.login_as_app_installation.assert_called_once()
self.assertEqual(result, mock)
@patch("github3.github.GitHub")
def test_auth_to_github_with_app(self, mock_gh):
"""
Test the auth_to_github function when app credentials are provided
"""
mock = mock_gh.return_value
mock.login_as_app_installation = MagicMock(return_value=True)
result = auth.auth_to_github(
"", "123", "123", b"123", "https://github.example.com", False
)
mock.login_as_app_installation.assert_called_once()
self.assertEqual(result, mock)
@patch("github3.apps.create_jwt_headers", MagicMock(return_value="gh_token"))
@patch("requests.post")
def test_get_github_app_installation_token(self, mock_post):
"""
Test the get_github_app_installation_token function.
"""
dummy_token = "dummytoken"
mock_response = MagicMock()
mock_response.raise_for_status.return_value = None
mock_response.json.return_value = {"token": dummy_token}
mock_post.return_value = mock_response
result = auth.get_github_app_installation_token(
b"ghe", "gh_private_token", "gh_app_id", "gh_installation_id"
)
self.assertEqual(result, dummy_token)
if __name__ == "__main__":

92
test_env.py
Просмотреть файл

@ -1,10 +1,12 @@
"""Test the get_env_vars function"""
import os
import random
import string
import unittest
from unittest.mock import patch
from env import EnvVars, get_env_vars
from env import MAX_BODY_LENGTH, MAX_TITLE_LENGTH, EnvVars, get_env_vars
BODY = "example CONTRIBUTING file contents"
ORGANIZATION = "Organization01"
@ -22,6 +24,7 @@ class TestEnv(unittest.TestCase):
"GH_APP_ID",
"GH_APP_INSTALLATION_ID",
"GH_APP_PRIVATE_KEY",
"GITHUB_APP_ENTERPRISE_ONLY",
"GH_ENTERPRISE_URL",
"GH_TOKEN",
"ORGANIZATION",
@ -40,6 +43,7 @@ class TestEnv(unittest.TestCase):
"GH_APP_ID": "",
"GH_APP_INSTALLATION_ID": "",
"GH_APP_PRIVATE_KEY": "",
"GITHUB_APP_ENTERPRISE_ONLY": "",
"GH_ENTERPRISE_URL": "",
"GH_TOKEN": TOKEN,
"ORGANIZATION": ORGANIZATION,
@ -54,6 +58,7 @@ class TestEnv(unittest.TestCase):
None,
None,
b"",
False,
"",
TOKEN,
ORGANIZATION,
@ -71,6 +76,7 @@ class TestEnv(unittest.TestCase):
"GH_APP_ID": "12345",
"GH_APP_INSTALLATION_ID": "678910",
"GH_APP_PRIVATE_KEY": "hello",
"GITHUB_APP_ENTERPRISE_ONLY": "",
"GH_ENTERPRISE_URL": "",
"GH_TOKEN": "",
"ORGANIZATION": ORGANIZATION,
@ -86,6 +92,7 @@ class TestEnv(unittest.TestCase):
12345,
678910,
b"hello",
False,
"",
"",
ORGANIZATION,
@ -103,6 +110,7 @@ class TestEnv(unittest.TestCase):
"GH_APP_ID": "",
"GH_APP_INSTALLATION_ID": "",
"GH_APP_PRIVATE_KEY": "",
"GITHUB_APP_ENTERPRISE_ONLY": "",
"GH_ENTERPRISE_URL": "testghe",
"GH_TOKEN": TOKEN,
"ORGANIZATION": ORGANIZATION,
@ -118,6 +126,7 @@ class TestEnv(unittest.TestCase):
None,
None,
b"",
False,
"testghe",
TOKEN,
ORGANIZATION,
@ -155,6 +164,87 @@ class TestEnv(unittest.TestCase):
"ORGANIZATION environment variable not set",
)
@patch.dict(
os.environ,
{
"ORGANIZATION": "my_organization",
"GH_APP_ID": "12345",
"GH_APP_INSTALLATION_ID": "",
"GH_APP_PRIVATE_KEY": "",
"GH_TOKEN": "",
},
clear=True,
)
def test_get_env_vars_auth_with_github_app_installation_missing_inputs(self):
"""Test that an error is raised when there are missing inputs for the gh app"""
with self.assertRaises(ValueError) as context_manager:
get_env_vars(True)
the_exception = context_manager.exception
self.assertEqual(
str(the_exception),
"GH_APP_ID set and GH_APP_INSTALLATION_ID or GH_APP_PRIVATE_KEY variable not set",
)
@patch.dict(
os.environ,
{
"ORGANIZATION": "",
"GH_TOKEN": "test",
},
clear=True,
)
def test_get_env_vars_no_organization_set(self):
"""Test that an error is raised whenthere are missing inputs for the gh app"""
with self.assertRaises(ValueError) as context_manager:
get_env_vars(True)
the_exception = context_manager.exception
self.assertEqual(
str(the_exception),
"ORGANIZATION environment variable not set",
)
@patch.dict(
os.environ,
{
"ORGANIZATION": "my_organization",
"GH_TOKEN": "test",
"PR_TITLE": "".join(
random.choices(string.ascii_letters, k=MAX_TITLE_LENGTH + 1)
),
},
clear=True,
)
def test_get_env_vars_pr_title_too_long(self):
"""Test that an error is raised when the PR_TITLE env variable has more than MAX_TITLE_LENGTH characters"""
with self.assertRaises(ValueError) as context_manager:
get_env_vars(True)
the_exception = context_manager.exception
self.assertEqual(
str(the_exception),
f"PR_TITLE environment variable is too long. Max {MAX_TITLE_LENGTH} characters",
)
@patch.dict(
os.environ,
{
"ORGANIZATION": "my_organization",
"GH_TOKEN": "test",
"PR_BODY": "".join(
random.choices(string.ascii_letters, k=MAX_BODY_LENGTH + 1)
),
},
clear=True,
)
def test_get_env_vars_pr_body_too_long(self):
"""Test that an error is raised when the PR_BODY env variable has more than MAX_BODY_LENGTH characters"""
with self.assertRaises(ValueError) as context_manager:
get_env_vars(True)
the_exception = context_manager.exception
self.assertEqual(
str(the_exception),
f"BODY environment variable is too long. Max {MAX_BODY_LENGTH} characters",
)
if __name__ == "__main__":
unittest.main()