diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 5c9fdd3..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,46 +0,0 @@ -# Contributing to LGTM Issue Tracker Example - -We welcome contributions to our example LGTM Issue Tracker. While we intend this project to remain a minimal pedagogical example, if you have an idea how it could be made clearer or more valuable to other users, then please go ahead an open a Pull Request! - -Before we accept your pull request, we will require that you have agreed to our Contributor License Agreement, this is not something that you need to do before you submit your pull request, but until you've done so, we will be unable to accept your contribution. - -## Using your personal data - -If you contribute to this project, we will record your name and email -address (as provided by you with your contributions) as part of the code -repositories, which might be made public. We might also use this information -to contact you in relation to your contributions, as well as in the -normal course of software development. We also store records of your -CLA agreements. Under GDPR legislation, we do this -on the basis of our legitimate interest in creating the QL product. - -Please do get in touch (privacy@semmle.com) if you have any questions about -this or our data protection policies. - -## Contributor License Agreement - -This Contributor License Agreement (“Agreement”) is entered into between Semmle Limited (“Semmle,” “we” or “us” etc.), and You (as defined and further identified below). - -Accordingly, You hereby agree to the following terms for Your present and future Contributions submitted to Semmle: - -1. **Definitions**. - - * "You" (or "Your") shall mean the Contribution copyright owner (whether an individual or organization) or legal entity authorized by the copyright owner that is making this Agreement with Semmle. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. - - * "Contribution(s)" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, submitted by You to Semmle for inclusion in, or documentation of, any of the products or projects owned or managed by Semmle (the "Work(s)"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to Semmle or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Semmle for the purpose of discussing and/or improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." - -2. **Grant of Copyright License**. You hereby grant to Semmle and to recipients of software distributed by Semmle a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. - -3. **Grant of Patent License**. You hereby grant to Semmle and to recipients of software distributed by Semmle a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that Your Contribution, or the Work to which You have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. - -4. **Ownership**. Except as set out above, You keep all right, title, and interest in Your Contribution. The rights that You grant to us under this Agreement are effective on the date You first submitted a Contribution to us, even if Your submission took place before the date You entered this Agreement. - -5. **Representations**. You represent and warrant that: (i) the Contributions are an original work and that You can legally grant the rights set out in this Agreement; (ii) the Contributions and Semmle’s exercise of any license rights granted hereunder, does not and will not, infringe the rights of any third party; (iii) You are not aware of any pending or threatened claims, suits, actions, or charges pertaining to the Contributions, including without limitation any claims or allegations that any or all of the Contributions infringes, violates, or misappropriate the intellectual property rights of any third party (You further agree that You will notify Semmle immediately if You become aware of any such actual or potential claims, suits, actions, allegations or charges). - -6. **Employer**. If Your employer(s) has rights to intellectual property that You create that includes Your Contributions, You represent and warrant that Your employer has waived such rights for Your Contributions to Semmle, or that You have received permission to make Contributions on behalf of that employer and that You are authorized to execute this Agreement on behalf of Your employer. - -7. **Inclusion of Code**. We determine the code that is in our Works. You understand that the decision to include the Contribution in any project or source repository is entirely that of Semmle, and this agreement does not guarantee that the Contributions will be included in any product. - -8. **Disclaimer**. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Except as set forth herein, and unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND. - -9. **General**. The failure of either party to enforce its rights under this Agreement for any period shall not be construed as a waiver of such rights. No changes or modifications or waivers to this Agreement will be effective unless in writing and signed by both parties. In the event that any provision of this Agreement shall be determined to be illegal or unenforceable, that provision will be limited or eliminated to the minimum extent necessary so that this Agreement shall otherwise remain in full force and effect and enforceable. This Agreement shall be governed by and construed in accordance with the laws of the State of California in the United States without regard to the conflicts of laws provisions thereof. In any action or proceeding to enforce rights under this Agreement, the prevailing party will be entitled to recover costs and attorneys’ fees. \ No newline at end of file diff --git a/README.md b/README.md index c2bcee9..ee45337 100644 --- a/README.md +++ b/README.md @@ -1,50 +1,125 @@ -# LGTM issue tracker integration example -[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black) [![Total alerts](https://img.shields.io/lgtm/alerts/g/Semmle/lgtm-issue-tracker-example.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Semmle/lgtm-issue-tracker-example/alerts/) +# `gh2jira` - Synchronize GitHub Code Scanning alerts and JIRA issues -## Issue tracking in LGTM Enterprise - -[LGTM Enterprise](https://semmle.com/lgtm) gives customers the option of exporting alerts to any issue tracker, by sending webhook POST requests to the external service. Semmle provides an existing full-featured [add-on for Atlassian Jira](https://github.com/Semmle/lgtm-jira-addon), but other issue trackers need a lightweight application to act as translator for LGTM. This repository provides a basic example of how such an application might be implemented. - -Integration with external issue trackers is not available to users of [LGTM.com](https://lgtm.com). +[GitHub's REST API](https://docs.github.com/en/rest) and [webhooks](https://docs.github.com/en/developers/webhooks-and-events/about-webhooks) give customers the option of exporting alerts to any issue tracker, by allowing users to fetch the data via API endpoints and / or by receiving webhook POST requests to a hosted server. ## This repository -This project gives a quick illustrative example of how to integrate LGTM Enterprise with a third-party issue tracker. This code is intended as a proof-of-concept only, showing the basic operations necessary to handle incoming requests from LGTM. It is not intended for production use. Please feel free to use this as a starting point for your own integration, but if you are using Atlassian Jira see also the [LGTM Jira add-on](https://github.com/Semmle/lgtm-jira-addon). +This repository gives a quick illustrative example of how to integrate GitHub Code Scanning with a third-party issue tracker - in this case JIRA Server. The example does not work with the cloud version of JIRA! The code is intended as a proof-of-concept, showing the basic operations necessary to handle incoming requests from GitHub. It is not intended for production use. Please feel free to use this as a starting point for your own integration. -We use a lightweight `Flask` server to handle incoming requests, which in turn writes to the issue tracker of a specified GitHub repository. When not run in debug mode, incoming requests are verified using the secret specified when configuring the integration. For a more detailed explanation please see the associated [tutorial](tutorial.md). +## Using the GitHub Action -For instructions on configuring your LGTM Enterprise instance, please see the relevant [LGTM help pages](https://help.semmle.com/lgtm-enterprise/admin/help/adding-issue-trackers.html). +The easiest way to use this tool is via its GitHub Action, which you can add to your workflows. Here is what you need before you can start: -Integration with the GitHub issue tracker requires an access token for the GitHub installation, with appropriate permissions. +* A GitHub repository with Code Scanning enabled and a few alerts. Follow [this guide](https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/setting-up-code-scanning-for-a-repository) to set up Code Scanning. +* A GitHub `personal access token`, so that the action can fetch alerts from your repository. It might be sufficient to use `${{ secrets.GITHUB_TOKEN }}`, which is a token that GitHub automatically generates for your workflows. If not, follow [this guide](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) to obtain a dedicated token. It will have to have at least the `security_events` scope. +* The URL of your JIRA Server instance. +* A [JIRA project](https://confluence.atlassian.com/adminjiraserver/creating-a-project-938846813.html) to store your issues. You will need to provide its `project key` to the action. +* A JIRA Server account (username + password) with the following permissions for the abovementioned project: + * `Browse Projects` + * `Close Issues` + * `Create Issues` + * `Delete Issues` + * `Edit Issues` + * `Transition Issues` +* Depending on where you run your workflow, the JIRA Server instance must be accessible from either the [GitHub.com IP addresses](https://docs.github.com/en/github/authenticating-to-github/about-githubs-ip-addresses) or the address of your GitHub Enterprise Server instance. -## Configuration +Make sure you safely store all credentials as [GitHub Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets). Finally, set up the following workflow in your repository, e.g. by adding the file `.github/workflows/jira-sync.yml`: -When run through `pipenv` the app pull its configuration from the `.env` file, for which an example is provided: -```bash -FLASK_APP=issues.py -FLASK_DEBUG=0 +```yaml +name: "Sync with JIRA" -GIT_REPO_URL=https://github.com/api/v3/repos/USERNAME/REPO/issues -GIT_ACCESS_TOKEN=PERSONAL_ACCESS_TOKEN +on: + schedule: + - cron: '*/10 * * * *' # trigger synchronization every 10 minutes -LGTM_SECRET=SECRET_AS_SPECIFIED_IN_LGTM_INTEGRATION_PANEL +jobs: + test_job: + runs-on: ubuntu-latest + steps: + - name: Sync with JIRA + uses: johnlugton/lgtm-issue-tracker-example@master + with: + github_token: '${{ secrets.GITHUB_TOKEN }}' + jira_url: '' + jira_user: '${{ secrets.JIRA_USER }}' + jira_token: '${{ secrets.JIRA_TOKEN }}' + jira_project: '' + sync_direction: 'gh2jira' ``` -## Running -The easiest way to get the app running is with `pipenv`. Obviously a proper deployment would require something other than the built-in `Flask` development server. +This action will push any changes (new alerts, alerts deleted, alert states changed) to JIRA, by creating, deleting or changing the state of the corresponding JIRA issues. If you set `sync_direction` to `jira2gh`, it will synchronize the other way. Currently, two-way integration is not yet possible via the action. If you need it, use the CLI's `serve` command (see below). -Note: This example project requires a minimum of `python3.5`. +## Using the CLI's `sync` command + +### Installation + +The easiest way to get the CLI running is with `pipenv`: ```bash pipenv install -pipenv run flask run +pipenv ./gh2jira --help ``` +Note: `gh2jira` requires a minimum of `python3.5`. + +In addition to the [usual requirements](#using-the-github-action) you also need the URL for the GitHub API, which is: +* https://api.github.com if your repository is located on GitHub.com +* https://your-hostname/api/v3/ if your repository is located on a GitHub Server instance + +```bash +pipenv ./gh2jira sync + --gh-url "" + --gh-token "" + --gh-org "" + --gh-repo "" + --jira-url "" + --jira-user "" + --jira-token "" + --jira-project "" + --direction gh2jira +``` + +Note: Instead of the `--gh-token` and `--jira-token` options, you may also set the `GH2JIRA_GH_TOKEN` and `GH2JIRA_JIRA_TOKEN` environment variables. +The above command could be invoked via a cronjob every X minutes, to make sure issues and alerts are kept in sync. Currently, two-way integration is not yet possible via this command. If you need it, use the CLI's `serve` command (see below). + +## Using the CLI's `serve` command + +The following method is the most involved one, but currently the only one which allows two-way integration (i.e. changes to Code Scanning alerts trigger changes to JIRA issues and vice versa). It uses a lightweight `Flask` server to handle incoming JIRA and GitHub webhooks. The server is meant to be an example and not production-ready. + +In addition to the [usual requirements](#using-the-github-action) you also need: +* A machine with an address that can be reached from GitHub.com or your GitHub Enterprise Server instance and your JIRA Server instance. This machine will run the server. +* Webhooks set up, both, on GitHub and JIRA. On GitHub only repository or organization owners can do so. On JIRA it requires administrator access. +* A secret which will be used to verify webhook requests. + +First, [create a GitHub webhook](https://docs.github.com/en/developers/webhooks-and-events/creating-webhooks) with the following event triggers: +* [Code scanning alerts](https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads#code_scanning_alert) +* [Repositories](https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads#repository) + +This can be either a repository or an organization-wide hook. Set the `Payload URL` to `https:///github`, the `Content type` to `application/json` and insert your webhook `Secret`. Make sure to `Enable SSL verification`. + +Second, [register a webhook on JIRA](https://developer.atlassian.com/server/jira/platform/webhooks/#registering-a-webhook). Give your webhook a `Name` and enter the `URL`: `https:///jira?secret_token=`. In the `Events` section specify `All issues` and mark the boxes `created`, `updated` and `deleted`. Click `Save`. + +Finally, start the server: + +```bash +pipenv ./gh2jira serve + --gh-url "" + --gh-token "" + --jira-url "" + --jira-user "" + --jira-token "" + --jira-project "" + --secret "" + --port 5000 + --direction both +``` + +This will enable two-way integration between GitHub and JIRA. Note: Instead of the `--secret` option, you may also set the `GH2JIRA_SECRET` environment variable. + ## Contributing -We welcome contributions to our example LGTM issue tracker integration. While we intend this project to remain a minimal pedagogical example, if you have an idea how it could be made clearer or more valuable to other users, then please go ahead an open a pull request! Before you do, though, please take the time to read our [contributing guidelines](CONTRIBUTING.md). +To be determined. ## License -The LGTM issue tracker integration example is licensed under [Apache License 2.0](LICENSE) by [Semmle](https://semmle.com). - +To be determined. diff --git a/tutorial.md b/tutorial.md deleted file mode 100644 index 75487c6..0000000 --- a/tutorial.md +++ /dev/null @@ -1,296 +0,0 @@ - -# Integrating a third-party issue tracker with LGTM - -To enable the creation of tickets in third-party issue trackers, LGTM Enterprise offers a lightweight webhook integration. When enabled, this will send outgoing POST requests to a specified endpoint, detailing new and changed alerts. - -We are going to outline a barebones implementation of a webapp that will process incoming requests from LGTM Enterprise, create issues in a specified GitHub repository and pass an appropriate response back to LGTM. We will implement our example for Python 3.5+ and use two third-party modules, `Flask` (a micro web framework) and `requests` (a user-friendly HTTP library). Both are extremely widely used and can be easily installed using `pip`. -```bash -pip install flask -pip install requests -``` - -## Basic Flask app - -For production-ready deployment it is important to consider both robustness and security, but in this tutorial we will focus on a minimal functioning implementation, demonstrating how to process the data from LGTM and integrate this with an example issue tracker. Similarly, we will avoid validation and error handling in this basic tutorial, and just assume for now that all the requests to the webhook are valid. - -The following is a basic 'hello world'-esque Flask app, which accepts POST requests and echoes the incoming JSON back to the sender. - -```python -from flask import Flask, request - -app = Flask(__name__) - -@app.route('/', methods=["POST"]) -def issues_webhook(): - return request.data, 200 - -if __name__ == "__main__": - app.run() -``` -Let's assume you've saved this file as `flask_testing.py`. Using the built-in WSGI development server that comes with Flask, you can run the application by simply executing the Python file. -```bash -python flask_testing.py -``` -When successfully executed, a service will be operating on localhost:5000 that will echo all POSTed JSON. - -## Format of webhook request - -All requests from LGTM to the specified webhook endpoint are of the HTTP method POST, and they fall into three categories. -- `create` -- `close` -- `reopen` -- `suppress` -- `unsuppress` - -For this tutorial we will focus just on the basic operations of opening and closing tickets. - -### Creating a new ticket - -A full example of a `create` request payload is given below. - -```json -{ - "transition": "create", - "project": { - "id": 1000001, - "url-identifier": "Git/example_user/example_repo", - "name": "example_user/example_repo", - "url": "http://lgtm.com/projects/Git/example_user/example_repo" - }, - "alert": { - "file": "/example.py", - "message": "Import of \"re\" is not used.\n", - "url": "http://lgtm.com/issues/1000001/python/8cdXzW+PyA3qiHBbWFomoMGtiIE=", - "query": { - "name": "Unused import", - "url": "http://lgtm.com/rules/1000678" - } - } -} -``` -With Flask, the payload of an incoming request can be accessed using the following utility function. -```python -json_dict = request.get_json() -``` -The GitHub API expects JSON with fields `title`, `body` and `labels`, and the body of the ticket can be formatted as markdown. For our example application, the following function takes `alert` and `project` from the LGTM payload and creates a dictionary that can be JSON serialized and then sent on to the correct GitHub endpoint. In this case we choose to just apply the single default label `LGTM` to all tickets. - -```python -def get_issue_dict(alert, project): - - title = "%s (%s)" % (alert["query"]["name"], project["name"]) - - lines = [] - lines.append("[%s](%s)" % (alert["query"]["name"], alert["query"]["url"])) - lines.append("") - lines.append("In %s:" % alert["file"]) - lines.append("> " + "\n> ".join(alert["message"].split("\n"))) - lines.append("[View alert on LGTM](%s)" % alert["url"]) - - return {"title": title, "body": "\n".join(lines), "labels": ["LGTM"]} -``` - -To interact with the GitHub API we use the `requests` module, and define the following details for the target GitHub repository, pulling the access token from an environment variable. -```python -URL = 'https://github.com/api/v3/repos/user/repo/issues' -HEADERS = {'content-type': 'application/json', - 'Authorization': 'Bearer %s' % os.getenv("GIT_ACCESS_TOKEN") - } -``` - -LGTM expects a 2XX HTTP response of the form shown below, where the `issue_id` provided will be stored and used in future requests to change the state of the ticket. -```json -{ - "issue-id": external_issue_id -} -``` - -Putting all of this together, to allow it to handle incoming `create` requests, our example app becomes: - -```python -import os -from flask import Flask, request, jsonify -import requests - -app = Flask(__name__) - -URL = 'https://github.com/api/v3/repos/user/repo/issues' -HEADERS = {'content-type': 'application/json', - 'Authorization': 'Bearer %s' % os.getenv("GIT_ACCESS_TOKEN") - } - -def get_issue_dict(alert, project): - - title = "%s (%s)" % (alert["query"]["name"], project["name"]) - - lines = [] - lines.append("[%s](%s)" % (alert["query"]["name"], alert["query"]["url"])) - lines.append("") - lines.append("In %s:" % alert["file"]) - lines.append("> " + "\n> ".join(alert["message"].split("\n"))) - lines.append("[View alert on LGTM](%s)" % alert["url"]) - - return {"title": title, "body": "\n".join(lines), "labels": ["LGTM"]} - -@app.route('/', methods=["POST"]) -def issues_webhook(): - - json_dict = request.get_json() - - transition = json_dict.get('transition') - - if transition == 'create': - - data = get_issue_dict(json_dict.get('alert'), json_dict.get('project')) - - r = requests.post(URL, json=data, headers=HEADERS) - - issue_id = r.json()['number'] - - return jsonify({'issue-id': issue_id}), r.status_code - -if __name__ == "__main__": - app.run() -``` - -### Changing an existing ticket - -When closing an existing ticket, LGTM sends a request of the form: - -```json -{ - "issue-id": external_issue_id, - "transition": "close" -} -``` -This can be handled by sending a `PATCH` request to the existing Github issue. -```python -if transition == 'create': - ######## -if transition == 'close': - - issue_id = json_dict.get('issue-id') - - r = requests.patch(URL + '/' + issue_id, - json={"state": transition}, - headers=HEADERS) - - return jsonify({'issue-id': issue_id}), r.status_code -``` -When reopening a ticket, the request will be of the same form, except with the transition `reopen`. This can be handled in a similar way to closing a ticket, but with Github expecting the state to be given as `open`. -```python -if transition == 'create': - ######## -if transition == 'close': - ######## -if transition == 'reopen': - - issue_id = json_dict.get('issue-id') - - r = requests.patch(URL + '/' + issue_id, - json={"state": 'open'}, # github expects `open` - headers=HEADERS) - - return jsonify({'issue-id': issue_id}), r.status_code -``` - -### Authorization - -When setting up the issue tracker integration a secret key is automatically generated, and this is used to crytographically sign all outgoing requests. These are signed in the same way as callbacks for pull request integrations—for more information, see [verify-callback-signature](https://lgtm.com/help/lgtm/api/run-code-review#verify-callback-signature) in the LGTM help. Verification of the incoming requests can therefore be easily achieved as follows. - -```python -import hashlib -import hmac - -KEY = os.getenv("LGTM_SECRET", '').encode('utf-8') - -digest = hmac.new(KEY, request.data, hashlib.sha1).hexdigest() -signature = request.headers.get('X-LGTM-Signature', "not-provided") - -if not hmac.compare_digest(signature, digest): - return jsonify({'message': "Unauthorized"}), 401 -``` - -## Full example - -Finally, putting all these pieces together, we have the following example Flask app, which handles webhook requests from the LGTM issue tracker integration, and creates tickets in the issue tracker of a specified GitHub repository. - -```python -import os -from flask import Flask, request, jsonify -import requests -import hashlib -import hmac - -app = Flask(__name__) - -URL = 'https://github.com/api/v3/repos/user/repo/issues' -HEADERS = {'content-type': 'application/json', - 'Authorization': 'Bearer %s' % os.getenv("GIT_ACCESS_TOKEN")} -KEY = os.getenv("LGTM_SECRET", '').encode('utf-8') - -def get_issue_dict(alert, project): - - title = "%s (%s)" % (alert["query"]["name"], project["name"]) - - lines = [] - lines.append("[%s](%s)" % (alert["query"]["name"], alert["query"]["url"])) - lines.append("") - lines.append("In %s:" % alert["file"]) - lines.append("> " + "\n> ".join(alert["message"].split("\n"))) - lines.append("[View alert on LGTM](%s)" % alert["url"]) - - return {"title": title, "body": "\n".join(lines), "labels": ["LGTM"]} - -@app.route('/', methods=["POST"]) -def issues_webhook(): - - digest = hmac.new(KEY, request.data, hashlib.sha1).hexdigest() - signature = request.headers.get('X-LGTM-Signature', "not-provided") - - if not hmac.compare_digest(signature, digest): - return jsonify({'message': "Unauthorized"}), 401 - - json_dict = request.get_json() - - transition = json_dict.get('transition') - - if transition == 'create': - - data = get_issue_dict(json_dict.get('alert'), json_dict.get('project')) - - r = requests.post(URL, json=data, headers=HEADERS) - - issue_id = r.json()['number'] - - return jsonify({'issue-id': issue_id}), r.status_code - - if transition == 'close': - - issue_id = json_dict.get('issue-id') - - r = requests.patch(URL + '/' + issue_id, - json={"state": transition}, - headers=HEADERS) - - return jsonify({'issue-id': issue_id}), r.status_code - - if transition == 'reopen': - - issue_id = json_dict.get('issue-id') - - r = requests.patch(URL + '/' + issue_id, - json={"state": 'open'}, # github expects `open` - headers=HEADERS) - - return jsonify({'issue-id': issue_id}), r.status_code - - # this example only supports the above three transition types - # if transition is unmatched we return an error response - return ( - jsonify({"code": 400, "error": "unknown transition type - %s" % transition}), - 400, - ) - -if __name__ == "__main__": - app.run() -```