Remove GitHub API and unused files.utils helpers (#10412)
Fixes #10411 This removes unused files.utils helpers `zip_folder_content` and `repack` along the way. Once this has been merged, ops can remove `GITHUB_API_USER` and `GITHUB_API_TOKEN` from the environment. I actually wonder where they have been generated, e.g to get them revoked?
This commit is contained in:
Christopher Grebs
GitHub
Родитель
6e85f809d4
Коммит
300eb84353
|
|
@ -1,41 +0,0 @@
|
|||
===============
|
||||
GitHub Webhooks
|
||||
===============
|
||||
|
||||
.. note::
|
||||
|
||||
These APIs are experimental and are currently being worked on. Endpoints
|
||||
may change without warning. Consider the :ref:`v3 API<api-stable-v3>`
|
||||
if you need stability.
|
||||
|
||||
This API provides an endpoint that works with GitHub to provide add-on validation as a GitHub webhook. This end point is designed to be called specifically from GitHub and will only send API responses back to `api.github.com`.
|
||||
|
||||
To set this up on a GitHub repository you will need to:
|
||||
|
||||
* Go to `Settings > Webhooks & Services`
|
||||
* Add a new Webhook with Payload URL of `https://addons.mozilla.org/api/v4/github/validate/`
|
||||
* Click `Send me everything`
|
||||
* Click `Update webhook`
|
||||
|
||||
At this point the validator will be able to get the data, but won't be able to write a response to GitHub. To enable responses to GitHub:
|
||||
|
||||
* Go to `Settings > Collaborators`
|
||||
* Enter `addons-robot` and select the entry
|
||||
* Click `Add collaborator`
|
||||
* You will have to wait for a Mozilla person to respond to the invite
|
||||
|
||||
If this service proves useful and this service transitions from its Experimental API state, we will remove as many of these steps as possible.
|
||||
|
||||
The validator will run when you create or alter a pull request.
|
||||
|
||||
.. http:post:: /api/v4/github/validate/
|
||||
|
||||
**Request:**
|
||||
|
||||
A `GitHub API webhook <https://developer.github.com/v4/repos/hooks/>`_ body. Currently only `pull_request` events are processed, all others are ignored.
|
||||
|
||||
**Response:**
|
||||
|
||||
:statuscode 201: request has been processed and a pending message sent back to GitHub.
|
||||
:statuscode 200: request is not a `pull_request`, it's been accepted.
|
||||
:statuscode 422: body is invalid.
|
||||
|
|
@ -43,4 +43,3 @@ using the API.
|
|||
ratings
|
||||
reviewers
|
||||
signing
|
||||
github
|
||||
|
|
|
|||
|
|
@ -265,6 +265,7 @@ The documentation for `v3` can be accessed at: :ref:`v3-api-index`
|
|||
v4 API changelog
|
||||
----------------
|
||||
|
||||
* 2019-01-16: Removed /api/{v3,v4,v4dev}/github api entirely. They have been marked as experimental. https://github.com/mozilla/addons-server/issues/10411
|
||||
* 2019-01-11: added new /reviewers/browse/ endpoint. https://github.com/mozilla/addons-server/issues/10322
|
||||
* 2018-05-18: renamed /reviews/ endpoint to /ratings/ https://github.com/mozilla/addons-server/issues/6849
|
||||
* 2018-05-25: renamed ``rating.rating`` property to ``rating.score`` https://github.com/mozilla/addons-server/pull/8332
|
||||
|
|
|
|||
|
|
@ -1,39 +0,0 @@
|
|||
===============
|
||||
GitHub Webhooks
|
||||
===============
|
||||
|
||||
.. note::
|
||||
|
||||
This is an Experimental API and can change at any time.
|
||||
|
||||
This API provides an endpoint that works with GitHub to provide add-on validation as a GitHub webhook. This end point is designed to be called specifically from GitHub and will only send API responses back to `api.github.com`.
|
||||
|
||||
To set this up on a GitHub repository you will need to:
|
||||
|
||||
* Go to `Settings > Webhooks & Services`
|
||||
* Add a new Webhook with Payload URL of `https://addons.mozilla.org/api/v3/github/validate/`
|
||||
* Click `Send me everything`
|
||||
* Click `Update webhook`
|
||||
|
||||
At this point the validator will be able to get the data, but won't be able to write a response to GitHub. To enable responses to GitHub:
|
||||
|
||||
* Go to `Settings > Collaborators`
|
||||
* Enter `addons-robot` and select the entry
|
||||
* Click `Add collaborator`
|
||||
* You will have to wait for a Mozilla person to respond to the invite
|
||||
|
||||
If this service proves useful and this service transitions from its Experimental API state, we will remove as many of these steps as possible.
|
||||
|
||||
The validator will run when you create or alter a pull request.
|
||||
|
||||
.. http:post:: /api/v3/github/validate/
|
||||
|
||||
**Request:**
|
||||
|
||||
A `GitHub API webhook <https://developer.github.com/v3/repos/hooks/>`_ body. Currently only `pull_request` events are processed, all others are ignored.
|
||||
|
||||
**Response:**
|
||||
|
||||
:statuscode 201: request has been processed and a pending message sent back to GitHub.
|
||||
:statuscode 200: request is not a `pull_request`, it's been accepted.
|
||||
:statuscode 422: body is invalid.
|
||||
|
|
@ -45,4 +45,3 @@ using the API.
|
|||
reviews
|
||||
reviewers
|
||||
signing
|
||||
github
|
||||
|
|
|
|||
|
|
@ -12,7 +12,6 @@ v3_api_urls = [
|
|||
url(r'^reviewers/', include('olympia.reviewers.api_urls')),
|
||||
url(r'^', include('olympia.signing.urls')),
|
||||
url(r'^activity/', include('olympia.activity.urls')),
|
||||
url(r'^github/', include('olympia.github.urls')),
|
||||
]
|
||||
|
||||
v4_api_urls = [
|
||||
|
|
@ -24,7 +23,6 @@ v4_api_urls = [
|
|||
url(r'^reviewers/', include('olympia.reviewers.api_urls')),
|
||||
url(r'^', include('olympia.signing.urls')),
|
||||
url(r'^activity/', include('olympia.activity.urls')),
|
||||
url(r'^github/', include('olympia.github.urls')),
|
||||
]
|
||||
|
||||
urlpatterns = [
|
||||
|
|
|
|||
Двоичные данные
src/olympia/files/fixtures/files/github-repo.xpi
Двоичные данные
src/olympia/files/fixtures/files/github-repo.xpi
Двоичный файл не отображается.
|
|
@ -612,48 +612,6 @@ class TestManifestJSONExtractorStaticTheme(TestManifestJSONExtractor):
|
|||
pass # Irrelevant for static themes.
|
||||
|
||||
|
||||
def test_zip_folder_content():
|
||||
extension_file = 'src/olympia/files/fixtures/files/extension.xpi'
|
||||
temp_filename, temp_folder = None, None
|
||||
try:
|
||||
temp_folder = utils.extract_zip(extension_file)
|
||||
assert sorted(os.listdir(temp_folder)) == [
|
||||
'chrome', 'chrome.manifest', 'install.rdf']
|
||||
temp_filename = amo.tests.get_temp_filename()
|
||||
utils.zip_folder_content(temp_folder, temp_filename)
|
||||
# Make sure the zipped files contain the same files.
|
||||
with zipfile.ZipFile(temp_filename, mode='r') as new:
|
||||
with zipfile.ZipFile(extension_file, mode='r') as orig:
|
||||
assert sorted(new.namelist()) == sorted(orig.namelist())
|
||||
finally:
|
||||
if temp_folder is not None and os.path.exists(temp_folder):
|
||||
amo.utils.rm_local_tmp_dir(temp_folder)
|
||||
if temp_filename is not None and os.path.exists(temp_filename):
|
||||
os.unlink(temp_filename)
|
||||
|
||||
|
||||
def test_repack():
|
||||
# Warning: context managers all the way down. Because they're awesome.
|
||||
extension_file = 'src/olympia/files/fixtures/files/extension.xpi'
|
||||
# We don't want to overwrite our fixture, so use a copy.
|
||||
with amo.tests.copy_file_to_temp(extension_file) as temp_filename:
|
||||
# This is where we're really testing the repack helper.
|
||||
with utils.repack(temp_filename) as folder_path:
|
||||
# Temporary folder contains the unzipped XPI.
|
||||
assert sorted(os.listdir(folder_path)) == [
|
||||
'chrome', 'chrome.manifest', 'install.rdf']
|
||||
# Add a file, which should end up in the repacked file.
|
||||
with open(os.path.join(folder_path, 'foo.bar'), 'w') as file_:
|
||||
file_.write('foobar')
|
||||
# Once we're done with the repack, the temporary folder is removed.
|
||||
assert not os.path.exists(folder_path)
|
||||
# And the repacked file has the added file.
|
||||
assert os.path.exists(temp_filename)
|
||||
with zipfile.ZipFile(temp_filename, mode='r') as zf:
|
||||
assert 'foo.bar' in zf.namelist()
|
||||
assert zf.read('foo.bar') == 'foobar'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('filename, expected_files', [
|
||||
('webextension_no_id.xpi', [
|
||||
'README.md', 'beasts', 'button', 'content_scripts', 'manifest.json',
|
||||
|
|
|
|||
|
|
@ -29,7 +29,6 @@ from django.utils.translation import ugettext
|
|||
|
||||
import flufl.lock
|
||||
import rdflib
|
||||
import scandir
|
||||
import six
|
||||
|
||||
from signing_clients.apps import get_signer_organizational_unit_name
|
||||
|
|
@ -1097,41 +1096,6 @@ def get_sha256(file_obj, block_size=io.DEFAULT_BUFFER_SIZE):
|
|||
return hash_.hexdigest()
|
||||
|
||||
|
||||
def zip_folder_content(folder, filename):
|
||||
"""Compress the _content_ of a folder."""
|
||||
with zipfile.ZipFile(filename, 'w', zipfile.ZIP_DEFLATED) as dest:
|
||||
# Add each file/folder from the folder to the zip file.
|
||||
for root, dirs, files in scandir.walk(folder):
|
||||
relative_dir = os.path.relpath(root, folder)
|
||||
for file_ in files:
|
||||
dest.write(os.path.join(root, file_),
|
||||
# We want the relative paths for the files.
|
||||
arcname=os.path.join(relative_dir, file_))
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def repack(xpi_path):
|
||||
"""Unpack the XPI, yield the temp folder, and repack on exit.
|
||||
|
||||
Usage:
|
||||
with repack('foo.xpi') as temp_folder:
|
||||
# 'foo.xpi' files are extracted to the temp_folder.
|
||||
modify_files(temp_folder) # Modify the files in the temp_folder.
|
||||
# The 'foo.xpi' extension is now repacked, with the file changes.
|
||||
"""
|
||||
# Unpack.
|
||||
tempdir = extract_zip(xpi_path, remove=False)
|
||||
yield tempdir
|
||||
try:
|
||||
# Repack.
|
||||
repacked = u'{0}.repacked'.format(xpi_path) # Temporary file.
|
||||
zip_folder_content(tempdir, repacked)
|
||||
# Overwrite the initial file with the repacked one.
|
||||
shutil.move(repacked, xpi_path)
|
||||
finally:
|
||||
rm_local_tmp_dir(tempdir)
|
||||
|
||||
|
||||
def update_version_number(file_obj, new_version_number):
|
||||
"""Update the manifest to have the new version number."""
|
||||
# Create a new xpi with the updated version.
|
||||
|
|
|
|||
|
|
@ -1,55 +0,0 @@
|
|||
import json
|
||||
|
||||
import olympia.core.logger
|
||||
|
||||
from olympia.amo.celery import task
|
||||
from olympia.amo.templatetags.jinja_helpers import absolutify
|
||||
from olympia.amo.urlresolvers import reverse
|
||||
from olympia.devhub.tasks import validate
|
||||
from olympia.files.models import FileUpload
|
||||
from olympia.github.utils import GithubCallback, rezip_file
|
||||
|
||||
|
||||
log = olympia.core.logger.getLogger('z.github')
|
||||
|
||||
|
||||
@task
|
||||
def process_webhook(upload_pk, callbacks):
|
||||
log.info('Processing webhook for: {}'.format(upload_pk))
|
||||
upload = FileUpload.objects.get(pk=upload_pk)
|
||||
github = GithubCallback(callbacks)
|
||||
res = github.get()
|
||||
|
||||
upload.name = '{}-github-webhook.xpi'.format(upload.pk)
|
||||
upload.path = rezip_file(res, upload.pk)
|
||||
upload.save()
|
||||
|
||||
log.info('Validating: {}'.format(upload_pk))
|
||||
validate(
|
||||
upload,
|
||||
listed=True,
|
||||
subtask=process_results.si(upload_pk, callbacks)
|
||||
)
|
||||
|
||||
|
||||
@task
|
||||
def process_results(upload_pk, callbacks):
|
||||
log.info('Processing validation results for: {}'.format(upload_pk))
|
||||
upload = FileUpload.objects.get(pk=upload_pk)
|
||||
validation = json.loads(upload.validation) if upload.validation else {}
|
||||
github = GithubCallback(callbacks)
|
||||
url = absolutify(
|
||||
reverse('devhub.upload_detail', args=[upload.uuid]))
|
||||
|
||||
if not validation:
|
||||
log.error('Validation not written: {}'.format(upload_pk))
|
||||
github.failure()
|
||||
return
|
||||
|
||||
if validation.get('success'):
|
||||
log.info('Notifying success for: {}'.format(upload_pk))
|
||||
github.success(url)
|
||||
return
|
||||
|
||||
log.info('Notifying errors for: {}'.format(upload_pk))
|
||||
github.error(url)
|
||||
|
|
@ -1,141 +0,0 @@
|
|||
import zipfile
|
||||
|
||||
from copy import deepcopy
|
||||
|
||||
from django import forms
|
||||
from django.test.utils import override_settings
|
||||
|
||||
import mock
|
||||
|
||||
from olympia.amo.tests import AMOPaths, TestCase
|
||||
from olympia.github.utils import GithubCallback, GithubRequest, rezip_file
|
||||
|
||||
|
||||
example_root = 'https://api.github.com/repos/org/repo'
|
||||
example_pull_request = {
|
||||
'pull_request': {
|
||||
'head': {
|
||||
'repo': {
|
||||
'archive_url': example_root + '/{archive_format}{/ref}',
|
||||
'statuses_url': example_root + '/statuses/{sha}',
|
||||
'pulls_url': example_root + '/pulls{/number}'
|
||||
},
|
||||
'sha': 'abc'
|
||||
},
|
||||
'number': 1,
|
||||
},
|
||||
'repository': {
|
||||
'commits_url': example_root + '/commits{/sha}'
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class TestGithub(TestCase):
|
||||
|
||||
def test_github(self):
|
||||
form = GithubRequest(data=example_pull_request)
|
||||
assert form.is_valid(), form.errors
|
||||
assert (
|
||||
form.cleaned_data['status_url'] ==
|
||||
'https://api.github.com/repos/org/repo/statuses/abc')
|
||||
assert (
|
||||
form.cleaned_data['zip_url'] ==
|
||||
'https://api.github.com/repos/org/repo/zipball/abc')
|
||||
|
||||
def test_invalid(self):
|
||||
example = deepcopy(example_pull_request)
|
||||
del example['pull_request']['head']
|
||||
form = GithubRequest(data=example)
|
||||
assert not form.is_valid()
|
||||
|
||||
def test_url_wrong(self):
|
||||
example = deepcopy(example_pull_request)
|
||||
example['pull_request']['head']['repo'] = 'http://a.m.o'
|
||||
form = GithubRequest(data=example)
|
||||
assert not form.is_valid()
|
||||
|
||||
|
||||
@override_settings(GITHUB_API_USER='key', GITHUB_API_TOKEN='token')
|
||||
class GithubBaseTestCase(TestCase):
|
||||
|
||||
def setUp(self):
|
||||
super(GithubBaseTestCase, self).setUp()
|
||||
patch = mock.patch('olympia.github.utils.requests', autospec=True)
|
||||
self.addCleanup(patch.stop)
|
||||
self.requests = patch.start()
|
||||
self.data = {
|
||||
'type': 'github',
|
||||
'status_url': 'https://github/status',
|
||||
'zip_url': 'https://github/zip',
|
||||
'sha': 'some:sha'
|
||||
}
|
||||
self.github = GithubCallback(self.data)
|
||||
|
||||
def check_status(self, status, call=None, url=None, **kw):
|
||||
url = url or self.data['status_url']
|
||||
body = {'context': 'mozilla/addons-linter'}
|
||||
if status != 'comment':
|
||||
body['state'] = status
|
||||
|
||||
body.update(**kw)
|
||||
if not call:
|
||||
call = self.requests.post.call_args_list
|
||||
if len(call) != 1:
|
||||
# If you don't specify a call to test, we'll get the last
|
||||
# one off the stack, if there's more than one, that's a
|
||||
# problem.
|
||||
raise AssertionError('More than one call to requests.post')
|
||||
call = call[0]
|
||||
assert call == mock.call(url, json=body, auth=('key', 'token'))
|
||||
|
||||
|
||||
class TestCallback(GithubBaseTestCase):
|
||||
|
||||
def test_create_not_github(self):
|
||||
with self.assertRaises(ValueError):
|
||||
GithubCallback({'type': 'bitbucket'})
|
||||
|
||||
def test_pending(self):
|
||||
self.github.pending()
|
||||
self.check_status('pending')
|
||||
|
||||
def test_success(self):
|
||||
self.github.success('http://a.m.o/')
|
||||
self.check_status('success', target_url='http://a.m.o/')
|
||||
|
||||
def test_error(self):
|
||||
self.github.error('http://a.m.o/')
|
||||
self.check_status(
|
||||
'error', description=mock.ANY, target_url='http://a.m.o/')
|
||||
|
||||
def test_failure(self):
|
||||
self.github.failure()
|
||||
self.check_status(
|
||||
'failure', description=mock.ANY)
|
||||
|
||||
def test_get(self):
|
||||
self.github.get()
|
||||
self.requests.get.assert_called_with(
|
||||
'https://github/zip'
|
||||
)
|
||||
|
||||
|
||||
class TestRezip(AMOPaths, TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.response = mock.Mock()
|
||||
self.response.content = open(self.xpi_path('github-repo')).read()
|
||||
|
||||
def test_rezip(self):
|
||||
new_path = rezip_file(self.response, 1)
|
||||
with open(new_path, 'r') as new_file:
|
||||
new_zip = zipfile.ZipFile(new_file)
|
||||
self.assertSetEqual(
|
||||
set([f.filename for f in new_zip.filelist]),
|
||||
set(['manifest.json', 'index.js'])
|
||||
)
|
||||
|
||||
def test_badzip(self):
|
||||
with self.settings(FILE_UNZIP_SIZE_LIMIT=5):
|
||||
with self.assertRaises(forms.ValidationError):
|
||||
rezip_file(self.response, 1)
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
import json
|
||||
|
||||
from django.test.utils import override_settings
|
||||
|
||||
import mock
|
||||
|
||||
from olympia.amo.templatetags.jinja_helpers import absolutify
|
||||
from olympia.amo.tests import AMOPaths
|
||||
from olympia.amo.urlresolvers import reverse
|
||||
from olympia.files.models import FileUpload
|
||||
from olympia.github.tasks import process_results, process_webhook
|
||||
from olympia.github.tests.test_github import GithubBaseTestCase
|
||||
|
||||
|
||||
@override_settings(GITHUB_API_USER='key', GITHUB_API_TOKEN='token')
|
||||
class TestGithub(AMOPaths, GithubBaseTestCase):
|
||||
|
||||
def get_url(self, upload_uuid):
|
||||
return absolutify(
|
||||
reverse('devhub.upload_detail', args=[upload_uuid]))
|
||||
|
||||
def test_good_results(self):
|
||||
upload = FileUpload.objects.create(
|
||||
validation=json.dumps({'success': True, 'errors': 0})
|
||||
)
|
||||
process_results(upload.pk, self.data)
|
||||
self.check_status('success', target_url=self.get_url(upload.uuid))
|
||||
|
||||
def test_failed_results(self):
|
||||
upload = FileUpload.objects.create()
|
||||
process_results(upload.pk, self.data)
|
||||
self.check_status('failure', description=mock.ANY)
|
||||
|
||||
def test_error_results(self):
|
||||
upload = FileUpload.objects.create(
|
||||
validation=json.dumps({
|
||||
'errors': 1,
|
||||
'messages': [{
|
||||
'description': ['foo'],
|
||||
'file': 'some/file',
|
||||
'line': 3,
|
||||
'type': 'error'
|
||||
}]
|
||||
})
|
||||
)
|
||||
process_results(upload.pk, self.data)
|
||||
error = self.requests.post.call_args_list[0]
|
||||
self.check_status(
|
||||
'error',
|
||||
call=error, description=mock.ANY,
|
||||
target_url=self.get_url(upload.uuid))
|
||||
|
||||
def test_webhook(self):
|
||||
upload = FileUpload.objects.create()
|
||||
|
||||
self.response = mock.Mock()
|
||||
self.response.content = open(self.xpi_path('github-repo')).read()
|
||||
self.requests.get.return_value = self.response
|
||||
|
||||
process_webhook(upload.pk, self.data)
|
||||
self.check_status('success', target_url=self.get_url(upload.uuid))
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
import json
|
||||
|
||||
from django.utils.http import urlencode
|
||||
|
||||
import mock
|
||||
import requests
|
||||
|
||||
from olympia.amo.tests import AMOPaths, TestCase, reverse_ns
|
||||
from olympia.files.models import FileUpload
|
||||
from olympia.github.tests.test_github import (
|
||||
GithubBaseTestCase, example_pull_request)
|
||||
|
||||
|
||||
class TestGithubView(AMOPaths, GithubBaseTestCase, TestCase):
|
||||
|
||||
def setUp(self):
|
||||
super(TestGithubView, self).setUp()
|
||||
self.url = reverse_ns('github.validate')
|
||||
|
||||
def post(self, data, header=None, data_type=None):
|
||||
data_type = data_type or 'application/json'
|
||||
if (data_type == 'application/json'):
|
||||
data = json.dumps(data)
|
||||
elif (data_type == 'application/x-www-form-urlencoded'):
|
||||
data = urlencode({'payload': json.dumps(data)})
|
||||
return self.client.post(
|
||||
self.url, data=data,
|
||||
content_type=data_type,
|
||||
HTTP_X_GITHUB_EVENT=header or 'pull_request'
|
||||
)
|
||||
|
||||
def complete(self):
|
||||
pending, success = self.requests.post.call_args_list
|
||||
self.check_status(
|
||||
'pending',
|
||||
call=pending,
|
||||
url='https://api.github.com/repos/org/repo/statuses/abc'
|
||||
)
|
||||
self.check_status(
|
||||
'success',
|
||||
call=success,
|
||||
url='https://api.github.com/repos/org/repo/statuses/abc',
|
||||
target_url=mock.ANY
|
||||
)
|
||||
|
||||
assert FileUpload.objects.get()
|
||||
|
||||
def test_not_pull_request(self):
|
||||
assert self.post({}, header='meh').status_code == 200
|
||||
|
||||
def test_bad_pull_request(self):
|
||||
assert self.post({'pull_request': {}}).status_code == 422
|
||||
|
||||
def setup_xpi(self):
|
||||
self.response = mock.Mock()
|
||||
self.response.content = open(self.xpi_path('github-repo')).read()
|
||||
self.requests.get.return_value = self.response
|
||||
|
||||
def test_pending_fails(self):
|
||||
self.setup_xpi()
|
||||
|
||||
post = mock.Mock()
|
||||
# GitHub returns a 404 when the addons-robot account does not
|
||||
# have write access.
|
||||
post.status_code = 404
|
||||
post.raise_for_status.side_effect = requests.HTTPError(response=post)
|
||||
self.requests.post.return_value = post
|
||||
|
||||
res = self.post(example_pull_request)
|
||||
assert 'write access' in json.loads(res.content)['details']
|
||||
|
||||
def test_good_not_json(self):
|
||||
self.setup_xpi()
|
||||
assert self.post(
|
||||
example_pull_request,
|
||||
data_type='application/x-www-form-urlencoded').status_code == 201
|
||||
self.complete()
|
||||
|
||||
def test_good(self):
|
||||
self.setup_xpi()
|
||||
assert self.post(example_pull_request).status_code == 201
|
||||
self.complete()
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
from django.conf.urls import url
|
||||
|
||||
from olympia.github.views import GithubView
|
||||
|
||||
|
||||
urlpatterns = [
|
||||
url(r'^validate/$', GithubView.as_view(), name='github.validate'),
|
||||
]
|
||||
|
|
@ -1,153 +0,0 @@
|
|||
import os
|
||||
import uuid
|
||||
import zipfile
|
||||
|
||||
from django import forms
|
||||
from django.conf import settings
|
||||
from django.core.files.storage import default_storage as storage
|
||||
|
||||
import requests
|
||||
|
||||
from django_statsd.clients import statsd
|
||||
|
||||
import olympia.core.logger
|
||||
|
||||
from olympia.amo.templatetags.jinja_helpers import user_media_path
|
||||
from olympia.files.utils import SafeZip
|
||||
|
||||
|
||||
log = olympia.core.logger.getLogger('z.github')
|
||||
|
||||
|
||||
class GithubCallback(object):
|
||||
|
||||
def __init__(self, data):
|
||||
if data['type'] != 'github':
|
||||
raise ValueError('Not a github callback.')
|
||||
self.data = data
|
||||
|
||||
def get(self):
|
||||
log.info('Getting zip from github: {}'.format(self.data['zip_url']))
|
||||
with statsd.timer('github.zip'):
|
||||
res = requests.get(self.data['zip_url'])
|
||||
res.raise_for_status()
|
||||
return res
|
||||
|
||||
def post(self, url, data):
|
||||
msg = data.get('state', 'comment')
|
||||
log.info('Setting github to: {} at: {}'.format(msg, url))
|
||||
with statsd.timer('github.{}'.format(msg)):
|
||||
data['context'] = 'mozilla/addons-linter'
|
||||
log.info('Body: {}'.format(data))
|
||||
res = requests.post(
|
||||
url,
|
||||
json=data,
|
||||
auth=(settings.GITHUB_API_USER, settings.GITHUB_API_TOKEN))
|
||||
log.info('Response: {}'.format(res.content))
|
||||
res.raise_for_status()
|
||||
|
||||
def pending(self):
|
||||
self.post(self.data['status_url'], data={'state': 'pending'})
|
||||
|
||||
def success(self, url):
|
||||
self.post(self.data['status_url'], data={
|
||||
'state': 'success',
|
||||
'target_url': url
|
||||
})
|
||||
|
||||
def error(self, url):
|
||||
self.post(self.data['status_url'], data={
|
||||
'state': 'error',
|
||||
# Not localising because we aren't sure what locale to localise to.
|
||||
# I would like to pass a longer string here that shows more details
|
||||
# however, we are limited to "A short description of the status."
|
||||
# Which means all the fancy things I wanted to do got truncated.
|
||||
'description': 'This add-on did not validate.',
|
||||
'target_url': url
|
||||
})
|
||||
|
||||
def failure(self):
|
||||
data = {
|
||||
'state': 'failure',
|
||||
# Not localising because we aren't sure what locale to localise to.
|
||||
'description': 'The validator failed to run correctly.'
|
||||
}
|
||||
self.post(self.data['status_url'], data=data)
|
||||
|
||||
|
||||
class GithubRequest(forms.Form):
|
||||
status_url = forms.URLField(required=False)
|
||||
zip_url = forms.URLField(required=False)
|
||||
sha = forms.CharField(required=False)
|
||||
|
||||
@property
|
||||
def repo(self):
|
||||
return self.data['pull_request']['head']['repo']
|
||||
|
||||
@property
|
||||
def sha(self):
|
||||
return self.data['pull_request']['head']['sha']
|
||||
|
||||
def get_status(self):
|
||||
return self.repo['statuses_url'].replace('{sha}', self.sha)
|
||||
|
||||
def get_zip(self):
|
||||
return (
|
||||
self.repo['archive_url']
|
||||
.replace('{archive_format}', 'zipball')
|
||||
.replace('{/ref}', '/' + self.sha))
|
||||
|
||||
def validate_url(self, url):
|
||||
if not url.startswith('https://api.github.com/'):
|
||||
raise forms.ValidationError('Invalid URL: {}'.format(url))
|
||||
return url
|
||||
|
||||
def clean(self):
|
||||
fields = (
|
||||
('status_url', self.get_status),
|
||||
('zip_url', self.get_zip),
|
||||
)
|
||||
for url, method in fields:
|
||||
try:
|
||||
self.cleaned_data[url] = self.validate_url(method())
|
||||
except Exception:
|
||||
log.error('Invalid data in processing JSON')
|
||||
raise forms.ValidationError('Invalid data')
|
||||
|
||||
self.cleaned_data['sha'] = self.data['pull_request']['head']['sha']
|
||||
self.cleaned_data['type'] = 'github'
|
||||
return self.cleaned_data
|
||||
|
||||
|
||||
def rezip_file(response, pk):
|
||||
# An .xpi does not have a directory inside the zip, yet zips from github
|
||||
# do, so we'll need to rezip the file before passing it through to the
|
||||
# validator.
|
||||
loc = os.path.join(user_media_path('addons'), 'temp', uuid.uuid4().hex)
|
||||
old_filename = '{}_github_webhook.zip'.format(pk)
|
||||
old_path = os.path.join(loc, old_filename)
|
||||
|
||||
with storage.open(old_path, 'wb') as old:
|
||||
old.write(response.content)
|
||||
|
||||
new_filename = '{}_github_webhook.xpi'.format(pk)
|
||||
new_path = os.path.join(loc, new_filename)
|
||||
|
||||
old_zip = SafeZip(old_path)
|
||||
if not old_zip.is_valid:
|
||||
raise
|
||||
|
||||
with storage.open(new_path, 'w') as new:
|
||||
new_zip = zipfile.ZipFile(new, 'w')
|
||||
|
||||
for obj in old_zip.filelist:
|
||||
# Basically strip off the leading directory.
|
||||
new_filename = obj.filename.partition('/')[-1]
|
||||
if not new_filename:
|
||||
continue
|
||||
new_zip.writestr(new_filename, old_zip.read(obj.filename))
|
||||
|
||||
new_zip.close()
|
||||
|
||||
old_zip.close()
|
||||
return new_path
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
import json
|
||||
|
||||
import requests
|
||||
|
||||
from rest_framework import status
|
||||
from rest_framework.response import Response
|
||||
from rest_framework.views import APIView
|
||||
|
||||
import olympia.core.logger
|
||||
|
||||
from olympia.files.models import FileUpload
|
||||
from olympia.github.tasks import process_webhook
|
||||
from olympia.github.utils import GithubCallback, GithubRequest
|
||||
|
||||
|
||||
log = olympia.core.logger.getLogger('z.github')
|
||||
|
||||
|
||||
class GithubView(APIView):
|
||||
|
||||
def post(self, request):
|
||||
if request.META.get('HTTP_X_GITHUB_EVENT') != 'pull_request':
|
||||
# That's ok, we are just going to ignore it, we'll return a 2xx
|
||||
# response so github doesn't report it as an error.
|
||||
return Response({}, status=status.HTTP_200_OK)
|
||||
|
||||
# If the content-type is form-urlencoded the JSON is sent in the
|
||||
# payload parameter.
|
||||
#
|
||||
# See: https://developer.github.com/webhooks/creating/#content-type
|
||||
if (request.META.get('CONTENT_TYPE') ==
|
||||
'application/x-www-form-urlencoded'):
|
||||
data = json.loads(request.data['payload'])
|
||||
else:
|
||||
data = request.data
|
||||
|
||||
github = GithubRequest(data=data)
|
||||
if not github.is_valid():
|
||||
return Response({}, status=422)
|
||||
|
||||
data = github.cleaned_data
|
||||
upload = FileUpload.objects.create()
|
||||
log.info('Created FileUpload from github api: {}'.format(upload.pk))
|
||||
|
||||
try:
|
||||
github = GithubCallback(data)
|
||||
github.pending()
|
||||
except requests.HTTPError as err:
|
||||
# This is a common error, where the user hasn't set up add-ons
|
||||
# robot as a contributor, so we'll get a 404 from GitHub. Let's
|
||||
# try and get a nice error message back into the GitHub UI.
|
||||
if err.response.status_code == status.HTTP_404_NOT_FOUND:
|
||||
return Response({
|
||||
'details': (
|
||||
'Writing pending status failed. Please ensure '
|
||||
'the addons.mozilla.org GitHub account has write '
|
||||
'access to this repository.')},
|
||||
status=status.HTTP_404_NOT_FOUND)
|
||||
|
||||
process_webhook.delay(upload.pk, data)
|
||||
return Response({}, status=status.HTTP_201_CREATED)
|
||||
|
|
@ -475,7 +475,6 @@ INSTALLED_APPS = (
|
|||
'olympia.devhub',
|
||||
'olympia.discovery',
|
||||
'olympia.files',
|
||||
'olympia.github',
|
||||
'olympia.legacy_api',
|
||||
'olympia.legacy_discovery',
|
||||
'olympia.lib.es',
|
||||
|
|
@ -1227,10 +1226,6 @@ CELERY_TASK_ROUTES = {
|
|||
'olympia.zadmin.tasks.admin_email': {'queue': 'zadmin'},
|
||||
'olympia.zadmin.tasks.celery_error': {'queue': 'zadmin'},
|
||||
|
||||
# Github API
|
||||
'olympia.github.tasks.process_results': {'queue': 'devhub'},
|
||||
'olympia.github.tasks.process_webhook': {'queue': 'devhub'},
|
||||
|
||||
# Temporary tasks to crush existing images.
|
||||
# Go in the addons queue to leave the 'devhub' queue free to process
|
||||
# validations etc.
|
||||
|
|
@ -1853,10 +1848,6 @@ FXA_SQS_AWS_WAIT_TIME = 20 # Seconds.
|
|||
AWS_STATS_S3_BUCKET = env('AWS_STATS_S3_BUCKET', default=None)
|
||||
AWS_STATS_S3_PREFIX = env('AWS_STATS_S3_PREFIX', default='amo_stats')
|
||||
|
||||
# For the Github webhook API.
|
||||
GITHUB_API_USER = env('GITHUB_API_USER', default='')
|
||||
GITHUB_API_TOKEN = env('GITHUB_API_TOKEN', default='')
|
||||
|
||||
MIGRATED_LWT_DEFAULT_OWNER_EMAIL = 'addons-team+landfill-account@mozilla.com'
|
||||
|
||||
MIGRATED_LWT_UPDATES_ENABLED = False
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче