Граф коммитов

57 Коммитов

Автор SHA1 Сообщение Дата
Mathieu Pillard 34f21aa8f8
Stop storing ip_address in abuse reports - lookup country via geoip instead (#10745)
Stop storing ip_address in abuse reports - lookup country via geoip instead
2019-02-22 12:32:37 +01:00
Christopher Grebs f26a150086
Remove amo-validator, remove support for uploading legacy-addons. (#10553)
* Remove amo-validator, remove more support for uploading legacy-addons.

This is dropping most of our compatibility checks for legacy add-ons and
completely blocks them from being uploaded.

We workaround this block for mozilla-signed add-ons by issuing the same
error, just as a warning instead of an actual error.

This affected *loads* of our tests so the PR contains more test-changes
then I'd like.

In addition to that this PR adds file extension validation for uploads,
we only did that in our JavaScript and never in our code. So from now
on, we have a hard limit to only allow whatever is in
`VALID_EXTENSIONS`.

Further...

* drop amo-validator and related dependencies
* remove `dump_apps` management command
* removed unused fixture files from the repo

Fixes #10502

Future improvement idea for our test-suite: Unify add-on file fixtures
under one single folder to avoid any duplication.

There is most certainly some more improvements that could be made to
code and test structure but I went for the least changes, it's way too
huge already :-/

* Fix syntax error in JavaScript, thus ui tests

* Switch back to passing 'channel' around

* Move valid addon file extensions to constants/base

* Fix validate() call

* Fix more tests

* Update src/olympia/devhub/tests/test_views_validation.py

Co-Authored-By: EnTeQuAk <cg@webshox.org>

* Remove old comment

* Issue description specific only to Thunderbird and Seamonkey if targeting thunderbird or seamonkey
2019-02-01 08:52:04 +01:00
Mathieu Pillard af842876b7 Install Python3 in development container
This also adds support for specifiying which version to use when installing
packages, though at the moment it's not possible to switch python versions
without reinstalling everything, because both versions share /usr/local/bin
2019-01-14 16:19:47 +01:00
Christopher Grebs 8f7331bed7
Don't explicitly install npm, update Python to 2.7.15
Our docker builds were broken because of some inter-dependency weirdness
going on when you're installing npm and nodejs in one apt-get call. To
be honest I was too lazy to look up the actual root cause given the fix
is quite simple and npm is part of nodejs anyway.

This PR updates the Python version to 2.7.15 along the way.
2019-01-02 11:54:19 +01:00
Christopher Grebs 5655295e05
Run validator via subprocess, huge cleanup along the way (#9682)
This ports all amo-validator related code to run via subprocess. This
makes us less blocked by it not running on Python 3, there may still be
some installation / execution related bits to actually run it via python
2 instead of python 3 but that's something for later.

* Cleans up how and where we're setting _BIN settings for all the
binaries we're calling
* Backports `unicodehelper` from validator
* Removes `requeue_uploads` since it's been broken and unused for at
least 3-4 years
* Remove Standalone Add-on Compatibility Checker (/developers/addon/check-compatibility/)
* Remove compat-check related bits from standalone add-on upload
* Remove more bulk compat report stuff that isn't used anymore
* Removed zadmin application_versions.json view, unused

Fixes #9670
"Fixes" #8857
2018-10-16 08:30:16 +02:00
Christopher Grebs d6e85cea5a
Add pygit2 dependency. (#9497)
* Add debian stretch backports repo as discussed with @autrilla
* Add libgit2-dev dependency that's needed by pygit2 (via debian stretch
to get the proper version that's required by up-to-date pygit2)
* Add pygit 0.27.1

Fixes #9496
2018-09-24 13:15:49 +02:00
Christopher Grebs 0b807e2a5e
Add uuid to the list of packages installed in our Dockerfile 2018-08-30 09:48:26 +02:00
bqbn f922c5e1f2
Upgrade to use node 8 (#9188) 2018-08-17 10:35:25 -07:00
Andrew Williamson 3faf451dca
remove Stylus (#7820) 2018-03-19 22:19:31 +08:00
Mathieu Pillard 3ff221a52a Run new image uploads through pngcrush, add a command to handle existing ones
Icons and previews for both themes and non-themes should go through pngcrush.
For the existing content, the command `crush_images_for_top_addons` focuses
on the content displayed on landing pages only and should only be run once.
2018-02-27 01:52:45 +01:00
Christopher Grebs 50203d149e
don't remove things in update_deps then? 2018-02-19 18:59:29 +01:00
Christopher Grebs 07dee02dde
Use docker-compose restart instead of supervisorctl.
* Make also use of setting PYTHONDONTWRITEBYTECODE inside the docker
  image to reduce it's size considerably
* Remove some outdated code-parts
* Rename `olympia` inside our worker image to `olympia-worker`
* remove all build and cache folders after every `make update_deps`
  (which might help avoid any more package update/install ambiquities)

Fixes #7518
2018-02-19 17:47:51 +01:00
Andrew Williamson c05e4cfdf1
Generate static theme Preview on version submit (#7475) 2018-02-14 22:27:16 +08:00
Philip Cristiano cb51f696df Revert "Merge pull request #7432 from mozilla/revert-docker-changes"
Revert for local work and eventual merge, once this is fixed.

This reverts commit 5307f487a7, reversing
changes made to 49389bd642.
2018-02-05 09:30:36 -05:00
Christopher Grebs 22f864bd14
Revert our centos -> debian docker image changes.
Revert "docker: Fix Debian build issue for python package"

This reverts commit 2239e8483e.

Revert "docker: Include uwsgi plugin"

This reverts commit 807420bf16.

Revert "docker: Include uwsgi"

This reverts commit ceb36b5890.

Revert "docker: Update deploy to use Docker Python image"

This reverts commit 916547d2b5.

Revert "docker: Remove ‘centos’ from docker-indicating file"

This reverts commit d34b92e0e8.

Revert "docker: Update build to use Docker Python image"

This reverts commit b08b51375d.
2018-01-29 19:38:04 +01:00
Philip Cristiano b08b51375d docker: Update build to use Docker Python image
This provides an updated Python (2.7.14) as part of
https://bugzilla.mozilla.org/show_bug.cgi?id=1390789

The Docker images are not based on Centos so package
names needed to be updated. Comparable packages were
added, but may not contain the same versions or even
components.
2018-01-22 16:12:25 -05:00
Philip Cristiano d34b92e0e8 docker: Remove ‘centos’ from docker-indicating file
Moving from Centos to the Docker Python image
2018-01-22 15:40:50 -05:00
Christopher Grebs 6be6d89782
Remove not working and duplicate supervisor package installation from Dockerfile.
Fixes #6679
2017-10-17 16:32:33 +02:00
Christopher Grebs 57410ab97e Pin all our dependencies, including pip, six, ipython and ipdb. (#6663)
* Pin all our dependencies, including pip, six, ipython and ipdb.

This removes all dynamically installed python packages from the
installation pipeline and should result in a cleaner system install.

Fixes #6001

* Cleanup docs, prod and merge a few dependencies with system.

* Move more dependencies from prod to system :-/

* Finally fix docs task
2017-10-13 06:29:40 +02:00
bqbn 8826016c01 Add back nodesource v6 repo (#6036)
* Revert "Remove old node 4 repo"

This reverts commit 7d221f0814.

* Add back nodesource v6 repo

We need to use nodesource v6 repo for latest security fixes. EPEL updates
are often delayed.
2017-07-28 13:51:18 -07:00
Christopher Grebs ceba72be46 Remove translate toolkit, improve denis-cmd linting, compile locales before tests are run (#5655)
* Compile locales before we run tests, add gettext dependency

* Make compile-mo fail on linting errors.

* Fix tox compilation

* More tox fixes

* Fix path

* remove usedevelop for now

* re-add plural forms
2017-06-20 19:43:11 +08:00
Christopher Grebs e3e020d3a1 Install six before we upgrade setuptools. (#5541)
* Install six before we upgrade setuptools.

See pypa/setuptools#964 for a few more details. Currently quite a few
travis and circleci jobs are failing because of that.

* Explicitly upgrade pip and setuptools in 'make update_deps' so for travis too.

* Potentially fix circleci

* Fix setuptools and potentially docs environments

* More explicitly install six and setuptools

* Add more requirements for docs

* Add pyparsing dependency
2017-06-01 19:20:12 +08:00
Christopher Grebs af33ee67e1 Make sure we are using an up-to-date pip and setuptools version.
Fixes #5515
2017-05-30 17:23:51 +08:00
Andreas Wagner 8bd272258c Fix installing node 2017-05-10 22:39:09 +02:00
Andreas Wagner cc6ef397b0 Restrict ipython to 5.x because we use python 2.7 2017-05-10 20:59:43 +02:00
Andreas Wagner 7d221f0814 Remove old node 4 repo 2017-05-10 14:40:11 +02:00
Rémy HUBSCHER 75f057c578 Use jingo-minify 0.6.0 release. (#4732)
* Use jingo-minify 0.6.0 release.

* Use a newer version of git.

* @EnTeQuAk review.

* @jasonthomas review.

* Too many spaces.
2017-02-28 14:35:30 +01:00
Jason Thomas cee688ef06 Do not run yum update during docker build 2016-12-13 13:12:12 -05:00
Jason Thomas f811d043a7 compile locale definition files for utf-8 2016-12-13 13:00:15 -05:00
bqbn 0f84b2a6d0 Update sha256sum values for a few packages (#2884)
* Update sha256sum values for a few packages

pypi use different packaging format for a few packages which results in
different sha256sum values.

* Further remove pyrepo dependency from other places.

This is to address https://github.com/mozilla/addons-server/issues/2885
as well, to remove pyrepo from docker build flow.

* Update sha256sum for a few dev packages

* Add swig to support compiling the m2crypto module

* Replace "sudo apt-get" with "addons.apt.packages"

This is because Travis' containerized build infrastructure doesn't
support sudo.

* Add swig to the Dockerfile as well

SWIG_FEATURES="-D__x86_64__" is needed by M2Crypto-0.22.3 on CentOS
platform.
2016-06-13 09:22:49 -07:00
Christopher Grebs 53060ba973 Simplify our node modules installation, fix paths
refs #2596, fixes #2696, fixes #2701
2016-05-19 11:27:22 +02:00
Christopher Grebs e7c49f8e13 Detect docker container and switch between index ordering accordingly to install correct packages. 2016-05-17 14:53:02 +02:00
Andy McKay 3c523d9f82 ensure src dependencies don't go into code 2016-05-12 14:55:32 -07:00
Christopher Grebs d7be555cbb Refactor the way we handle dependencies, use 'make update_deps' in tox and docker.
There's now only one place that defines how we install things, the Makefile.

This also merges flake8.txt, test.txt into dev.txt and removed the
unused load.txt and cleans up all installation related code in
Dockerfile and tox.

* Install our node dependencies in update_deps too, we do use at least addons-linter in the near future.
2016-05-12 18:31:40 +02:00
Christopher Grebs cf3b4f982d Upgrade pip before installing any dependencies.
Fixes #2536
2016-05-09 12:21:23 +02:00
Christopher Grebs 0cbb294201 Rename addons-validator to addons-linter and update to latest release.
Fixes #1742
2016-02-24 10:53:26 +01:00
Christopher Grebs f3d3808d00 Fix typo on pip install line in Dockerfile.
This resultet in olympia being installed as a sdist and
python-package afterwards.
2016-02-04 23:11:24 +01:00
Christopher Grebs 92918ac47d Install olympia after we mount /code 2016-02-04 21:52:37 +01:00
Christopher Grebs 4bdd97a3cd Fix syntax error in Dockerfile 2016-02-04 21:21:28 +01:00
Christopher Grebs 5b7ae06efd Update django-piston-oauth2, fix merges, more path/import fixes.
* Make migrations work again
* Fix paths in Makefile, make sure olympia get's installed as a module too
* Fix migrations path in 'initialize_db'
* Make sure settings won't be loaded on top in olympia.startup
2016-01-06 10:48:04 +01:00
Christopher Grebs 20ac1daeab Merge pull request #951 from EnTeQuAk/feature/881-integrate-addons-validator
Install addons-validator and configure it's path. Fixes #881
2015-12-04 00:48:20 +01:00
Christopher Grebs a0c68641d8 Update addons-validator to 0.0.8, fix path 2015-12-04 00:47:15 +01:00
Christopher Grebs 0734dbcf67 Install addons-validator and configure it's path. Fixes #881 2015-12-04 00:45:07 +01:00
Christopher Grebs 96319bde4a Create a new base image for addons/olympia.
* Upgrade to Centos 7
* Use gpg-keys for nodesource and mysql56 repositories
* Uses more prod-like package versions
* Update pyrepo links to use https://pyrepo.stage.mozaws.net/olympia/
  (this is what we're using on prod)
2015-12-04 00:11:23 +01:00
Andy McKay 9993acfbe3 move to new repo 2015-11-25 13:09:24 -08:00
Stuart Colville 5104fe4d47 Move npm installation to Dockerfile 2015-09-29 09:39:46 +01:00
Stuart Colville eb93fd7e78 Move package installation 2015-09-18 17:01:32 +01:00
Stuart Colville 5bf6773865 Use supervisord for runserver in docker 2015-09-18 12:59:08 +01:00
Kumar McMillan 7d769f87be Set the locale to UTF8 to fix file system encoding
The main thing this fixes are a few failures in `lib/crypto/tests.py` which were the result of Python treating file system encoding as ASCII.
2015-09-17 11:44:13 -05:00
Kumar McMillan 4a212dc576 No longer use /wheelhouse from docker
This fixes a regression from https://github.com/mozilla/olympia/pull/743

For some reason, m2crypto was not getting installed right when the /wheelhouse wheel was used.
2015-09-15 11:08:40 -05:00