This implements more advanced file-type detection based on `python-magic` thus we don't rely on the files extensions anymore but actually verify their content (header) before serving it to the client and reviewer.
Fixes#10685 (and actually rewrites it's meaning)
* Remove amo-validator, remove more support for uploading legacy-addons.
This is dropping most of our compatibility checks for legacy add-ons and
completely blocks them from being uploaded.
We workaround this block for mozilla-signed add-ons by issuing the same
error, just as a warning instead of an actual error.
This affected *loads* of our tests so the PR contains more test-changes
then I'd like.
In addition to that this PR adds file extension validation for uploads,
we only did that in our JavaScript and never in our code. So from now
on, we have a hard limit to only allow whatever is in
`VALID_EXTENSIONS`.
Further...
* drop amo-validator and related dependencies
* remove `dump_apps` management command
* removed unused fixture files from the repo
Fixes#10502
Future improvement idea for our test-suite: Unify add-on file fixtures
under one single folder to avoid any duplication.
There is most certainly some more improvements that could be made to
code and test structure but I went for the least changes, it's way too
huge already :-/
* Fix syntax error in JavaScript, thus ui tests
* Switch back to passing 'channel' around
* Move valid addon file extensions to constants/base
* Fix validate() call
* Fix more tests
* Update src/olympia/devhub/tests/test_views_validation.py
Co-Authored-By: EnTeQuAk <cg@webshox.org>
* Remove old comment
* Issue description specific only to Thunderbird and Seamonkey if targeting thunderbird or seamonkey
This also adds support for specifiying which version to use when installing
packages, though at the moment it's not possible to switch python versions
without reinstalling everything, because both versions share /usr/local/bin
Our docker builds were broken because of some inter-dependency weirdness
going on when you're installing npm and nodejs in one apt-get call. To
be honest I was too lazy to look up the actual root cause given the fix
is quite simple and npm is part of nodejs anyway.
This PR updates the Python version to 2.7.15 along the way.
This ports all amo-validator related code to run via subprocess. This
makes us less blocked by it not running on Python 3, there may still be
some installation / execution related bits to actually run it via python
2 instead of python 3 but that's something for later.
* Cleans up how and where we're setting _BIN settings for all the
binaries we're calling
* Backports `unicodehelper` from validator
* Removes `requeue_uploads` since it's been broken and unused for at
least 3-4 years
* Remove Standalone Add-on Compatibility Checker (/developers/addon/check-compatibility/)
* Remove compat-check related bits from standalone add-on upload
* Remove more bulk compat report stuff that isn't used anymore
* Removed zadmin application_versions.json view, unused
Fixes#9670
"Fixes" #8857
* Add debian stretch backports repo as discussed with @autrilla
* Add libgit2-dev dependency that's needed by pygit2 (via debian stretch
to get the proper version that's required by up-to-date pygit2)
* Add pygit 0.27.1
Fixes#9496
Icons and previews for both themes and non-themes should go through pngcrush.
For the existing content, the command `crush_images_for_top_addons` focuses
on the content displayed on landing pages only and should only be run once.
* Make also use of setting PYTHONDONTWRITEBYTECODE inside the docker
image to reduce it's size considerably
* Remove some outdated code-parts
* Rename `olympia` inside our worker image to `olympia-worker`
* remove all build and cache folders after every `make update_deps`
(which might help avoid any more package update/install ambiquities)
Fixes#7518
Revert "docker: Fix Debian build issue for python package"
This reverts commit 2239e8483e.
Revert "docker: Include uwsgi plugin"
This reverts commit 807420bf16.
Revert "docker: Include uwsgi"
This reverts commit ceb36b5890.
Revert "docker: Update deploy to use Docker Python image"
This reverts commit 916547d2b5.
Revert "docker: Remove ‘centos’ from docker-indicating file"
This reverts commit d34b92e0e8.
Revert "docker: Update build to use Docker Python image"
This reverts commit b08b51375d.
This provides an updated Python (2.7.14) as part of
https://bugzilla.mozilla.org/show_bug.cgi?id=1390789
The Docker images are not based on Centos so package
names needed to be updated. Comparable packages were
added, but may not contain the same versions or even
components.
* Pin all our dependencies, including pip, six, ipython and ipdb.
This removes all dynamically installed python packages from the
installation pipeline and should result in a cleaner system install.
Fixes#6001
* Cleanup docs, prod and merge a few dependencies with system.
* Move more dependencies from prod to system :-/
* Finally fix docs task
* Revert "Remove old node 4 repo"
This reverts commit 7d221f0814.
* Add back nodesource v6 repo
We need to use nodesource v6 repo for latest security fixes. EPEL updates
are often delayed.
* Compile locales before we run tests, add gettext dependency
* Make compile-mo fail on linting errors.
* Fix tox compilation
* More tox fixes
* Fix path
* remove usedevelop for now
* re-add plural forms
* Install six before we upgrade setuptools.
See pypa/setuptools#964 for a few more details. Currently quite a few
travis and circleci jobs are failing because of that.
* Explicitly upgrade pip and setuptools in 'make update_deps' so for travis too.
* Potentially fix circleci
* Fix setuptools and potentially docs environments
* More explicitly install six and setuptools
* Add more requirements for docs
* Add pyparsing dependency
* Update sha256sum values for a few packages
pypi use different packaging format for a few packages which results in
different sha256sum values.
* Further remove pyrepo dependency from other places.
This is to address https://github.com/mozilla/addons-server/issues/2885
as well, to remove pyrepo from docker build flow.
* Update sha256sum for a few dev packages
* Add swig to support compiling the m2crypto module
* Replace "sudo apt-get" with "addons.apt.packages"
This is because Travis' containerized build infrastructure doesn't
support sudo.
* Add swig to the Dockerfile as well
SWIG_FEATURES="-D__x86_64__" is needed by M2Crypto-0.22.3 on CentOS
platform.
There's now only one place that defines how we install things, the Makefile.
This also merges flake8.txt, test.txt into dev.txt and removed the
unused load.txt and cleans up all installation related code in
Dockerfile and tox.
* Install our node dependencies in update_deps too, we do use at least addons-linter in the near future.
* Make migrations work again
* Fix paths in Makefile, make sure olympia get's installed as a module too
* Fix migrations path in 'initialize_db'
* Make sure settings won't be loaded on top in olympia.startup
* Upgrade to Centos 7
* Use gpg-keys for nodesource and mysql56 repositories
* Uses more prod-like package versions
* Update pyrepo links to use https://pyrepo.stage.mozaws.net/olympia/
(this is what we're using on prod)
Andrew Williamson
Mathieu Pillard
pyup.io bot
Christopher Grebs
Mathieu Pillard
bqbn
Philip Cristiano
Andreas Wagner
Rémy HUBSCHER
Jason Thomas
Andy McKay