Move install bits to hacking_howto; cleanup

* clean up the settings_local.py example
* move the redis install and configuration bits to a separate chapter
* add links to the advanced installation section
* moved some bits that were in installation chapter to the hacking_howto
* fixed the note about Elastic Search 0.19.9 so it's correct.
This commit is contained in:
Will Kahn-Greene 2012-12-11 18:03:22 -08:00
Родитель 0d31c226d2
Коммит 93eabd94fe
10 изменённых файлов: 182 добавлений и 443 удалений

Просмотреть файл

@ -1,32 +1,35 @@
.. _celery-chapter:
=================
Celery and Rabbit
=================
Kitsune uses `Celery <http://celeryproject.org/>`_ to enable offline task
processing for long-running jobs like sending email notifications and
re-rendering the Knowledge Base.
Kitsune uses `Celery <http://celeryproject.org/>`_ to enable offline
task processing for long-running jobs like sending email notifications
and re-rendering the Knowledge Base.
Though Celery supports multiple message backends, we use, and recommend that
you use, `RabbitMQ <http://www.rabbitmq.com/>`_. RabbitMQ is an AMQP message
broker written in Erlang.
Though Celery supports multiple message backends, we use, and
recommend that you use, `RabbitMQ
<http://www.rabbitmq.com/>`_. RabbitMQ is an AMQP message broker
written in Erlang.
When is Celery Appropriate
==========================
You can use Celery to do any processing that doesn't need to happen in the
current request-response cycle. Examples are generating thumbnails, sending out
notification emails, updating content that isn't about to be displayed to the
user, and others.
You can use Celery to do any processing that doesn't need to happen in
the current request-response cycle. Examples are generating
thumbnails, sending out notification emails, updating content that
isn't about to be displayed to the user, and others.
Ask yourself the question: "Is the user going to need this data on the page I'm
about to send them?" If not, using a Celery task may be a good choice.
Ask yourself the question: "Is the user going to need this data on the
page I'm about to send them?" If not, using a Celery task may be a
good choice.
RabbitMQ
========
Installing
----------

Просмотреть файл

@ -44,19 +44,19 @@ Linux
-----
We know these work in Debian Testing (Wheezy) and Ubuntu 12.04 and
will probably work in other distributions. It's likely that you'll encounter
some steps that are slightly different. If you run into problems, let
us know.
will probably work in other distributions. It's likely that you'll
encounter some steps that are slightly different. If you run into
problems, let us know.
Requirements
============
For the minimum installation, you'll need the following:
These are required for the minimum installation:
* git
* Python 2.6 or 2.7
* `pip <http://www.pip-installer.org/en/latest/>`_
* pip: `<http://www.pip-installer.org/en/latest/>`_
* MySQL server and client headers
* Memcached Server
* libxml and headers
@ -64,6 +64,12 @@ For the minimum installation, you'll need the following:
* libjpeg and headers
* zlib and headers
These are optional:
* Redis
* LESS
* ElasticSearch: :ref:`search-chapter`
Installation for these is very system dependent. Using a package
manager, like yum, aptitude, or brew, is encouraged.
@ -78,7 +84,7 @@ Grab the source from Github using::
.. Note::
If you forgot to add ``--recursive``, you can get all the
If you forgot to add ``--recursive``, you can still get all the
submodules with::
$ git submodule update --init --recursive
@ -144,14 +150,11 @@ Start by creating a file named ``settings_local.py`` in the
CACHES = {
'default': {
'BACKEND': 'caching.backends.memcached.CacheClass',
'LOCATION': 'localhost:11211'
'LOCATION': 'localhost:11211',
'PREFIX': 'sumo:',
}
}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = ''
CACHE_MIDDLEWARE_SECONDS = 600
# Basic database configuration for development.
DATABASES = {
'default': {
@ -166,19 +169,11 @@ Start by creating a file named ``settings_local.py`` in the
},
}
REDIS_BACKENDS = {
'default': 'redis://localhost:6379?socket_timeout=0.5&db=0',
'karma': 'redis://localhost:6381?socket_timeout=0.5&db=0',
'helpfulvotes': 'redis://localhost:6379?socket_timeout=0.\
5&db=1',
}
REDIS_BACKEND = REDIS_BACKENDS['default']
LESS_PREPROCESS = True
LESS_BIN = '/path/to/kitsune/node_modules/less/bin/lessc'
Don't forget to change ``<YOUR_PASSWORD>``!
Now you can copy and modify any settings from ``settings.py`` into
``settings_local.py`` and the value will override the default.
@ -186,8 +181,8 @@ Now you can copy and modify any settings from ``settings.py`` into
memcached
---------
If you are running Red Hat/CentOS/Fedora, once you have installed memcached you
can start it and configure it to run on startup using::
If you are running RedHat/CentOS/Fedora, once you have installed
memcached you can start it and configure it to run on startup using::
$ chkconfig memcached on
$ /etc/init.d/memcached start
@ -213,7 +208,8 @@ Now install LESS using::
$ npm install less
Ensure that LESS_BIN was configured correctly in your local settings.
Ensure that `LESS_BIN` was configured correctly in your local
settings.
Database
@ -306,15 +302,15 @@ For more information, see the :ref:`test documentation
Advanced install
================
This page covers a minimal install. This minimal install lets you run
Kitsune and work on many parts of Kitsune. However, it's missing some
components.
The above covers a minimal install which will let you run most of
Kitsune. In order to get everything working, you'll need to install
some additional bits.
The following aren't installed in this guide:
See the following chapters for installing those additional bits:
* Redis
* RabbitMQ
* Elastic Search
* Redis: :ref:`redis-chapter`
* RabbitMQ: :ref:`celery-chapter`
* Elastic Search: :ref:`search-chapter`
For installing and configuring those components, you should look at
the more comprehensive :ref:`installation-chapter`.
If you want to install Kitsune on an Apache server in a mod_wsgi
environment, see :ref:`wsgi-chapter`.

Просмотреть файл

@ -19,12 +19,12 @@ Part 2: Developer's Guide
.. toctree::
:maxdepth: 2
installation
conventions
patching
development
tests
celery
redis
wsgi
email
localization

Просмотреть файл

@ -1,350 +0,0 @@
.. _installation-chapter:
============
Installation
============
Requirements
============
To run everything and make all the tests pass locally, you'll need the
following things (in addition to Git, of course).
* Python 2.6 or 2.7.
* `setuptools <http://pypi.python.org/pypi/setuptools#downloads>`_
or `pip <http://pip.openplans.org/>`_.
* MySQL Server and client headers.
* Memcached Server.
* RabbitMQ.
* ``libxml`` and headers.
* ``libxslt`` and headers.
* ``libjpeg`` and headers.
* ``zlib`` and headers.
* LESS
* `Redis <http://redis.io>`_
* Several Python packages. See `Installing the Packages`_.
* Elastic Search. :ref:`search-chapter` covers installation,
configuration, and running.
Installation for these is very system dependent. Using a package manager, like
yum, aptitude, or brew, is encouraged.
Additional Requirements
-----------------------
If you want to use Apache, instead of the dev server (not strictly required but
it's more like our production environment) you'll also need:
* Apache HTTPD Server.
* ``mod_wsgi``
See the documentation on :ref:`WSGI <wsgi-chapter>` for more
information and instructions.
Getting the Source
==================
Grab the source from Github using::
$ git clone --recursive git://github.com/mozilla/kitsune.git
$ cd kitsune
If you forgot to add ``--recursive``, you can get all the submodules with::
$ git submodule update --init --recursive
Installing the Packages
=======================
Compiled Packages
-----------------
There are a small number of compiled packages, including the MySQL Python
client. You can install these using ``pip`` (if you don't have ``pip``, you
can get it with ``easy_install pip``) or via a package manager.
To use ``pip``, you only need to do this::
$ sudo pip install -r requirements/compiled.txt
Python Packages
---------------
All the pure-Python requirements are provided in the ``vendor`` directory, also
known as the "vendor library". This makes the packages available to Python
without installing them globally and keeps them pinned to known-compatible
versions.
See the :ref:`vendor library <vendor-chapter>` documentation for more
information on getting the vendor lib and keeping it up to date.
Configuration
=============
Start by creating a file named ``settings_local.py`` in the kitsune
directory, and putting this line in it::
from settings import *
Now you can copy and modify any settings from ``settings.py`` into
``settings_local.py`` and the value will override the default.
For local development you will want to add the following settings::
DEBUG = True
TEMPLATE_DEBUG = DEBUG
SESSION_COOKIE_SECURE = False
# Allows you to run Kitsune without running Celery---all tasks
# will be done synchronously.
CELERY_ALWAYS_EAGER = False
# Allows you to specify waffle settings in the querystring.
WAFFLE_OVERRIDE = True
# Change this to True if you're going to be doing search-related
# work.
ES_LIVE_INDEXING = False
# Basic cache configuration for development.
CACHES = {
'default': {
'BACKEND': 'caching.backends.memcached.CacheClass',
'LOCATION': 'localhost:11211'
}
}
CACHE_BACKEND = 'caching.backends.memcached://localhost:11211'
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = ''
CACHE_MIDDLEWARE_SECONDS = 600
CACHE_PREFIX = 'sumo:'
# Basic database configuration for development.
DATABASES = {
'default': {
'NAME': 'kitsune',
'ENGINE': 'django.db.backends.mysql',
'HOST': 'localhost',
'USER': 'kitsune',
'PASSWORD': '<YOUR_PASSWORD>',
'OPTIONS': {'init_command': 'SET storage_engine=InnoDB'},
'TEST_CHARSET': 'utf8',
'TEST_COLLATION': 'utf8_unicode_ci',
},
}
REDIS_BACKENDS = {
'default': 'redis://localhost:6379?socket_timeout=0.5&db=0',
'karma': 'redis://localhost:6381?socket_timeout=0.5&db=0',
'helpfulvotes': 'redis://localhost:6379?socket_timeout=0.\
5&db=1',
}
REDIS_BACKEND = REDIS_BACKENDS['default']
LESS_PREPROCESS = True
LESS_BIN = '/path/to/kitsune/node_modules/less/bin/lessc'
Redis
-----
You need to copy the ``REDIS_BACKEND`` section from ``settings.py``
into your ``settings_local.py``. After doing that, uncomment the
three lines in each section.
There are three ``.conf`` files in ``configs/redis/``. One is for
testing and is used in ``settings_test.py``. The other two are used
for the sections in ``REDIS_BACKEND``.
There are two ways to set this up. First is to set it up like in
``settings.py`` and run all three redis servers. The second is to set
it up differently, tweak the settings in ``settings_local.py``
accordingly, and run Redis using just the test configuration.
memcache
--------
.. Note::
This should probably be somewhere else, but the easy way to flush
your cache is something like this::
echo "flush_all" | nc localhost 11211
Assuming you have memcache configured to listen to 11211.
Database
--------
At a minimum, you will need to define a database connection. An example
configuration is::
DATABASES = {
'default': {
'NAME': 'kitsune',
'ENGINE': 'django.db.backends.mysql',
'HOST': 'localhost',
'USER': 'kitsune',
'PASSWORD': '<YOUR_PASSWORD>',
'OPTIONS': {'init_command': 'SET storage_engine=InnoDB'},
'TEST_CHARSET': 'utf8',
'TEST_COLLATION': 'utf8_unicode_ci',
},
}
Note the two settings ``TEST_CHARSET`` and ``TEST_COLLATION``. Without these,
the test suite will use MySQL's (moronic) defaults when creating the test
database (see below) and lots of tests will fail. Hundreds.
Create the database and grant permissions to the user, based on your database
settings. For example, using the settings above::
$ mysql -u root -p
mysql> CREATE DATABASE kitsune;
mysql> GRANT ALL ON kitsune.* TO kitsune@localhost IDENTIFIED BY '<YOUR_PASSWORD>';
To load the latest database schema, use ``scripts/schema.sql`` and
``schematic``::
$ mysql -u kitsune -p kitsune < scripts/schema.sql
$ ./vendor/src/schematic/schematic migrations/
You'll now have an empty but up-to-date database!
Finally, you'll probably want to create a superuser. Just use Django's
``createsuperuser`` management command::
$ ./manage.py createsuperuser
And follow the prompts. After logging in, you can create a profile for the
user by going to ``/users/edit`` in your browser.
See also the :ref:`important wiki documents <wiki-chapter>` documentation.
Product Details Initialization
------------------------------
One of the packages Kitsune uses, ``product_details``, needs to fetch JSON
files containing historical Firefox version data and write them within its
package directory. To set this up, run this command to do the initial fetch::
$ ./manage.py update_product_details
LESS
----
To install LESS you will first need to `install Node.js and NPM
<https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager>`_.
Now install LESS using::
$ npm install less
Ensure that LESS_BIN was configured correctly in your local settings.
Running redis
-------------
This script runs all three servers---one for each configuration.
I (Will) put that in a script that creates the needed directories in
``/var/redis/`` and kicks off the three redis servers::
#!/bin/bash
set -e
# Adjust these according to your setup!
REDISBIN=/usr/bin/redis-server
CONFFILE=/path/to/conf/files/
if test ! -e /var/redis/sumo/
then
echo "creating /var/redis/sumo/"
mkdir -p /var/redis/sumo/
fi
if test ! -e /var/redis/sumo-test/
then
echo "creating /var/redis/sumo-test/"
mkdir -p /var/redis/sumo-test/
fi
if test ! -e /var/redis/sumo-persistent/
then
echo "creating /var/redis/sumo-persistent/"
mkdir -p /var/redis/sumo-persistent/
fi
$REDISBIN $CONFFILE/redis-persistent.conf
$REDISBIN $CONFFILE/redis-test.conf
$REDISBIN $CONFFILE/redis-volatile.conf
Elastic search
--------------
Elastic Search. :ref:`search-chapter` covers installation,
configuration, and running.
.. todo:: The installation side of these two should get moved here.
Testing it Out
==============
To start the dev server, run ``./manage.py runserver``, then open up
``http://localhost:8000``. If everything's working, you should see a somewhat
empty version of the SUMO home page!
Running the Tests
-----------------
A great way to check that everything really is working is to run the test
suite. You'll need to add an extra grant in MySQL for your database user::
GRANT ALL ON test_NAME.* TO USER@localhost;
Where ``NAME`` and ``USER`` are the same as the values in your database
configuration.
The test suite will create and use this database, to keep any data in your
development database safe from tests.
Running the test suite is easy::
$ ./manage.py test -s --noinput --logging-clear-handlers
For more information, see the :ref:`test documentation <tests-chapter>`.
.. Note::
Some of us use `nose-progressive
<https://github.com/erikrose/nose-progressive>`_ because it makes
tests easier to run and debug.

Просмотреть файл

@ -16,7 +16,7 @@ fairly straight-forward workflow in git for submitting patches. This is
outlined below.
You should run the tests before submitting a pull request. You can find help
for getting set up in the :ref:`installation docs <installation-chapter>`.
for getting set up in the :ref:`installation docs <hacking-howto-chapter>`.
If you ever find yourself stuck, come look for us in `#sumodev
<ircs://irc.mozilla.org/sumodev>`_ on Mozilla's IRC network. We're happy to

84
docs/redis.rst Normal file
Просмотреть файл

@ -0,0 +1,84 @@
.. _redis-chapter:
=====
Redis
=====
This covers installing `Redis <http://redis.io/>`_.
Installation
============
Install Redis on your machine.
There are three ``.conf`` files in ``configs/redis/``. One is for
testing and is used in ``settings_test.py``. The other two are used
for the sections in ``REDIS_BACKEND``.
There are two ways to set this up. First is to set it up using the
configuration below and run three separate Redis servers. The second
is to set it up differently, tweak the settings in
``settings_local.py`` accordingly, and run Redis using just the test
configuration.
Configuration
=============
If you want to run three separate Redis servers, add this to
``settings_local.py``::
REDIS_BACKENDS = {
'default': 'redis://localhost:6379?socket_timeout=0.5&db=0',
'karma': 'redis://localhost:6381?socket_timeout=0.5&db=0',
'helpfulvotes': 'redis://localhost:6379?socket_timeout=0.\
5&db=1',
}
REDIS_BACKEND = REDIS_BACKENDS['default']
Otherwise adjust the above accordingly.
Running redis
=============
Running redis
-------------
This script runs all three servers---one for each configuration.
I (Will) put that in a script that creates the needed directories in
``/var/redis/`` and kicks off the three redis servers::
#!/bin/bash
set -e
# Adjust these according to your setup!
REDISBIN=/usr/bin/redis-server
CONFFILE=/path/to/conf/files/
if test ! -e /var/redis/sumo/
then
echo "creating /var/redis/sumo/"
mkdir -p /var/redis/sumo/
fi
if test ! -e /var/redis/sumo-test/
then
echo "creating /var/redis/sumo-test/"
mkdir -p /var/redis/sumo-test/
fi
if test ! -e /var/redis/sumo-persistent/
then
echo "creating /var/redis/sumo-persistent/"
mkdir -p /var/redis/sumo-persistent/
fi
$REDISBIN $CONFFILE/redis-persistent.conf
$REDISBIN $CONFFILE/redis-test.conf
$REDISBIN $CONFFILE/redis-volatile.conf

Просмотреть файл

@ -32,18 +32,18 @@ or 0.19.8.
.. Note::
**Don't** install 0.19.9 or later. They've changed bits of the API in
a way that doesn't yet work with Kitsune.
**Don't** install 0.19.9 or 0.19.10. They have bugs that don't work
in Kitsune.
The directory you install Elastic in will hereafter be referred to as
``ELASTICDIR``.
The directory you install Elastic Search in will hereafter be referred
to as ``ELASTICDIR``.
You can configure Elastic Search with the configuration file at
``ELASTICDIR/config/elasticsearch.yml``.
Elastic Search uses several settings in ``settings.py`` that you
should override in ``settings_local.py``::
need to override in ``settings_local.py``. Here's an example::
# Connection information for Elastic
ES_HOSTS = ['127.0.0.1:9200']
@ -51,6 +51,8 @@ should override in ``settings_local.py``::
ES_WRITE_INDEXES = ES_INDEXES
These settings explained:
``ES_HOSTS``
No default.
@ -286,7 +288,6 @@ On the command line, you can do::
$ ./manage.py esdelete <index-name>
Tools
-----

Просмотреть файл

@ -13,7 +13,7 @@ Running the Test Suite
======================
If you followed the steps in :ref:`the installation docs
<installation-chapter>`, then all you should need to do to run the
<hacking-howto-chapter>`, then all you should need to do to run the
test suite is::
./manage.py test

Просмотреть файл

@ -8,21 +8,22 @@ To help make setup faster and deployment easier, we pull all of our
pure-Python dependencies into a "vendor library" (``kitsune/vendor``)
in the kitsune repository and add them to the path in ``manage.py``.
The vendor library used to be optional, with a virtualenv option available, as
well. While it's still possible to install the compiled requirements in a
virtualenv, we've decided to simplify docs/setup/tooling and encourage
environments to be as similar to production as possible, by settling on the
vendor library as the only method for managing dependencies.
The vendor library used to be optional, with a virtualenv option
available, as well. While it's still possible to install the compiled
requirements in a virtualenv, we've decided to simplify
docs/setup/tooling and encourage environments to be as similar to
production as possible, by settling on the vendor library as the only
method for managing dependencies.
Getting the Vendor Library and Keeping Up-to-Date
=================================================
If you cloned Kitsune with ``--recursive``, you already have the vendor library
in ``vendor/``.
If you cloned Kitsune with ``--recursive``, you already have the
vendor library in ``vendor/``.
If you didn't clone with ``--recursive``, or need to update the vendor library
(or other submodules), just run::
If you didn't clone with ``--recursive``, or need to update the vendor
library (or other submodules), just run::
$ git submodule update --init --recursive
@ -32,25 +33,25 @@ Aliasing that to something short (e.g. ``gsu``) is recommended.
Updating the Vendor Library
===========================
From time to time we need to update libraries, either for new versions of
libraries or to add a new library. There are two ways to do that. The easiest
and prefered way is pure git.
From time to time we need to update libraries, either for new versions
of libraries or to add a new library. There are two ways to do
that. The easiest and prefered way is pure git.
Using Git Submodules
--------------------
Using git submodules is prefered because it is much easier to maintain, and it
keeps the repository size small. Upgrading is as simple as updating a
submodule.
Using git submodules is prefered because it is much easier to
maintain, and it keeps the repository size small. Upgrading is as
simple as updating a submodule.
Updating a Library with Git Submodules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If the library is in ``vendor/src``, it was pulled directly from version
control, and if that version control was git, updating the submodule is as easy
as::
If the library is in ``vendor/src``, it was pulled directly from
version control, and if that version control was git, updating the
submodule is as easy as::
$ cd vendor/src/$LIBRARY
$ git fetch origin
@ -65,8 +66,9 @@ Easy! Just like updating any other git submodule.
Adding a New Library with Git Submodules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Technically this can be done with ``pip install --no-install`` but there's an
even easier method when installing a new library from a git repo::
Technically this can be done with ``pip install --no-install`` but
there's an even easier method when installing a new library from a git
repo::
$ cd vendor/src
$ git clone git://<repo>
@ -86,15 +88,16 @@ even easier method when installing a new library from a git repo::
Using PyPI
----------
Sometimes a library isn't in a git repository. It, sadly, happens. Maybe you
can find a git mirror? If not, it might as well be installed from PyPI.
Sometimes a library isn't in a git repository. It, sadly,
happens. Maybe you can find a git mirror? If not, it might as well be
installed from PyPI.
Updating a Library from PyPI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The easiest way to update a library from PyPI is to remove it completely and
then install the new version.
The easiest way to update a library from PyPI is to remove it
completely and then install the new version.
::
@ -139,16 +142,16 @@ Make sure you add any dependencies from the new library, as well.
Requirements Files
==================
There are a few requirements that are not included in the vendor library
because they need to be (or can be, for performance benefits) compiled (or have
compiled dependencies themselves).
There are a few requirements that are not included in the vendor
library because they need to be (or can be, for performance benefits)
compiled (or have compiled dependencies themselves).
You can :ref:`install <installation-chapter>` these in a virtualenv or at the
system level by running::
You can :ref:`install <hacking-howto-chapter>` these in a virtualenv
or at the system level by running::
$ pip install -r requirements/compiled.txt
If you want to run coverage builds or are having issues with tests, you can
run::
If you want to run coverage builds or are having issues with tests,
you can run::
$ pip install -r requirements/tests-compiled.txt

Просмотреть файл

@ -8,7 +8,7 @@ Running Kitsune with mod_wsgi
Requirements
============
* See :ref:`the installation docs <installation-chapter>`.
* See :ref:`the installation docs <hacking-howto-chapter>`.
* `Apache HTTP server <http://httpd.apache.org/>`_
* mod_rewrite
* mod_headers
@ -20,9 +20,10 @@ Requirements
Overview
========
Setting up Kitsune to run as a WSGI application is fairly straightforward. You
will need to install the requirements and clone the vendor repo as described in
:ref:`the installation chapter <installation-chapter>`.
Setting up Kitsune to run as a WSGI application is fairly
straightforward. You will need to install the requirements and clone
the vendor repo as described in :ref:`the installation chapter
<hacking-howto-chapter>`.
There are 3 steps once Kitsune is installed:
@ -35,9 +36,10 @@ There are 3 steps once Kitsune is installed:
Apache Modules
--------------
Most of the Apache modules are part of a default Apache install, but may need
to be activated. If they aren't installed, all of them, including ``mod_wsgi``
should be installable via your favorite package manager.
Most of the Apache modules are part of a default Apache install, but
may need to be activated. If they aren't installed, all of them,
including ``mod_wsgi`` should be installable via your favorite package
manager.
WSGI Configuration