* add document_id to index
* update to elasticutils master tip -- this picks up pyelasticsearch
and ditches pyes
* nix get_indexing_es (it was dumb)
* remove bunch of now-unused settings (also dumb)
* tweak test scaffolding -- this should make tests run faster
for kitsune in a VM and on slower machines
* clean up docs
Note: This doesn't remove pyes. For the purposes of making this easier
to review, that'll get done in a future commit.
Note: This adds a new field to the mapping, but it's only used in indexing
and it's obviously a string. Thus it's ok for ES to infer what it is
and we don't have to do the "change the mapping" two-step dance.
* Make LESS required
* Move LESS and memcached instructions to the Requirements section.
* Tweaked and moved the NOTE about empty CSS to the part where
the issue may be encountered.
* URL for the add-on is now a setting, to change it easily.
* AMO will provide automatic updates to the add-on. We could have done
this in SUMO, but it is not trivial.
* The url used here will automatically point to the latest version.
* Remove the old add-on from the media directory.t
* Add docs about the add-on.
* The version of the add-on in AMO will work on -dev and -stage.
This adds:
1. xx locale which is Pirate
2. scripts/poxx.py which translates a .po file into Pirate
3. scripts/test_locale.sh which extracts strings and runs poxx.py
The xx locale is set up so that it doesn't show up in the language
chooser lists, but you can explicitly specify it in the url:
http://localhost:8000/xx/home
The locale is only available translated locally--it will show up
as English on -dev, -stage, and -prod. It'll show up as English
if you don't run scripts/test_locale.sh.
Nixed settings_local.prod.py from the codebase. It never gets updated,
it was old and stale and stinky like the floor of your local dive bar,
and we have better docs already elsewhere.
* clean up the settings_local.py example
* move the redis install and configuration bits to a separate chapter
* add links to the advanced installation section
* moved some bits that were in installation chapter to the hacking_howto
* fixed the note about Elastic Search 0.19.9 so it's correct.
We don't have it set up like this on our machines and if you don't have
redis running, then this causes a bunch of odd problems.
Better to nix it from the docs.
I nixed the Redis section. It shouldn't be in the hacking howto.
I also tweaked the text at the top to be clearer that if you're
running a production instance, you should read all the way through.
This allows us to reindex a specific doctype from the cli and the
admin. This helps when we only need to reindex the kb--now we don't
have to reindex everything with it.
* Splits the docs into three sections---trying to reduce the "omg!
there are a million sections!" overwhelming feeling
* Moves the test-related things to the tests chapter so they're all
in one place
* Turned "coding" into "conventions"---that's what it really is.
* Fleshed out the git commit message conventions---it keeps coming up
and we pointed to a guide which pointed to a guide and the links
were buried in both places.
After talking with Matt, we decided to use the terms "regular" and
"advanced" for searches. This updates the documentation accordingly
and also cleans up some of the other text.
This removes all the Sphinx-related stuff.
You may need to do:
git rm --cached vendors/src/oedipus vendor/src/sphinxapi
to remove the vendor git submodule stuff.
* add --delete to reindex cmd: By default, it **won't** wipe the index.
It now only wipes and rebuilds the index from scratch if you tell it to.
* add note about fixing Redis scoreboard to the docs: If you're doing dev
work and you purge the celery tasks, you also need to reset the Redis
scoreboard that keeps track of outstanding tasks. This adds docs for that.
* add delete-then-index to admin: This reworks the reindex section so that
it has a "delete-then-index" button and makes it clearer which button
does what and why you'd use it.
* adds the DANGER style to admin page.
* reworks indexing to be parallelized when run from the admin
* changes the admin/search page to show outstanding tasks
* removes all the old progress-y stuff
* fixes the --checkindex flag for esstatus
* adds note to the ES documentation about purging celery--that's super
helpful
Since the weights weren't actually being applied since they were specified
wrong, I removed them a couple of commits ago.
Given that, I'm updating the docs to reflect that.
* fixed some typos
* add hacking howto, contact us, and contributor docs
* added an AUTHORS file
The hacking howto probably doesn't work, but it's close. It needs someone
to go through it and fix issues in the docs and in the code to make it
work better.
* Ricky suggested using schematic to tell us the version number
rather than hit the database for it.
* Ricky suggested adding a note for making sure to insert the
content type and default permissions for new models.
* Then I went through and fixed the wrapping for a bunch of
lines
* Splits searching and indexing so they use different index
settings.
* Rename eswhazzup to esstatus and make it more useful.
* Make search admin more useful so it's easier to spot problems.
* Updates eswhazzup to be more useful and nixes the cluster status
* Add x per 1000 documents calculations to reindexing output.
* Fixes wiki document indexing so that when doing a reindexing, we're
not calling unindex for things.
* Add esdelete command.
* Cleans up some code to make it more obvious and less errory.
* Update documentation.
* Update default settings.
This adds documentation for search scoring as it's currently implemented.
Additionally, it adds some minor notes about where the ES-related code is
and links to the search view where the filters are.
Also adds a link to elasticsearch-head.
* update the docs regarding hudson -> jenkins naming of the irc bot
* add note about model makers to coding docs
* fixed a header (cosmetic)
* I added the email/name setting to the quick section since if you miss
this step (which is not unlikely for new contributors), you have
to do some fancy footwork to fix the commits you've made.
* the ES connection already has code for forcing bulk, so we don't need to
repeat that. this changes the code to push the setting to ES.
* this also tweaks the estimation code so that it shows minutes and seconds and
shows the total delta later. Now I can stop running
"time ./manage.py esreindex".
* fix esreindex so that you can specify doctypes. This will appropriately
create/delete indexes so that what you don't want to delete won't get
deleted.
* adds basic handling for bad data.
This does a log.exception, but we really should log more than that and/or
make it more obvious to developers that there's bad data out there.
In the meantime, this allows us to continue indexing.
* reduced memory usage of indexing by iterating over ids---now it runs on
my laptop.
* ghanges _get_index() to get_es_index(). We use it so often it might as
well be part of the "public API".
* fixed create/delete indexes so that to switch doctypes to their own index
is now just a change in settings---no code changes needed.
* fix DEBUG = True case by reseting queries
* this also adds a bunch of helpful comments, moves reindex_model to
SearchMixin.index_all, and has some other cosmetic code cleanup.
End result of this is that indexing doesn't die if it hits bad data, indexing
takes much less memory to run, you can specify specific doctypes to index at
the command line, and the code is better.
This reduces the triplification of everything in es_search modules.
This reduces the triplification of everything in es_search modules.
This reduces the triplification of everything in es_search modules.
Also, I introduced ES_INDEXING_TIMEOUT, fixed documentation, and
cleaned up some other things while I was poking around.
Also also, I nixed all the ES mapping constants. Calling put_mapping
with typos will kick up an error, so there's no need to additionally
have constants around to prevent typos.
This tweaks the language to make it clearer that the vendor stuff
is in the kitsune repository and not a separate one.
It also turns a caveat thing into a warning so it shows up better.
KarmaManager() gets created before we stomp on the REDIS_BACKENDS settings
with REDIS_TEST_BACKENDS. This switches all that testy stuff around to use
settings_test.py which stomps on everything before modules get imported.
The advanced search works, but the front-page search and the searching from within the questions app are still buggy (returning 0 results). Thus, we'll leave the waffle flags off for now.
* unhardcodes indexes
* ditches ElasticMeta--elasticutils uses Model._meta.db_table for
queries, so we should use that for building indexes, too
* implements incremental updates for Question and adds test
* adds some test harness code for future elastic tests
* adds basic docs for setting up Elastic
* Videos are identified by title and locale
* Depends on an update in py-wikimarkup which adds support for <video> and <source> tags (py-wikimarkup commit cc06e6d264622891b6b018e8670c9ef4bb12d618)
* Attaches all the _hook_*s to the WikiParser class, because they need a contextual locale.
* Adds locale support for any of the hooks that do document lookup.
* Uses SWFobject JS lib to support flash fallback for video.
* Adds a migration for unique ('locale', 'title') on gallery_video and gallery_image
* Adds a WIKI_VIDEO_WIDTH|HEIGHT constant that may be used as MAX_WIDTH|HEIGHT in the future, once we get video thumbnails.
* Raise FixtureMissingError in test helpers, e.g. on creating media gallery image and wiki revision, instead of creating testuser
* Add ImageUploadForm and VideoUploadForm
* Simplify gallery app's urls.py
* Adds a bunch of gallery utils to create and upload media
* Adds two views up_media_async, del_media_async
* Rename upload_images to upload_imageattachments
* Define a more generic upload_media function to be used by the gallery app
* Document some of our coding conventions
* Add a migration for the video model
* Define a MAX_FILESIZE for video uploads (16 megabytes), overwritable in settings_local.py
Includes:
* migrations for the models
* a shared abstract model Media for the concrete Image and Video models
* restricting locale choices in the gallery and wiki models
* some check.py complaints and cleanup
* a documentation section for production regarding upload paths
* Tags are added for full and tenths-place variants (e.g., 4.0.1 and 4.0) of known Firefox versions.
* An OS will be auto-tagged if there exists a tag with its name (case insensitive). (I posit ease of administration outweighs risk of abuse.)
* Finished off questions_config based on Cww's etherpad.
* Show user agent string in System Details pop-up.
Will Kahn-Greene
Mike Cooper
Ricky Rosario
Börni
Jen Fong-Adwent
Rehan Dalal
Berker Peksag
Kadir Topal
Cesar Carruitero
Erik Rose
James Socol
Tanay Gavankar
Paul Craciunoiu
Joshua Gross