mozilla
/
Bugzilla-ETL
зеркало из https://github.com/mozilla/Bugzilla-ETL.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

lib updates

This commit is contained in:
Kyle Lahnakoski 2018-03-26 10:46:44 -04:00
Родитель 0119ad06a5
Коммит b16238982f
134 изменённых файлов: 1801 добавлений и 7281 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

373
vendor/LICENSE.txt поставляемый
Просмотреть файл

@ -1,373 +0,0 @@
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.

21
vendor/MANIFEST.in сгенерированный поставляемый
Просмотреть файл

@ -1,21 +0,0 @@
graft tests
graft docs
prune .git
prune .svn
exclude build
prune dist
prune .idea
exclude *.egg-info
exclude *.iml
exclude *.txt
exclude *.tab
exclude README.md
exclude *.json
exclude .git*
recursive-exclude . *.pyc
recursive-include . __init__.py
include README.txt
exclude MANIFEST.in

133
vendor/README.md поставляемый
Просмотреть файл

@ -1,133 +0,0 @@
pyLibrary
=========
A library of wonderful Python things!
Motivation
----------
This library is born from my version of the `utils` library everyone makes.
Only, instead of being utilities that are specific to the task, these utilities
are for programming in general: They assume logs should be structured,
all data should be JSONizable, and OO is preferred, and more.
### Python is a Little Crufty ###
Python is awesome now, but it was originally a procedural language invented
before pure functional semantics, before OO, and even before the
discovery of vowels. As a consequence there are many procedures that alter
their own parameters, or have disemvoweled names. This library puts a facade
over these relics of the past and uses convention to name methods.
Installing pyLibrary
--------------------
Python packages are easy to install, assuming you have Python (see below).
pip install pyLibrary
Installing for Development
--------------------------
* Download from Github:
git clone https://github.com/klahnakoski/pyLibrary.git
* Install requirements:
python setup.py develop
Windows 7 Install Instructions for Python
-----------------------------------------
Updated November 2014, for Python 2.7.8
Python was really made for Linux, and installation will be easier there.
Technically, Python works on Windows too, but there are a few gotchas you can
avoid by following these instructions.
* Download Python 2.7
* 32bit ONLY!!! Many native libs are 32 bit
* Varsion 2.7.8 or higher (includes pip, so install is easier)
* Install Python at ```c:\Python27``` (The space in the "Program Files" may screw up installs of native libs)
* Add to you path: ```c:\Python27;c:\Python27\scripts;```
* Download ```https://bootstrap.pypa.io/get-pip.py```
CALL python get-pip.py
CALL pip install virtualenv
* Many "Python Powered" native installs require a pointer to the python installation, but they have no idea where to
look in 64bit windows. You must alter the registry ([http://stackoverflow.com/questions/3652625/installing-setuptools-on-64-bit-windows](http://stackoverflow.com/questions/3652625/installing-setuptools-on-64-bit-windows)):
SET HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPath = "C:\Python27"
###Using virtualenv
```virtualenv``` allows you to have multiple python projects on the same
machine, even if they use different versions of the same libraries.
```virtualenv``` does this by making a copy of the main python directory and
using it to hold the specific versions required.
* New environment: ```virtualenv <name_of_dir>```
* Activate environment: ```<name_of_dir>\scripts\activate```
* Exit environment: ```deactivate```
If you have more than one project on your dev box I suggest you do all your
work inside a virtual environment.
### PyPy and Virtual Environments
```virtualenv``` can be used with PyPy, but it is a bit more involved. The
paths must be explict, and some copying is required.
#### New environment:
The first call to virtualenv will make the directory, to which you copy the
PyPy core libraries, and the second call finishes the install.
c:\PyPy27\bin\virtualenv <name_of_dir>
copy c:\PyPy27\bin\lib_pypy <name_of_dir>
copy c:\PyPy27\bin\lib_python <name_of_dir>
c:\PyPy27\bin\virtualenv <name_of_dir>
#### Activate environment:
With CPython ```virtualenv``` places it's executables in ```Scripts```. The
PyPy version uses ```bin```
<name_of_dir>\bin\activate
#### Using PIP in PyPy:
PyPy does not share any libraries with CPython. You must install the PyPy libraries using
C:\pypy\bin\pip.exe
The `pip` found in your `%PATH%` probably points to `C:\python27\Scripts\pip.exe`.
#### Using PIP in PyPy virtualenv:
Do **NOT** use the ```<name_of_dir>\Scripts``` directory: It installs to your
main PyPy installation. Pip install is done using the `bin` directory:
<name_of_dir>\bin\pip.exe
#### Exit environment:
Deactivation is like normal
deactivate
### CPython Binaries and Virtual Environments
If you plan to use any binary packages, ```virtualenv``` will not work
directly. Instead, install the binary (32 bit only!!) to the main python
installation. Then copy any newly installed files/directories from
```C:\Python27\Lib\site-packages``` to ```<name_of_dir>\Lib\site-packages```.
### Binaries and PyPy
This strategy for installing binaries into Virtual Environments is almost
identical to installing binaries into your PyPy environment: Install Numpy
and Scipy to your CPython installation using a windows installer (which has
pre-compiled binaries), and then copy the ```C:\Python27\Lib\site-packages\<package>```
to ```c:\PyPy\site-packages\```; note lack of ```Lib``` subdirectory.

0
vendor/__init__.py поставляемый
Просмотреть файл

Двоичные данные
vendor/docs/Libraries.pptx поставляемый
Просмотреть файл

Двоичный файл не отображается.

29
vendor/jx_elasticsearch/__init__.py поставляемый
Просмотреть файл

@ -19,6 +19,9 @@ from mo_logs import Log
from mo_logs.url import URL
from pyLibrary.env import http
DEBUG = False
known_hosts = {}
@ -43,7 +46,7 @@ def new_instance(
url = URL(host)
url.port = port
status = http.get_json(text_type(url))
status = http.get_json(text_type(url), stream=False)
version = status.version.number
if version.startswith("1."):
from jx_elasticsearch.es14 import ES14
@ -61,3 +64,27 @@ def new_instance(
Log.error("No jx interpreter for Elasticsearch {{version}}", version=version)
except Exception as e:
Log.error("Can not make an interpreter for Elasticsearch", cause=e)
# SCRUB THE QUERY SO IT IS VALID
# REPORT ERROR IF OUTPUT APEARS TO HAVE HIT GIVEN limit
def post(es, es_query, limit):
post_result = None
try:
if not es_query.sort:
es_query.sort = None
post_result = es.search(es_query)
for facetName, f in post_result.facets.items():
if f._type == "statistical":
continue
if not f.terms:
continue
if not DEBUG and not limit and len(f.terms) == limit:
Log.error("Not all data delivered (" + str(len(f.terms)) + "/" + str(f.total) + ") try smaller range")
except Exception as e:
Log.error("Error with FromES", e)
return post_result

2
vendor/jx_elasticsearch/es09/aggop.py поставляемый
Просмотреть файл

@ -14,7 +14,7 @@ from __future__ import unicode_literals
from jx_base.queries import is_variable_name
from jx_elasticsearch import es09
from jx_elasticsearch.es09.util import aggregates, fix_es_stats, build_es_query
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es52.expressions import Variable
from jx_python.containers.cube import Cube
from jx_python.expressions import jx_expression_to_function

5
vendor/jx_elasticsearch/es09/expressions.py поставляемый
Просмотреть файл

@ -15,6 +15,7 @@ from collections import Mapping
from datetime import datetime
import re
from mo_future import text_type
from pyLibrary import convert
from mo_collections import reverse
from mo_logs import Log
@ -48,7 +49,7 @@ class _MVEL(object):
body = "var output = \"\";\n" + \
code.replace(
"<CODE>",
"if (" + _where(whereClause, lambda(v): self._translate(v)) + "){\n" +
"if (" + _where(whereClause, lambda v: self._translate(v)) + "){\n" +
select.body +
"}\n"
) + \
@ -584,7 +585,7 @@ FUNCTIONS = {
"}};\n",
"Value2Pipe":
'var Value2Pipe = function(value){\n' + # SPACES ARE IMPORTANT BETWEEN "="
'var Value2Pipe = function(value){\n' + # SPACES ARE IMPORTANT BETWEEN "=".
"if (value==null){ \"0\" }else " +
"if (value is ArrayList || value is org.elasticsearch.common.mvel2.util.FastList){" +
"var out = \"\";\n" +

2
vendor/jx_elasticsearch/es09/setop.py поставляемый
Просмотреть файл

@ -19,7 +19,7 @@ from jx_base.queries import is_variable_name
from jx_elasticsearch import es09
from jx_elasticsearch.es09.expressions import unpack_terms
from jx_elasticsearch.es09.util import aggregates
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_python.containers.cube import Cube
from mo_collections.matrix import Matrix
from mo_dots import coalesce, split_field, Data, wrap

2
vendor/jx_elasticsearch/es09/terms.py поставляемый
Просмотреть файл

@ -12,7 +12,7 @@ from __future__ import division
from __future__ import unicode_literals
from jx_elasticsearch.es09.util import aggregates, build_es_query, compileEdges2Term
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_python import jx
from jx_python.containers.cube import Cube
from mo_collections.matrix import Matrix

24
vendor/jx_elasticsearch/es09/util.py поставляемый
Просмотреть файл

@ -26,31 +26,9 @@ from jx_base import domains
from jx_elasticsearch.es09.expressions import value2MVEL, isKeyword
from mo_times import durations
TrueFilter = {"match_all": {}}
DEBUG = False
# SCRUB THE QUERY SO IT IS VALID
# REPORT ERROR IF OUTPUT APEARS TO HAVE HIT GIVEN limit
def post(es, es_query, limit):
post_result = None
try:
if not es_query.sort:
es_query.sort = None
post_result = es.search(es_query)
for facetName, f in post_result.facets.items():
if f._type == "statistical":
continue
if not f.terms:
continue
if not DEBUG and not limit and len(f.terms) == limit:
Log.error("Not all data delivered (" + str(len(f.terms)) + "/" + str(f.total) + ") try smaller range")
except Exception as e:
Log.error("Error with FromES", e)
return post_result
def build_es_query(query):
output = wrap({

2
vendor/jx_elasticsearch/es14/__init__.py поставляемый
Просмотреть файл

@ -126,7 +126,7 @@ class ES14(Container):
def query(self, _query):
try:
query = QueryOp.wrap(_query, schema=self)
query = QueryOp.wrap(_query, _query.frum, schema=self)
for n in self.namespaces:
query = n.convert(query)

2
vendor/jx_elasticsearch/es14/aggs.py поставляемый
Просмотреть файл

@ -16,7 +16,7 @@ from mo_future import text_type
from jx_base.domains import SetDomain
from jx_base.expressions import TupleOp, NULL
from jx_base.query import DEFAULT_LIMIT, MAX_LIMIT
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es14.decoders import DefaultDecoder, AggsDecoder, ObjectDecoder
from jx_elasticsearch.es14.decoders import DimFieldListDecoder
from jx_elasticsearch.es14.expressions import split_expression_by_depth, AndOp, Variable, NullOp

2
vendor/jx_elasticsearch/es14/deep.py поставляемый
Просмотреть файл

@ -14,7 +14,7 @@ from __future__ import unicode_literals
from jx_base import STRUCT, NESTED, EXISTS
from jx_base.expressions import NULL
from jx_base.query import DEFAULT_LIMIT
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es14.expressions import split_expression_by_depth, AndOp, Variable, LeavesOp
from jx_elasticsearch.es14.setop import format_dispatch, get_pull_function, get_pull
from jx_elasticsearch.es14.util import jx_sort_to_es_sort, es_query_template

17
vendor/jx_elasticsearch/es14/expressions.py поставляемый
Просмотреть файл

@ -22,9 +22,9 @@ from jx_base.expressions import Variable, TupleOp, LeavesOp, BinaryOp, OrOp, Scr
PrefixOp, NotLeftOp, InOp, CaseOp, AndOp, \
ConcatOp, IsNumberOp, Expression, BasicIndexOfOp, MaxOp, MinOp, BasicEqOp, BooleanOp, IntegerOp, BasicSubstringOp, ZERO, NULL, FirstOp, FALSE, TRUE, simplified
from mo_dots import coalesce, wrap, Null, unwraplist, set_default, literal_field
from mo_json import quote
from mo_logs import Log, suppress_exception
from mo_logs.strings import expand_template
from mo_logs.strings import expand_template, quote
from mo_math import MAX, OR
from pyLibrary.convert import string2regexp
@ -139,7 +139,18 @@ def to_ruby(self, schema):
@extend(CaseOp)
def to_esfilter(self, schema):
return ScriptOp("script", self.to_ruby(schema).script(schema)).to_esfilter(schema)
if self.type == BOOLEAN:
return OrOp(
"or",
[
AndOp("and", [w.when, w.then])
for w in self.whens[:-1]
] +
self.whens[-1:]
).partial_eval().to_esfilter(schema)
else:
Log.error("do not know how to handle")
return ScriptOp("script", self.to_ruby(schema).script(schema)).to_esfilter(schema)
@extend(ConcatOp)

2
vendor/jx_elasticsearch/es14/setop.py поставляемый
Просмотреть файл

@ -17,7 +17,7 @@ from jx_base import NESTED
from jx_base.domains import ALGEBRAIC
from jx_base.expressions import IDENTITY
from jx_base.query import DEFAULT_LIMIT
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es14.expressions import Variable, LeavesOp
from jx_elasticsearch.es14.util import jx_sort_to_es_sort, es_query_template
from jx_python.containers.cube import Cube

17
vendor/jx_elasticsearch/es52/__init__.py поставляемый
Просмотреть файл

@ -26,16 +26,11 @@ from jx_elasticsearch.es52.setop import is_setop, es_setop
from jx_elasticsearch.es52.util import aggregates
from jx_elasticsearch.meta import FromESMetadata
from jx_python import jx
from mo_dots import Data, Null, unwrap
from mo_dots import coalesce, split_field, literal_field, unwraplist, join_field
from mo_dots import wrap, listwrap
from mo_dots.lists import FlatList
from mo_json import scrub
from mo_dots import Data, Null, unwrap, coalesce, split_field, literal_field, unwraplist, join_field, wrap, listwrap, FlatList
from mo_json import scrub, value2json
from mo_json.typed_encoder import TYPE_PREFIX
from mo_kwargs import override
from mo_logs import Log
from mo_logs.exceptions import Except
from pyLibrary import convert
from mo_logs import Log, Except
from pyLibrary.env import elasticsearch, http
@ -127,13 +122,13 @@ class ES52(Container):
def query(self, _query):
try:
query = QueryOp.wrap(_query, table=self)
query = QueryOp.wrap(_query, table=self, schema=self.schema)
for n in self.namespaces:
query = n.convert(query)
for s in listwrap(query.select):
if not aggregates.get(s.aggregate):
if s.aggregate != None and not aggregates.get(s.aggregate):
Log.error(
"ES can not aggregate {{name}} because {{aggregate|quote}} is not a recognized aggregate",
name=s.name,
@ -222,7 +217,7 @@ class ES52(Container):
for s in scripts:
updates.append({"update": {"_id": h._id, "_routing": unwraplist(h.fields[literal_field(schema._routing.path)])}})
updates.append(s)
content = ("\n".join(convert.value2json(c) for c in updates) + "\n").encode('utf-8')
content = ("\n".join(value2json(c) for c in updates) + "\n")
response = self._es.cluster.post(
self._es.path + "/_bulk",
data=content,

99
vendor/jx_elasticsearch/es52/aggs.py поставляемый
Просмотреть файл

@ -11,28 +11,73 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from mo_future import text_type
from jx_base import OBJECT, EXISTS
from jx_base import EXISTS
from jx_base.domains import SetDomain
from jx_base.expressions import TupleOp, NULL
from jx_base.query import DEFAULT_LIMIT
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch.es52.decoders import DefaultDecoder, AggsDecoder, ObjectDecoder
from jx_elasticsearch.es52.decoders import DimFieldListDecoder
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es52.decoders import DefaultDecoder, AggsDecoder, ObjectDecoder, DimFieldListDecoder
from jx_elasticsearch.es52.expressions import split_expression_by_depth, AndOp, Variable, NullOp
from jx_elasticsearch.es52.setop import get_pull_stats
from jx_elasticsearch.es52.util import aggregates
from jx_python import jx
from jx_python.expressions import jx_expression_to_function
from mo_dots import listwrap, Data, wrap, literal_field, set_default, coalesce, Null, split_field, FlatList, unwrap, unwraplist
from mo_future import text_type
from mo_json.typed_encoder import encode_property
from mo_logs import Log
from mo_logs.strings import quote
from mo_logs.strings import quote, expand_template
from mo_math import Math, MAX, UNION
from mo_times.timer import Timer
COMPARE_TUPLE = """
(a, b)->{
int i=0;
for (dummy in a){ //ONLY THIS FOR LOOP IS ACCEPTED (ALL OTHER FORMS THROW NullPointerException)
if (a[i]==null) return -1*({{dir}});
if (b[i]==null) return 1*({{dir}});
if (a[i]!=b[i]) {
if (a[i] instanceof Boolean){
if (b[i] instanceof Boolean){
int cmp = Boolean.compare(a[i], b[i]);
if (cmp != 0) return cmp;
} else {
return -1;
}//endif
}else if (a[i] instanceof Number) {
if (b[i] instanceof Boolean) {
return 1
} else if (b[i] instanceof Number) {
int cmp = Double.compare(a[i], b[i]);
if (cmp != 0) return cmp;
} else {
return -1;
}//endif
}else {
if (b[i] instanceof Boolean) {
return 1;
} else if (b[i] instanceof Number) {
return 1;
} else {
int cmp = ((String)a[i]).compareTo((String)b[i]);
if (cmp != 0) return cmp;
}//endif
}//endif
}//endif
i=i+1;
}//for
return 0;
}
"""
MAX_OF_TUPLE = """
(Object[])Arrays.asList(new Object[]{{{expr1}}, {{expr2}}}).stream().{{op}}("""+COMPARE_TUPLE+""").get()
"""
def is_aggsop(es, query):
es.cluster.get_metadata()
if query.edges or query.groupby or any(a != None and a != "none" for a in listwrap(query.select).aggregate):
@ -61,7 +106,7 @@ def get_decoders_by_depth(query):
edge = edge.copy()
vars_ = edge.value.vars()
for v in vars_:
if not schema.leaves(v, meta=True):
if not schema.leaves(v.var, meta=True):
Log.error("{{var}} does not exist in schema", var=v)
elif edge.range:
vars_ = edge.range.min.vars() | edge.range.max.vars()
@ -79,7 +124,7 @@ def get_decoders_by_depth(query):
try:
vars_ |= edge.value.vars()
depths = set(len(c.nested_path) - 1 for v in vars_ for c in schema.leaves(v))
depths = set(len(c.nested_path) - 1 for v in vars_ for c in schema.leaves(v.var))
if -1 in depths:
Log.error(
"Do not know of column {{column}}",
@ -137,7 +182,7 @@ def es_aggsop(es, frum, query):
new_select["count_"+literal_field(s.value.var)] += [s]
else:
new_select[literal_field(s.value.var)] += [s]
else:
elif s.aggregate:
formula.append(s)
for canonical_name, many in new_select.items():
@ -243,8 +288,34 @@ def es_aggsop(es, frum, query):
if s.aggregate == "count":
# TUPLES ALWAYS EXIST, SO COUNTING THEM IS EASY
s.pull = "doc_count"
elif s.aggregate in ('max', 'maximum', 'min', 'minimum'):
if s.aggregate in ('max', 'maximum'):
dir = 1
op = "max"
else:
dir = -1
op = 'min'
nully = TupleOp("tuple", [NULL]*len(s.value.terms)).partial_eval().to_painless(schema).expr
selfy = s.value.partial_eval().to_painless(schema).expr
script = {"scripted_metric": {
'init_script': 'params._agg.best = ' + nully + ';',
'map_script': 'params._agg.best = ' + expand_template(MAX_OF_TUPLE, {"expr1": "params._agg.best", "expr2": selfy, "dir": dir, "op": op}) + ";",
'combine_script': 'return params._agg.best',
'reduce_script': 'return params._aggs.stream().max(' + expand_template(COMPARE_TUPLE, {"dir": dir, "op": op}) + ').get()',
}}
if schema.query_path[0] == ".":
es_query.aggs[canonical_name] = script
s.pull = jx_expression_to_function(literal_field(canonical_name) + ".value")
else:
es_query.aggs[canonical_name] = {
"nested": {"path": schema.query_path[0]},
"aggs": {"_nested": script}
}
s.pull = jx_expression_to_function(literal_field(canonical_name) + "._nested.value")
else:
Log.error("{{agg}} is not a supported aggregate over a tuple", agg=s.aggregate)
Log.error("{{agg}} is not a supported aggregate over a tuple", agg=s.aggregate)
elif s.aggregate == "count":
es_query.aggs[literal_field(canonical_name)].value_count.script = s.value.partial_eval().to_painless(schema).script(schema)
s.pull = jx_expression_to_function(literal_field(canonical_name) + ".value")
@ -280,7 +351,7 @@ def es_aggsop(es, frum, query):
es_query.aggs[median_name].percentiles.percents += [50]
s.pull = get_pull_stats(stats_name, median_name)
elif s.aggregate=="union":
elif s.aggregate == "union":
# USE TERMS AGGREGATE TO SIMULATE union
stats_name = literal_field(canonical_name)
es_query.aggs[stats_name].terms.script_field = s.value.to_painless(schema).script(schema)
@ -382,6 +453,7 @@ def drill(agg):
deeper = agg.get("_filter") or agg.get("_nested")
return agg
def aggs_iterator(aggs, decoders, coord=True):
"""
DIG INTO ES'S RECURSIVE aggs DATA-STRUCTURE:
@ -436,13 +508,14 @@ def aggs_iterator(aggs, decoders, coord=True):
if coord:
for a, parts in _aggs_iterator(unwrap(aggs), depth - 1):
coord = tuple(d.get_index(parts) for d in decoders)
if any(c is None for c in coord):
continue
yield parts, coord, a
else:
for a, parts in _aggs_iterator(unwrap(aggs), depth - 1):
yield parts, None, a
def count_dim(aggs, decoders):
if any(isinstance(d, (DefaultDecoder, DimFieldListDecoder, ObjectDecoder)) for d in decoders):
# ENUMERATE THE DOMAINS, IF UNKNOWN AT QUERY TIME

50
vendor/jx_elasticsearch/es52/decoders.py поставляемый
Просмотреть файл

@ -19,13 +19,13 @@ from jx_base.dimensions import Dimension
from jx_base.domains import SimpleSetDomain, DefaultDomain, PARTITION
from jx_base.expressions import TupleOp, value2json
from jx_base.query import MAX_LIMIT, DEFAULT_LIMIT
from jx_elasticsearch.es52.expressions import Variable, NotOp, InOp, Literal, OrOp, AndOp, InequalityOp, LeavesOp
from jx_elasticsearch.es52.expressions import Variable, NotOp, InOp, Literal, OrOp, AndOp, InequalityOp, LeavesOp, LIST_TO_PIPE
from jx_python import jx
from mo_dots import set_default, coalesce, literal_field, Data, relative_field, unwraplist
from mo_dots import wrap
from mo_json.typed_encoder import untype_path
from mo_logs import Log
from mo_logs.strings import quote
from mo_logs.strings import quote, expand_template
from mo_math import MAX, MIN
from mo_math import Math
@ -38,7 +38,7 @@ class AggsDecoder(object):
# if query.groupby:
# return object.__new__(DefaultDecoder, e)
if isinstance(e.value, (text_type, binary_type)):
if isinstance(e.value, text_type):
Log.error("Expecting Variable or Expression, not plain string")
if isinstance(e.value, LeavesOp):
@ -143,18 +143,17 @@ class SetDecoder(AggsDecoder):
def __init__(self, edge, query, limit):
AggsDecoder.__init__(self, edge, query, limit)
domain = self.domain = edge.domain
self.sorted = None
# WE ASSUME IF THE VARIABLES MATCH, THEN THE SORT TERM AND EDGE TERM MATCH, AND WE SORT BY TERM
# self.sorted = {1: "asc", -1: "desc", None: None}[getattr(edge.domain, 'sort', None)]
edge_var = edge.value.vars()
edge_var = set(v.var for v in edge.value.vars())
if query.sort:
for s in query.sort:
if not edge_var - s.value.vars():
if not edge_var - set(v.var for v in s.value.vars()):
self.sorted = {1: "asc", -1: "desc"}[s.sort]
parts = jx.sort(domain.partitions, {"value": domain.key, "sort": s.sort})
edge.domain = self.domain = SimpleSetDomain(key=domain.key, label=domain.label, partitions=parts)
else:
self.sorted = None
def append_query(self, es_query, start):
self.start = start
@ -465,17 +464,19 @@ class MultivalueDecoder(SetDecoder):
self.start = start
es_field = self.query.frum.schema.leaves(self.var)[0].es_column
for i, v in enumerate(self.values):
es_query = wrap({"aggs": {
"_match": set_default({"terms": {
"script": 'doc['+quote(es_field)+'].values.contains(' + value2json(v) + ') ? 1 : 0'
}}, es_query)
}})
es_query = wrap({"aggs": {
"_match": set_default({"terms": {
"script": expand_template(LIST_TO_PIPE, {"expr": 'doc[' + quote(es_field) + '].values'})
}}, es_query)
}})
return es_query
def get_value_from_row(self, row):
return unwraplist([v for v, p in zip(self.values, row[self.start:self.start + self.num_columns:]) if p["key"] == '1'])
values = row[self.start]['key'].replace("||", "\b").split("|")
if len(values) == 2:
return None
return unwraplist([v.replace("\b", "|") for v in values[1:-1]])
def get_index(self, row):
find = self.get_value_from_row(row)
@ -487,7 +488,7 @@ class MultivalueDecoder(SetDecoder):
@property
def num_columns(self):
return len(self.values)
return 1
class ObjectDecoder(AggsDecoder):
@ -695,6 +696,7 @@ class DefaultDecoder(SetDecoder):
class DimFieldListDecoder(SetDecoder):
def __init__(self, edge, query, limit):
AggsDecoder.__init__(self, edge, query, limit)
edge.allowNulls = False
self.fields = edge.domain.dimension.fields
self.domain = self.edge.domain
self.domain.limit = Math.min(coalesce(self.domain.limit, query.limit, 10), MAX_LIMIT)
@ -712,11 +714,10 @@ class DimFieldListDecoder(SetDecoder):
"size": self.domain.limit
}}, es_query)}
}}})
if self.edge.allowNulls:
nest.aggs._missing = set_default(
{"filter": NotOp("not", exists).to_esfilter(self.schema)},
es_query
)
nest.aggs._missing = set_default(
{"filter": NotOp("not", exists).to_esfilter(self.schema)},
es_query
)
es_query = nest
if self.domain.where:
@ -743,9 +744,12 @@ class DimFieldListDecoder(SetDecoder):
)
def get_index(self, row):
find = tuple(p.get("key") for p in row[self.start:self.start + self.num_columns:])
return self.domain.getIndexByKey(find)
part = row[self.start:self.start + len(self.fields):]
if part[0]['doc_count']==0:
return None
find = tuple(p.get("key") for p in part)
output = self.domain.getIndexByKey(find)
return output
@property
def num_columns(self):
return len(self.fields)

2
vendor/jx_elasticsearch/es52/deep.py поставляемый
Просмотреть файл

@ -14,7 +14,7 @@ from __future__ import unicode_literals
from jx_base import STRUCT, NESTED
from jx_base.expressions import NULL
from jx_base.query import DEFAULT_LIMIT
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es52.expressions import split_expression_by_depth, AndOp, Variable, LeavesOp
from jx_elasticsearch.es52.setop import format_dispatch, get_pull_function, get_pull
from jx_elasticsearch.es52.util import jx_sort_to_es_sort, es_query_template

137
vendor/jx_elasticsearch/es52/expressions.py поставляемый
Просмотреть файл

@ -20,20 +20,39 @@ from jx_base.expressions import Variable, TupleOp, LeavesOp, BinaryOp, OrOp, Scr
WhenOp, InequalityOp, extend, Literal, NullOp, TrueOp, FalseOp, DivOp, FloorOp, \
EqOp, NeOp, NotOp, LengthOp, NumberOp, StringOp, CountOp, MultiOp, RegExpOp, CoalesceOp, MissingOp, ExistsOp, \
PrefixOp, NotLeftOp, InOp, CaseOp, AndOp, \
ConcatOp, IsNumberOp, Expression, BasicIndexOfOp, MaxOp, MinOp, BasicEqOp, BooleanOp, IntegerOp, BasicSubstringOp, ZERO, NULL, FirstOp, FALSE, TRUE
ConcatOp, IsNumberOp, Expression, BasicIndexOfOp, MaxOp, MinOp, BasicEqOp, BooleanOp, IntegerOp, BasicSubstringOp, ZERO, NULL, FirstOp, FALSE, TRUE, SuffixOp, simplified
from mo_dots import coalesce, wrap, Null, unwraplist, set_default, literal_field
from mo_logs import Log, suppress_exception
from mo_logs.strings import expand_template, quote
from mo_math import MAX, OR
from pyLibrary.convert import string2regexp
TO_STRING = """Optional.of({{expr}}).map(
value -> {
String output = String.valueOf(value);
if (output.endsWith(".0")) output = output.substring(0, output.length() - 2);
return output;
}
).orElse(null)"""
NUMBER_TO_STRING = """
Optional.of({{expr}}).map(
value -> {
String output = String.valueOf(value);
if (output.endsWith(".0")) output = output.substring(0, output.length() - 2);
return output;
}
).orElse(null)
"""
LIST_TO_PIPE = """
StringBuffer output=new StringBuffer();
for(String s : {{expr}}){
output.append("|");
String sep2="";
StringTokenizer parts = new StringTokenizer(s, "|");
while (parts.hasMoreTokens()){
output.append(sep2);
output.append(parts.nextToken());
sep2="||";
}//for
}//for
output.append("|");
return output.toString()
"""
class Painless(Expression):
@ -132,7 +151,18 @@ def to_painless(self, schema):
@extend(CaseOp)
def to_esfilter(self, schema):
return ScriptOp("script", self.to_painless(schema).script(schema)).to_esfilter(schema)
if self.type == BOOLEAN:
return OrOp(
"or",
[
AndOp("and", [w.when, w.then])
for w in self.whens[:-1]
] +
self.whens[-1:]
).partial_eval().to_esfilter(schema)
else:
Log.error("do not know how to handle")
return ScriptOp("script", self.to_ruby(schema).script(schema)).to_esfilter(schema)
@extend(ConcatOp)
@ -315,6 +345,19 @@ def to_esfilter(self, schema):
Log.error("not supported")
@extend(TupleOp)
def to_painless(self, schema):
terms = [FirstOp("first", t).partial_eval().to_painless(schema) for t in self.terms]
expr = 'new Object[]{'+','.join(t.expr for t in terms)+'}'
return Painless(
type=OBJECT,
expr=expr,
miss=FALSE,
many=FALSE,
frum=self
)
@extend(LeavesOp)
def to_painless(self, schema):
Log.error("not supported")
@ -404,6 +447,15 @@ def to_esfilter(self, schema):
Log.error("Logic error")
@simplified
@extend(EqOp)
def partial_eval(self):
lhs = self.lhs.partial_eval()
rhs = self.rhs.partial_eval()
return EqOp("eq", [lhs, rhs])
@extend(EqOp)
def to_painless(self, schema):
return CaseOp("case", [
@ -659,6 +711,16 @@ def to_painless(self, schema):
@extend(FirstOp)
def to_painless(self, schema):
if isinstance(self.term, Variable):
columns = schema.values(self.term.var)
if len(columns) == 1:
return Painless(
miss=MissingOp("missing", self.term),
type=self.term.type,
expr="doc[" + quote(columns[0].es_column) + "].value",
frum=self
)
term = self.term.to_painless(schema)
if isinstance(term.frum, CoalesceOp):
@ -840,13 +902,22 @@ def to_painless(self, schema):
)
_painless_operators = {
"add": (" + ", "0"), # (operator, zero-array default value) PAIR
"sum": (" + ", "0"),
"mul": (" * ", "1"),
"mult": (" * ", "1"),
"multiply": (" * ", "1")
}
@extend(MultiOp)
def to_painless(self, schema):
op, unit = MultiOp.operators[self.op]
op, unit = _painless_operators[self.op]
if self.nulls:
calc = op.join(
"((" + t.missing().to_painless(schema).expr + ") ? " + unit + " : (" + NumberOp("number", t).partial_eval().to_painless(schema).expr + "))" for
t in self.terms
"((" + t.missing().to_painless(schema).expr + ") ? " + unit + " : (" + NumberOp("number", t).partial_eval().to_painless(schema).expr + "))"
for t in self.terms
)
return WhenOp(
"when",
@ -905,7 +976,7 @@ def to_painless(self, schema):
return Painless(
miss=self.term.missing().partial_eval(),
type=STRING,
expr=expand_template(TO_STRING, {"expr":value.expr}),
expr=expand_template(NUMBER_TO_STRING, {"expr":value.expr}),
frum=self
)
elif value.type == STRING:
@ -914,7 +985,7 @@ def to_painless(self, schema):
return Painless(
miss=self.term.missing().partial_eval(),
type=STRING,
expr=expand_template(TO_STRING, {"expr":value.expr}),
expr=expand_template(NUMBER_TO_STRING, {"expr":value.expr}),
frum=self
)
@ -942,14 +1013,32 @@ def to_painless(self, schema):
@extend(PrefixOp)
def to_esfilter(self, schema):
if not self.field:
if not self.expr:
return {"match_all": {}}
elif isinstance(self.field, Variable) and isinstance(self.prefix, Literal):
var = schema.leaves(self.field.var)[0].es_column
elif isinstance(self.expr, Variable) and isinstance(self.prefix, Literal):
var = schema.leaves(self.expr.var)[0].es_column
return {"prefix": {var: self.prefix.value}}
else:
return ScriptOp("script", self.to_painless(schema).script(schema)).to_esfilter(schema)
@extend(SuffixOp)
def to_painless(self, schema):
if not self.suffix:
return "true"
else:
return "(" + self.expr.to_painless(schema) + ").endsWith(" + self.suffix.to_painless(schema) + ")"
@extend(SuffixOp)
def to_esfilter(self, schema):
if not self.suffix:
return {"match_all": {}}
elif isinstance(self.expr, Variable) and isinstance(self.suffix, Literal):
var = schema.leaves(self.expr.var)[0].es_column
return {"regexp": {var: ".*"+string2regexp(self.suffix.value)}}
else:
return ScriptOp("script", self.to_painless(schema).script(schema)).to_esfilter(schema)
@extend(InOp)
def to_painless(self, schema):
@ -1001,7 +1090,7 @@ def to_painless(self, schema):
acc.append(Painless(
miss=frum.missing(),
type=c.type,
expr="doc[" + q + "].values",
expr="doc[" + q + "].values" if c.type!=BOOLEAN else "doc[" + q + "].value",
frum=frum,
many=True
))
@ -1282,13 +1371,13 @@ def split_expression_by_depth(where, schema, output=None, var_to_depth=None):
if not vars_:
return Null
# MAP VARIABLE NAMES TO HOW DEEP THEY ARE
var_to_depth = {v: len(c.nested_path) - 1 for v in vars_ for c in schema[v]}
var_to_depth = {v: max(len(c.nested_path) - 1, 0) for v in vars_ for c in schema[v]}
all_depths = set(var_to_depth.values())
if -1 in all_depths:
Log.error(
"Can not find column with name {{column|quote}}",
column=unwraplist([k for k, v in var_to_depth.items() if v == -1])
)
# if -1 in all_depths:
# Log.error(
# "Can not find column with name {{column|quote}}",
# column=unwraplist([k for k, v in var_to_depth.items() if v == -1])
# )
if len(all_depths) == 0:
all_depths = {0}
output = wrap([[] for _ in range(MAX(all_depths) + 1)])

6
vendor/jx_elasticsearch/es52/format.py поставляемый
Просмотреть файл

@ -284,12 +284,6 @@ def format_list_from_aggop(decoders, aggs, start, query, select):
})
def format_line(decoders, aggs, start, query, select):
list = format_list(decoders, aggs, start, query, select)

4
vendor/jx_elasticsearch/es52/setop.py поставляемый
Просмотреть файл

@ -17,7 +17,7 @@ from jx_base import NESTED
from jx_base.domains import ALGEBRAIC
from jx_base.expressions import IDENTITY
from jx_base.query import DEFAULT_LIMIT
from jx_elasticsearch.es09.util import post as es_post
from jx_elasticsearch import post as es_post
from jx_elasticsearch.es52.expressions import Variable, LeavesOp
from jx_elasticsearch.es52.util import jx_sort_to_es_sort, es_query_template
from jx_python.containers.cube import Cube
@ -75,7 +75,7 @@ def es_setop(es, query):
# IF THERE IS A *, THEN INSERT THE EXTRA COLUMNS
if isinstance(select.value, LeavesOp) and isinstance(select.value.term, Variable):
term = select.value.term
leaves = schema.leaves(term.var)
leaves = schema.values(term.var)
for c in leaves:
full_name = concat_field(select.name, relative_field(untype_path(c.names["."]), term.var))
if c.type == NESTED:

136
vendor/jx_elasticsearch/meta.py поставляемый
Просмотреть файл

@ -15,27 +15,19 @@ import itertools
from copy import copy
from itertools import product
from jx_base import STRUCT
from jx_base import STRUCT, Table
from jx_base.query import QueryOp
from jx_base.schema import Schema
from jx_python import jx
from jx_python import meta as jx_base_meta
from jx_python import jx, meta as jx_base_meta
from jx_python.containers.list_usingPythonList import ListContainer
from jx_python.meta import ColumnList, metadata_columns, metadata_tables, Column, Table
from mo_dots import Data, relative_field, concat_field, SELF_PATH
from mo_dots import coalesce, set_default, Null, split_field, join_field
from mo_dots import wrap
from jx_python.meta import ColumnList, Column
from mo_dots import Data, relative_field, concat_field, SELF_PATH, ROOT_PATH, coalesce, set_default, Null, split_field, join_field, wrap
from mo_json.typed_encoder import EXISTS_TYPE
from mo_kwargs import override
from mo_logs import Log
from mo_logs.strings import quote
from mo_threads import Queue
from mo_threads import THREAD_STOP
from mo_threads import Thread
from mo_threads import Till
from mo_times.dates import Date
from mo_times.durations import HOUR, MINUTE
from mo_times.timer import Timer
from mo_threads import Queue, THREAD_STOP, Thread, Till
from mo_times import HOUR, MINUTE, Timer, Date
from pyLibrary.env import elasticsearch
from pyLibrary.env.elasticsearch import es_type_to_json_type
@ -60,7 +52,7 @@ class FromESMetadata(Schema):
return jx_base_meta.singlton
@override
def __init__(self, host, index, alias=None, name=None, port=9200, kwargs=None):
def __init__(self, host, index, sql_file='metadata.sqlite', alias=None, name=None, port=9200, kwargs=None):
if hasattr(self, "settings"):
return
@ -75,7 +67,7 @@ class FromESMetadata(Schema):
self.abs_columns = set()
self.last_es_metadata = Date.now()-OLD_METADATA
self.meta=Data()
self.meta = Data()
table_columns = metadata_tables()
column_columns = metadata_columns()
self.meta.tables = ListContainer("meta.tables", [], wrap({c.names["."]: c for c in table_columns}))
@ -210,7 +202,7 @@ class FromESMetadata(Schema):
table = Table(
name=es_index_name,
url=None,
query_path=None,
query_path=['.'],
timestamp=Date.now()
)
with self.meta.tables.locker:
@ -292,6 +284,13 @@ class FromESMetadata(Schema):
count = result.hits.total
cardinality = 1001
multi = 1001
elif column.es_column == "_id":
result = self.default_es.post("/" + es_index + "/_search", data={
"query": {"match_all": {}},
"size": 0
})
count = cardinality = result.hits.total
multi = 1
else:
result = self.default_es.post("/" + es_index + "/_search", data={
"aggs": {
@ -300,10 +299,10 @@ class FromESMetadata(Schema):
},
"size": 0
})
r = result.aggregations.count
agg_results = result.aggregations
count = result.hits.total
cardinality = coalesce(r.value, r._nested.value, r.doc_count)
multi = coalesce(r.multi.value, 1)
cardinality = coalesce(agg_results.count.value, agg_results.count._nested.value, agg_results.count.doc_count)
multi = int(coalesce(agg_results.multi.value, 1))
if cardinality == None:
Log.error("logic error")
@ -513,3 +512,100 @@ def _counting_query(c):
"field": c.es_column
}}
def metadata_columns():
return wrap(
[
Column(
names={".":c},
es_index="meta.columns",
es_column=c,
type="string",
nested_path=ROOT_PATH
)
for c in [
"type",
"nested_path",
"es_column",
"es_index"
]
] + [
Column(
es_index="meta.columns",
names={".":c},
es_column=c,
type="object",
nested_path=ROOT_PATH
)
for c in [
"names",
"domain",
"partitions"
]
] + [
Column(
names={".": c},
es_index="meta.columns",
es_column=c,
type="long",
nested_path=ROOT_PATH
)
for c in [
"count",
"cardinality"
]
] + [
Column(
names={".": "last_updated"},
es_index="meta.columns",
es_column="last_updated",
type="time",
nested_path=ROOT_PATH
)
]
)
def metadata_tables():
return wrap(
[
Column(
names={".": c},
es_index="meta.tables",
es_column=c,
type="string",
nested_path=ROOT_PATH
)
for c in [
"name",
"url",
"query_path"
]
]+[
Column(
names={".": "timestamp"},
es_index="meta.tables",
es_column="timestamp",
type="integer",
nested_path=ROOT_PATH
)
]
)
def init_database(sql):
sql.execute("""
CREATE TABLE tables AS (
table_name VARCHAR(200),
alias CHAR
)
""")

2
vendor/jx_python/__init__.py поставляемый
Просмотреть файл

@ -63,7 +63,7 @@ def wrap_from(frum, schema=None):
if isinstance(frum, text_type):
if not container.config.default.settings:
Log.error("expecting jx_base.query.config.default.settings to contain default elasticsearch connection info")
Log.error("expecting jx_base.container.config.default.settings to contain default elasticsearch connection info")
type_ = None
index = frum

28
vendor/jx_python/containers/list_usingPythonList.py поставляемый
Просмотреть файл

@ -33,7 +33,7 @@ _get = object.__getattribute__
class ListContainer(Container):
def __init__(self, name, data, schema=None):
#TODO: STORE THIS LIKE A CUBE FOR FASTER ACCESS AND TRANSFORMATION
# TODO: STORE THIS LIKE A CUBE FOR FASTER ACCESS AND TRANSFORMATION
data = list(unwrap(data))
Container.__init__(self, data, schema)
if schema == None:
@ -86,8 +86,30 @@ class ListContainer(Container):
output.window(param)
if q.format:
if q.format=="list":
return Data(data=output.data)
if q.format == "list":
return Data(data=output.data, meta={"format": "list"})
elif q.format == "table":
head = [c.names['.'] for c in output.schema.columns]
data = [
[r[h] for h in head]
for r in output.data
]
return Data(header=head, data=data, meta={"format": "table"})
elif q.format == "cube":
head = [c.names['.'] for c in output.schema.columns]
rows = [
[r[h] for h in head]
for r in output.data
]
data = {h: c for h, c in zip(head, zip(*rows))}
return Data(
data=data,
meta={"format": "cube"},
edges=[{
"name": "rownum",
"domain": {"type": "rownum", "min": 0, "max": len(rows), "interval": 1}
}]
)
else:
Log.error("unknown format {{format}}", format=q.format)
else:

35
vendor/jx_python/expressions.py поставляемый
Просмотреть файл

@ -25,7 +25,7 @@ from jx_base.expressions import Variable, DateOp, TupleOp, LeavesOp, BinaryOp, O
InequalityOp, extend, RowsOp, OffsetOp, GetOp, Literal, NullOp, TrueOp, FalseOp, DivOp, FloorOp, \
EqOp, NeOp, NotOp, LengthOp, NumberOp, StringOp, CountOp, MultiOp, RegExpOp, CoalesceOp, MissingOp, ExistsOp, \
PrefixOp, NotLeftOp, RightOp, NotRightOp, FindOp, BetweenOp, RangeOp, CaseOp, AndOp, \
ConcatOp, InOp, jx_expression, Expression, WhenOp, MaxOp, SplitOp, NULL, SelectOp, SuffixOp
ConcatOp, InOp, jx_expression, Expression, WhenOp, MaxOp, SplitOp, NULL, SelectOp, SuffixOp, LastOp
from jx_python.expression_compiler import compile_expression
from mo_times.dates import Date
@ -100,11 +100,17 @@ def to_python(self, not_null=False, boolean=False, many=False):
return "listwrap("+obj+")[" + code + "]"
@extend(LastOp)
def to_python(self, not_null=False, boolean=False, many=False):
term = self.term.to_python()
return "listwrap(" + term + ").last()"
@extend(SelectOp)
def to_python(self, not_null=False, boolean=False, many=False):
return (
"wrap_leaves({" +
",\n".join(
','.join(
quote(t['name']) + ":" + t['value'].to_python() for t in self.terms
) +
"})"
@ -148,7 +154,7 @@ def to_python(self, not_null=False, boolean=False, many=False):
elif len(self.terms) == 1:
return "(" + self.terms[0].to_python() + ",)"
else:
return "(" + (",".join(t.to_python() for t in self.terms)) + ")"
return "(" + (','.join(t.to_python() for t in self.terms)) + ")"
@extend(LeavesOp)
@ -239,7 +245,18 @@ def to_python(self, not_null=False, boolean=False, many=False):
@extend(MaxOp)
def to_python(self, not_null=False, boolean=False, many=False):
return "max(["+(",".join(t.to_python() for t in self.terms))+"])"
return "max(["+(','.join(t.to_python() for t in self.terms))+"])"
_python_operators = {
"add": (" + ", "0"), # (operator, zero-array default value) PAIR
"sum": (" + ", "0"),
"mul": (" * ", "1"),
"mult": (" * ", "1"),
"multiply": (" * ", "1")
}
@extend(MultiOp)
@ -247,9 +264,9 @@ def to_python(self, not_null=False, boolean=False, many=False):
if len(self.terms) == 0:
return self.default.to_python()
elif self.default is NULL:
return MultiOp.operators[self.op][0].join("(" + t.to_python() + ")" for t in self.terms)
return _python_operators[self.op][0].join("(" + t.to_python() + ")" for t in self.terms)
else:
return "coalesce(" + MultiOp.operators[self.op][0].join("(" + t.to_python() + ")" for t in self.terms) + ", " + self.default.to_python() + ")"
return "coalesce(" + _python_operators[self.op][0].join("(" + t.to_python() + ")" for t in self.terms) + ", " + self.default.to_python() + ")"
@extend(RegExpOp)
def to_python(self, not_null=False, boolean=False, many=False):
@ -258,7 +275,7 @@ def to_python(self, not_null=False, boolean=False, many=False):
@extend(CoalesceOp)
def to_python(self, not_null=False, boolean=False, many=False):
return "coalesce(" + (",".join(t.to_python() for t in self.terms)) + ")"
return "coalesce(" + (', '.join(t.to_python() for t in self.terms)) + ")"
@extend(MissingOp)
@ -273,12 +290,12 @@ def to_python(self, not_null=False, boolean=False, many=False):
@extend(PrefixOp)
def to_python(self, not_null=False, boolean=False, many=False):
return "(" + self.field.to_python() + ").startswith(" + self.prefix.to_python() + ")"
return "(" + self.expr.to_python() + ").startswith(" + self.prefix.to_python() + ")"
@extend(SuffixOp)
def to_python(self, not_null=False, boolean=False, many=False):
return "(" + self.field.to_python() + ").endswith(" + self.suffix.to_python() + ")"
return "(" + self.expr.to_python() + ").endswith(" + self.suffix.to_python() + ")"
@extend(ConcatOp)

57
vendor/jx_python/jx.py поставляемый
Просмотреть файл

@ -23,7 +23,7 @@ from jx_python import expressions as _expressions
from jx_python import flat_list, group_by
from mo_dots import listwrap, wrap, unwrap, FlatList, NullType
from mo_dots import set_default, Null, Data, split_field, coalesce, join_field
from mo_future import text_type, boolean_type, none_type, long, generator_types, sort_using_cmp
from mo_future import text_type, boolean_type, none_type, long, generator_types, sort_using_cmp, PY2
from mo_logs import Log
from mo_math import Math
from mo_math import UNION, MIN
@ -66,10 +66,10 @@ def run(query, frum=Null):
BUT IT IS ALSO PROCESSING A list CONTAINER; SEPARATE TO A ListContainer
"""
if frum == None:
query_op = QueryOp.wrap(query)
frum = query_op.frum
frum = wrap(query)['from']
query_op = QueryOp.wrap(query, table=frum, schema=frum.schema)
else:
query_op = QueryOp.wrap(query, frum.schema)
query_op = QueryOp.wrap(query, frum, frum.schema)
if frum == None:
from jx_python.containers.list_usingPythonList import DUAL
@ -89,12 +89,6 @@ def run(query, frum=Null):
if is_aggs(query_op):
frum = list_aggs(frum, query_op)
else: # SETOP
# try:
# if query.filter != None or query.esfilter != None:
# Log.error("use 'where' clause")
# except AttributeError:
# pass
if query_op.where is not TRUE:
frum = filter(frum, query_op.where)
@ -565,60 +559,66 @@ def count(values):
return sum((1 if v!=None else 0) for v in values)
def value_compare(l, r, ordering=1):
def value_compare(left, right, ordering=1):
"""
SORT VALUES, NULL IS THE LEAST VALUE
:param l: LHS
:param r: RHS
:param left: LHS
:param right: RHS
:param ordering: (-1, 0, 0) TO AFFECT SORT ORDER
:return: The return value is negative if x < y, zero if x == y and strictly positive if x > y.
"""
try:
if isinstance(l, list) or isinstance(r, list):
for a, b in zip(listwrap(l), listwrap(r)):
if isinstance(left, list) or isinstance(right, list):
if left == None:
return ordering
elif right == None:
return - ordering
left = listwrap(left)
right = listwrap(right)
for a, b in zip(left, right):
c = value_compare(a, b) * ordering
if c != 0:
return c
if len(l) < len(r):
if len(left) < len(right):
return - ordering
elif len(l) > len(r):
elif len(left) > len(right):
return ordering
else:
return 0
ltype = type(l)
rtype = type(r)
ltype = type(left)
rtype = type(right)
type_diff = TYPE_ORDER.get(ltype, 10) - TYPE_ORDER.get(rtype, 10)
if type_diff != 0:
return ordering if type_diff > 0 else -ordering
if ltype is builtin_tuple:
for a, b in zip(l, r):
for a, b in zip(left, right):
c = value_compare(a, b)
if c != 0:
return c * ordering
return 0
elif ltype in (dict, Data):
for k in sorted(set(l.keys()) | set(r.keys())):
c = value_compare(l.get(k), r.get(k)) * ordering
for k in sorted(set(left.keys()) | set(right.keys())):
c = value_compare(left.get(k), right.get(k)) * ordering
if c != 0:
return c
return 0
elif l > r:
elif left > right:
return ordering
elif l < r:
elif left < right:
return -ordering
else:
return 0
except Exception as e:
Log.error("Can not compare values {{left}} to {{right}}", left=l, right=r, cause=e)
Log.error("Can not compare values {{left}} to {{right}}", left=left, right=right, cause=e)
TYPE_ORDER = {
boolean_type: 0,
int: 1,
long: 1,
float: 1,
Date: 1,
text_type: 2,
@ -631,9 +631,8 @@ TYPE_ORDER = {
NullOp: 9
}
if PY2:
TYPE_ORDER[long] = 1
def pairwise(values):

133
vendor/jx_python/meta.py поставляемый
Просмотреть файл

@ -11,144 +11,27 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from copy import copy
from datetime import date
from datetime import datetime
from mo_future import text_type, long
from jx_base import STRUCT
from jx_base import STRUCT, Column
from jx_base.container import Container
from jx_base.schema import Schema
from jx_python import jx
from mo_collections import UniqueIndex
from mo_dots import Data, concat_field, get_attr, listwrap, unwraplist, NullType, FlatList
from mo_dots import split_field, join_field, ROOT_PATH
from mo_dots import wrap
from mo_future import none_type
from mo_future import text_type, long, PY2
from mo_json.typed_encoder import untype_path, unnest_path
from mo_logs import Log
from mo_threads import Lock
from mo_future import none_type
from jx_base.container import Container
from jx_base.schema import Schema
from mo_times.dates import Date
from pyLibrary.meta import DataClass
singlton = None
def metadata_columns():
return wrap(
[
Column(
names={".":c},
es_index="meta.columns",
es_column=c,
type="string",
nested_path=ROOT_PATH
)
for c in [
"type",
"nested_path",
"es_column",
"es_index"
]
] + [
Column(
es_index="meta.columns",
names={".":c},
es_column=c,
type="object",
nested_path=ROOT_PATH
)
for c in [
"names",
"domain",
"partitions"
]
] + [
Column(
names={".": c},
es_index="meta.columns",
es_column=c,
type="long",
nested_path=ROOT_PATH
)
for c in [
"count",
"cardinality"
]
] + [
Column(
names={".": "last_updated"},
es_index="meta.columns",
es_column="last_updated",
type="time",
nested_path=ROOT_PATH
)
]
)
def metadata_tables():
return wrap(
[
Column(
names={".": c},
es_index="meta.tables",
es_column=c,
type="string",
nested_path=ROOT_PATH
)
for c in [
"name",
"url",
"query_path"
]
]+[
Column(
names={".": "timestamp"},
es_index="meta.tables",
es_column="timestamp",
type="integer",
nested_path=ROOT_PATH
)
]
)
class Table(DataClass("Table", [
"name",
"url",
"query_path",
"timestamp"
])):
@property
def columns(self):
return singlton.get_columns(table_name=self.name)
Column = DataClass(
"Column",
[
# "table",
"names", # MAP FROM TABLE NAME TO COLUMN NAME (ONE COLUMN CAN HAVE MULTIPLE NAMES)
"es_column",
"es_index",
# "es_type",
"type",
{"name": "useSource", "default": False},
{"name": "nested_path", "nulls": True}, # AN ARRAY OF PATHS (FROM DEEPEST TO SHALLOWEST) INDICATING THE JSON SUB-ARRAYS
{"name": "count", "nulls": True},
{"name": "cardinality", "nulls": True},
{"name": "multi", "nulls": True},
{"name": "partitions", "nulls": True},
{"name": "last_updated", "nulls": True}
]
)
class ColumnList(Container):
"""
OPTIMIZED FOR THE PARTICULAR ACCESS PATTERNS USED
@ -262,7 +145,7 @@ def get_schema_from_list(table_name, frum):
"""
columns = UniqueIndex(keys=("names.\\.",))
_get_schema_from_list(frum, ".", prefix_path=[], nested_path=ROOT_PATH, columns=columns)
return Schema(table_name=table_name, columns=columns)
return Schema(table_name=table_name, columns=list(columns))
def _get_schema_from_list(frum, table_name, prefix_path, nested_path, columns):
@ -332,7 +215,6 @@ _type_to_name = {
str: "string",
text_type: "string",
int: "integer",
long: "integer",
float: "double",
Data: "object",
dict: "object",
@ -344,6 +226,9 @@ _type_to_name = {
date: "double"
}
if PY2:
_type_to_name[long] = "integer"
_merge_type = {
"undefined": {
"undefined": "undefined",

4
vendor/mo_dots/__init__.py поставляемый
Просмотреть файл

@ -326,8 +326,8 @@ def _get_attr(obj, path):
# TRY FILESYSTEM
File = get_module("mo_files").File
possible_error = None
python_file = File.new_instance(File(obj.__file__).parent, attr_name).set_extension("py")
python_module = File.new_instance(File(obj.__file__).parent, attr_name, "__init__.py")
python_file = (File(obj.__file__).parent / attr_name).set_extension("py")
python_module = (File(obj.__file__).parent / attr_name / "__init__.py")
if python_file.exists or python_module.exists:
try:
# THIS CASE IS WHEN THE __init__.py DOES NOT IMPORT THE SUBDIR FILE

2
vendor/mo_dots/datas.py поставляемый
Просмотреть файл

@ -218,7 +218,7 @@ class Data(MutableMapping):
def values(self):
d = _get(self, "_dict")
return listwrap(d.values())
return listwrap(list(d.values()))
def clear(self):
get_logger().error("clear() not supported")

61
vendor/mo_files/__init__.py поставляемый
Просмотреть файл

@ -19,6 +19,7 @@ import os
from mo_future import text_type, binary_type
from mo_dots import get_module, coalesce
from mo_logs import Log, Except
from mo_threads import Thread, Till
mime = MimeTypes()
@ -136,7 +137,11 @@ class File(object):
@property
def mime_type(self):
if not self._mime_type:
if self.abspath.endswith(".json"):
if self.abspath.endswith(".js"):
self._mime_type = "application/javascript"
elif self.abspath.endswith(".css"):
self._mime_type = "text/css"
elif self.abspath.endswith(".json"):
self._mime_type = "application/json"
else:
self._mime_type, _ = mime.guess_type(self.abspath)
@ -402,7 +407,7 @@ class TempDirectory(File):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.delete()
Thread.run("delete "+self.name, delete_daemon, file=self)
class TempFile(File):
@ -418,13 +423,13 @@ class TempFile(File):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.delete()
Thread.run("delete "+self.name, delete_daemon, file=self)
def _copy(from_, to_):
if from_.is_directory():
for c in os.listdir(from_.abspath):
_copy(File.new_instance(from_, c), File.new_instance(to_, c))
_copy(from_ / c, to_ / c)
else:
File.new_instance(to_).write_bytes(File.new_instance(from_).read_bytes())
@ -443,19 +448,29 @@ def datetime2string(value, format="%Y-%m-%d %H:%M:%S"):
Log.error(u"Can not format {{value}} with {{format}}", value=value, format=format, cause=e)
def join_path(*path):
def scrub(i, p):
if isinstance(p, File):
p = p.abspath
if p == "/":
return "."
p = p.replace(os.sep, "/")
if p[-1] == b'/':
if p in ('', '/'):
return "."
if p[-1] == '/':
p = p[:-1]
if i > 0 and p[0] == b'/':
if i > 0 and p[0] == '/':
p = p[1:]
return p
path = [p._filename if isinstance(p, File) else p for p in path]
abs_prefix = ''
if path and path[0]:
if path[0][0] == '/':
abs_prefix = '/'
path[0] = path[0][1:]
elif os.sep == '\\' and path[0][1:].startswith(':/'):
# If windows, then look for the "c:/" prefix
abs_prefix = path[0][0:3]
path[0] = path[0][3:]
scrubbed = []
for i, p in enumerate(path):
scrubbed.extend(scrub(i, p).split("/"))
@ -465,14 +480,34 @@ def join_path(*path):
pass
elif s == "..":
if simpler:
simpler.pop()
if simpler[-1] == '..':
simpler.append(s)
else:
simpler.pop()
elif abs_prefix:
raise Exception("can not get parent of root")
else:
simpler.append(s)
else:
simpler.append(s)
if not simpler:
joined = "."
if abs_prefix:
joined = abs_prefix
else:
joined = "."
else:
joined = '/'.join(simpler)
joined = abs_prefix + ('/'.join(simpler))
return joined
def delete_daemon(file, please_stop):
# WINDOWS WILL HANG ONTO A FILE FOR A BIT AFTER WE CLOSED IT
while not please_stop:
try:
file.delete()
return
except Exception as e:
Log.warning(u"problem deleting file {{file}}", file=file.abspath, cause=e)
Till(seconds=1).wait()

16
vendor/mo_future/__init__.py поставляемый
Просмотреть файл

@ -24,6 +24,9 @@ if PY3:
import collections
from functools import cmp_to_key
from configparser import ConfigParser
from itertools import zip_longest
izip = zip
text_type = str
string_types = str
@ -31,17 +34,23 @@ if PY3:
integer_types = int
number_types = (int, float)
long = int
unichr = chr
xrange = range
filter_type = type(filter(lambda x: True, []))
generator_types = (collections.Iterable, filter_type)
unichr = chr
round = round
from html.parser import HTMLParser
from urllib.parse import urlparse
from io import StringIO
from io import BytesIO
from _thread import allocate_lock, get_ident, start_new_thread, interrupt_main
def transpose(*args):
return list(zip(*args))
def get_function_name(func):
return func.__name__
@ -75,7 +84,9 @@ else:
import __builtin__
from types import GeneratorType
from ConfigParser import ConfigParser
from itertools import izip_longest as zip_longest
from __builtin__ import zip as transpose
from itertools import izip
text_type = __builtin__.unicode
string_types = (str, unicode)
@ -83,14 +94,17 @@ else:
integer_types = (int, long)
number_types = (int, long, float)
long = __builtin__.long
unichr = __builtin__.unichr
xrange = __builtin__.xrange
generator_types = (GeneratorType,)
unichr = __builtin__.unichr
round = __builtin__.round
import HTMLParser
from urlparse import urlparse
from StringIO import StringIO
from io import BytesIO
from thread import allocate_lock, get_ident, start_new_thread, interrupt_main
def get_function_name(func):

40
vendor/mo_json/__init__.py поставляемый
Просмотреть файл

@ -25,8 +25,10 @@ from mo_logs.strings import expand_template
from mo_times import Date, Duration
FIND_LOOPS = False
SNAP_TO_BASE_10 = True # Identify floats near a round base10 value (has 000 or 999) and shorten
CAN_NOT_DECODE_JSON = "Can not decode JSON"
_get = object.__getattribute__
@ -61,21 +63,35 @@ def float2json(value):
sign = "-" if value < 0 else ""
value = abs(value)
sci = value.__format__(".15e")
mantissa, exp = sci.split("e")
exp = int(exp)
if 0 <= exp:
digits = u"".join(mantissa.split("."))
return sign+(digits[:1+exp]+u"."+digits[1+exp:].rstrip('0')).rstrip(".")
elif -4 < exp:
digits = ("0"*(-exp))+u"".join(mantissa.split("."))
return sign+(digits[:1]+u"."+digits[1:].rstrip('0')).rstrip(".")
mantissa, str_exp = sci.split("e")
int_exp = int(str_exp)
digits = _snap_to_base_10(mantissa)
if int_exp > 15:
return sign + digits[0] + '.' + (digits[1:].rstrip('0') or '0') + u"e" + text_type(int_exp)
elif int_exp >= 0:
return sign + (digits[:1 + int_exp] + '.' + digits[1 + int_exp:].rstrip('0')).rstrip('.')
elif -4 < int_exp:
digits = ("0" * (-int_exp)) + digits
return sign + (digits[:1] + '.' + digits[1:].rstrip('0')).rstrip('.')
else:
return sign+mantissa.rstrip("0")+u"e"+text_type(exp)
return sign + digits[0] + '.' + (digits[1:].rstrip('0') or '0') + u"e" + text_type(int_exp)
except Exception as e:
from mo_logs import Log
Log.error("not expected", e)
def _snap_to_base_10(mantissa):
digits = ''.join(mantissa.split('.'))
if SNAP_TO_BASE_10:
f9 = strings.find(digits, '999', start=1)
f0 = strings.find(digits, '000')
if f9 < f0:
digits = digits[:f9 - 1] + text_type(int(digits[f9 - 1]) + 1) + ('0' * (16 - f9)) # we know the last digit is not 9
else:
digits = digits[:f0]+('0'*(16-f0))
return digits
def _scrub_number(value):
d = float(value)
i_d = int(d)
@ -154,8 +170,8 @@ def _scrub(value, is_done, stack, scrub_text, scrub_number):
pass
elif isinstance(k, binary_type):
k = k.decode('utf8')
elif hasattr(k, "__unicode__"):
k = text_type(k)
# elif hasattr(k, "__unicode__"):
# k = text_type(k)
else:
Log.error("keys must be strings")
v = _scrub(v, is_done, stack, scrub_text, scrub_number)
@ -322,7 +338,7 @@ def json2value(json_string, params=Null, flexible=False, leaves=False):
def bytes2hex(value, separator=" "):
return separator.join('{:02x}'.format(x) for x in value)
return separator.join('{:02X}'.format(ord(x)) for x in value)
def utf82unicode(value):

10
vendor/mo_json/encoder.py поставляемый
Просмотреть файл

@ -18,12 +18,11 @@ import time
from collections import Mapping
from datetime import datetime, date, timedelta
from decimal import Decimal
from json.encoder import encode_basestring
from math import floor
from past.builtins import xrange
from mo_dots import Data, FlatList, NullType, Null
from mo_future import text_type, binary_type, long, utf8_json_encoder, sort_using_key
from mo_future import text_type, binary_type, long, utf8_json_encoder, sort_using_key, xrange
from mo_json import ESCAPE_DCT, scrub, float2json
from mo_logs import Except
from mo_logs.strings import utf82unicode, quote
@ -111,9 +110,6 @@ def pypy_json_encode(value, pretty=False):
_dealing_with_problem = False
almost_pattern = r"(?:\.(\d*)999)|(?:\.(\d*)000)"
class cPythonJSONEncoder(object):
def __init__(self, sort_keys=True):
object.__init__(self)
@ -291,7 +287,7 @@ def pretty_json(value):
elif isinstance(value, Mapping):
try:
items = sort_using_key(list(value.items()), lambda r: r[0])
values = [quote(k) + PRETTY_COLON + indent(pretty_json(v)).strip() for k, v in items if v != None]
values = [encode_basestring(k) + PRETTY_COLON + indent(pretty_json(v)).strip() for k, v in items if v != None]
if not values:
return "{}"
elif len(values) == 1:

37
vendor/mo_logs/__init__.py поставляемый
Просмотреть файл

@ -13,16 +13,16 @@ from __future__ import unicode_literals
import os
import platform
import sys
from collections import Mapping
from datetime import datetime
import sys
from mo_dots import coalesce, listwrap, wrap, unwrap, unwraplist, set_default, FlatList
from mo_future import text_type
from mo_future import text_type, PY3
from mo_logs import constants
from mo_logs.exceptions import Except, suppress_exception
from mo_logs.strings import indent
from mo_logs import constants
_Thread = None
@ -98,7 +98,7 @@ class Log(object):
for log in listwrap(settings.log):
Log.add_log(Log.new_instance(log))
if settings.cprofile.enabled==True:
if settings.cprofile.enabled == True:
Log.alert("cprofiling is enabled, writing to {{filename}}", filename=os.path.abspath(settings.cprofile.filename))
@classmethod
@ -325,7 +325,7 @@ class Log(object):
:return:
"""
if not isinstance(template, text_type):
Log.error("Log.note was expecting a unicode template")
Log.error("Log.warning was expecting a unicode template")
if isinstance(default_params, BaseException):
cause = default_params
@ -378,24 +378,25 @@ class Log(object):
add_to_trace = False
if cause == None:
pass
causes = None
elif isinstance(cause, list):
cause = []
causes = []
for c in listwrap(cause): # CAN NOT USE LIST-COMPREHENSION IN PYTHON3 (EXTRA STACK DEPTH FROM THE IN-LINED GENERATOR)
cause.append(Except.wrap(c, stack_depth=1))
cause = FlatList(cause)
causes.append(Except.wrap(c, stack_depth=1))
causes = FlatList(causes)
elif isinstance(cause, BaseException):
cause = Except.wrap(cause, stack_depth=1)
causes = Except.wrap(cause, stack_depth=1)
else:
Log.error("can only accept Exception , or list of exceptions")
causes = None
Log.error("can only accept Exception, or list of exceptions")
trace = exceptions.extract_stack(stack_depth + 1)
if add_to_trace:
cause[0].trace.extend(trace[1:])
e = Except(exceptions.ERROR, template, params, cause, trace)
raise e
e = Except(exceptions.ERROR, template, params, causes, trace)
raise_from_none(e)
@classmethod
def fatal(
@ -453,6 +454,7 @@ def write_profile(profile_settings, stats):
from pyLibrary import convert
from mo_files import File
Log.note("aggregating {{num}} profile stats", num=len(stats))
acc = stats[0]
for s in stats[1:]:
acc.add(s)
@ -482,6 +484,13 @@ machine_metadata = wrap({
})
def raise_from_none(e):
raise e
if PY3:
exec("def raise_from_none(e):\n raise e from None\n", globals(), locals())
from mo_logs.log_usingFile import StructuredLogger_usingFile
from mo_logs.log_usingMulti import StructuredLogger_usingMulti
from mo_logs.log_usingStream import StructuredLogger_usingStream

5
vendor/mo_logs/log_usingElasticSearch.py поставляемый
Просмотреть файл

@ -13,11 +13,10 @@ from __future__ import unicode_literals
from collections import Mapping
from mo_future import text_type, binary_type
import mo_json
from jx_python import jx
from mo_dots import wrap, coalesce, FlatList
from mo_future import text_type, binary_type, number_types
from mo_json import value2json
from mo_kwargs import override
from mo_logs import Log, strings
@ -116,7 +115,7 @@ def _deep_json_to_string(value, depth):
return {k: _deep_json_to_string(v, depth - 1) for k, v in value.items()}
elif isinstance(value, (list, FlatList)):
return strings.limit(value2json(value), LOG_STRING_LENGTH)
elif isinstance(value, (float, int, long)):
elif isinstance(value, number_types):
return value
elif isinstance(value, text_type):
return strings.limit(value, LOG_STRING_LENGTH)

50
vendor/mo_logs/log_usingEmail.py поставляемый
Просмотреть файл

@ -13,12 +13,13 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from mo_dots import listwrap, get_module, set_default, literal_field, Data
from mo_dots import listwrap, literal_field, Data
from mo_kwargs import override
from mo_logs import Log
from mo_logs.exceptions import ALARM, NOTE
from mo_logs.log_usingNothing import StructuredLogger
from mo_logs.strings import expand_template
from mo_math.randoms import Random
from mo_threads import Lock
from mo_times import Date, Duration, HOUR, MINUTE
from pyLibrary.env.emailer import Emailer
@ -39,7 +40,7 @@ class StructuredLogger_usingEmail(StructuredLogger):
use_ssl=1,
cc=None,
log_type="email",
max_interval=HOUR,
average_interval=HOUR,
kwargs=None
):
"""
@ -59,7 +60,6 @@ class StructuredLogger_usingEmail(StructuredLogger):
"password": "password",
"use_ssl": 1
}
"""
assert kwargs.log_type == "email", "Expecing settings to be of type 'email'"
self.settings = kwargs
@ -67,7 +67,7 @@ class StructuredLogger_usingEmail(StructuredLogger):
self.cc = listwrap(cc)
self.next_send = Date.now() + MINUTE
self.locker = Lock()
self.settings.max_interval = Duration(kwargs.max_interval)
self.settings.average_interval = Duration(kwargs.average_interval)
def write(self, template, params):
with self.locker:
@ -83,28 +83,30 @@ class StructuredLogger_usingEmail(StructuredLogger):
def _send_email(self):
try:
if self.accumulation:
with Emailer(self.settings) as emailer:
# WHO ARE WE SENDING TO
emails = Data()
for template, params in self.accumulation:
content = expand_template(template, params)
emails[literal_field(self.settings.to_address)] += [content]
for c in self.cc:
if any(d in params.params.error for d in c.contains):
emails[literal_field(c.to_address)] += [content]
if not self.accumulation:
return
with Emailer(self.settings) as emailer:
# WHO ARE WE SENDING TO
emails = Data()
for template, params in self.accumulation:
content = expand_template(template, params)
emails[literal_field(self.settings.to_address)] += [content]
for c in self.cc:
if any(d in params.params.error for d in c.contains):
emails[literal_field(c.to_address)] += [content]
# SEND TO EACH
for to_address, content in emails.items():
emailer.send_email(
from_address=self.settings.from_address,
to_address=listwrap(to_address),
subject=self.settings.subject,
text_data="\n\n".join(content)
)
# SEND TO EACH
for to_address, content in emails.items():
emailer.send_email(
from_address=self.settings.from_address,
to_address=listwrap(to_address),
subject=self.settings.subject,
text_data="\n\n".join(content)
)
self.next_send = Date.now() + self.settings.max_interval
self.accumulation = []
except Exception as e:
self.next_send = Date.now() + self.settings.max_interval
Log.warning("Could not send", e)
finally:
self.next_send = Date.now() + self.settings.average_interval * (2 * Random.float())

49
vendor/mo_logs/log_usingSES.py поставляемый
Просмотреть файл

@ -15,12 +15,13 @@ from __future__ import unicode_literals
from boto.ses import connect_to_region
from mo_dots import listwrap, unwrap, get_module, set_default, literal_field, Data, wrap
from mo_dots import listwrap, unwrap, literal_field, Data
from mo_kwargs import override
from mo_logs import Log, suppress_exception
from mo_logs.exceptions import ALARM, NOTE
from mo_logs.log_usingNothing import StructuredLogger
from mo_logs.strings import expand_template
from mo_math.randoms import Random
from mo_threads import Lock
from mo_times import Date, Duration, HOUR, MINUTE
@ -38,16 +39,32 @@ class StructuredLogger_usingSES(StructuredLogger):
aws_secret_access_key=None,
cc=None,
log_type="ses",
max_interval=HOUR,
average_interval=HOUR,
kwargs=None
):
"""
SEND WARNINGS AND ERRORS VIA EMAIL
settings = {
"log_type": "ses",
"from_address": "klahnakoski@mozilla.com",
"to_address": "klahnakoski@mozilla.com",
"cc":[
{"to_address":"me@example.com", "where":{"eq":{"template":"gr"}}}
],
"subject": "[ALERT][STAGING] Problem in ETL",
"aws_access_key_id": "userkey"
"aws_secret_access_key": "secret"
"region":"us-west-2"
}
"""
assert kwargs.log_type == "ses", "Expecing settings to be of type 'ses'"
self.settings = kwargs
self.accumulation = []
self.cc = listwrap(cc)
self.next_send = Date.now() + MINUTE
self.locker = Lock()
self.settings.max_interval = Duration(kwargs.max_interval)
self.settings.average_interval = Duration(kwargs.average_interval)
def write(self, template, params):
with self.locker:
@ -65,24 +82,19 @@ class StructuredLogger_usingSES(StructuredLogger):
try:
if not self.accumulation:
return
with Closer(connect_to_region(
self.settings.region,
aws_access_key_id=unwrap(self.settings.aws_access_key_id),
aws_secret_access_key=unwrap(self.settings.aws_secret_access_key)
)) as conn:
with Emailer(self.settings) as emailer:
# WHO ARE WE SENDING TO
emails = Data()
for template, params in self.accumulation:
content = expand_template(template, params)
emails[literal_field(self.settings.to_address)] += [content]
for c in self.cc:
if any(c in params.params.error for c in c.contains):
if any(d in params.params.error for d in c.contains):
emails[literal_field(c.to_address)] += [content]
# SEND TO EACH
for to_address, content in emails.items():
conn.send_email(
emailer.send_email(
source=self.settings.from_address,
to_addresses=listwrap(to_address),
subject=self.settings.subject,
@ -90,17 +102,20 @@ class StructuredLogger_usingSES(StructuredLogger):
format="text"
)
self.next_send = Date.now() + self.settings.max_interval
self.accumulation = []
except Exception as e:
self.next_send = Date.now() + self.settings.max_interval
Log.warning("Could not send", e)
finally:
self.next_send = Date.now() + self.settings.average_interval * (2 * Random.float())
class Closer(object):
def __init__(self, resource):
self.resource = resource
class Emailer(object):
def __init__(self, settings):
self.resource = connect_to_region(
settings.region,
aws_access_key_id=unwrap(settings.aws_access_key_id),
aws_secret_access_key=unwrap(settings.aws_secret_access_key)
)
def __enter__(self):
return self

3
vendor/mo_logs/log_usingThread.py поставляемый
Просмотреть файл

@ -13,8 +13,7 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from mo_logs.exceptions import suppress_exception, Except
from mo_logs import Log
from mo_logs import Log, Except, suppress_exception
from mo_logs.log_usingNothing import StructuredLogger
from mo_threads import Thread, Queue, Till, THREAD_STOP

2
vendor/mo_logs/log_usingThreadedStream.py поставляемый
Просмотреть файл

@ -77,7 +77,7 @@ class StructuredLogger_usingThreadedStream(StructuredLogger):
try:
self.queue.close()
except Exception, f:
except Exception as f:
if DEBUG_LOGGING:
raise f

1
vendor/mo_logs/profiles.py поставляемый
Просмотреть файл

@ -142,4 +142,5 @@ class CProfiler(object):
self.cprofiler.disable()
_Log.cprofiler_stats.add(pstats.Stats(self.cprofiler))
del self.cprofiler
_Log.note("done cprofile")

32
vendor/mo_logs/startup.py поставляемый
Просмотреть файл

@ -39,19 +39,32 @@ from mo_dots import listwrap, wrap, unwrap
# dest - The name of the attribute to be added to the object returned by parse_args().
class _ArgParser(_argparse.ArgumentParser):
def error(self, message):
Log.error("argparse error: {{error}}", error=message)
def argparse(defs):
parser = _argparse.ArgumentParser()
parser = _ArgParser()
for d in listwrap(defs):
args = d.copy()
name = args.name
args.name = None
parser.add_argument(*unwrap(listwrap(name)), **args)
namespace = parser.parse_args()
namespace, unknown = parser.parse_known_args()
if unknown:
Log.warning("Ignoring arguments: {{unknown|json}}", unknown=unknown)
output = {k: getattr(namespace, k) for k in vars(namespace)}
return wrap(output)
def read_settings(filename=None, defs=None):
def read_settings(filename=None, defs=None, env_filename=None):
"""
:param filename: Force load a file
:param defs: arguments you want to accept
:param env_filename: A config file from an environment variable (a fallback config file, if no other provided)
:return:
"""
# READ SETTINGS
if filename:
settings_file = File(filename)
@ -66,7 +79,7 @@ def read_settings(filename=None, defs=None):
else:
defs = listwrap(defs)
defs.append({
"name": ["--settings", "--settings-file", "--settings_file"],
"name": ["--config", "--settings", "--settings-file", "--settings_file"],
"help": "path to JSON file with settings",
"type": str,
"dest": "filename",
@ -74,6 +87,9 @@ def read_settings(filename=None, defs=None):
"required": False
})
args = argparse(defs)
if env_filename:
args.filename = env_filename
settings = mo_json_config.get("file://" + args.filename.replace(os.sep, "/"))
settings.args = args
return settings
@ -123,10 +139,10 @@ class SingleInstance:
fcntl.lockf(self.fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
Log.note(
"\n"
"**********************************************************************\n"
"** Another instance is already running, quitting.\n"
"**********************************************************************\n"
"\n" +
"**********************************************************************\n" +
"** Another instance is already running, quitting.\n" +
"****************************************************** ****************\n"
)
sys.exit(-1)
self.initialized = True

234
vendor/mo_logs/strings.py поставляемый
Просмотреть файл

@ -23,10 +23,12 @@ from datetime import timedelta, date
from json.encoder import encode_basestring
from mo_dots import coalesce, wrap, get_module
from mo_future import text_type, xrange, binary_type, round as _round, PY3
from mo_future import text_type, xrange, binary_type, round as _round, PY3, get_function_name
from mo_logs.convert import datetime2unix, datetime2string, value2json, milli2datetime, unix2datetime
from mo_logs.url import value2url_param
FORMATTERS = {}
_json_encoder = None
_Log = None
_Except = None
@ -53,20 +55,21 @@ def _late_import():
_ = _Duration
def expand_template(template, value):
def formatter(func):
"""
:param template: A UNICODE STRING WITH VARIABLE NAMES IN MOUSTACHES `{{}}`
:param value: Data HOLDING THE PARAMTER VALUES
:return: UNICODE STRING WITH VARIABLES EXPANDED
register formatters
"""
value = wrap(value)
if isinstance(template, text_type):
return _simple_expand(template, (value,))
return _expand(template, (value,))
FORMATTERS[get_function_name(func)]=func
return func
@formatter
def datetime(value):
"""
Convert from unix timestamp to GMT string
:param value: unix timestamp
:return: string with GMT time
"""
if isinstance(value, (date, builtin_datetime)):
pass
elif value < 10000000000:
@ -77,13 +80,25 @@ def datetime(value):
return datetime2string(value, "%Y-%m-%d %H:%M:%S")
@formatter
def unicode(value):
"""
Convert to a unicode string
:param value: any value
:return: unicode
"""
if value == None:
return ""
return text_type(value)
@formatter
def unix(value):
"""
Convert a date, or datetime to unix timestamp
:param value:
:return:
"""
if isinstance(value, (date, builtin_datetime)):
pass
elif value < 10000000000:
@ -94,6 +109,7 @@ def unix(value):
return str(datetime2unix(value))
@formatter
def url(value):
"""
convert FROM dict OR string TO URL PARAMETERS
@ -101,6 +117,7 @@ def url(value):
return value2url_param(value)
@formatter
def html(value):
"""
convert FROM unicode TO HTML OF THE SAME
@ -108,14 +125,27 @@ def html(value):
return cgi.escape(value)
@formatter
def upper(value):
"""
convert to uppercase
:param value:
:return:
"""
return value.upper()
@formatter
def lower(value):
"""
convert to lowercase
:param value:
:return:
"""
return value.lower()
@formatter
def newline(value):
"""
ADD NEWLINE, IF SOMETHING
@ -123,28 +153,57 @@ def newline(value):
return "\n" + toString(value).lstrip("\n")
@formatter
def replace(value, find, replace):
"""
:param value: focal value
:param find: string to find
:param replace: string to replace with
:return:
"""
return value.replace(find, replace)
@formatter
def json(value, pretty=True):
"""
convert value to JSON
:param value:
:param pretty:
:return:
"""
if not _Duration:
_late_import()
return _json_encoder(value, pretty=pretty)
@formatter
def tab(value):
"""
convert single value to tab-delimited form, including a header
:param value:
:return:
"""
if isinstance(value, Mapping):
h, d = zip(*wrap(value).leaves())
return \
"\t".join(map(value2json, h)) + \
"\n" + \
return (
"\t".join(map(value2json, h)) +
"\n" +
"\t".join(map(value2json, d))
)
else:
text_type(value)
@formatter
def indent(value, prefix=u"\t", indent=None):
"""
indent given string, using prefix * indent as prefix for each line
:param value:
:param prefix:
:param indent:
:return:
"""
if indent != None:
prefix = prefix * indent
@ -158,7 +217,13 @@ def indent(value, prefix=u"\t", indent=None):
raise Exception(u"Problem with indent of value (" + e.message + u")\n" + text_type(toString(value)))
@formatter
def outdent(value):
"""
remove common whitespace prefix from lines
:param value:
:return:
"""
try:
num = 100
lines = toString(value).splitlines()
@ -174,6 +239,7 @@ def outdent(value):
_Log.error("can not outdent value", e)
@formatter
def round(value, decimal=None, digits=None, places=None):
"""
:param value: THE VALUE TO ROUND
@ -196,7 +262,16 @@ def round(value, decimal=None, digits=None, places=None):
return format.format(_round(value, decimal))
@formatter
def percent(value, decimal=None, digits=None, places=None):
"""
display value as a percent (1 = 100%)
:param value:
:param decimal:
:param digits:
:param places:
:return:
"""
value = float(value)
if value == 0.0:
return "0%"
@ -212,9 +287,14 @@ def percent(value, decimal=None, digits=None, places=None):
return format.format(_round(value, decimal + 2))
@formatter
def find(value, find, start=0):
"""
MUCH MORE USEFUL VERSION OF string.find()
Return index of `find` in `value` beginning at `start`
:param value:
:param find:
:param start:
:return: If NOT found, return the length of `value` string
"""
l = len(value)
if isinstance(find, list):
@ -232,6 +312,7 @@ def find(value, find, start=0):
return i
@formatter
def strip(value):
"""
REMOVE WHITESPACE (INCLUDING CONTROL CHARACTERS)
@ -255,11 +336,26 @@ def strip(value):
return ""
@formatter
def trim(value):
"""
Alias for `strip`
:param value:
:return:
"""
return strip(value)
@formatter
def between(value, prefix, suffix, start=0):
"""
Return first substring between `prefix` and `suffix`
:param value:
:param prefix: if None then return the prefix that ends with `suffix`
:param suffix: if None then return the suffix that begins with `prefix`
:param start: where to start the search
:return:
"""
value = toString(value)
if prefix == None:
e = value.find(suffix, start)
@ -282,13 +378,26 @@ def between(value, prefix, suffix, start=0):
return value[s:e]
@formatter
def right(value, len):
"""
Return the `len` last characters of a string
:param value:
:param len:
:return:
"""
if len <= 0:
return u""
return value[-len:]
@formatter
def right_align(value, length):
"""
:param value: string to right align
:param length: the number of characters to output (spaces added to left)
:return:
"""
if length <= 0:
return u""
@ -300,6 +409,7 @@ def right_align(value, length):
return value[-length:]
@formatter
def left_align(value, length):
if length <= 0:
return u""
@ -312,12 +422,20 @@ def left_align(value, length):
return value[:length]
@formatter
def left(value, len):
"""
return the `len` left-most characters in value
:param value:
:param len:
:return:
"""
if len <= 0:
return u""
return value[0:len]
@formatter
def comma(value):
"""
FORMAT WITH THOUSANDS COMMA (,) SEPARATOR
@ -333,7 +451,13 @@ def comma(value):
return output
@formatter
def quote(value):
"""
return JSON-quoted value
:param value:
:return:
"""
if value == None:
output = ""
elif isinstance(value, text_type):
@ -343,16 +467,22 @@ def quote(value):
return output
@formatter
def hex(value):
"""
return `value` in hex format
:param value:
:return:
"""
return hex(value)
_SNIP = "...<snip>..."
@formatter
def limit(value, length):
# LIMIT THE STRING value TO GIVEN LENGTH
# LIMIT THE STRING value TO GIVEN LENGTH, CHOPPING OUT THE MIDDLE IF REQUIRED
if len(value) <= length:
return value
elif length < len(_SNIP) * 2:
@ -363,6 +493,7 @@ def limit(value, length):
return value[:lhs] + _SNIP + value[-rhs:]
@formatter
def split(value, sep="\n"):
# GENERATOR VERSION OF split()
# SOMETHING TERRIBLE HAPPENS, SOMETIMES, IN PYPY
@ -376,6 +507,23 @@ def split(value, sep="\n"):
yield value[s:]
value = None
"""
THE REST OF THIS FILE IS TEMPLATE EXPANSION CODE USED BY mo-logs
"""
def expand_template(template, value):
"""
:param template: A UNICODE STRING WITH VARIABLE NAMES IN MOUSTACHES `{{}}`
:param value: Data HOLDING THE PARAMTER VALUES
:return: UNICODE STRING WITH VARIABLES EXPANDED
"""
value = wrap(value)
if isinstance(template, text_type):
return _simple_expand(template, (value,))
return _expand(template, (value,))
def common_prefix(*args):
prefix = args[0]
@ -401,7 +549,29 @@ def is_hex(value):
return all(c in string.hexdigits for c in value)
pattern = re.compile(r"\{\{([\w_\.]+(\|[^\}^\|]+)*)\}\}")
if PY3:
delchars = "".join(c for c in map(chr, range(256)) if not c.isalnum())
else:
delchars = "".join(c.decode("latin1") for c in map(chr, range(256)) if not c.decode("latin1").isalnum())
def deformat(value):
"""
REMOVE NON-ALPHANUMERIC CHARACTERS
FOR SOME REASON translate CAN NOT BE CALLED:
ERROR: translate() takes exactly one argument (2 given)
File "C:\Python27\lib\string.py", line 493, in translate
"""
output = []
for c in value:
if c in delchars:
continue
output.append(c)
return "".join(output)
_variable_pattern = re.compile(r"\{\{([\w_\.]+(\|[^\}^\|]+)*)\}\}")
def _expand(template, seq):
@ -454,7 +624,7 @@ def _simple_expand(template, seq):
if len(parts) > 1:
val = eval(parts[0] + "(val, " + ("(".join(parts[1::])))
else:
val = globals()[func_name](val)
val = FORMATTERS[func_name](val)
val = toString(val)
return val
except Exception as e:
@ -477,29 +647,7 @@ def _simple_expand(template, seq):
)
return "[template expansion error: (" + str(e.message) + ")]"
return pattern.sub(replacer, template)
if PY3:
delchars = "".join(c for c in map(chr, range(256)) if not c.isalnum())
else:
delchars = "".join(c.decode("latin1") for c in map(chr, range(256)) if not c.decode("latin1").isalnum())
def deformat(value):
"""
REMOVE NON-ALPHANUMERIC CHARACTERS
FOR SOME REASON translate CAN NOT BE CALLED:
ERROR: translate() takes exactly one argument (2 given)
File "C:\Python27\lib\string.py", line 493, in translate
"""
output = []
for c in value:
if c in delchars:
continue
output.append(c)
return "".join(output)
return _variable_pattern.sub(replacer, template)
def toString(val):
@ -632,6 +780,10 @@ def apply_diff(text, diff, reverse=False):
return text
def unicode2utf8(value):
return value.encode('utf8')
def utf82unicode(value):
"""
WITH EXPLANATION FOR FAILURE

6
vendor/mo_logs/url.py поставляемый
Просмотреть файл

@ -63,7 +63,7 @@ class URL(object):
if value.startswith("file://") or value.startswith("//"):
# urlparse DOES NOT WORK IN THESE CASES
scheme, suffix = value.split("//", 2)
scheme, suffix = value.split("//", 1)
self.scheme = scheme.rstrip(":")
parse(self, suffix, 0, 1)
self.query = wrap(url_param2value(self.query))
@ -79,7 +79,7 @@ class URL(object):
if not _Log:
_late_import()
_Log.error("problem parsing {{value}} to URL", value=value, cause=e)
_Log.error(u"problem parsing {{value}} to URL", value=value, cause=e)
def __nonzero__(self):
if self.scheme or self.host or self.port or self.path or self.query or self.fragment:
@ -201,7 +201,7 @@ def value2url_param(value):
_late_import()
if value == None:
_Log.error("Can not encode None into a URL")
_Log.error(u"Can not encode None into a URL")
if isinstance(value, Mapping):
value_ = wrap(value)

4
vendor/mo_math/__init__.py поставляемый
Просмотреть файл

@ -153,8 +153,8 @@ class Math(object):
def round(value, decimal=7, digits=None):
"""
ROUND TO GIVEN NUMBER OF DIGITS, OR GIVEN NUMBER OF DECIMAL PLACES
decimal - NUMBER OF SIGNIFICANT DIGITS (LESS THAN 1 IS INVALID)
digits - NUMBER OF DIGITS AFTER DECIMAL POINT (NEGATIVE IS VALID)
decimal - NUMBER OF DIGITS AFTER DECIMAL POINT (NEGATIVE IS VALID)
digits - NUMBER OF SIGNIFICANT DIGITS (LESS THAN 1 IS INVALID)
"""
if value == None:
return None

27
vendor/mo_testing/fuzzytestcase.py поставляемый
Просмотреть файл

@ -17,8 +17,8 @@ import mo_dots
from mo_collections.unique_index import UniqueIndex
from mo_dots import coalesce, literal_field, unwrap, wrap
from mo_future import text_type
from mo_logs import Log
from mo_logs.exceptions import suppress_exception, Except
from mo_future import zip_longest
from mo_logs import Log, Except, suppress_exception
from mo_logs.strings import expand_template
from mo_math import Math
@ -76,25 +76,6 @@ class FuzzyTestCase(unittest.TestCase):
Log.error("Expecting an exception to be raised")
def zipall(*args):
"""
LOOP THROUGH LONGEST OF THE LISTS, None-FILL THE REMAINDER
"""
iters = [iter(a) for a in args]
def _next(_iter):
try:
return False, _iter.next()
except:
return True, None
while True:
is_done, value = zip(*(_next(a) for a in iters))
if all(is_done):
return
else:
yield value
def assertAlmostEqual(test, expected, digits=None, places=None, msg=None, delta=None):
show_detail = True
@ -145,13 +126,13 @@ def assertAlmostEqual(test, expected, digits=None, places=None, msg=None, delta=
return
if expected == None:
expected = [] # REPRESENT NOTHING
for a, b in zipall(test, expected):
for a, b in zip_longest(test, expected):
assertAlmostEqual(a, b, msg=msg, digits=digits, places=places, delta=delta)
else:
assertAlmostEqualValue(test, expected, msg=msg, digits=digits, places=places, delta=delta)
except Exception as e:
Log.error(
"{{test|json}} does not match expected {{expected|json}}",
"{{test|json|limit(10000)}} does not match expected {{expected|json|limit(10000)}}",
test=test if show_detail else "[can not show]",
expected=expected if show_detail else "[can not show]",
cause=e

55
vendor/mo_threads/__init__.py поставляемый
Просмотреть файл

@ -15,6 +15,8 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from mo_future import get_function_name
from mo_threads.lock import Lock
from mo_threads.signal import Signal
from mo_threads.till import Till
@ -24,4 +26,55 @@ from mo_threads.queues import ThreadedQueue
from mo_threads.multiprocess import Process
# from threading import Thread as _threading_Thread
# _temp = _threading_Thread.setDaemon
#
# fixes = []
# # WE NOW ADD A FIX FOR EACH KNOWN BAD ACTOR
# try:
# from paramiko import Transport
#
# def fix(self):
# if isinstance(self, Transport):
# self.stop = self.close # WE KNOW Transport DOES NOT HAVE A stop() METHOD, SO ADDING SHOULD BE FINE
# parent = Thread.current()
# parent.add_child(self)
# return True
#
# fixes.append(fix)
# except Exception:
# pass
#
#
# _known_daemons = [
# ('thread_handling', 17), # fabric/thread_handling.py
# ('pydevd_comm.py', 285), # plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py",
# ]
#
#
# # WE WRAP THE setDaemon METHOD TO APPLY THE FIX WHEN CALLED
# def _setDaemon(self, daemonic):
# for fix in fixes:
# if fix(self):
# break
# else:
# from mo_logs import Log
# from mo_logs.exceptions import extract_stack
# from mo_files import File
#
# get_function_name(self.__target)
#
# stack = extract_stack(1)[0]
# uid = (File(stack['file']).name, stack['line'])
# if uid in _known_daemons:
# pass
# else:
# _known_daemons.append(uid)
# Log.warning("daemons in threading.Thread do not shutdown clean. {{type}} not handled.", type=repr(self))
#
# _temp(self, daemonic)
#
#
# _threading_Thread.setDaemon = _setDaemon
#
#

8
vendor/mo_threads/multiprocess.py поставляемый
Просмотреть файл

@ -13,8 +13,8 @@ from __future__ import unicode_literals
import os
import subprocess
from mo_dots import set_default, unwrap, NullType
from mo_future import none_type, binary_type, text_type
from mo_dots import set_default, NullType
from mo_future import none_type
from mo_logs import Log, strings
from mo_logs.exceptions import Except
from mo_threads.lock import Lock
@ -41,8 +41,8 @@ class Process(object):
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=bufsize,
cwd=cwd if isinstance(cwd, (text_type, binary_type, NullType, none_type)) else cwd.abspath,
env=unwrap(set_default(env, os.environ)),
cwd=cwd if isinstance(cwd, (str, NullType, none_type)) else cwd.abspath,
env={str(k): str(v) for k, v in set_default(env, os.environ).items()},
shell=shell
)

20
vendor/mo_threads/signal.py поставляемый
Просмотреть файл

@ -34,7 +34,7 @@ class Signal(object):
__slots__ = ["_name", "lock", "_go", "job_queue", "waiting_threads"]
def __init__(self, name=None):
if DEBUG:
if DEBUG and name:
Log.note("New signal {{name|quote}}", name=name)
self._name = name
self.lock = _allocate_lock()
@ -68,10 +68,10 @@ class Signal(object):
else:
self.waiting_threads.append(stopper)
if DEBUG:
if DEBUG and self._name:
Log.note("wait for go {{name|quote}}", name=self.name)
stopper.acquire()
if DEBUG:
if DEBUG and self._name:
Log.note("GOing! {{name|quote}}", name=self.name)
return True
@ -79,7 +79,7 @@ class Signal(object):
"""
ACTIVATE SIGNAL (DOES NOTHING IF SIGNAL IS ALREADY ACTIVATED)
"""
if DEBUG:
if DEBUG and self._name:
Log.note("GO! {{name|quote}}", name=self.name)
if self._go:
@ -90,13 +90,13 @@ class Signal(object):
return
self._go = True
if DEBUG:
if DEBUG and self._name:
Log.note("internal GO! {{name|quote}}", name=self.name)
jobs, self.job_queue = self.job_queue, None
threads, self.waiting_threads = self.waiting_threads, None
if threads:
if DEBUG:
if DEBUG and self._name:
Log.note("Release {{num}} threads", num=len(threads))
for t in threads:
t.release()
@ -117,7 +117,7 @@ class Signal(object):
with self.lock:
if not self._go:
if DEBUG:
if DEBUG and self._name:
Log.note("Adding target to signal {{name|quote}}", name=self.name)
if not self.job_queue:
self.job_queue = [target]
@ -176,7 +176,7 @@ class Signal(object):
if not isinstance(other, Signal):
Log.error("Expecting OR with other signal")
if DEBUG:
if DEBUG and self._name:
output = Signal(self.name + " and " + other.name)
else:
output = Signal(self.name + " and " + other.name)
@ -207,3 +207,7 @@ class AndSignals(object):
remaining = self.remaining
if not remaining:
self.signal.go()
DONE = Signal()
DONE.go()

109
vendor/mo_threads/threads.py поставляемый
Просмотреть файл

@ -203,67 +203,66 @@ class Thread(object):
pass
def _run(self):
with CProfiler():
self.id = get_ident()
with ALL_LOCK:
ALL[self.id] = self
self.id = get_ident()
with ALL_LOCK:
ALL[self.id] = self
try:
if self.target is not None:
a, k, self.args, self.kwargs = self.args, self.kwargs, None, None
try:
if self.target is not None:
a, k, self.args, self.kwargs = self.args, self.kwargs, None, None
with CProfiler(): # PROFILE IN HERE SO THAT __exit__() IS RUN BEFORE THREAD MARKED AS stopped
response = self.target(*a, **k)
with self.synch_lock:
self.end_of_thread = Data(response=response)
else:
with self.synch_lock:
self.end_of_thread = Null
except Exception as e:
e = Except.wrap(e)
with self.synch_lock:
self.end_of_thread = Data(exception=e)
if self not in self.parent.children:
# THREAD FAILURES ARE A PROBLEM ONLY IF NO ONE WILL BE JOINING WITH IT
try:
Log.fatal("Problem in thread {{name|quote}}", name=self.name, cause=e)
except Exception:
sys.stderr.write(b"ERROR in thread: " + str(self.name) + b" " + str(e) + b"\n")
finally:
self.end_of_thread = Data(response=response)
else:
with self.synch_lock:
self.end_of_thread = Null
except Exception as e:
e = Except.wrap(e)
with self.synch_lock:
self.end_of_thread = Data(exception=e)
if self not in self.parent.children:
# THREAD FAILURES ARE A PROBLEM ONLY IF NO ONE WILL BE JOINING WITH IT
try:
children = copy(self.children)
for c in children:
try:
if DEBUG:
sys.stdout.write(b"Stopping thread " + str(c.name) + b"\n")
c.stop()
except Exception as e:
Log.warning("Problem stopping thread {{thread}}", thread=c.name, cause=e)
Log.fatal("Problem in thread {{name|quote}}", name=self.name, cause=e)
except Exception:
sys.stderr.write(b"ERROR in thread: " + str(self.name) + b" " + str(e) + b"\n")
finally:
try:
children = copy(self.children)
for c in children:
try:
if DEBUG:
sys.stdout.write(b"Stopping thread " + str(c.name) + b"\n")
c.stop()
except Exception as e:
Log.warning("Problem stopping thread {{thread}}", thread=c.name, cause=e)
for c in children:
try:
if DEBUG:
sys.stdout.write(b"Joining on thread " + str(c.name) + b"\n")
c.join()
except Exception as e:
Log.warning("Problem joining thread {{thread}}", thread=c.name, cause=e)
finally:
if DEBUG:
sys.stdout.write(b"Joined on thread " + str(c.name) + b"\n")
for c in children:
try:
if DEBUG:
sys.stdout.write(b"Joining on thread " + str(c.name) + b"\n")
c.join()
except Exception as e:
Log.warning("Problem joining thread {{thread}}", thread=c.name, cause=e)
finally:
if DEBUG:
sys.stdout.write(b"Joined on thread " + str(c.name) + b"\n")
self.stopped.go()
if DEBUG:
Log.note("thread {{name|quote}} stopping", name=self.name)
del self.target, self.args, self.kwargs
with ALL_LOCK:
del ALL[self.id]
self.stopped.go()
if DEBUG:
Log.note("thread {{name|quote}} stopping", name=self.name)
del self.target, self.args, self.kwargs
with ALL_LOCK:
del ALL[self.id]
except Exception as e:
if DEBUG:
Log.warning("problem with thread {{name|quote}}", cause=e, name=self.name)
finally:
self.stopped.go()
if DEBUG:
Log.note("thread {{name|quote}} is done", name=self.name)
except Exception as e:
if DEBUG:
Log.warning("problem with thread {{name|quote}}", cause=e, name=self.name)
finally:
self.stopped.go()
if DEBUG:
Log.note("thread {{name|quote}} is done", name=self.name)
def is_alive(self):
return not self.stopped
@ -423,5 +422,5 @@ ALL_LOCK = Lock("threads ALL_LOCK")
ALL = dict()
ALL[get_ident()] = MAIN_THREAD
MAIN_THREAD.timers = Thread.run("timers", till.daemon)
MAIN_THREAD.timers = Thread.run("timers daemon", till.daemon)
MAIN_THREAD.children.remove(MAIN_THREAD.timers)

12
vendor/mo_times/dates.py поставляемый
Просмотреть файл

@ -18,9 +18,8 @@ from datetime import datetime, date, timedelta
from decimal import Decimal
from time import time as _time
from mo_future import text_type, PY3, long
from mo_dots import Null
from mo_future import unichr, text_type, long
from mo_logs import Except
from mo_logs.strings import deformat
@ -440,19 +439,12 @@ def _unix2Date(unix):
return output
if PY3:
delchars = "".join(c for c in map(chr, range(256)) if not c.isalnum())
else:
delchars = "".join(c.decode("latin1") for c in map(chr, range(256)) if not c.decode("latin1").isalnum())
delchars = "".join(c for c in map(unichr, range(256)) if not c.isalnum())
def deformat(value):
"""
REMOVE NON-ALPHANUMERIC CHARACTERS
FOR SOME REASON translate CAN NOT BE CALLED:
ERROR: translate() takes exactly one argument (2 given)
File "C:\\Python27\\lib\\string.py", line 493, in translate
"""
output = []
for c in value:

1
vendor/mo_times/durations.py поставляемый
Просмотреть файл

@ -17,6 +17,7 @@ import re
from mo_future import text_type
from mo_dots import get_module, wrap
from mo_future import text_type
from mo_math import MIN, Math
from mo_times.vendor.dateutil.relativedelta import relativedelta

10
vendor/mo_times/vendor/dateutil/parser.py поставляемый
Просмотреть файл

@ -9,21 +9,17 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
__license__ = "Simplified BSD"
import collections
import datetime
import string
import time
import collections
from mo_future import text_type, StringIO, integer_types, binary_type
from mo_future import text_type, integer_types, binary_type, StringIO
from . import relativedelta
from . import tz
__license__ = "Simplified BSD"
__all__ = ["parse", "parserinfo"]

7
vendor/mo_times/vendor/dateutil/relativedelta.py поставляемый
Просмотреть файл

@ -4,15 +4,12 @@ Copyright (c) 2003-2010 Gustavo Niemeyer <gustavo@niemeyer.net>
This module offers extensions to the standard Python
datetime module.
"""
__license__ = "Simplified BSD"
import calendar
import datetime
from mo_future import long
integer_types = (int, long)
from mo_future import integer_types
__license__ = "Simplified BSD"
__all__ = ["relativedelta", "MO", "TU", "WE", "TH", "FR", "SA", "SU"]
class weekday(object):

9
vendor/mo_times/vendor/dateutil/tz.py поставляемый
Просмотреть файл

@ -5,8 +5,6 @@ This module offers extensions to the standard Python
datetime module.
"""
__license__ = "Simplified BSD"
import datetime
import os
import struct
@ -15,13 +13,14 @@ import time
from mo_future import PY3, string_types
__license__ = "Simplified BSD"
__all__ = ["tzutc", "tzoffset", "tzlocal", "tzfile", "tzrange",
"tzstr", "tzical", "tzwin", "tzwinlocal", "gettz"]
relativedelta = None
parser = None
rrule = None
__all__ = ["tzutc", "tzoffset", "tzlocal", "tzfile", "tzrange",
"tzstr", "tzical", "tzwin", "tzwinlocal", "gettz"]
try:
from dateutil.tzwin import tzwin, tzwinlocal
except (ImportError, OSError):

55
vendor/moz_sql_parser/__init__.py поставляемый
Просмотреть файл

@ -1,55 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import json
from mo_future import text_type
from pyparsing import ParseException
from moz_sql_parser.sql_parser import SQLParser, all_exceptions
def parse(sql):
try:
parse_result = SQLParser.parseString(sql, parseAll=True)
except Exception as e:
if e.msg == "Expected end of text":
problems = all_exceptions[e.loc]
expecting = [
f
for f in (set(p.msg.lstrip("Expected").strip() for p in problems)-{"Found unwanted token"})
if not f.startswith("{")
]
raise ParseException(sql, e.loc, "Expecting one of (" + (", ".join(expecting)) + ")")
else:
raise e
return _scrub(parse_result)
def _scrub(result):
if isinstance(result, (str, text_type, int, float)):
return result
elif not result:
return {}
elif isinstance(result, list) or not result.items():
if not result:
return None
elif len(result) == 1:
return _scrub(result[0])
else:
return [_scrub(r) for r in result]
else:
return {k: _scrub(v) for k, v in result.items()}
_ = json.dumps

347
vendor/moz_sql_parser/sql_parser.py поставляемый
Просмотреть файл

@ -1,347 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import ast
import sys
from pyparsing import \
CaselessLiteral, Word, delimitedList, Optional, Combine, Group, alphas, nums, alphanums, Forward, restOfLine, Keyword, Literal, ParserElement, infixNotation, opAssoc, Regex, MatchFirst, ZeroOrMore, _ustr
ParserElement.enablePackrat()
# THE PARSING DEPTH IS NASTY
sys.setrecursionlimit(1500)
DEBUG = False
END = None
all_exceptions = {}
def record_exception(instring, loc, expr, exc):
# if DEBUG:
# print ("Exception raised:" + _ustr(exc))
es = all_exceptions.setdefault(loc, [])
es.append(exc)
def nothing(*args):
pass
if DEBUG:
debug = (None, None, None)
else:
debug = (nothing, nothing, record_exception)
keywords = [
"and",
"as",
"between",
"case",
"collate nocase",
"cross join",
"desc",
"else",
"end",
"from",
"group by",
"having",
"in",
"inner join",
"is",
"join",
"limit",
"on",
"or",
"order by",
"select",
"then",
"union",
"when",
"where",
"with"
]
locs = locals()
reserved = []
for k in keywords:
name = k.upper().replace(" ", "")
locs[name] = value = Keyword(k, caseless=True).setName(k.lower()).setDebugActions(*debug)
reserved.append(value)
RESERVED = MatchFirst(reserved)
KNOWN_OPS = [
(BETWEEN, AND),
Literal("||").setName("concat").setDebugActions(*debug),
Literal("*").setName("mult").setDebugActions(*debug),
Literal("/").setName("div").setDebugActions(*debug),
Literal("+").setName("add").setDebugActions(*debug),
Literal("-").setName("sub").setDebugActions(*debug),
Literal("<>").setName("neq").setDebugActions(*debug),
Literal(">").setName("gt").setDebugActions(*debug),
Literal("<").setName("lt").setDebugActions(*debug),
Literal(">=").setName("gte").setDebugActions(*debug),
Literal("<=").setName("lte").setDebugActions(*debug),
IN.setName("in").setDebugActions(*debug),
IS.setName("eq").setDebugActions(*debug),
Literal("=").setName("eq").setDebugActions(*debug),
Literal("==").setName("eq").setDebugActions(*debug),
Literal("!=").setName("neq").setDebugActions(*debug),
OR.setName("or").setDebugActions(*debug),
AND.setName("and").setDebugActions(*debug)
]
def to_json_operator(instring, tokensStart, retTokens):
# ARRANGE INTO {op: params} FORMAT
tok = retTokens[0]
for o in KNOWN_OPS:
if isinstance(o, tuple):
if o[0].match == tok[1]:
op = o[0].name
break
elif o.match == tok[1]:
op = o.name
break
else:
if tok[1] == COLLATENOCASE.match:
op = COLLATENOCASE.name
return {op: tok[0]}
else:
raise "not found"
if op == "eq":
if tok[2] == "null":
return {"missing": tok[0]}
elif tok[0] == "null":
return {"missing": tok[2]}
elif op == "neq":
if tok[2] == "null":
return {"exists": tok[0]}
elif tok[0] == "null":
return {"exists": tok[2]}
return {op: [tok[i * 2] for i in range(int((len(tok) + 1) / 2))]}
def to_json_call(instring, tokensStart, retTokens):
# ARRANGE INTO {op: params} FORMAT
tok = retTokens
op = tok.op.lower()
if op == "-":
op = "neg"
params = tok.params
if not params:
params = None
elif len(params) == 1:
params = params[0]
return {op: params}
def to_case_call(instring, tokensStart, retTokens):
tok = retTokens
cases = list(tok.case)
elze = getattr(tok, "else", None)
if elze:
cases.append(elze)
return {"case": cases}
def to_when_call(instring, tokensStart, retTokens):
tok = retTokens
return {"when": tok.when, "then":tok.then}
def to_join_call(instring, tokensStart, retTokens):
tok = retTokens
output = {tok.op: tok.join}
if tok.on:
output['on'] = tok.on
return output
def to_select_call(instring, tokensStart, retTokens):
# toks = datawrap(retTokens)
# return {
# "select": toks.select,
# "from": toks['from'],
# "where": toks.where,
# "groupby": toks.groupby,
# "having": toks.having,
# "limit": toks.limit
#
# }
return retTokens
def to_union_call(instring, tokensStart, retTokens):
tok = retTokens[0].asDict()
unions = tok['from']['union']
if len(unions) == 1:
output = unions[0]
if tok.get('orderby'):
output["orderby"] = tok.get('orderby')
if tok.get('limit'):
output["limit"] = tok.get('limit')
return output
else:
if not tok.get('orderby') and not tok.get('limit'):
return tok['from']
else:
return {
"from": {"union": unions},
"orderby": tok.get('orderby') if tok.get('orderby') else None,
"limit": tok.get('limit') if tok.get('limit') else None
}
def unquote(instring, tokensStart, retTokens):
val = retTokens[0]
if val.startswith("'") and val.endswith("'"):
val = "'"+val[1:-1].replace("''", "\\'")+"'"
# val = val.replace(".", "\\.")
elif val.startswith('"') and val.endswith('"'):
val = '"'+val[1:-1].replace('""', '\\"')+'"'
# val = val.replace(".", "\\.")
elif val.startswith("+"):
val = val[1:]
un = ast.literal_eval(val)
return un
def to_string(instring, tokensStart, retTokens):
val = retTokens[0]
val = "'"+val[1:-1].replace("''", "\\'")+"'"
return {"literal": ast.literal_eval(val)}
# NUMBERS
E = CaselessLiteral("E")
# binop = oneOf("= != < > >= <= eq ne lt le gt ge", caseless=True)
arithSign = Word("+-", exact=1)
realNum = Combine(
Optional(arithSign) +
(Word(nums) + "." + Optional(Word(nums)) | ("." + Word(nums))) +
Optional(E + Optional(arithSign) + Word(nums))
).addParseAction(unquote)
intNum = Combine(
Optional(arithSign) +
Word(nums) +
Optional(E + Optional("+") + Word(nums))
).addParseAction(unquote)
# STRINGS, NUMBERS, VARIABLES
sqlString = Combine(Regex(r"\'(\'\'|\\.|[^'])*\'")).addParseAction(to_string)
identString = Combine(Regex(r'\"(\"\"|\\.|[^"])*\"')).addParseAction(unquote)
ident = Combine(~RESERVED + (delimitedList(Literal("*") | Word(alphas + "_", alphanums + "_$") | identString, delim=".", combine=True))).setName("identifier")
# EXPRESSIONS
expr = Forward()
# CASE
case = (
CASE +
Group(ZeroOrMore((WHEN + expr("when") + THEN + expr("then")).addParseAction(to_when_call)))("case") +
Optional(ELSE + expr("else")) +
END
).addParseAction(to_case_call)
selectStmt = Forward()
compound = (
(Literal("-")("op").setDebugActions(*debug) + expr("params")).addParseAction(to_json_call) |
(Keyword("not", caseless=True)("op").setDebugActions(*debug) + expr("params")).addParseAction(to_json_call) |
(Keyword("distinct", caseless=True)("op").setDebugActions(*debug) + expr("params")).addParseAction(to_json_call) |
Keyword("null", caseless=True).setName("null").setDebugActions(*debug) |
case |
(Literal("(").setDebugActions(*debug).suppress() + selectStmt + Literal(")").suppress()) |
(Literal("(").setDebugActions(*debug).suppress() + Group(delimitedList(expr)) + Literal(")").suppress()) |
realNum.setName("float").setDebugActions(*debug) |
intNum.setName("int").setDebugActions(*debug) |
sqlString.setName("string").setDebugActions(*debug) |
(
Word(alphas)("op").setName("function name").setDebugActions(*debug) +
Literal("(").setName("func_param").setDebugActions(*debug) +
Optional(selectStmt | Group(delimitedList(expr)))("params") +
")"
).addParseAction(to_json_call).setDebugActions(*debug) |
ident.copy().setName("variable").setDebugActions(*debug)
)
expr << Group(infixNotation(
compound,
[
(
o,
3 if isinstance(o, tuple) else 2,
opAssoc.LEFT,
to_json_operator
)
for o in KNOWN_OPS
]+[
(
COLLATENOCASE,
1,
opAssoc.LEFT,
to_json_operator
)
]
).setName("expression").setDebugActions(*debug))
# SQL STATEMENT
selectColumn = Group(
Group(expr).setName("expression1")("value").setDebugActions(*debug) + Optional(Optional(AS) + ident.copy().setName("column_name1")("name").setDebugActions(*debug)) |
Literal('*')("value").setDebugActions(*debug)
).setName("column")
tableName = (
ident("value").setName("table name").setDebugActions(*debug) +
Optional(AS) +
ident("name").setName("table alias").setDebugActions(*debug) |
ident.setName("table name").setDebugActions(*debug)
)
join = ((CROSSJOIN | INNERJOIN | JOIN)("op") + tableName("join") + Optional(ON + expr("on"))).addParseAction(to_join_call)
sortColumn = expr("value").setName("sort1").setDebugActions(*debug) + Optional(DESC("sort")) | \
expr("value").setName("sort2").setDebugActions(*debug)
# define SQL tokens
selectStmt << Group(
Group(Group(
delimitedList(
Group(
SELECT.suppress().setDebugActions(*debug) + delimitedList(selectColumn)("select") +
Optional(
FROM.suppress().setDebugActions(*debug) + (delimitedList(Group(tableName)) + ZeroOrMore(join))("from") +
Optional(WHERE.suppress().setDebugActions(*debug) + expr.setName("where"))("where") +
Optional(GROUPBY.suppress().setDebugActions(*debug) + delimitedList(Group(selectColumn))("groupby").setName("groupby")) +
Optional(HAVING.suppress().setDebugActions(*debug) + expr("having").setName("having")) +
Optional(LIMIT.suppress().setDebugActions(*debug) + expr("limit"))
)
),
delim=UNION
)
)("union"))("from") +
Optional(ORDERBY.suppress().setDebugActions(*debug) + delimitedList(Group(sortColumn))("orderby").setName("orderby")) +
Optional(LIMIT.suppress().setDebugActions(*debug) + expr("limit"))
).addParseAction(to_union_call)
SQLParser = selectStmt
# IGNORE SOME COMMENTS
oracleSqlComment = Literal("--") + restOfLine
mySqlComment = Literal("#") + restOfLine
SQLParser.ignore(oracleSqlComment | mySqlComment)

9
vendor/pyLibrary/aws/s3.py поставляемый
Просмотреть файл

@ -24,6 +24,7 @@ from mo_future import text_type
from mo_dots import wrap, Null, coalesce, unwrap, Data
from mo_kwargs import override
from mo_logs import Log, Except
from mo_logs.strings import utf82unicode, unicode2utf8
from mo_logs.url import value2url_param
from mo_times.dates import Date
from mo_times.timer import Timer
@ -264,7 +265,7 @@ class Bucket(object):
elif source.key.endswith(".gz"):
json = convert.zip2bytes(json)
return convert.utf82unicode(json)
return utf82unicode(json)
def read_bytes(self, key):
source = self.get_meta(key)
@ -278,7 +279,7 @@ class Bucket(object):
if source.key.endswith(".gz"):
return LazyLines(ibytes2ilines(scompressed2ibytes(source)))
else:
return convert.utf82unicode(source.read()).split("\n")
return utf82unicode(source.read()).split("\n")
if source.key.endswith(".gz"):
return LazyLines(ibytes2ilines(scompressed2ibytes(source)))
@ -314,7 +315,7 @@ class Bucket(object):
value = convert.bytes2zip(value)
key += ".json.gz"
else:
value = convert.bytes2zip(convert.unicode2utf8(value))
value = convert.bytes2zip(unicode2utf8(value))
key += ".json.gz"
else:
@ -446,7 +447,7 @@ class PublicBucket(object):
def more():
xml = http.get(self.url + "?" + value2url_param(state)).content
data = BeautifulSoup(xml)
data = BeautifulSoup(xml, 'xml')
state.get_more = data.find("istruncated").contents[0] == "true"
contents = data.findAll("contents")

12
vendor/pyLibrary/convert.py поставляемый
Просмотреть файл

@ -405,14 +405,6 @@ def value2number(v):
Log.error("Not a number ({{value}})", value= v, cause=e)
def utf82unicode(value):
return value.decode('utf8')
def unicode2utf8(value):
return value.encode('utf8')
def latin12unicode(value):
if isinstance(value, text_type):
Log.error("can not convert unicode from latin1")
@ -478,9 +470,9 @@ def ini2value(ini_content):
"""
INI FILE CONTENT TO Data
"""
from ConfigParser import ConfigParser
from mo_future import ConfigParser, StringIO
buff = StringIO.StringIO(ini_content)
buff = StringIO(ini_content)
config = ConfigParser()
config._read(buff, "dummy")

20
vendor/pyLibrary/env/README.md поставляемый
Просмотреть файл

@ -1,27 +1,23 @@
Environment
===========
# Environment
This directory is for connecting to other systems. Generally, these
classes are facades that assume content is UTF-8 encoded JSON.
emailer
-------
## emailer
A simple emailer, the primary purpose is to accept a [Data](../dot/README.md)
A simple emailer, the primary purpose is to accept a [Data](https://github.com/klahnakoski/mo-dots/blob/dev/docs/README.md)
of settings.
pulse
-----
## pulse
For connecting clients to [Mozilla's Pulse](https://pulse.mozilla.org/).
elasticsearch
-------------
## elasticsearch
This module handles the lifecycle of an Elasticsearch index in the context of
ETL. You only need this module if you are creating and retiring indexes. You
@ -45,9 +41,7 @@ selecting only the properties it requires.
Cluster
-------
## Cluster
Index
-----
## Index

24
vendor/pyLibrary/env/big_data.py поставляемый
Просмотреть файл

@ -11,12 +11,15 @@ from __future__ import division
from __future__ import absolute_import
import gzip
import struct
from io import BytesIO
from tempfile import TemporaryFile
import zipfile
import zlib
from mo_future import text_type, PY3
import time
from mo_future import text_type, PY3, long
from mo_logs.exceptions import suppress_exception
from mo_logs import Log
@ -318,6 +321,25 @@ def ibytes2ilines(generator, encoding="utf8", flexible=False, closer=None):
e = _buffer.find(b"\n", s)
def ibytes2icompressed(source):
yield (
b'\037\213\010\000' + # Gzip file, deflate, no filename
struct.pack('<L', long(time.time())) + # compression start time
b'\002\377' # maximum compression, no OS specified
)
crc = zlib.crc32(b"")
length = 0
compressor = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL, 0)
for d in source:
crc = zlib.crc32(d, crc) & 0xffffffff
length += len(d)
chunk = compressor.compress(d)
if chunk:
yield chunk
yield compressor.flush()
yield struct.pack("<2L", crc, length & 0xffffffff)
class GzipLines(CompressedLines):
"""

205
vendor/pyLibrary/env/elasticsearch.py поставляемый
Просмотреть файл

@ -15,27 +15,22 @@ import re
from collections import Mapping
from copy import deepcopy
import mo_json
from jx_python import jx
from jx_python.expressions import jx_expression_to_function
from jx_python.meta import Column
from mo_dots import coalesce, Null, Data, set_default, listwrap, literal_field, ROOT_PATH, concat_field, split_field
from mo_dots import wrap
from mo_dots.lists import FlatList
from mo_dots import wrap, FlatList
from mo_future import text_type, binary_type
from mo_json import value2json
from mo_json import value2json, json2value
from mo_json.typed_encoder import EXISTS_TYPE, BOOLEAN_TYPE, STRING_TYPE, NUMBER_TYPE, NESTED_TYPE, TYPE_PREFIX
from mo_kwargs import override
from mo_logs import Log, strings
from mo_logs.exceptions import Except
from mo_logs.strings import utf82unicode
from mo_logs.strings import utf82unicode, unicode2utf8
from mo_math import Math
from mo_math.randoms import Random
from mo_threads import Lock
from mo_threads import ThreadedQueue
from mo_threads import Till
from mo_times.dates import Date
from mo_times.timer import Timer
from mo_threads import Lock, ThreadedQueue, Till
from mo_times import Date, Timer
from pyLibrary import convert
from pyLibrary.env import http
@ -44,6 +39,8 @@ ES_NUMERIC_TYPES = ["long", "integer", "double", "float"]
ES_PRIMITIVE_TYPES = ["string", "boolean", "integer", "date", "long", "double"]
INDEX_DATE_FORMAT = "%Y%m%d_%H%M%S"
DATA_KEY = text_type("data")
class Features(object):
pass
@ -73,7 +70,7 @@ class Index(Features):
alias=None,
explore_metadata=True, # PROBING THE CLUSTER FOR METADATA IS ALLOWED
read_only=True,
tjson=False, # STORED AS TYPED JSON
tjson=None, # STORED AS TYPED JSON
timeout=None, # NUMBER OF SECONDS TO WAIT FOR RESPONSE, OR SECONDS TO WAIT FOR DOWNLOAD (PASSED TO requests)
consistency="one", # ES WRITE CONSISTENCY (https://www.elastic.co/guide/en/elasticsearch/reference/1.7/docs-index_.html#index-consistency)
debug=False, # DO NOT SHOW THE DEBUG STATEMENTS
@ -124,13 +121,14 @@ class Index(Features):
if self.debug:
Log.alert("elasticsearch debugging for {{url}} is on", url=self.url)
if kwargs.tjson:
if tjson:
from pyLibrary.env.typed_inserter import TypedInserter
self.encode = TypedInserter(self, id_column).typed_encode
else:
if not kwargs.read_only:
Log.warning("{{index}} is not typed", index=self.settings.index)
if tjson == None and not read_only:
kwargs.tjson = False
Log.warning("{{index}} is not typed tjson={{tjson}}", index=self.settings.index, tjson=self.settings.tjson)
self.encode = get_encoder(id_column)
@property
@ -190,7 +188,8 @@ class Index(Features):
{"add": {"index": self.settings.index, "alias": alias}}
]
},
timeout=coalesce(self.settings.timeout, 30)
timeout=coalesce(self.settings.timeout, 30),
stream=False
)
self.settings.alias = alias
@ -292,8 +291,8 @@ class Index(Features):
try:
for r in records:
rec = self.encode(r)
json_bytes = rec['json'].encode('utf8')
lines.append(b'{"index":{"_id": ' + convert.value2json(rec['id']).encode("utf8") + b'}}')
json_bytes = rec['json']
lines.append('{"index":{"_id": ' + convert.value2json(rec['id']) + '}}')
lines.append(json_bytes)
del records
@ -303,7 +302,7 @@ class Index(Features):
with Timer("Add {{num}} documents to {{index}}", {"num": len(lines) / 2, "index":self.settings.index}, debug=self.debug):
try:
data_bytes = b"\n".join(l for l in lines) + b"\n"
data_string = "\n".join(l for l in lines) + "\n"
except Exception as e:
raise Log.error("can not make request body from\n{{lines|indent}}", lines=lines, cause=e)
@ -314,7 +313,7 @@ class Index(Features):
response = self.cluster.post(
self.path + "/_bulk",
data=data_bytes,
data=data_string,
headers={"Content-Type": "application/x-ndjson"},
timeout=self.settings.timeout,
retry=self.settings.retry,
@ -342,7 +341,7 @@ class Index(Features):
status=items[i].index.status,
error=items[i].index.error,
some=len(fails) - 1,
line=strings.limit(lines[i * 2 + 1].decode('utf8'), 500 if not self.debug else 100000),
line=strings.limit(lines[i * 2 + 1], 500 if not self.debug else 100000),
index=self.settings.index,
tjson=self.settings.tjson,
id=items[i].index._id
@ -356,7 +355,7 @@ class Index(Features):
status=items[i].index.status,
error=items[i].index.error,
some=len(fails) - 1,
line=strings.limit(lines[i * 2 + 1].decode('utf8'), 500 if not self.debug else 100000),
line=strings.limit(lines[i * 2 + 1], 500 if not self.debug else 100000),
index=self.settings.index,
tjson=self.settings.tjson,
id=items[i].index._id
@ -364,9 +363,10 @@ class Index(Features):
Log.error("Problems with insert", cause=cause)
except Exception as e:
e = Except.wrap(e)
if e.message.startswith("sequence item "):
Log.error("problem with {{data}}", data=text_type(repr(lines[int(e.message[14:16].strip())])), cause=e)
Log.error("problem sending to ES", e)
Log.error("problem sending to ES", cause=e)
# RECORDS MUST HAVE id AND json AS A STRING OR
# HAVE id AND value AS AN OBJECT
@ -398,7 +398,7 @@ class Index(Features):
**kwargs
)
result = mo_json.json2value(utf82unicode(response.all_content))
result = json2value(utf82unicode(response.all_content))
if not result.ok:
Log.error("Can not set refresh interval ({{error}})", {
"error": utf82unicode(response.all_content)
@ -406,7 +406,7 @@ class Index(Features):
elif self.cluster.version.startswith(("1.4.", "1.5.", "1.6.", "1.7.", "5.", "6.")):
result = self.cluster.put(
"/" + self.settings.index + "/_settings",
data=convert.unicode2utf8('{"index":{"refresh_interval":' + value2json(interval) + '}}'),
data=unicode2utf8('{"index":{"refresh_interval":' + value2json(interval) + '}}'),
**kwargs
)
@ -529,7 +529,7 @@ class Cluster(object):
schema=None,
limit_replicas=None,
read_only=False,
tjson=False,
tjson=None,
kwargs=None
):
best = self._get_best(kwargs)
@ -546,17 +546,18 @@ class Cluster(object):
index = kwargs.index
meta = self.get_metadata()
columns = parse_properties(index, ".", meta.indices[index].mappings.values()[0].properties)
tjson = kwargs.tjson
if len(columns) != 0:
kwargs.tjson = tjson or any(
c.names["."].startswith(TYPE_PREFIX) or
c.names["."].find("." + TYPE_PREFIX) != -1
for c in columns
)
if not kwargs.tjson:
if tjson is None and not kwargs.tjson:
Log.warning("Not typed index, columns are:\n{{columns|json}}", columns=columns)
return Index(kwargs)
return Index(kwargs=kwargs, cluster=self)
def _get_best(self, settings):
aliases = self.get_aliases()
@ -585,7 +586,7 @@ class Cluster(object):
kwargs.index = match.index
else:
Log.error("Can not find index {{index_name}}", index_name=kwargs.index)
return Index(kwargs)
return Index(kwargs=kwargs, cluster=self)
else:
# GET BEST MATCH, INCLUDING PROTOTYPE
best = self._get_best(kwargs)
@ -603,7 +604,7 @@ class Cluster(object):
metadata = self.get_metadata()
metadata[kwargs.index]
return Index(kwargs)
return Index(kwargs=kwargs, cluster=self)
def get_alias(self, alias):
"""
@ -615,7 +616,7 @@ class Cluster(object):
settings = self.settings.copy()
settings.alias = alias
settings.index = alias
return Index(read_only=True, kwargs=settings)
return Index(read_only=True, kwargs=settings, cluster=self)
Log.error("Can not find any index with alias {{alias_name}}", alias_name= alias)
def get_prototype(self, alias):
@ -657,24 +658,33 @@ class Cluster(object):
Log.error("Expecting a schema")
elif isinstance(schema, text_type):
Log.error("Expecting a schema")
elif self.version.startswith(("5.", "6.")):
schema = mo_json.json2value(value2json(schema), leaves=True)
else:
schema = retro_schema(mo_json.json2value(value2json(schema), leaves=True))
for m in schema.mappings.values():
if tjson:
m.properties[EXISTS_TYPE] = {"type": "long", "store": True}
m.dynamic_templates = DEFAULT_DYNAMIC_TEMPLATES + m.dynamic_templates + [{
"default_all": {
"mapping": {"store": True},
"match": "*"
}
}]
m.dynamic_templates = (
DEFAULT_DYNAMIC_TEMPLATES +
m.dynamic_templates #+
# [{
# "default_all": {
# "mapping": {"store": True},
# "match": "*"
# }
# }]
)
if self.version.startswith("5."):
schema.settings.index.max_inner_result_window = None # NOT ACCEPTED BY ES5
schema = json2value(value2json(schema), leaves=True)
elif self.version.startswith("6."):
schema = json2value(value2json(schema), leaves=True)
else:
schema = retro_schema(json2value(value2json(schema), leaves=True))
if limit_replicas:
# DO NOT ASK FOR TOO MANY REPLICAS
health = self.get("/_cluster/health")
health = self.get("/_cluster/health", stream=False)
if schema.settings.index.number_of_replicas >= health.number_of_nodes:
if limit_replicas_warning:
Log.warning(
@ -687,13 +697,14 @@ class Cluster(object):
self.put(
"/" + index,
data=schema,
headers={"Content-Type": "application/json"}
headers={text_type("Content-Type"): text_type("application/json")},
stream=False
)
# CONFIRM INDEX EXISTS
while True:
try:
state = self.get("/_cluster/state", retry={"times": 5}, timeout=3)
state = self.get("/_cluster/state", retry={"times": 5}, timeout=3, stream=False)
if index in state.metadata.indices:
self._metadata = None
break
@ -703,7 +714,7 @@ class Cluster(object):
Till(seconds=1).wait()
Log.alert("Made new index {{index|quote}}", index=index)
es = Index(kwargs=kwargs)
es = Index(kwargs=kwargs, cluster=self)
return es
def delete_index(self, index_name):
@ -726,7 +737,7 @@ class Cluster(object):
response = http.delete(url)
if response.status_code != 200:
Log.error("Expecting a 200, got {{code}}", code=response.status_code)
details = mo_json.json2value(utf82unicode(response.content))
details = json2value(utf82unicode(response.content))
if self.debug:
Log.note("delete response {{response}}", response=details)
return response
@ -738,7 +749,7 @@ class Cluster(object):
RETURN LIST OF {"alias":a, "index":i} PAIRS
ALL INDEXES INCLUDED, EVEN IF NO ALIAS {"alias":Null}
"""
data = self.get("/_aliases", retry={"times": 5}, timeout=3)
data = self.get("/_aliases", retry={"times": 5}, timeout=3, stream=False)
output = []
for index, desc in data.items():
if not desc["aliases"]:
@ -753,7 +764,7 @@ class Cluster(object):
Log.error("Metadata exploration has been disabled")
if not self._metadata or force:
response = self.get("/_cluster/state", retry={"times": 3}, timeout=30)
response = self.get("/_cluster/state", retry={"times": 3}, timeout=30, stream=False)
with self.metadata_locker:
self._metadata = wrap(response.metadata)
# REPLICATE MAPPING OVER ALL ALIASES
@ -763,7 +774,7 @@ class Cluster(object):
for a in m.aliases:
if not indices[a]:
indices[a] = m
self.cluster_state = wrap(self.get("/"))
self.cluster_state = wrap(self.get("/", stream=False))
self.version = self.cluster_state.version.number
return self._metadata
@ -777,16 +788,18 @@ class Cluster(object):
heads["Accept-Encoding"] = "gzip,deflate"
heads["Content-Type"] = "application/json"
data = kwargs.get(b'data')
data = kwargs.get(DATA_KEY)
if data == None:
pass
elif isinstance(data, Mapping):
kwargs[b'data'] = data = convert.unicode2utf8(value2json(data))
elif not isinstance(kwargs[b"data"], str):
kwargs[DATA_KEY] = unicode2utf8(value2json(data))
elif isinstance(data, text_type):
kwargs[DATA_KEY] = unicode2utf8(data)
else:
Log.error("data must be utf8 encoded string")
if self.debug:
sample = kwargs.get(b'data', "")[:300]
sample = kwargs.get(DATA_KEY, b"")[:300]
Log.note("{{url}}:\n{{data|indent}}", url=url, data=sample)
if self.debug:
@ -796,7 +809,7 @@ class Cluster(object):
Log.error(response.reason.decode("latin1") + ": " + strings.limit(response.content.decode("latin1"), 100 if self.debug else 10000))
if self.debug:
Log.note("response: {{response}}", response=utf82unicode(response.content)[:130])
details = mo_json.json2value(utf82unicode(response.content))
details = json2value(utf82unicode(response.content))
if details.error:
Log.error(convert.quote2string(details.error))
if details._shards.failed > 0:
@ -811,11 +824,11 @@ class Cluster(object):
else:
suggestion = ""
if kwargs.get("data"):
if kwargs.get(DATA_KEY):
Log.error(
"Problem with call to {{url}}" + suggestion + "\n{{body|left(10000)}}",
url=url,
body=strings.limit(kwargs["data"].decode('utf8'), 100 if self.debug else 10000),
body=strings.limit(kwargs[DATA_KEY], 100 if self.debug else 10000),
cause=e
)
else:
@ -829,7 +842,7 @@ class Cluster(object):
Log.error(response.reason+": "+response.all_content)
if self.debug:
Log.note("response: {{response}}", response=strings.limit(utf82unicode(response.all_content), 130))
details = wrap(mo_json.json2value(utf82unicode(response.all_content)))
details = wrap(json2value(utf82unicode(response.all_content)))
if details.error:
Log.error(details.error)
return details
@ -846,7 +859,7 @@ class Cluster(object):
Log.error(response.reason + ": " + response.all_content)
if self.debug:
Log.note("response: {{response}}", response=strings.limit(utf82unicode(response.all_content), 130))
details = wrap(mo_json.json2value(utf82unicode(response.all_content)))
details = wrap(json2value(utf82unicode(response.all_content)))
if details.error:
Log.error(details.error)
return details
@ -862,7 +875,7 @@ class Cluster(object):
if self.debug:
Log.note("response: {{response}}", response=strings.limit(utf82unicode(response.all_content), 130))
if response.all_content:
details = wrap(mo_json.json2value(utf82unicode(response.all_content)))
details = wrap(json2value(utf82unicode(response.all_content)))
if details.error:
Log.error(details.error)
return details
@ -875,35 +888,37 @@ class Cluster(object):
url = self.settings.host + ":" + text_type(self.settings.port) + path
heads = wrap(kwargs).headers
heads[b"Accept-Encoding"] = b"gzip,deflate"
heads[b"Content-Type"] = b"application/json"
heads[text_type("Accept-Encoding")] = text_type("gzip,deflate")
heads[text_type("Content-Type")] = text_type("application/json")
data = kwargs.get(b'data')
data = kwargs.get(DATA_KEY)
if data == None:
pass
elif isinstance(data, Mapping):
kwargs[b'data'] = data = convert.unicode2utf8(convert.value2json(data))
elif not isinstance(kwargs["data"], str):
kwargs[DATA_KEY] = unicode2utf8(convert.value2json(data))
elif isinstance(kwargs[DATA_KEY], text_type):
pass
else:
Log.error("data must be utf8 encoded string")
if self.debug:
sample = kwargs.get(b'data', "")[:1000]
sample = kwargs.get(DATA_KEY, "")[:1000]
Log.note("{{url}}:\n{{data|indent}}", url=url, data=sample)
try:
response = http.put(url, **kwargs)
if response.status_code not in [200]:
Log.error(response.reason+": "+response.all_content)
Log.error(response.reason + ": " + utf82unicode(response.all_content))
if self.debug:
Log.note("response: {{response}}", response= utf82unicode(response.all_content)[0:300:])
Log.note("response: {{response}}", response=utf82unicode(response.all_content)[0:300:])
details = mo_json.json2value(utf82unicode(response.content))
details = json2value(utf82unicode(response.content))
if details.error:
Log.error(convert.quote2string(details.error))
if details._shards.failed > 0:
Log.error("Shard failures {{failures|indent}}",
failures="---\n".join(r.replace(";", ";\n") for r in details._shards.failures.reason)
)
Log.error(
"Shard failures {{failures|indent}}",
failures="---\n".join(r.replace(";", ";\n") for r in details._shards.failures.reason)
)
return details
except Exception as e:
Log.error("Problem with call to {{url}}", url=url, cause=e)
@ -1280,23 +1295,10 @@ def retro_schema(schema):
output = wrap({
"mappings":{
typename: {
"dynamic_templates": (
[
retro_dynamic_template(*(t.items()[0])) for t in details.dynamic_templates
] + [
{
"default_strings": {
"mapping": {
"index": "not_analyzed",
"type": "keyword",
"store": True
},
"match_mapping_type": "string",
"match": "*"
}
}
]
),
"dynamic_templates": [
retro_dynamic_template(*(t.items()[0]))
for t in details.dynamic_templates
],
"properties": retro_properties(details.properties)
}
for typename, details in schema.mappings.items()
@ -1314,6 +1316,9 @@ def retro_dynamic_template(name, template):
elif template.mapping.type == "text":
template.mapping.type = "string"
template.mapping.index = "analyzed"
elif template.mapping.type == "string":
template.mapping.type = "string"
template.mapping.index = "analyzed"
return {name: template}
@ -1333,37 +1338,49 @@ def retro_properties(properties):
if v.properties:
v.properties = retro_properties(v.properties)
if v.fields:
v.fields = retro_properties(v.fields)
v.fields[k] = {
"type": v.type,
"index": v.index,
"doc_values": v.doc_values,
"analyzer": v.analyzer
}
v.type = "multi_field"
v.index = None
v.doc_values = None
v.analyzer = None
output[k] = v
return output
DEFAULT_DYNAMIC_TEMPLATES = wrap([
{
"default_boolean": {
"default_typed_boolean": {
"mapping": {"type": "boolean", "store": True},
"match": BOOLEAN_TYPE
}
},
{
"default_number": {
"default_typed_number": {
"mapping": {"type": "double", "store": True},
"match": NUMBER_TYPE
}
},
{
"default_string": {
"default_typed_string": {
"mapping": {"type": "keyword", "store": True},
"match": STRING_TYPE
}
},
{
"default_exist": {
"default_typed_exist": {
"mapping": {"type": "long", "store": True},
"match": EXISTS_TYPE
}
},
{
"default_nested": {
"default_typed_nested": {
"mapping": {"type": "nested", "store": True},
"match": NESTED_TYPE
}

67
vendor/pyLibrary/env/flask_wrappers.py поставляемый Normal file
Просмотреть файл

@ -0,0 +1,67 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import flask
from mo_dots import coalesce
from mo_future import binary_type
from pyLibrary.env.big_data import ibytes2icompressed
TOO_SMALL_TO_COMPRESS = 510 # DO NOT COMPRESS DATA WITH LESS THAN THIS NUMBER OF BYTES
def gzip_wrapper(func, compress_lower_limit=None):
compress_lower_limit = coalesce(compress_lower_limit, TOO_SMALL_TO_COMPRESS)
def output(*args, **kwargs):
response = func(*args, **kwargs)
accept_encoding = flask.request.headers.get('Accept-Encoding', '')
if 'gzip' not in accept_encoding.lower():
return response
resp = response.data
if isinstance(resp, binary_type) and len(resp) > compress_lower_limit:
response.headers['Content-Encoding'] = 'gzip'
response.set_data(b''.join(ibytes2icompressed([resp])))
return response
return output
def cors_wrapper(func):
"""
Decorator for CORS
:param func: Flask method that handles requests and returns a response
:return: Same, but with permissive CORS headers set
"""
def _setdefault(obj, key, value):
if value == None:
return
obj.setdefault(key, value)
def output(*args, **kwargs):
response = func(*args, **kwargs)
headers = response.headers
_setdefault(headers, "Access-Control-Allow-Origin", "*")
_setdefault(headers, "Access-Control-Allow-Headers", flask.request.headers.get("Access-Control-Request-Headers"))
_setdefault(headers, "Access-Control-Allow-Methods", flask.request.headers.get("Access-Control-Request-Methods"))
_setdefault(headers, "Content-Type", "application/json")
_setdefault(headers, "Strict-Transport-Security", "max-age=31536000; includeSubDomains; preload")
return response
output.provide_automatic_options = False
output.__name__ = func.__name__
return output

234
vendor/pyLibrary/env/http.py поставляемый
Просмотреть файл

@ -18,27 +18,27 @@
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from contextlib import closing
from copy import copy
from mmap import mmap
from numbers import Number
from tempfile import TemporaryFile
from mo_future import text_type
from requests import sessions, Response
from jx_python import jx
from mo_dots import Data, coalesce, wrap, set_default, unwrap
from mo_json import value2json
from mo_dots import Data, coalesce, wrap, set_default, unwrap, Null
from mo_future import text_type, PY2
from mo_json import value2json, json2value
from mo_logs import Log
from mo_logs.strings import utf82unicode, unicode2utf8
from mo_logs.exceptions import Except
from mo_math import Math
from mo_threads import Lock
from mo_threads import Till
from pyLibrary import convert
from requests import sessions, Response
import mo_json
from mo_logs.exceptions import Except
from mo_times.durations import Duration
from pyLibrary import convert
from pyLibrary.env.big_data import safe_size, ibytes2ilines, icompressed2ibytes
DEBUG = False
@ -50,6 +50,8 @@ default_timeout = 600
_warning_sent = False
request_count = 0
def request(method, url, zip=None, retry=None, **kwargs):
"""
@ -69,14 +71,16 @@ def request(method, url, zip=None, retry=None, **kwargs):
INCLUDES url AND headers
"""
global _warning_sent
global request_count
if not default_headers and not _warning_sent:
_warning_sent = True
Log.warning(
"The pyLibrary.env.http module was meant to add extra "
"default headers to all requests, specifically the 'Referer' "
"header with a URL to the project. Use the `pyLibrary.debug.constants.set()` "
Log.warning(text_type(
"The pyLibrary.env.http module was meant to add extra " +
"default headers to all requests, specifically the 'Referer' " +
"header with a URL to the project. Use the `pyLibrary.debug.constants.set()` " +
"function to set `pyLibrary.env.http.default_headers`"
)
))
if isinstance(url, list):
# TRY MANY URLS
@ -91,90 +95,97 @@ def request(method, url, zip=None, retry=None, **kwargs):
except Exception as e:
e = Except.wrap(e)
failures.append(e)
Log.error("Tried {{num}} urls", num=len(url), cause=failures)
Log.error(u"Tried {{num}} urls", num=len(url), cause=failures)
if b"session" in kwargs:
session = kwargs[b"session"]
del kwargs[b"session"]
if 'session' in kwargs:
session = kwargs['session']
del kwargs['session']
sess = Null
else:
session = sessions.Session()
sess = session = sessions.Session()
session.headers.update(default_headers)
if zip is None:
zip = ZIP_REQUEST
with closing(sess):
if zip is None:
zip = ZIP_REQUEST
if isinstance(url, text_type):
# httplib.py WILL **FREAK OUT** IF IT SEES ANY UNICODE
url = url.encode("ascii")
if isinstance(url, text_type):
# httplib.py WILL **FREAK OUT** IF IT SEES ANY UNICODE
url = url.encode('ascii')
_to_ascii_dict(kwargs)
timeout = kwargs[b'timeout'] = coalesce(kwargs.get(b'timeout'), default_timeout)
_to_ascii_dict(kwargs)
timeout = kwargs['timeout'] = coalesce(kwargs.get('timeout'), default_timeout)
if retry == None:
retry = Data(times=1, sleep=0)
elif isinstance(retry, Number):
retry = Data(times=retry, sleep=1)
else:
retry = wrap(retry)
if isinstance(retry.sleep, Duration):
retry.sleep = retry.sleep.seconds
set_default(retry, {"times": 1, "sleep": 0})
if b'json' in kwargs:
kwargs[b'data'] = value2json(kwargs[b'json']).encode("utf8")
del kwargs[b'json']
try:
headers = kwargs[b"headers"] = unwrap(coalesce(kwargs.get(b'headers'), {}))
set_default(headers, {b"Accept-Encoding": b"compress, gzip"})
if zip and len(coalesce(kwargs.get(b"data"))) > 1000:
compressed = convert.bytes2zip(kwargs[b"data"])
headers[b'content-encoding'] = b'gzip'
kwargs[b"data"] = compressed
_to_ascii_dict(headers)
if retry == None:
retry = Data(times=1, sleep=0)
elif isinstance(retry, Number):
retry = Data(times=retry, sleep=1)
else:
_to_ascii_dict(headers)
except Exception as e:
Log.error("Request setup failure on {{url}}", url=url, cause=e)
retry = wrap(retry)
if isinstance(retry.sleep, Duration):
retry.sleep = retry.sleep.seconds
set_default(retry, {"times": 1, "sleep": 0})
errors = []
for r in range(retry.times):
if r:
Till(seconds=retry.sleep).wait()
if 'json' in kwargs:
kwargs['data'] = value2json(kwargs['json']).encode('utf8')
del kwargs['json']
try:
if DEBUG:
Log.note("http {{method}} to {{url}}", method=method, url=url)
return session.request(method=method, url=url, **kwargs)
except Exception as e:
errors.append(Except.wrap(e))
headers = kwargs['headers'] = unwrap(coalesce(kwargs.get('headers'), {}))
set_default(headers, {'Accept-Encoding': 'compress, gzip'})
if " Read timed out." in errors[0]:
Log.error("Tried {{times}} times: Timeout failure (timeout was {{timeout}}", timeout=timeout, times=retry.times, cause=errors[0])
else:
Log.error("Tried {{times}} times: Request failure of {{url}}", url=url, times=retry.times, cause=errors[0])
if zip and len(coalesce(kwargs.get('data'))) > 1000:
compressed = convert.bytes2zip(kwargs['data'])
headers['content-encoding'] = 'gzip'
kwargs['data'] = compressed
def _to_ascii_dict(headers):
if headers is None:
return
for k, v in copy(headers).items():
if isinstance(k, text_type):
del headers[k]
if isinstance(v, text_type):
headers[k.encode("ascii")] = v.encode("ascii")
_to_ascii_dict(headers)
else:
headers[k.encode("ascii")] = v
elif isinstance(v, text_type):
headers[k] = v.encode("ascii")
_to_ascii_dict(headers)
except Exception as e:
Log.error(u"Request setup failure on {{url}}", url=url, cause=e)
errors = []
for r in range(retry.times):
if r:
Till(seconds=retry.sleep).wait()
try:
if DEBUG:
Log.note(u"http {{method}} to {{url}}", method=method, url=url)
request_count += 1
return session.request(method=method, url=url, **kwargs)
except Exception as e:
errors.append(Except.wrap(e))
if " Read timed out." in errors[0]:
Log.error(u"Tried {{times}} times: Timeout failure (timeout was {{timeout}}", timeout=timeout, times=retry.times, cause=errors[0])
else:
Log.error(u"Tried {{times}} times: Request failure of {{url}}", url=url, times=retry.times, cause=errors[0])
if PY2:
def _to_ascii_dict(headers):
if headers is None:
return
for k, v in copy(headers).items():
if isinstance(k, text_type):
del headers[k]
if isinstance(v, text_type):
headers[k.encode('ascii')] = v.encode('ascii')
else:
headers[k.encode('ascii')] = v
elif isinstance(v, text_type):
headers[k] = v.encode('ascii')
else:
def _to_ascii_dict(headers):
pass
def get(url, **kwargs):
kwargs.setdefault(b'allow_redirects', True)
kwargs[b"stream"] = True
return HttpResponse(request(b'get', url, **kwargs))
kwargs.setdefault('allow_redirects', True)
kwargs.setdefault('stream', True)
return HttpResponse(request('get', url, **kwargs))
def get_json(url, **kwargs):
@ -182,66 +193,73 @@ def get_json(url, **kwargs):
ASSUME RESPONSE IN IN JSON
"""
response = get(url, **kwargs)
c = response.all_content
return mo_json.json2value(convert.utf82unicode(c))
try:
c = response.all_content
return json2value(utf82unicode(c))
except Exception as e:
if Math.round(response.status_code, decimal=-2) in [400, 500]:
Log.error(u"Bad GET response: {{code}}", code=response.status_code)
else:
Log.error(u"Good GET requests, but bad JSON", cause=e)
def options(url, **kwargs):
kwargs.setdefault(b'allow_redirects', True)
kwargs[b"stream"] = True
return HttpResponse(request(b'options', url, **kwargs))
kwargs.setdefault('allow_redirects', True)
kwargs.setdefault('stream', True)
return HttpResponse(request('options', url, **kwargs))
def head(url, **kwargs):
kwargs.setdefault(b'allow_redirects', False)
kwargs[b"stream"] = True
return HttpResponse(request(b'head', url, **kwargs))
kwargs.setdefault('allow_redirects', False)
kwargs.setdefault('stream', True)
return HttpResponse(request('head', url, **kwargs))
def post(url, **kwargs):
kwargs[b"stream"] = True
return HttpResponse(request(b'post', url, **kwargs))
kwargs.setdefault('stream', True)
return HttpResponse(request('post', url, **kwargs))
def delete(url, **kwargs):
return HttpResponse(request(b'delete', url, **kwargs))
return HttpResponse(request('delete', url, **kwargs))
def post_json(url, **kwargs):
"""
ASSUME RESPONSE IN IN JSON
"""
if b"json" in kwargs:
kwargs[b"data"] = convert.unicode2utf8(value2json(kwargs[b"json"]))
elif b'data' in kwargs:
kwargs[b"data"] = convert.unicode2utf8(value2json(kwargs[b"data"]))
if 'json' in kwargs:
kwargs['data'] = unicode2utf8(value2json(kwargs['json']))
elif 'data' in kwargs:
kwargs['data'] = unicode2utf8(value2json(kwargs['data']))
else:
Log.error("Expecting `json` parameter")
Log.error(u"Expecting `json` parameter")
response = post(url, **kwargs)
c = response.content
try:
details = mo_json.json2value(convert.utf82unicode(c))
details = json2value(utf82unicode(c))
except Exception as e:
Log.error("Unexpected return value {{content}}", content=c, cause=e)
Log.error(u"Unexpected return value {{content}}", content=c, cause=e)
if response.status_code not in [200, 201]:
Log.error("Bad response", cause=Except.wrap(details))
Log.error(u"Bad response", cause=Except.wrap(details))
return details
def put(url, **kwargs):
return HttpResponse(request(b'put', url, **kwargs))
return HttpResponse(request('put', url, **kwargs))
def patch(url, **kwargs):
kwargs[b"stream"] = True
return HttpResponse(request(b'patch', url, **kwargs))
kwargs.setdefault('stream', True)
return HttpResponse(request('patch', url, **kwargs))
def delete(url, **kwargs):
kwargs[b"stream"] = True
return HttpResponse(request(b'delete', url, **kwargs))
kwargs.setdefault('stream', False)
return HttpResponse(request('delete', url, **kwargs))
class HttpResponse(Response):
@ -278,7 +296,7 @@ class HttpResponse(Response):
def all_lines(self):
return self.get_all_lines()
def get_all_lines(self, encoding="utf8", flexible=False):
def get_all_lines(self, encoding='utf8', flexible=False):
try:
iterator = self.raw.stream(4096, decode_content=False)
@ -286,12 +304,12 @@ class HttpResponse(Response):
return ibytes2ilines(icompressed2ibytes(iterator), encoding=encoding, flexible=flexible)
elif self.headers.get('content-type') == 'application/zip':
return ibytes2ilines(icompressed2ibytes(iterator), encoding=encoding, flexible=flexible)
elif self.url.endswith(".gz"):
elif self.url.endswith('.gz'):
return ibytes2ilines(icompressed2ibytes(iterator), encoding=encoding, flexible=flexible)
else:
return ibytes2ilines(iterator, encoding=encoding, flexible=flexible, closer=self.close)
except Exception as e:
Log.error("Can not read content", cause=e)
Log.error(u"Can not read content", cause=e)
class Generator_usingStream(object):

2
vendor/pyLibrary/env/pulse.py поставляемый
Просмотреть файл

@ -218,6 +218,6 @@ class ModifiedGenericConsumer(GenericConsumer):
Log.warning("timeout! Restarting {{name}} pulse consumer.", name=self.exchange, cause=e)
try:
self.disconnect()
except Exception, f:
except Exception as f:
Log.warning("Problem with disconnect()", cause=f)
break

33
vendor/pyLibrary/env/rollover_index.py поставляемый
Просмотреть файл

@ -11,16 +11,16 @@ from __future__ import unicode_literals
from activedata_etl import etl2path
from activedata_etl import key2etl
from jx_python import jx
from jx_python.containers.list_usingPythonList import ListContainer
from mo_dots import coalesce, wrap, Null
from mo_hg.hg_mozilla_org import minimize_repo
from mo_json import json2value, value2json, CAN_NOT_DECODE_JSON
from mo_kwargs import override
from mo_logs import Log, strings
from mo_threads import Lock
from mo_hg.hg_mozilla_org import minimize_repo
from mo_logs import Log
from mo_logs.exceptions import suppress_exception
from mo_math.randoms import Random
from mo_testing.fuzzytestcase import assertAlmostEqual
from mo_threads import Lock
from mo_times.dates import Date, unicode2Date, unix2Date
from mo_times.durations import Duration
from mo_times.timer import Timer
@ -37,16 +37,19 @@ class RolloverIndex(object):
AND THREADED QUEUE AND SPLIT DATA BY
"""
@override
def __init__(self, rollover_field, rollover_interval, rollover_max, queue_size=10000, batch_size=5000, kwargs=None):
"""
:param rollover_field: the FIELD with a timestamp to use for determining which index to push to
:param rollover_interval: duration between roll-over to new index
:param rollover_max: remove old indexes, do not add old records
:param queue_size: number of documents to queue in memory
:param batch_size: number of documents to push at once
:param kwargs: plus additional ES settings
:return:
"""
def __init__(
self,
rollover_field, # the FIELD with a timestamp to use for determining which index to push to
rollover_interval, # duration between roll-over to new index
rollover_max, # remove old indexes, do not add old records
queue_size=10000, # number of documents to queue in memory
batch_size=5000, # number of documents to push at once
tjson=None, # indicate if we are expected typed json
kwargs=None # plus additional ES settings
):
if tjson == None:
Log.error("not expected")
self.settings = kwargs
self.locker = Lock("lock for rollover_index")
self.rollover_field = jx.get(rollover_field)
@ -74,7 +77,7 @@ class RolloverIndex(object):
queue = self.known_queues.get(rounded_timestamp.unix)
if queue == None:
candidates = jx.run({
"from": self.cluster.get_aliases(),
"from": ListContainer('.', self.cluster.get_aliases()),
"where": {"regex": {"index": self.settings.index + "\d\d\d\d\d\d\d\d_\d\d\d\d\d\d"}},
"sort": "index"
})

77
vendor/pyLibrary/env/typed_inserter.py поставляемый
Просмотреть файл

@ -17,7 +17,8 @@ from collections import Mapping
from datetime import datetime, date, timedelta
from decimal import Decimal
from future.utils import text_type, binary_type
from jx_python.expressions import jx_expression_to_function
from mo_future import text_type, binary_type
from jx_python.meta import Column
from jx_base import python_type_to_json_type, INTEGER, NUMBER, EXISTS, NESTED, STRING, BOOLEAN, STRUCT, OBJECT
@ -30,7 +31,7 @@ from mo_logs import Log
from mo_logs.strings import utf82unicode, quote
from mo_times.dates import Date
from mo_times.durations import Duration
from pyLibrary.env.elasticsearch import parse_properties, random_id
from pyLibrary.env.elasticsearch import parse_properties, random_id, es_type_to_json_type
append = UnicodeBuilder.append
@ -52,15 +53,16 @@ json_type_to_inserter_type = {
class TypedInserter(object):
def __init__(self, es=None, id_column="_id"):
def __init__(self, es=None, id_expression="_id"):
self.es = es
self.id_column = id_column
self.remove_id = True if id_column == "_id" else False
self.id_column = id_expression
self.get_id = jx_expression_to_function(id_expression)
self.remove_id = True if id_expression == "_id" else False
if es:
_schema = Data()
for c in parse_properties(es.settings.alias, ".", es.get_properties()):
if c.type != OBJECT:
if c.type not in (OBJECT, NESTED):
_schema[c.names["."]] = c
self.schema = unwrap(_schema)
else:
@ -85,9 +87,9 @@ class TypedInserter(object):
net_new_properties = []
path = []
if isinstance(value, Mapping):
given_id = value.get(self.id_column)
given_id = self.get_id(value)
if self.remove_id:
value[self.id_column] = None
value['_id'] = None
else:
given_id = None
@ -125,6 +127,21 @@ class TypedInserter(object):
def _typed_encode(self, value, sub_schema, path, net_new_properties, _buffer):
try:
if isinstance(sub_schema, Column):
value_json_type = python_type_to_json_type[value.__class__]
column_json_type = es_type_to_json_type[sub_schema.type]
if value_json_type == column_json_type:
pass # ok
elif value_json_type == NESTED and all(python_type_to_json_type[v.__class__] == column_json_type for v in value if v != None):
pass # empty arrays can be anything
else:
from mo_logs import Log
Log.error("Can not store {{value}} in {{column|quote}}", value=value, column=sub_schema.names['.'])
sub_schema = {json_type_to_inserter_type[value_json_type]: sub_schema}
if value is None:
append(_buffer, '{}')
return
@ -181,24 +198,14 @@ class TypedInserter(object):
append(_buffer, ESCAPE_DCT.get(c, c))
append(_buffer, '"}')
elif _type is text_type:
if isinstance(sub_schema, Column):
# WE WILL NOT COMPLAIN IF ELASTICSEARCH HAS A PROPERTY FOR THIS ALREADY
if sub_schema.type not in ["keyword", "text", "string"]:
from mo_logs import Log
Log.warning("this is going to fail!")
append(_buffer, '"')
for c in value:
append(_buffer, ESCAPE_DCT.get(c, c))
append(_buffer, '"')
else:
if STRING_TYPE not in sub_schema:
sub_schema[STRING_TYPE] = True
net_new_properties.append(path + [STRING_TYPE])
if STRING_TYPE not in sub_schema:
sub_schema[STRING_TYPE] = True
net_new_properties.append(path + [STRING_TYPE])
append(_buffer, '{'+QUOTED_STRING_TYPE+COLON+'"')
for c in value:
append(_buffer, ESCAPE_DCT.get(c, c))
append(_buffer, '"}')
append(_buffer, '{'+QUOTED_STRING_TYPE+COLON+'"')
for c in value:
append(_buffer, ESCAPE_DCT.get(c, c))
append(_buffer, '"}')
elif _type in (int, long, Decimal):
if NUMBER_TYPE not in sub_schema:
sub_schema[NUMBER_TYPE] = True
@ -226,17 +233,21 @@ class TypedInserter(object):
append(_buffer, '}')
else:
# ALLOW PRIMITIVE MULTIVALUES
value = [v for v in value if v != None]
types = list(set(python_type_to_json_type[v.__class__] for v in value))
if len(types) > 1:
if len(types) == 0: # HANDLE LISTS WITH Nones IN THEM
append(_buffer, '{'+QUOTED_NESTED_TYPE+COLON+'[]}')
elif len(types) > 1:
from mo_logs import Log
Log.error("Can not handle multi-typed multivalues")
element_type = json_type_to_inserter_type[types[0]]
if element_type not in sub_schema:
sub_schema[element_type] = True
net_new_properties.append(path + [element_type])
append(_buffer, '{'+quote(element_type)+COLON)
self._multivalue2json(value, sub_schema[element_type], path+[element_type], net_new_properties, _buffer)
append(_buffer, '}')
else:
element_type = json_type_to_inserter_type[types[0]]
if element_type not in sub_schema:
sub_schema[element_type] = True
net_new_properties.append(path + [element_type])
append(_buffer, '{'+quote(element_type)+COLON)
self._multivalue2json(value, sub_schema[element_type], path + [element_type], net_new_properties, _buffer)
append(_buffer, '}')
elif _type is date:
if NUMBER_TYPE not in sub_schema:
sub_schema[NUMBER_TYPE] = True

154
vendor/pyLibrary/meta.py поставляемый
Просмотреть файл

@ -11,20 +11,15 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import mo_json
from mo_future import text_type, get_function_arguments
from mo_dots import set_default, wrap, _get_attr, Null, coalesce
from mo_json import value2json
from mo_logs import Log
from mo_threads import Lock
from pyLibrary import convert
from types import FunctionType
from jx_python.expressions import jx_expression
import mo_json
from mo_dots import set_default, _get_attr, Null
from mo_future import text_type, get_function_arguments
from mo_logs import Log
from mo_logs.exceptions import Except
from mo_logs.strings import expand_template
from mo_math.randoms import Random
from mo_threads import Lock
from mo_times.dates import Date
from mo_times.durations import DAY
@ -198,145 +193,6 @@ class _FakeLock():
pass
def DataClass(name, columns, constraint=True):
"""
Use the DataClass to define a class, but with some extra features:
1. restrict the datatype of property
2. restrict if `required`, or if `nulls` are allowed
3. generic constraints on object properties
It is expected that this class become a real class (or be removed) in the
long term because it is expensive to use and should only be good for
verifying program correctness, not user input.
:param name: Name of the class we are creating
:param columns: Each columns[i] has properties {
"name", - (required) name of the property
"required", - False if it must be defined (even if None)
"nulls", - True if property can be None, or missing
"default", - A default value, if none is provided
"type" - a Python datatype
}
:param constraint: a JSON query Expression for extra constraints
:return: The class that has been created
"""
columns = wrap([{"name": c, "required": True, "nulls": False, "type": object} if isinstance(c, text_type) else c for c in columns])
slots = columns.name
required = wrap(filter(lambda c: c.required and not c.nulls and not c.default, columns)).name
nulls = wrap(filter(lambda c: c.nulls, columns)).name
defaults = {c.name: coalesce(c.default, None) for c in columns}
types = {c.name: coalesce(c.type, object) for c in columns}
code = expand_template(
"""
from __future__ import unicode_literals
from collections import Mapping
meta = None
types_ = {{types}}
defaults_ = {{defaults}}
class {{class_name}}(Mapping):
__slots__ = {{slots}}
def _constraint(row, rownum, rows):
return {{constraint_expr}}
def __init__(self, **kwargs):
if not kwargs:
return
for s in {{slots}}:
object.__setattr__(self, s, kwargs.get(s, {{defaults}}.get(s, None)))
missed = {{required}}-set(kwargs.keys())
if missed:
Log.error("Expecting properties {"+"{missed}}", missed=missed)
illegal = set(kwargs.keys())-set({{slots}})
if illegal:
Log.error("{"+"{names}} are not a valid properties", names=illegal)
if not self._constraint(0, [self]):
Log.error("constraint not satisfied {"+"{expect}}\\n{"+"{value|indent}}", expect={{constraint}}, value=self)
def __getitem__(self, item):
return getattr(self, item)
def __setitem__(self, item, value):
setattr(self, item, value)
return self
def __setattr__(self, item, value):
if item not in {{slots}}:
Log.error("{"+"{item|quote}} not valid attribute", item=item)
object.__setattr__(self, item, value)
if not self._constraint(0, [self]):
Log.error("constraint not satisfied {"+"{expect}}\\n{"+"{value|indent}}", expect={{constraint}}, value=self)
def __getattr__(self, item):
Log.error("{"+"{item|quote}} not valid attribute", item=item)
def __hash__(self):
return object.__hash__(self)
def __eq__(self, other):
if isinstance(other, {{class_name}}) and dict(self)==dict(other) and self is not other:
Log.error("expecting to be same object")
return self is other
def __dict__(self):
return {k: getattr(self, k) for k in {{slots}}}
def items(self):
return ((k, getattr(self, k)) for k in {{slots}})
def __copy__(self):
_set = object.__setattr__
output = object.__new__({{class_name}})
{{assign}}
return output
def __iter__(self):
return {{slots}}.__iter__()
def __len__(self):
return {{len_slots}}
def __str__(self):
return str({{dict}})
""",
{
"class_name": name,
"slots": "(" + (", ".join(convert.value2quote(s) for s in slots)) + ")",
"required": "{" + (", ".join(convert.value2quote(s) for s in required)) + "}",
"nulls": "{" + (", ".join(convert.value2quote(s) for s in nulls)) + "}",
"defaults": jx_expression({"literal": defaults}).to_python(),
"len_slots": len(slots),
"dict": "{" + (", ".join(convert.value2quote(s) + ": self." + s for s in slots)) + "}",
"assign": "; ".join("_set(output, "+convert.value2quote(s)+", self."+s+")" for s in slots),
"types": "{" + (",".join(convert.string2quote(k) + ": " + v.__name__ for k, v in types.items())) + "}",
"constraint_expr": jx_expression(constraint).to_python(),
"constraint": value2json(constraint)
}
)
return _exec(code, name)
def _exec(code, name):
try:
globs = globals()
fake_locals = {}
exec(code, globs, fake_locals)
temp = globs[name] = fake_locals[name]
return temp
except Exception as e:
Log.error("Can not make class\n{{code}}", code=code, cause=e)
def value2quote(value):
# RETURN PRETTY PYTHON CODE FOR THE SAME

69
vendor/pyLibrary/queries/jx_usingMySQL.py поставляемый
Просмотреть файл

@ -13,6 +13,10 @@ from __future__ import unicode_literals
from collections import Mapping
import mo_json
from jx_base import IS_NULL
from mo_future import text_type
from mo_logs import Log
from mo_logs.exceptions import suppress_exception
from mo_logs.strings import indent, expand_template
@ -22,7 +26,7 @@ from mo_dots.lists import FlatList
from pyLibrary import convert
from mo_collections.matrix import Matrix
from mo_kwargs import override
from pyLibrary.sql import SQL
from pyLibrary.sql import SQL, SQL_IS_NULL, SQL_AND, SQL_IS_NOT_NULL, SQL_ORDERBY, SQL_LIMIT, SQL_COMMA, sql_iso, sql_list, SQL_TRUE, sql_alias
from pyLibrary.sql.mysql import int_list_packer
@ -77,7 +81,6 @@ class MySQL(object):
"where": self._where2sql(query.where)
})
def _subquery(self, query, isolate=True, stacked=False):
if isinstance(query, text_type):
return self.db.quote_column(query), None
@ -114,10 +117,10 @@ class MySQL(object):
if e.domain.type != "default":
Log.error("domain of type {{type}} not supported, yet", type=e.domain.type)
groups.append(e.value)
selects.append(e.value + " AS " + self.db.quote_column(e.name))
selects.append(sql_alias(e.value, self.db.quote_column(e.name)))
for s in select:
selects.append(aggregates[s.aggregate].replace("{{code}}", s.value) + " AS " + self.db.quote_column(s.name))
selects.append(sql_alias(aggregates[s.aggregate].replace("{{code}}", s.value), self.db.quote_column(s.name)))
sql = expand_template("""
SELECT
@ -181,7 +184,7 @@ class MySQL(object):
selects = FlatList()
for s in query.select:
selects.append(aggregates[s.aggregate].replace("{{code}}", s.value) + " AS " + self.db.quote_column(s.name))
selects.append(sql_alias(aggregates[s.aggregate].replace("{{code}}", s.value),self.db.quote_column(s.name)))
sql = expand_template("""
SELECT
@ -202,7 +205,7 @@ class MySQL(object):
if s0.aggregate not in aggregates:
Log.error("Expecting all columns to have an aggregate: {{select}}", select=s0)
select = aggregates[s0.aggregate].replace("{{code}}", s0.value) + " AS " + self.db.quote_column(s0.name)
select = sql_alias(aggregates[s0.aggregate].replace("{{code}}", s0.value) , self.db.quote_column(s0.name))
sql = expand_template("""
SELECT
@ -232,12 +235,12 @@ class MySQL(object):
for s in listwrap(query.select):
if isinstance(s.value, Mapping):
for k, v in s.value.items:
selects.append(v + " AS " + self.db.quote_column(s.name + "." + k))
selects.append(sql_alias(v, self.db.quote_column(s.name + "." + k)))
if isinstance(s.value, list):
for i, ss in enumerate(s.value):
selects.append(s.value + " AS " + self.db.quote_column(s.name + "," + str(i)))
selects.append(sql_alias(s.value, self.db.quote_column(s.name + "," + str(i))))
else:
selects.append(s.value + " AS " + self.db.quote_column(s.name))
selects.append(sql_alias(s.value, self.db.quote_column(s.name)))
sql = expand_template("""
SELECT
@ -282,7 +285,7 @@ class MySQL(object):
select = "*"
else:
name = query.select.name
select = query.select.value + " AS " + self.db.quote_column(name)
select = sql_alias(query.select.value, self.db.quote_column(name))
sql = expand_template("""
SELECT
@ -316,11 +319,10 @@ class MySQL(object):
"""
if not sort:
return ""
return SQL("ORDER BY " + ",\n".join([self.db.quote_column(o.field) + (" DESC" if o.sort == -1 else "") for o in sort]))
return SQL_ORDERBY + sql_list([self.db.quote_column(o.field) + (" DESC" if o.sort == -1 else "") for o in sort])
def _limit2sql(self, limit):
return SQL("" if not limit else "LIMIT " + str(limit))
return SQL("" if not limit else SQL_LIMIT + str(limit))
def _where2sql(self, where):
if where == None:
@ -336,10 +338,10 @@ def _isolate(separator, list):
return list[0]
except Exception as e:
Log.error("Programming problem: separator={{separator}}, list={{list}",
list=list,
separator=separator,
cause=e
)
list=list,
separator=separator,
cause=e
)
def esfilter2sqlwhere(db, esfilter):
@ -354,15 +356,15 @@ def _esfilter2sqlwhere(db, esfilter):
esfilter = wrap(esfilter)
if esfilter is True:
return "1=1"
return SQL_TRUE
elif esfilter["and"]:
return _isolate("AND", [esfilter2sqlwhere(db, a) for a in esfilter["and"]])
return _isolate(SQL_AND, [esfilter2sqlwhere(db, a) for a in esfilter["and"]])
elif esfilter["or"]:
return _isolate("OR", [esfilter2sqlwhere(db, a) for a in esfilter["or"]])
elif esfilter["not"]:
return "NOT (" + esfilter2sqlwhere(db, esfilter["not"]) + ")"
return "NOT " + sql_iso(esfilter2sqlwhere(db, esfilter["not"]))
elif esfilter.term:
return _isolate("AND", [db.quote_column(col) + SQL("=") + db.quote_value(val) for col, val in esfilter.term.items()])
return _isolate(SQL_AND, [db.quote_column(col) + SQL("=") + db.quote_value(val) for col, val in esfilter.term.items()])
elif esfilter.terms:
for col, v in esfilter.terms.items():
if len(v) == 0:
@ -386,9 +388,9 @@ def _esfilter2sqlwhere(db, esfilter):
return esfilter2sqlwhere(db, {"missing": col})
else:
return "false"
return db.quote_column(col) + SQL(" in (" + ",\n".join([db.quote_value(val) for val in v]) + ")")
return db.quote_column(col) + " in " + sql_iso(sql_list([db.quote_value(val) for val in v]))
elif esfilter.script:
return "(" + esfilter.script + ")"
return sql_iso(esfilter.script)
elif esfilter.range:
name2sign = {
"gt": SQL(">"),
@ -402,30 +404,30 @@ def _esfilter2sqlwhere(db, esfilter):
max = coalesce(r["lte"], r["<="])
if min != None and max != None:
# SPECIAL CASE (BETWEEN)
sql = db.quote_column(col) + SQL(" BETWEEN ") + db.quote_value(min) + SQL(" AND ") + db.quote_value(max)
sql = db.quote_column(col) + SQL(" BETWEEN ") + db.quote_value(min) + SQL_AND + db.quote_value(max)
else:
sql = SQL(" AND ").join(
sql = SQL_AND.join(
db.quote_column(col) + name2sign[sign] + db.quote_value(value)
for sign, value in r.items()
)
return sql
output = _isolate("AND", [single(col, ranges) for col, ranges in esfilter.range.items()])
output = _isolate(SQL_AND, [single(col, ranges) for col, ranges in esfilter.range.items()])
return output
elif esfilter.missing:
if isinstance(esfilter.missing, text_type):
return "(" + db.quote_column(esfilter.missing) + " IS Null)"
return sql_iso(db.quote_column(esfilter.missing) + SQL_IS_NULL)
else:
return "(" + db.quote_column(esfilter.missing.field) + " IS Null)"
return sql_iso(db.quote_column(esfilter.missing.field) + SQL_IS_NULL)
elif esfilter.exists:
if isinstance(esfilter.exists, text_type):
return "(" + db.quote_column(esfilter.exists) + " IS NOT Null)"
return sql_iso(db.quote_column(esfilter.exists) + SQL_IS_NOT_NULL)
else:
return "(" + db.quote_column(esfilter.exists.field) + " IS NOT Null)"
return sql_iso(db.quote_column(esfilter.exists.field) + SQL_IS_NOT_NULL)
elif esfilter.match_all:
return "1=1"
return SQL_TRUE
elif esfilter.instr:
return _isolate("AND", ["instr(" + db.quote_column(col) + ", " + db.quote_value(val) + ")>0" for col, val in esfilter.instr.items()])
return _isolate(SQL_AND, ["instr" + sql_iso(db.quote_column(col) + ", " + db.quote_value(val)) + ">0" for col, val in esfilter.instr.items()])
else:
Log.error("Can not convert esfilter to SQL: {{esfilter}}", esfilter=esfilter)
@ -440,7 +442,6 @@ def expand_json(rows):
r[k] = value
# MAP NAME TO SQL FUNCTION
aggregates = {
"one": "COUNT({{code}})",
@ -464,6 +465,6 @@ aggregates = {
"variance": "POWER(STDDEV({{code}}), 2)"
}
from jx_base.container import type2container
type2container["mysql"] = MySQL

68
vendor/pyLibrary/sql/__init__.py поставляемый
Просмотреть файл

@ -46,10 +46,11 @@ class SQL(text_type):
else:
return SQL(other.sql + self.sql)
def join(self, list):
if not all(isinstance(s, SQL) for s in list):
def join(self, list_):
list_ = list(list_)
if not all(isinstance(s, SQL) for s in list_):
Log.error("Can only join other SQL")
return SQL(self.sql.join(list))
return SQL(self.sql.join(list_))
if PY3:
def __bytes__(self):
@ -59,6 +60,42 @@ class SQL(text_type):
Log.error("do not do this")
SQL_STAR = SQL(" * ")
SQL_AND = SQL(" AND ")
SQL_OR = SQL(" OR ")
SQL_ON = SQL(" ON ")
SQL_CASE = SQL(" CASE ")
SQL_WHEN = SQL(" WHEN ")
SQL_THEN = SQL(" THEN ")
SQL_ELSE = SQL(" ELSE ")
SQL_END = SQL(" END ")
SQL_COMMA = SQL(", ")
SQL_UNION_ALL = SQL("\nUNION ALL\n")
SQL_UNION = SQL("\nUNION\n")
SQL_LEFT_JOIN = SQL("\nLEFT JOIN\n")
SQL_INNER_JOIN = SQL("\nJOIN\n")
SQL_EMPTY_STRING = SQL("''")
SQL_TRUE = SQL(" 1 ")
SQL_FALSE = SQL(" 0 ")
SQL_ONE = SQL(" 1 ")
SQL_ZERO = SQL(" 0 ")
SQL_NULL = SQL(" NULL ")
SQL_IS_NULL = SQL(" IS NULL ")
SQL_IS_NOT_NULL = SQL(" IS NOT NULL ")
SQL_SELECT = SQL("\nSELECT\n")
SQL_FROM = SQL("\nFROM\n")
SQL_WHERE = SQL("\nWHERE\n")
SQL_GROUPBY = SQL("\nGROUP BY\n")
SQL_ORDERBY = SQL("\nORDER BY\n")
SQL_DESC = SQL(" DESC ")
SQL_ASC = SQL(" ASC ")
SQL_LIMIT = SQL("\nLIMIT\n")
class DB(object):
def quote_column(self, column_name, table=None):
@ -67,3 +104,28 @@ class DB(object):
def db_type_to_json_type(self, type):
raise NotImplementedError()
def sql_list(list_):
list_ = list(list_)
if not all(isinstance(s, SQL) for s in list_):
Log.error("Can only join other SQL")
return SQL(", ".join(l.template for l in list_))
def sql_iso(sql):
return "("+sql+")"
def sql_count(sql):
return "COUNT(" + sql + ")"
def sql_concat(list_):
return SQL(" || ").join(sql_iso(l) for l in list_)
def sql_alias(value, alias):
return SQL(value.template + " AS " + alias.template)
def sql_coalesce(list_):
return "COALESCE(" + SQL_COMMA.join(list_) + ")"

147
vendor/pyLibrary/sql/mysql.py поставляемый
Просмотреть файл

@ -31,12 +31,12 @@ from mo_logs.strings import indent
from mo_logs.strings import outdent
from mo_math import Math
from mo_times import Date
from pyLibrary.sql import SQL
from pyLibrary.sql import SQL, SQL_NULL, SQL_SELECT, SQL_LIMIT, SQL_WHERE, SQL_LEFT_JOIN, SQL_COMMA, SQL_FROM, SQL_AND, sql_list, sql_iso, SQL_ASC, SQL_TRUE, SQL_ONE, SQL_DESC, SQL_IS_NULL, sql_alias
from pyLibrary.sql.sqlite import join_column
DEBUG = False
MAX_BATCH_SIZE = 100
EXECUTE_TIMEOUT = 5*600*1000 # in milliseconds
EXECUTE_TIMEOUT = 5 * 600 * 1000 # in milliseconds
all_db = []
@ -46,6 +46,7 @@ class MySQL(object):
Parameterize SQL by name rather than by position. Return records as objects
rather than tuples.
"""
@override
def __init__(
self,
@ -96,7 +97,7 @@ class MySQL(object):
user=coalesce(self.settings.username, self.settings.user),
passwd=coalesce(self.settings.password, self.settings.passwd),
db=coalesce(self.settings.schema, self.settings.db),
read_timeout=coalesce(self.settings.read_timeout, (EXECUTE_TIMEOUT/1000)-10),
read_timeout=coalesce(self.settings.read_timeout, (EXECUTE_TIMEOUT / 1000) - 10),
charset=u"utf8",
use_unicode=True,
ssl=coalesce(self.settings.ssl, None),
@ -105,16 +106,16 @@ class MySQL(object):
except Exception as e:
if self.settings.host.find("://") == -1:
Log.error(u"Failure to connect to {{host}}:{{port}}",
host= self.settings.host,
port= self.settings.port,
cause=e
)
host=self.settings.host,
port=self.settings.port,
cause=e
)
else:
Log.error(u"Failure to connect. PROTOCOL PREFIX IS PROBABLY BAD", e)
self.cursor = None
self.partial_rollback = False
self.transaction_level = 0
self.backlog = [] # accumulate the write commands so they are sent at once
self.backlog = [] # accumulate the write commands so they are sent at once
if self.readonly:
self.begin()
@ -146,7 +147,6 @@ class MySQL(object):
finally:
self.close()
def transaction(self):
"""
return not-started transaction (for with statement)
@ -160,7 +160,6 @@ class MySQL(object):
self.execute("SET TIME_ZONE='+00:00'")
self.execute("SET MAX_EXECUTION_TIME=" + text_type(EXECUTE_TIMEOUT))
def close(self):
if self.transaction_level > 0:
if self.readonly:
@ -217,7 +216,7 @@ class MySQL(object):
Log.error("Can not flush", e)
def rollback(self):
self.backlog = [] # YAY! FREE!
self.backlog = [] # YAY! FREE!
if self.transaction_level == 0:
Log.error("No transaction has begun")
elif self.transaction_level == 1:
@ -273,7 +272,7 @@ class MySQL(object):
except Exception as e:
if isinstance(e, InterfaceError) or e.message.find("InterfaceError") >= 0:
Log.error("Did you close the db connection?", e)
Log.error("Problem executing SQL:\n{{sql|indent}}", sql= sql, cause=e, stack_depth=1)
Log.error("Problem executing SQL:\n{{sql|indent}}", sql=sql, cause=e, stack_depth=1)
def column_query(self, sql, param=None):
"""
@ -282,7 +281,7 @@ class MySQL(object):
self._execute_backlog()
try:
old_cursor = self.cursor
if not old_cursor: # ALLOW NON-TRANSACTIONAL READS
if not old_cursor: # ALLOW NON-TRANSACTIONAL READS
self.cursor = self.db.cursor()
self.cursor.execute("SET TIME_ZONE='+00:00'")
self.cursor.close()
@ -299,7 +298,7 @@ class MySQL(object):
# columns = [utf8_to_unicode(d[0]) for d in coalesce(self.cursor.description, [])]
result = zip(*grid)
if not old_cursor: # CLEANUP AFTER NON-TRANSACTIONAL READS
if not old_cursor: # CLEANUP AFTER NON-TRANSACTIONAL READS
self.cursor.close()
self.cursor = None
@ -307,10 +306,7 @@ class MySQL(object):
except Exception as e:
if isinstance(e, InterfaceError) or e.message.find("InterfaceError") >= 0:
Log.error("Did you close the db connection?", e)
Log.error("Problem executing SQL:\n{{sql|indent}}", sql= sql, cause=e,stack_depth=1)
Log.error("Problem executing SQL:\n{{sql|indent}}", sql=sql, cause=e, stack_depth=1)
# EXECUTE GIVEN METHOD FOR ALL ROWS RETURNED
def forall(self, sql, param=None, _execute=None):
@ -320,14 +316,14 @@ class MySQL(object):
self._execute_backlog()
try:
old_cursor = self.cursor
if not old_cursor: # ALLOW NON-TRANSACTIONAL READS
if not old_cursor: # ALLOW NON-TRANSACTIONAL READS
self.cursor = self.db.cursor()
if param:
sql = expand_template(sql, self.quote_param(param))
sql = self.preamble + outdent(sql)
if self.debug:
Log.note("Execute SQL:\n{{sql}}", sql=indent(sql))
Log.note("Execute SQL:\n{{sql}}", sql=indent(sql))
self.cursor.execute(sql)
columns = tuple([utf8_to_unicode(d[0]) for d in self.cursor.description])
@ -335,16 +331,15 @@ class MySQL(object):
num += 1
_execute(wrap(dict(zip(columns, [utf8_to_unicode(c) for c in r]))))
if not old_cursor: # CLEANUP AFTER NON-TRANSACTIONAL READS
if not old_cursor: # CLEANUP AFTER NON-TRANSACTIONAL READS
self.cursor.close()
self.cursor = None
except Exception as e:
Log.error("Problem executing SQL:\n{{sql|indent}}", sql= sql, cause=e, stack_depth=1)
Log.error("Problem executing SQL:\n{{sql|indent}}", sql=sql, cause=e, stack_depth=1)
return num
def execute(self, sql, param=None):
if self.transaction_level == 0:
Log.error("Expecting transaction to be started before issuing queries")
@ -356,7 +351,6 @@ class MySQL(object):
if self.debug or len(self.backlog) >= MAX_BATCH_SIZE:
self._execute_backlog()
@staticmethod
@override
def execute_sql(
@ -404,10 +398,10 @@ class MySQL(object):
if len(sql) > 10000:
sql = "<" + text_type(len(sql)) + " bytes of sql>"
Log.error("Unable to execute sql: return code {{return_code}}, {{output}}:\n {{sql}}\n",
sql=indent(sql),
return_code=proc.returncode,
output=output
)
sql=indent(sql),
return_code=proc.returncode,
output=output
)
@staticmethod
@override
@ -441,10 +435,10 @@ class MySQL(object):
sql = self.preamble + b
try:
if self.debug:
Log.note("Execute SQL:\n{{sql|indent}}", sql= sql)
Log.note("Execute SQL:\n{{sql|indent}}", sql=sql)
self.cursor.execute(b)
except Exception as e:
Log.error("Can not execute sql:\n{{sql}}", sql= sql, cause=e)
Log.error("Can not execute sql:\n{{sql}}", sql=sql, cause=e)
self.cursor.close()
self.cursor = self.db.cursor()
@ -453,13 +447,12 @@ class MySQL(object):
sql = self.preamble + ";\n".join(g)
try:
if self.debug:
Log.note("Execute block of SQL:\n{{sql|indent}}", sql= sql)
Log.note("Execute block of SQL:\n{{sql|indent}}", sql=sql)
self.cursor.execute(sql)
self.cursor.close()
self.cursor = self.db.cursor()
except Exception as e:
Log.error("Problem executing SQL:\n{{sql|indent}}", sql= sql, cause=e, stack_depth=1)
Log.error("Problem executing SQL:\n{{sql|indent}}", sql=sql, cause=e, stack_depth=1)
## Insert dictionary of values into table
def insert(self, table_name, record):
@ -467,39 +460,48 @@ class MySQL(object):
try:
command = (
"INSERT INTO " + self.quote_column(table_name) + "(" +
SQL(",").join([self.quote_column(k) for k in keys]) +
") VALUES (" +
SQL(",").join([self.quote_value(record[k]) for k in keys]) +
")"
"INSERT INTO " + self.quote_column(table_name) +
sql_iso(sql_list([self.quote_column(k) for k in keys])) +
" VALUES " +
sql_iso(sql_list([self.quote_value(record[k]) for k in keys]))
)
self.execute(command)
except Exception as e:
Log.error("problem with record: {{record}}", record= record, cause=e)
Log.error("problem with record: {{record}}", record=record, cause=e)
# candidate_key IS LIST OF COLUMNS THAT CAN BE USED AS UID (USUALLY PRIMARY KEY)
# ONLY INSERT IF THE candidate_key DOES NOT EXIST YET
def insert_new(self, table_name, candidate_key, new_record):
candidate_key = listwrap(candidate_key)
condition = " AND\n".join([self.quote_column(k) + "=" + self.quote_value(new_record[k]) if new_record[k] != None else self.quote_column(k) + " IS Null" for k in candidate_key])
condition = SQL_AND.join([
self.quote_column(k) + "=" + self.quote_value(new_record[k])
if new_record[k] != None
else self.quote_column(k) + SQL_IS_NULL
for k in candidate_key
])
command = (
"INSERT INTO " + self.quote_column(table_name) + " (" +
",".join([self.quote_column(k) for k in new_record.keys()]) +
")\n" +
"SELECT a.* FROM (SELECT " + ",".join([self.quote_value(v) + " " + self.quote_column(k) for k, v in new_record.items()]) + " FROM DUAL) a\n" +
"LEFT JOIN " +
"(SELECT 'dummy' exist FROM " + self.quote_column(table_name) + " WHERE " + condition + " LIMIT 1) b ON 1=1 WHERE exist IS Null"
"INSERT INTO " + self.quote_column(table_name) + sql_iso(sql_list(
self.quote_column(k) for k in new_record.keys()
)) +
SQL_SELECT + "a.*" + SQL_FROM + sql_iso(
SQL_SELECT + sql_list([self.quote_value(v) + " " + self.quote_column(k) for k, v in new_record.items()]) +
SQL_FROM + "DUAL"
) + " a" +
SQL_LEFT_JOIN + sql_iso(
SQL_SELECT + "'dummy' exist " +
SQL_FROM + self.quote_column(table_name) +
SQL_WHERE + condition +
SQL_LIMIT + SQL_ONE
) + " b ON " + SQL_TRUE + SQL_WHERE + " exist " + SQL_IS_NULL
)
self.execute(command, {})
# ONLY INSERT IF THE candidate_key DOES NOT EXIST YET
def insert_newlist(self, table_name, candidate_key, new_records):
for r in new_records:
self.insert_new(table_name, candidate_key, r)
def insert_list(self, table_name, records):
if not records:
return
@ -511,17 +513,16 @@ class MySQL(object):
try:
command = (
"INSERT INTO " + self.quote_column(table_name) + "(" +
",".join([self.quote_column(k) for k in keys]) +
") VALUES " + ",\n".join([
"(" + ",".join([self.quote_value(r[k]) for k in keys]) + ")"
for r in records
])
"INSERT INTO " + self.quote_column(table_name) +
sql_iso(sql_list([self.quote_column(k) for k in keys])) +
" VALUES " + sql_list([
sql_iso(sql_list([self.quote_value(r[k]) for k in keys]))
for r in records
])
)
self.execute(command)
except Exception as e:
Log.error("problem with record: {{record}}", record= records, cause=e)
Log.error("problem with record: {{record}}", record=records, cause=e)
def update(self, table_name, where_slice, new_values):
"""
@ -531,21 +532,20 @@ class MySQL(object):
"""
new_values = self.quote_param(new_values)
where_clause = " AND\n".join([
self.quote_column(k) + "=" + self.quote_value(v) if v != None else self.quote_column(k) + " IS NULL"
where_clause = SQL_AND.join([
self.quote_column(k) + "=" + self.quote_value(v) if v != None else self.quote_column(k) + SQL_IS_NULL
for k, v in where_slice.items()
])
command = (
"UPDATE " + self.quote_column(table_name) + "\n" +
"SET " +
",\n".join([self.quote_column(k) + "=" + v for k, v in new_values.items()]) + "\n" +
"WHERE " +
sql_list([self.quote_column(k) + "=" + v for k, v in new_values.items()]) +
SQL_WHERE +
where_clause
)
self.execute(command, {})
def quote_param(self, param):
return {k: self.quote_value(v) for k, v in param.items()}
@ -556,7 +556,7 @@ class MySQL(object):
"""
try:
if value == None:
return SQL("NULL")
return SQL_NULL
elif isinstance(value, SQL):
if not value.param:
# value.template CAN BE MORE THAN A TEMPLATE STRING
@ -572,7 +572,7 @@ class MySQL(object):
elif isinstance(value, datetime):
return SQL("str_to_date('" + value.strftime("%Y%m%d%H%M%S.%f") + "', '%Y%m%d%H%i%s.%f')")
elif isinstance(value, Date):
return SQL("str_to_date('"+value.format("%Y%m%d%H%M%S.%f")+"', '%Y%m%d%H%i%s.%f')")
return SQL("str_to_date('" + value.format("%Y%m%d%H%M%S.%f") + "', '%Y%m%d%H%i%s.%f')")
elif hasattr(value, '__iter__'):
return SQL(self.db.literal(json_encode(value)))
else:
@ -580,7 +580,6 @@ class MySQL(object):
except Exception as e:
Log.error("problem quoting SQL", e)
def quote_sql(self, value, param=None):
"""
USED TO EXPAND THE PARAMETERS TO THE SQL() OBJECT
@ -596,30 +595,30 @@ class MySQL(object):
elif isinstance(value, Mapping):
return self.db.literal(json_encode(value))
elif hasattr(value, '__iter__'):
return "(" + ",".join([self.quote_sql(vv) for vv in value]) + ")"
return sql_iso(sql_list([self.quote_sql(vv) for vv in value]))
else:
return text_type(value)
except Exception as e:
Log.error("problem quoting SQL", e)
def quote_column(self, column_name, table=None):
if column_name==None:
if column_name == None:
Log.error("missing column_name")
elif isinstance(column_name, text_type):
if table:
column_name = table + "." + column_name
return SQL("`" + column_name.replace(".", "`.`") + "`") # MY SQL QUOTE OF COLUMN NAMES
column_name = join_column(table, column_name)
return SQL("`" + column_name.replace(".", "`.`") + "`") # MY SQL QUOTE OF COLUMN NAMES
elif isinstance(column_name, list):
if table:
return SQL(", ".join([self.quote_column(table + "." + c) for c in column_name]))
return SQL(", ".join([self.quote_column(c) for c in column_name]))
return sql_list(join_column(table, c) for c in column_name)
return sql_list(self.quote_column(c) for c in column_name)
else:
# ASSUME {"name":name, "value":value} FORM
return SQL(column_name.value + " AS " + self.quote_column(column_name.name))
return SQL(sql_alias(column_name.value, self.quote_column(column_name.name)))
def sort2sqlorderby(self, sort):
sort = jx.normalize_sort_parameters(sort)
return ",\n".join([self.quote_column(s.field) + (" DESC" if s.sort == -1 else " ASC") for s in sort])
return sql_list([self.quote_column(s.field) + (SQL_DESC if s.sort == -1 else SQL_ASC) for s in sort])
def utf8_to_unicode(v):
@ -632,8 +631,6 @@ def utf8_to_unicode(v):
Log.error("not expected", e)
def int_list_packer(term, values):
"""
return singletons, ranges and exclusions

29
vendor/pyLibrary/sql/redshift.py поставляемый
Просмотреть файл

@ -86,11 +86,13 @@ class Redshift(object):
keys = record.keys()
try:
command = "INSERT INTO " + self.quote_column(table_name) + "(" + \
",".join([self.quote_column(k) for k in keys]) + \
") VALUES (" + \
",".join([self.quote_value(record[k]) for k in keys]) + \
")"
command = (
"INSERT INTO " + self.quote_column(table_name) + "(" +
",".join([self.quote_column(k) for k in keys]) +
") VALUES (" +
",".join([self.quote_value(record[k]) for k in keys]) +
")"
)
self.execute(command)
except Exception as e:
@ -112,13 +114,14 @@ class Redshift(object):
{"ids": self.quote_column([r["_id"] for r in records])}
)
command = \
"INSERT INTO " + self.quote_column(table_name) + "(" + \
",".join([self.quote_column(k) for k in columns]) + \
command = (
"INSERT INTO " + self.quote_column(table_name) + "(" +
",".join([self.quote_column(k) for k in columns]) +
") VALUES " + ",\n".join([
"(" + ",".join([self.quote_value(r.get(k, None)) for k in columns]) + ")"
for r in records
])
sql_iso(",".join([self.quote_value(r.get(k, None)) for k in columns]))
for r in records
])
)
self.execute(command)
except Exception as e:
Log.error("problem with insert", e)
@ -137,11 +140,11 @@ class Redshift(object):
def quote_column(self, name):
if isinstance(name, text_type):
return SQL('"' + name.replace('"', '""') + '"')
return SQL("(" + (", ".join(self.quote_value(v) for v in name)) + ")")
return SQL(sql_iso((", ".join(self.quote_value(v) for v in name))))
def quote_value(self, value):
if value ==None:
return SQL("NULL")
return SQL_NULL
if isinstance(value, list):
json = value2json(value)
return self.quote_value(json)

245
vendor/pyLibrary/sql/sqlite.py поставляемый
Просмотреть файл

@ -14,11 +14,10 @@ from __future__ import unicode_literals
import os
import re
import sqlite3
import sys
from collections import Mapping
from mo_future import text_type
from mo_future import allocate_lock as _allocate_lock, text_type, zip_longest
from mo_dots import Data, coalesce
from mo_files import File
from mo_logs import Log
@ -26,33 +25,53 @@ from mo_logs.exceptions import Except, extract_stack, ERROR
from mo_logs.strings import quote
from mo_math.stats import percentile
from mo_threads import Queue, Signal, Thread
from mo_threads.signal import DONE
from mo_times import Date, Duration
from mo_times.timer import Timer
from pyLibrary import convert
from pyLibrary.sql import DB, SQL
from pyLibrary.sql import DB, SQL, SQL_TRUE, SQL_FALSE, SQL_NULL, SQL_SELECT, sql_iso
DEBUG = False
TRACE = True
DEBUG_EXECUTE = False
DEBUG_INSERT = False
sqlite3 = None
_load_extension_warning_sent = False
_upgraded = False
def _upgrade():
global _upgraded
_upgraded = True
try:
import sys
global sqlite3
sqlite_dll = File.new_instance(sys.exec_prefix, "dlls/sqlite3.dll")
python_dll = File("pyLibrary/vendor/sqlite/sqlite3.dll")
if python_dll.read_bytes() != sqlite_dll.read_bytes():
backup = sqlite_dll.backup()
File.copy(python_dll, sqlite_dll)
try:
Log.note("sqlite not upgraded ")
# return
#
# import sys
# import platform
# if "windows" in platform.system().lower():
# original_dll = File.new_instance(sys.exec_prefix, "dlls/sqlite3.dll")
# if platform.architecture()[0]=='32bit':
# source_dll = File("vendor/pyLibrary/vendor/sqlite/sqlite3_32.dll")
# else:
# source_dll = File("vendor/pyLibrary/vendor/sqlite/sqlite3_64.dll")
#
# if not all(a == b for a, b in zip_longest(source_dll.read_bytes(), original_dll.read_bytes())):
# original_dll.backup()
# File.copy(source_dll, original_dll)
# else:
# pass
except Exception as e:
Log.warning("could not upgrade python's sqlite", cause=e)
import sqlite3
_ = sqlite3
_upgraded = True
class Sqlite(DB):
"""
@ -70,12 +89,13 @@ class Sqlite(DB):
if upgrade and not _upgraded:
_upgrade()
self.filename = filename
self.filename = File(filename).abspath
self.db = db
self.queue = Queue("sql commands") # HOLD (command, result, signal) PAIRS
self.worker = Thread.run("sqlite db thread", self._worker)
self.get_trace = DEBUG
self.get_trace = TRACE
self.upgrade = upgrade
self.closed = False
def _enhancements(self):
def regex(pattern, value):
@ -100,16 +120,35 @@ class Sqlite(DB):
COMMANDS WILL BE EXECUTED IN THE ORDER THEY ARE GIVEN
BUT CAN INTERLEAVE WITH OTHER TREAD COMMANDS
:param command: COMMAND FOR SQLITE
:return: None
:return: Signal FOR IF YOU WANT TO BE NOTIFIED WHEN DONE
"""
if DEBUG: # EXECUTE IMMEDIATELY FOR BETTER STACK TRACE
return self.query(command)
if self.closed:
Log.error("database is closed")
if DEBUG_EXECUTE: # EXECUTE IMMEDIATELY FOR BETTER STACK TRACE
self.query(command)
return DONE
if self.get_trace:
trace = extract_stack(1)
else:
trace = None
self.queue.add((command, None, None, trace))
is_done = Signal()
self.queue.add((command, None, is_done, trace))
return is_done
def commit(self):
"""
WILL BLOCK CALLING THREAD UNTIL ALL PREVIOUS execute() CALLS ARE COMPLETED
:return:
"""
if self.closed:
Log.error("database is closed")
signal = _allocate_lock()
signal.acquire()
self.queue.add((COMMIT, None, signal, None))
signal.acquire()
return
def query(self, command):
"""
@ -117,73 +156,115 @@ class Sqlite(DB):
:param command: COMMAND FOR SQLITE
:return: list OF RESULTS
"""
if self.closed:
Log.error("database is closed")
if not self.worker:
self.worker = Thread.run("sqlite db thread", self._worker)
signal = Signal()
signal = _allocate_lock()
signal.acquire()
result = Data()
self.queue.add((command, result, signal, None))
signal.wait()
signal.acquire()
if result.exception:
Log.error("Problem with Sqlite call", cause=result.exception)
return result
def close(self):
"""
OPTIONAL COMMIT-AND-CLOSE
IF THIS IS NOT DONE, THEN THE THREAD THAT SPAWNED THIS INSTANCE
:return:
"""
self.closed = True
signal = _allocate_lock()
signal.acquire()
self.queue.add((COMMIT, None, signal, None))
signal.acquire()
self.worker.please_stop.go()
return
def __enter__(self):
pass
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
def _worker(self, please_stop):
global _load_extension_warning_sent
if DEBUG:
Log.note("Sqlite version {{version}}", version=sqlite3.sqlite_version)
if Sqlite.canonical:
self.db = Sqlite.canonical
else:
self.db = sqlite3.connect(coalesce(self.filename, ':memory:'), check_same_thread = False)
library_loc = File.new_instance(sys.modules[__name__].__file__, "../..")
full_path = File.new_instance(library_loc, "vendor/sqlite/libsqlitefunctions.so").abspath
try:
trace = extract_stack(0)[0]
if self.upgrade:
if os.name == 'nt':
file = File.new_instance(trace["file"], "../../vendor/sqlite/libsqlitefunctions.so")
else:
file = File.new_instance(trace["file"], "../../vendor/sqlite/libsqlitefunctions")
full_path = file.abspath
self.db.enable_load_extension(True)
self.db.execute("SELECT load_extension(" + self.quote_value(full_path) + ")")
except Exception as e:
if not _load_extension_warning_sent:
_load_extension_warning_sent = True
Log.warning("Could not load {{file}}}, doing without. (no SQRT for you!)", file=full_path, cause=e)
try:
while not please_stop:
command, result, signal, trace = self.queue.pop(till=please_stop)
if DEBUG:
Log.note("Sqlite version {{version}}", version=sqlite3.sqlite_version)
if Sqlite.canonical:
self.db = Sqlite.canonical
else:
self.db = sqlite3.connect(coalesce(self.filename, ':memory:'), check_same_thread = False)
library_loc = File.new_instance(sys.modules[__name__].__file__, "../..")
full_path = File.new_instance(library_loc, "vendor/sqlite/libsqlitefunctions.so").abspath
try:
trace = extract_stack(0)[0]
if self.upgrade:
if os.name == 'nt':
file = File.new_instance(trace["file"], "../../vendor/sqlite/libsqlitefunctions.so")
else:
file = File.new_instance(trace["file"], "../../vendor/sqlite/libsqlitefunctions")
full_path = file.abspath
self.db.enable_load_extension(True)
self.db.execute(SQL_SELECT + "load_extension" + sql_iso(self.quote_value(full_path)))
except Exception as e:
if not _load_extension_warning_sent:
_load_extension_warning_sent = True
Log.warning("Could not load {{file}}}, doing without. (no SQRT for you!)", file=full_path, cause=e)
while not please_stop:
quad = self.queue.pop(till=please_stop)
if quad is None:
break
command, result, signal, trace = quad
show_timing = False
if DEBUG_INSERT and command.strip().lower().startswith("insert"):
Log.note("Running command\n{{command|indent}}", command=command)
Log.note("Running command\n{{command|limit(100)|indent}}", command=command)
show_timing = True
if DEBUG and not command.strip().lower().startswith("insert"):
Log.note("Running command\n{{command|indent}}", command=command)
with Timer("Run command", debug=DEBUG):
if signal is not None:
Log.note("Running command\n{{command|limit(100)|indent}}", command=command)
show_timing = True
with Timer("SQL Timing", silent=not show_timing):
if command is COMMIT:
self.db.commit()
signal.release()
elif signal is not None:
try:
curr = self.db.execute(command)
self.db.commit()
result.meta.format = "table"
result.header = [d[0] for d in curr.description] if curr.description else None
result.data = curr.fetchall()
if DEBUG and result.data:
text = convert.table2csv(list(result.data))
Log.note("Result:\n{{data}}", data=text)
if result is not None:
result.meta.format = "table"
result.header = [d[0] for d in curr.description] if curr.description else None
result.data = curr.fetchall()
if DEBUG and result.data:
text = convert.table2csv(list(result.data))
Log.note("Result:\n{{data}}", data=text)
except Exception as e:
e = Except.wrap(e)
result.exception = Except(ERROR, "Problem with\n{{command|indent}}", command=command, cause=e)
e.cause = Except(
type=ERROR,
template="Bad call to Sqlite",
trace=trace
)
if result is None:
Log.error("Problem with\n{{command|indent}}", command=command, cause=e)
else:
result.exception = Except(ERROR, "Problem with\n{{command|indent}}", command=command, cause=e)
finally:
signal.go()
if isinstance(signal, Signal):
signal.go()
else:
signal.release()
else:
try:
self.db.execute(command)
self.db.commit()
except Exception as e:
e = Except.wrap(e)
e.cause = Except(
@ -195,11 +276,11 @@ class Sqlite(DB):
except Exception as e:
if not please_stop:
Log.error("Problem with sql thread", e)
Log.warning("Problem with sql thread", cause=e)
finally:
self.closed = True
if DEBUG:
Log.note("Database is closed")
self.db.commit()
self.db.close()
def quote_column(self, column_name, table=None):
@ -220,36 +301,42 @@ _no_need_to_quote = re.compile(r"^\w+$", re.UNICODE)
def quote_column(column_name, table=None):
if isinstance(column_name, SQL):
return column_name
if not isinstance(column_name, text_type):
Log.error("expecting a name")
if table != None:
return SQL(quote(table) + "." + quote(column_name))
return SQL(" d" + quote(table) + "." + quote(column_name) + " ")
else:
if _no_need_to_quote.match(column_name):
return SQL(column_name)
return SQL(quote(column_name))
def quote_table(column):
if _no_need_to_quote.match(column):
return column
return quote(column)
return SQL(" " + column_name + " ")
return SQL(" " + quote(column_name) + " ")
def quote_value(value):
if isinstance(value, (Mapping, list)):
return "."
return SQL(".")
elif isinstance(value, Date):
return text_type(value.unix)
return SQL(text_type(value.unix))
elif isinstance(value, Duration):
return text_type(value.seconds)
return SQL(text_type(value.seconds))
elif isinstance(value, text_type):
return "'" + value.replace("'", "''") + "'"
return SQL("'" + value.replace("'", "''") + "'")
elif value == None:
return "NULL"
return SQL_NULL
elif value is True:
return "1"
return SQL_TRUE
elif value is False:
return "0"
return SQL_FALSE
else:
return text_type(value)
return SQL(text_type(value))
def join_column(a, b):
a = quote_column(a)
b = quote_column(b)
return SQL(a.template.rstrip() + "." + b.template.lstrip())
COMMIT = "commit"

0
vendor/pyLibrary/sql/sql.py → vendor/pyLibrary/sql/util.py поставляемый
Просмотреть файл

4
vendor/requirements.txt поставляемый
Просмотреть файл

@ -1,4 +0,0 @@
future
pymysql
requests
boto

62
vendor/setup.py поставляемый
Просмотреть файл

@ -1,62 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from distutils.util import convert_path
import os
from setuptools import setup
root = os.path.abspath(os.path.dirname(__file__))
path = lambda *p: os.path.join(root, *p)
try:
long_desc = open(path('README.txt')).read()
except Exception:
long_desc = "<Missing README.txt>"
print("Missing README.txt")
def find_packages(where='.', lib_prefix='', exclude=()):
"""
SNAGGED FROM distribute-0.6.49-py2.7.egg/setuptools/__init__.py
"""
out = []
stack=[(convert_path(where), lib_prefix)]
while stack:
where,prefix = stack.pop(0)
for name in os.listdir(where):
fn = os.path.join(where,name)
if ('.' not in name and os.path.isdir(fn) and
os.path.isfile(os.path.join(fn,'__init__.py'))
):
out.append(prefix+name); stack.append((fn,prefix+name+'.'))
for pat in list(exclude)+['ez_setup', 'distribute_setup']:
from fnmatch import fnmatchcase
out = [item for item in out if not fnmatchcase(item,pat)]
return out
setup(
name='pyLibrary',
version="1.4.17227",
description='Library of Wonderful Things',
long_description=long_desc,
author='Kyle Lahnakoski',
author_email='kyle@lahnakoski.com',
url='https://github.com/klahnakoski/pyLibrary',
license='MPL 2.0',
packages=find_packages(),
install_requires=["boto", "future", "mo_collections", "mo_dots", "mo_files", "mo_json", "mo_json_config", "mo_kwargs", "mo_logs", "mo_math", "mo_testing", "mo_threads", "mo_times", "pymysql", "requests"],
include_package_data=True,
zip_safe=False,
classifiers=[ #https://pypi.python.org/pypi?%3Aaction=list_classifiers
"Development Status :: 4 - Beta",
"Topic :: Software Development :: Libraries",
"Topic :: Software Development :: Libraries :: Python Modules",
"License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
]
)

0
vendor/tests/__init__.py поставляемый
Просмотреть файл

36
vendor/tests/test_cnv.py поставляемый
Просмотреть файл

@ -1,36 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
import datetime
import unittest
from mo_future import text_type
from pyLibrary import convert
class TestConvert(unittest.TestCase):
def test_datetime(self):
result = convert.datetime2milli(datetime.datetime(2012, 7, 24))
expected = 1343088000000
assert result == expected
result = convert.datetime2milli(datetime.date(2012, 7, 24))
expected = 1343088000000
assert result == expected
result = convert.datetime2milli(datetime.datetime(2014, 1, 7, 10, 21, 0))
expected = 1389090060000
assert result == expected
result = text_type(convert.datetime2milli(datetime.datetime(2014, 1, 7, 10, 21, 0)))
expected = u"1389090060000"
assert result == expected

0
vendor/tests/test_collections/__init__.py поставляемый
Просмотреть файл

64
vendor/tests/test_collections/test_unique_index.py поставляемый
Просмотреть файл

@ -1,64 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import unicode_literals
from mo_collections.unique_index import UniqueIndex
from mo_testing.fuzzytestcase import FuzzyTestCase
class TestUniqueIndex(FuzzyTestCase):
def test_single_key(self):
data = [
{"a": 1, "b": "w"},
{"a": 2, "b": "x"},
{"a": 3, "b": "y"},
{"a": 4, "b": "z"}
]
i = UniqueIndex(["a"], data=data)
s = UniqueIndex(["a"])
s.add({"a": 4, "b": "x"})
self.assertEqual(i - s, [
{"a": 1, "b": "w"},
{"a": 2, "b": "x"},
{"a": 3, "b": "y"}
])
self.assertEqual(i | s, data)
self.assertEqual(s | i, [
{"a": 1, "b": "w"},
{"a": 2, "b": "x"},
{"a": 3, "b": "y"},
{"a": 4, "b": "x"}
])
self.assertEqual(i & s, [{"a": 4, "b": "z"}])
def test_double_key(self):
data = [
{"a": 1, "b": "w"},
{"a": 2, "b": "x"},
{"a": 3, "b": "y"},
{"a": 4, "b": "z"}
]
i = UniqueIndex(["a", "b"], data=data)
s = UniqueIndex(["a", "b"])
s.add({"a": 4, "b": "x"})
self.assertEqual(i - s, data)
self.assertEqual(i | s, i |s)
self.assertEqual(i & s, [])

119
vendor/tests/test_db.py поставляемый
Просмотреть файл

@ -1,119 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import re
from mo_testing.fuzzytestcase import FuzzyTestCase
from pyLibrary import convert
from pyLibrary.queries.jx_usingMySQL import esfilter2sqlwhere
from pyLibrary.sql import mysql
from pyLibrary.sql.mysql import MySQL
class TestDB(FuzzyTestCase):
def test_int_packer(self):
v = [801000, 801001, 801002, 801003, 801004, 801005, 801006, 801007, 801008, 801009, 801010, 801011, 801012, 801013, 801014, 801015, 801016, 801017, 801018, 801019, 801020, 801021, 801022, 801023,
801024, 801025, 801026, 801028, 801029, 801030, 801032, 801034, 801035, 801036, 801037, 801038, 801040, 801042, 801043, 801044, 801045, 801047, 801048, 801049, 801050, 801051, 801052, 801053,
801055, 801056, 801057, 801058, 801059, 801060, 801061, 801062, 801063, 801065, 801066, 801067, 801068, 801069, 801070, 801071, 801072, 801073, 801074, 801075, 801076, 801077, 801078, 801079,
801080, 801081, 801083, 801084, 801085, 801086, 801087, 801088, 801089, 801090, 801091, 801092, 801093, 801094, 801095, 801096, 801097, 801098, 801099, 801100, 801101, 801102, 801103, 801104,
801105, 801106, 801107, 801108, 801109, 801110, 801111, 801112, 801113, 801116, 801117, 801118, 801119, 801120, 801121, 801122, 801123, 801124, 801125, 801126, 801128, 801129, 801130, 801131,
801132, 801133, 801134, 801135, 801136, 801137, 801138, 801139, 801140, 801141, 801142, 801143, 801144, 801145, 801146, 801147, 801148, 801149, 801150, 801151, 801152, 801153, 801154, 801155,
801156, 801158, 801159, 801160, 801161, 801162, 801163, 801164, 801165, 801166, 801167, 801168, 801169, 801170, 801171, 801172, 801173, 801174, 801175, 801176, 801177, 801178, 801179, 801180,
801181, 801182, 801183, 801184, 801185, 801186, 801187, 801188, 801189, 801190, 801191, 801192, 801193, 801194, 801196, 801197, 801198, 801199, 801200, 801201, 801202, 801203, 801204, 801205,
801207, 801208, 801209, 801210, 801211, 801212, 801213, 801214, 801215, 801216, 801217, 801218, 801219, 801222, 801223, 801224, 801225, 801226, 801229, 801230, 801231, 801232, 801233, 801235,
801236, 801237, 801238, 801239, 801240, 801241, 801242, 801243, 801244, 801245, 801246, 801247, 801253, 801254, 801255, 801256, 801257, 801258, 801260, 801261, 801262, 801263, 801264, 801265,
801266, 801267, 801268, 801269, 801270, 801271, 801272, 801273, 801274, 801275, 801276, 801277, 801278, 801279, 801280, 801281, 801282, 801283, 801284, 801285, 801286, 801287, 801288, 801289,
801290, 801291, 801292, 801293, 801294, 801295, 801296, 801297, 801298, 801300, 801301, 801303, 801304, 801305, 801306, 801307, 801308, 801309, 801310, 801311, 801312, 801313, 801314, 801315,
801316, 801317, 801318, 801319, 801320, 801321, 801322, 801323, 801324, 801325, 801326, 801327, 801328, 801329, 801331, 801332, 801333, 801334, 801336, 801337, 801338, 801339, 801340, 801341,
801343, 801344, 801345, 801346, 801347, 801348, 801349, 801350, 801351, 801352, 801353, 801354, 801355, 801356, 801357, 801358, 801359, 801360, 801361, 801362, 801363, 801364, 801365, 801367,
801368, 801369, 801370, 801371, 801372, 801373, 801374, 801375, 801376, 801377, 801378, 801379, 801380, 801381, 801382, 801383, 801384, 801385, 801386, 801387, 801388, 801389, 801390, 801391,
801393, 801394, 801395, 801396, 801397, 801398, 801399, 801400, 801401, 801402, 801403, 801404, 801405, 801406, 801407, 801408, 801409, 801410, 801411, 801412, 801413, 801414, 801415, 801416,
801417, 801418, 801419, 801420, 801421, 801422, 801423, 801424, 801425, 801426, 801427, 801428, 801429, 801430, 801431, 801432, 801433, 801434, 801435, 801436, 801437, 801439, 801440, 801441,
801442, 801443, 801444, 801445, 801446, 801447, 801448, 801449, 801450, 801451, 801452, 801453, 801454, 801455, 801456, 801457, 801458, 801459, 801460, 801461, 801462, 801463, 801464, 801465,
801466, 801467, 801468, 801469, 801470, 801471, 801472, 801473, 801474, 801475, 801476, 801477, 801478, 801479, 801480, 801481, 801482, 801483, 801484, 801485, 801486, 801487, 801488, 801489,
801490, 801491, 801492, 801493, 801494, 801495, 801496, 801497, 801498, 801499, 801500, 801501, 801502, 801503, 801505, 801506, 801507, 801508, 801509, 801510, 801511, 801512, 801513, 801514,
801515, 801516, 801517, 801518, 801519, 801520, 801521, 801522, 801523, 801524, 801525, 801526, 801527, 801528, 801529, 801530, 801531, 801532, 801533, 801534, 801535, 801536, 801537, 801538,
801539, 801540, 801541, 801542, 801543, 801544, 801545, 801546, 801547, 801548, 801549, 801550, 801551, 801552, 801553, 801554, 801555, 801556, 801557, 801558, 801559, 801560, 801561, 801562,
801563, 801564, 801565, 801566, 801567, 801568, 801569, 801571, 801572, 801573, 801574, 801575, 801576, 801577, 801578, 801579, 801580, 801581, 801582, 801583, 801584, 801585, 801587, 801589,
801590, 801591, 801592, 801593, 801595, 801596, 801597, 801598, 801599, 801600, 801601, 801602, 801603, 801604, 801605, 801606, 801607, 801608, 801609, 801610, 801611, 801612, 801613, 801614,
801615, 801616, 801617, 801618, 801619, 801620, 801621, 801622, 801623, 801624, 801625, 801626, 801627, 801628, 801629, 801630, 801631, 801632, 801633, 801634, 801635, 801636, 801637, 801639,
801640, 801642, 801643, 801644, 801645, 801646, 801647, 801649, 801650, 801651, 801652, 801653, 801655, 801656, 801658, 801659, 801660, 801661, 801662, 801663, 801664, 801665, 801666, 801667,
801668, 801669, 801670, 801671, 801672, 801674, 801675, 801676, 801677, 801678, 801679, 801680, 801681, 801682, 801683, 801684, 801685, 801686, 801687, 801688, 801690, 801691, 801693, 801694,
801695, 801696, 801697, 801698, 801699, 801701, 801702, 801703, 801705, 801706, 801707, 801708, 801710, 801711, 801712, 801713, 801714, 801715, 801716, 801717, 801718, 801719, 801720, 801721,
801722, 801723, 801724, 801725, 801726, 801727, 801729, 801730, 801731, 801732, 801733, 801734, 801737, 801738, 801739, 801740, 801741, 801742, 801743, 801744, 801745, 801747, 801748, 801749,
801750, 801752, 801753, 801754, 801755, 801756, 801757, 801758, 801759, 801760, 801761, 801763, 801764, 801765, 801766, 801767, 801769, 801771, 801773, 801774, 801775, 801776, 801778, 801779,
801780, 801781, 801782, 801783, 801784, 801785, 801786, 801787, 801788, 801789, 801791, 801792, 801793, 801794, 801795, 801796, 801797, 801798, 801799, 801800, 801801, 801802, 801803, 801804,
801805, 801806, 801807, 801808, 801809, 801810, 801811, 801812, 801813, 801814, 801816, 801817, 801818, 801819, 801820, 801821, 801823, 801824, 801825, 801826, 801827, 801828, 801829, 801830,
801832, 801833, 801834, 801835, 801836, 801837, 801838, 801839, 801840, 801841, 801842, 801843, 801844, 801845, 801846, 801847, 801849, 801850, 801852, 801855, 801856, 801857, 801858, 801859,
801860, 801862, 801869, 801872, 801874, 801880, 801882, 801883, 801884, 801885, 801891, 801892, 801895, 801897, 801898, 801899, 801902, 801905, 801906, 801909, 801911, 801912, 801913, 801914,
801915, 801916, 801917, 801918, 801919, 801920, 801921, 801922, 801923, 801924, 801925, 801926, 801927, 801928, 801929, 801930, 801931, 801932, 801934, 801935, 801936, 801937, 801938, 801941,
801942, 801943, 801944, 801945, 801946, 801947, 801948, 801949, 801950, 801951, 801952, 801953, 801954, 801955, 801956, 801957, 801958, 801960, 801961, 801962, 801964, 801965, 801966, 801967,
801969, 801970, 801971, 801972, 801973, 801974, 801976, 801977, 801978, 801979, 801980, 801981, 801982, 801983, 801984, 801985, 801986, 801987, 801988, 801989, 801990, 801991, 801993, 801994,
801995, 801996, 801998]
int_list = convert.value2intlist(v)
result = mysql.int_list_packer("bug_id", int_list)
reference = {"or": [{"terms": {"bug_id": [801869, 801872, 801874, 801880, 801882, 801883, 801884, 801885, 801891, 801892, 801895, 801897, 801898, 801899, 801902, 801905, 801906, 801909, 801911, 801912, 801913, 801914, 801915, 801916, 801917, 801918, 801919, 801920, 801921, 801922, 801923, 801924, 801925, 801926, 801927, 801928, 801929, 801930, 801931, 801932, 801934, 801935, 801936, 801937, 801938, 801941, 801942, 801943, 801944, 801945, 801946, 801947, 801948, 801949, 801950, 801951, 801952, 801953, 801954, 801955, 801956, 801957, 801958, 801960, 801961, 801962, 801964, 801965, 801966, 801967, 801969, 801970, 801971, 801972, 801973, 801974, 801976, 801977, 801978, 801979, 801980, 801981, 801982, 801983, 801984, 801985, 801986, 801987, 801988, 801989, 801990, 801991, 801993, 801994, 801995, 801996, 801998]}}, {"and": [{"or": [{"range": {"bug_id": {"gte": 801000, "lte": 801045}}}, {"range": {"bug_id": {"gte": 801047, "lte": 801247}}}, {"range": {"bug_id": {"gte": 801253, "lte": 801862}}}]}, {"not": {"terms": {"bug_id": [801027, 801031, 801033, 801039, 801041, 801054, 801064, 801082, 801114, 801115, 801127, 801157, 801195, 801206, 801220, 801221, 801227, 801228, 801234, 801259, 801299, 801302, 801330, 801335, 801342, 801366, 801392, 801438, 801504, 801570, 801586, 801588, 801594, 801638, 801641, 801648, 801654, 801657, 801673, 801689, 801692, 801700, 801704, 801709, 801728, 801735, 801736, 801746, 801751, 801762, 801768, 801770, 801772, 801777, 801790, 801815, 801822, 801831, 801848, 801851, 801853, 801854, 801861]}}}]}]}
self.assertEqual(result, reference)
def test_filter2where(self):
v = [856000, 856001, 856002, 856003, 856004, 856006, 856007, 856008, 856009, 856011, 856012, 856013, 856014, 856015, 856016, 856017, 856018, 856020, 856021, 856022, 856023, 856024, 856025, 856026,
856027, 856028, 856030, 856031, 856032, 856034, 856037, 856038, 856039, 856040, 856041, 856043, 856045, 856047, 856048, 856049, 856050, 856051, 856052, 856053, 856054, 856055, 856056, 856058,
856059, 856062, 856070, 856071, 856072, 856073, 856074, 856075, 856076, 856077, 856078, 856079, 856080, 856081, 856082, 856083, 856084, 856085, 856086, 856087, 856088, 856089, 856090, 856092,
856093, 856094, 856095, 856096, 856097, 856098, 856100, 856101, 856102, 856103, 856105, 856107, 856108, 856109, 856110, 856111, 856112, 856113, 856114, 856115, 856116, 856117, 856118, 856119,
856120, 856121, 856122, 856123, 856124, 856127, 856128, 856129, 856130, 856131, 856132, 856133, 856134, 856137, 856138, 856139, 856140, 856141, 856142, 856143, 856144, 856145, 856146, 856147,
856148, 856149, 856150, 856151, 856152, 856153, 856154, 856155, 856156, 856158, 856159, 856160, 856163, 856165, 856166, 856167, 856168, 856169, 856170, 856171, 856172, 856176, 856177, 856178,
856179, 856180, 856182, 856183, 856184, 856186, 856187, 856188, 856189, 856190, 856191, 856192, 856193, 856194, 856195, 856196, 856197, 856198, 856199, 856201, 856202, 856203, 856204, 856205,
856206, 856207, 856208, 856209, 856210, 856211, 856212, 856213, 856214, 856215, 856216, 856217, 856222, 856223, 856224, 856225, 856226, 856227, 856228, 856229, 856230, 856232, 856233, 856234,
856235, 856238, 856239, 856240, 856241, 856242, 856244, 856245, 856246, 856247, 856248, 856249, 856250, 856251, 856252, 856253, 856254, 856255, 856256, 856257, 856258, 856260, 856261, 856262,
856263, 856264, 856265, 856266, 856267, 856268, 856269, 856270, 856272, 856273, 856275, 856276, 856277, 856278, 856279, 856280, 856281, 856282, 856283, 856284, 856285, 856286, 856287, 856288,
856289, 856290, 856291, 856292, 856295, 856296, 856297, 856298, 856299, 856300, 856301, 856302, 856303, 856304, 856305, 856306, 856307, 856308, 856309, 856310, 856311, 856312, 856313, 856314,
856315, 856316, 856317, 856318, 856319, 856321, 856322, 856323, 856324, 856325, 856327, 856328, 856329, 856330, 856331, 856332, 856333, 856335, 856337, 856338, 856339, 856340, 856341, 856342,
856344, 856345, 856346, 856349, 856350, 856351, 856352, 856353, 856354, 856355, 856356, 856357, 856358, 856359, 856360, 856361, 856362, 856363, 856364, 856365, 856366, 856367, 856368, 856369,
856370, 856371, 856372, 856373, 856375, 856378, 856381, 856383, 856385, 856386, 856387, 856388, 856389, 856390, 856391, 856392, 856393, 856394, 856396, 856397, 856400, 856401, 856402, 856403,
856404, 856405, 856406, 856407, 856408, 856409, 856410, 856411, 856412, 856413, 856414, 856415, 856417, 856418, 856419, 856420, 856421, 856422, 856423, 856424, 856425, 856426, 856427, 856429,
856430, 856431, 856432, 856433, 856434, 856436, 856437, 856438, 856439, 856440, 856441, 856442, 856443, 856444, 856445, 856448, 856450, 856451, 856452, 856453, 856454, 856455, 856456, 856457,
856458, 856459, 856460, 856461, 856462, 856463, 856464, 856465, 856466, 856467, 856468, 856469, 856470, 856471, 856472, 856474, 856475, 856476, 856477, 856478, 856479, 856481, 856482, 856484,
856485, 856486, 856487, 856489, 856490, 856491, 856492, 856493, 856494, 856495, 856496, 856497, 856498, 856499, 856500, 856501, 856502, 856503, 856504, 856505, 856506, 856507, 856508, 856509,
856511, 856512, 856513, 856514, 856515, 856516, 856517, 856518, 856519, 856520, 856521, 856522, 856523, 856524, 856525, 856526, 856527, 856528, 856529, 856530, 856531, 856532, 856533, 856534,
856535, 856536, 856538, 856540, 856541, 856542, 856543, 856544, 856545, 856546, 856547, 856548, 856549, 856550, 856551, 856552, 856553, 856554, 856555, 856556, 856557, 856558, 856559, 856560,
856561, 856562, 856565, 856566, 856567, 856568, 856569, 856571, 856572, 856574, 856575, 856576, 856577, 856579, 856580, 856581, 856582, 856583, 856584, 856585, 856586, 856587, 856588, 856590,
856591, 856592, 856593, 856594, 856595, 856596, 856598, 856599, 856600, 856601, 856602, 856603, 856604, 856605, 856606, 856607, 856608, 856609, 856611, 856612, 856613, 856614, 856615, 856616,
856617, 856618, 856619, 856620, 856621, 856622, 856623, 856624, 856625, 856626, 856627, 856629, 856630, 856631, 856632, 856633, 856634, 856635, 856637, 856638, 856639, 856640, 856641, 856642,
856643, 856644, 856645, 856646, 856647, 856651, 856653, 856654, 856657, 856658, 856659, 856660, 856661, 856662, 856664, 856665, 856666, 856670, 856671, 856672, 856673, 856674, 856675, 856676,
856677, 856678, 856679, 856680, 856681, 856682, 856683, 856684, 856685, 856687, 856688, 856689, 856690, 856691, 856692, 856693, 856694, 856695, 856696, 856697, 856698, 856699, 856700, 856701,
856702, 856703, 856705, 856707, 856708, 856709, 856710, 856711, 856712, 856713, 856715, 856716, 856717, 856718, 856720, 856728, 856729, 856731, 856732, 856733, 856734, 856736, 856738, 856739,
856740, 856741, 856742, 856743]
where = esfilter2sqlwhere(MySQL(host="", port=1, username="", password=""), {"terms": {"bug_id": v}})
reference = """
(
`bug_id` in (856000, 856001, 856002, 856003, 856004, 856006, 856007, 856008, 856009, 856011, 856012, 856013, 856014, 856015, 856016, 856017, 856018, 856020, 856021, 856022, 856023, 856024, 856025, 856026, 856027, 856028, 856030, 856031, 856032, 856034, 856037, 856038, 856039, 856040, 856041, 856043, 856045, 856047, 856048, 856049, 856050, 856051, 856052, 856053, 856054, 856055, 856056, 856058, 856059, 856062, 856165, 856166, 856167, 856168, 856169, 856170, 856171, 856172, 856176, 856177, 856178, 856179, 856180, 856182, 856183, 856184, 856222, 856223, 856224, 856225, 856226, 856227, 856228, 856229, 856230, 856232, 856233, 856234, 856235, 856238, 856239, 856240, 856241, 856242, 856381, 856383, 856651, 856653, 856654, 856657, 856658, 856659, 856660, 856661, 856662, 856664, 856665, 856666, 856728, 856729, 856731, 856732, 856733, 856734, 856736, 856738, 856739, 856740, 856741, 856742, 856743) OR
(
(
`bug_id` BETWEEN 856070 AND 856163 OR
`bug_id` BETWEEN 856186 AND 856217 OR
`bug_id` BETWEEN 856244 AND 856378 OR
`bug_id` BETWEEN 856385 AND 856448 OR
`bug_id` BETWEEN 856450 AND 856647 OR
`bug_id` BETWEEN 856670 AND 856720
) AND
NOT (`bug_id` in (856091, 856099, 856104, 856106, 856125, 856126, 856135, 856136, 856157, 856161, 856162, 856200, 856259, 856271, 856274, 856293, 856294, 856320, 856326, 856334, 856336, 856343, 856347, 856348, 856374, 856376, 856377, 856395, 856398, 856399, 856416, 856428, 856435, 856446, 856447, 856473, 856480, 856483, 856488, 856510, 856537, 856539, 856563, 856564, 856570, 856573, 856578, 856589, 856597, 856610, 856628, 856636, 856686, 856704, 856706, 856714, 856719))
)
)"""
reference = re.sub(r"\s+", " ", reference).strip()
where = re.sub(r"\s+", " ", where).strip()
self.assertAlmostEqual(where, reference)

0
vendor/tests/test_dots/__init__.py поставляемый
Просмотреть файл

124
vendor/tests/test_dots/speedtest_wrap.py поставляемый
Просмотреть файл

@ -1,124 +0,0 @@
import gc
from math import log, floor
from mo_logs import Log
from mo_logs import profiles
from mo_logs.profiles import Profiler
from mo_math.randoms import Random
from mo_dots import Data, wrap
from mo_dots.lists import FlatList
def baseline(v):
return [v]
NUM_INPUT = 1000000
NUM_REPEAT = 10
def test_wrap_1():
switch = [
lambda: Data(i=Random.int(2000)),
lambda: {"i": Random.int(2000)},
lambda: FlatList([{"i": Random.int(2000)}]),
lambda: [{"i": Random.int(2000)}]
]
inputs = [switch[min(len(switch) - 1, int(floor(-log(Random.float(), 2))))]() for i in range(NUM_INPUT)]
for i in range(NUM_REPEAT):
results = []
gc.collect()
with Profiler("more struct: slow_wrap"):
for v in inputs:
results.append(slow_wrap(v))
results = []
gc.collect()
with Profiler("more struct: wrap"):
for v in inputs:
results.append(wrap(v))
results = []
gc.collect()
with Profiler("more struct: baseline"):
for v in inputs:
results.append(baseline(v))
Log.note("Done {{i}} of {{num}}", i= i, num= NUM_REPEAT)
def test_wrap_2():
switch = [
lambda: {"i": Random.int(2000)},
lambda: Data(i=Random.int(2000)),
lambda: FlatList([{"i": Random.int(2000)}]),
lambda: [{"i": Random.int(2000)}]
]
inputs = [switch[min(len(switch) - 1, int(floor(-log(Random.float(), 2))))]() for i in range(NUM_INPUT)]
for i in range(NUM_REPEAT):
results = []
gc.collect()
with Profiler("more dict: slow_wrap"):
for v in inputs:
results.append(slow_wrap(v))
results = []
gc.collect()
with Profiler("more dict: wrap"):
for v in inputs:
results.append(wrap(v))
results = []
gc.collect()
with Profiler("more dict: baseline"):
for v in inputs:
results.append(baseline(v))
Log.note("Done {{i}} of {{num}}", i= i, num= NUM_REPEAT)
def test_wrap_3():
switch = [
lambda: Random.string(20),
lambda: {"i": Random.int(2000)},
lambda: Data(i=Random.int(2000)),
lambda: FlatList([{"i": Random.int(2000)}]),
lambda: [{"i": Random.int(2000)}]
]
inputs = [switch[min(len(switch) - 1, int(floor(-log(Random.float(), 2))))]() for i in range(NUM_INPUT)]
for i in range(NUM_REPEAT):
results = []
gc.collect()
with Profiler("more string: slow_wrap"):
for v in inputs:
results.append(slow_wrap(v))
results = []
gc.collect()
with Profiler("more string: wrap"):
for v in inputs:
results.append(wrap(v))
results = []
gc.collect()
with Profiler("more string: baseline"):
for v in inputs:
results.append(baseline(v))
Log.note("Done {{i}} of {{num}}", i= i, num= NUM_REPEAT)
profiles.ON = True
Log.start()
test_wrap_1()
test_wrap_2()
test_wrap_3()
profiles.write(Data(filename="speedtest_wrap.tab"))
Log.stop()

597
vendor/tests/test_dots/test_dot.py поставляемый
Просмотреть файл

@ -1,597 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from collections import Mapping
from copy import deepcopy
from future.moves.collections import UserDict
from mo_logs import Log
from mo_math import MAX
from mo_testing.fuzzytestcase import FuzzyTestCase
from mo_dots import wrap, Null, set_default, unwrap, Data, literal_field, NullType
from mo_dots.objects import datawrap
class TestDot(FuzzyTestCase):
def test_set_union_w_null(self):
s = set('a')
s |= Null
self.assertAlmostEqual(s, set('a'))
def test_null_class(self):
self.assertFalse(isinstance(Null, Mapping))
def test_userdict(self):
def show_kwargs(**kwargs):
return kwargs
a = UserDict(a=1, b=2)
d = show_kwargs(**a)
self.assertAlmostEqual(d, {"a":1, "b":2})
def test_userdict(self):
def show_kwargs(**kwargs):
return kwargs
a = _UserDict()
a.data["a"] = 1
a.data["b"] = 2
d = show_kwargs(**a)
self.assertAlmostEqual(d, {"a":1, "b":2})
def test_dict_args(self):
def show_kwargs(**kwargs):
return kwargs
a = Data()
a["a"] = 1
a["b"] = 2
d = show_kwargs(**a)
self.assertAlmostEqual(d, {"a": 1, "b": 2})
def test_is_mapping(self):
self.assertTrue(isinstance(Data(), Mapping), "All Data must be Mappings")
def test_none(self):
a = 0
b = 0
c = None
d = None
if a == b:
pass
else:
Log.error("error")
if c == d:
pass
else:
Log.error("error")
if a == c:
Log.error("error")
if d == b:
Log.error("error")
if not c:
pass
else:
Log.error("error")
def test_null_access(self):
a = Data()
c = a.b[b'test']
self.assertTrue(c == None, "Expecting Null to accept str() for item access")
def test_null(self):
a = 0
b = 0
c = Null
d = Null
e = Data()
f = Data()
if a == b:
pass
else:
Log.error("error")
if c == d:
pass
else:
Log.error("error")
if a == c:
Log.error("error")
if d == b:
Log.error("error")
if c == None:
pass
else:
Log.error("error")
if not c:
pass
else:
Log.error("error")
if Null != Null:
Log.error("error")
if Null != None:
Log.error("error")
if None != Null:
Log.error("error")
if e.test != f.test:
Log.error("error")
def test_get_value(self):
a = wrap({"a": 1, "b": {}})
if a.a != 1:
Log.error("error")
if not isinstance(a.b, Mapping):
Log.error("error")
def test_get_class(self):
a = wrap({})
_type = a.__class__
if _type is not Data:
Log.error("error")
def test_int_null(self):
a = Data()
value = a.b*1000
assert value == Null
def test_dot_self(self):
a = Data(b=42)
assert a["."] == a
assert a["."].b == 42
a["."] = {"c": 42}
assert a.c == 42
assert a.b == None
def test_list(self):
if not []:
pass
else:
Log.error("error")
if []:
Log.error("error")
if not [0]:
Log.error("error")
def test_assign1(self):
a = {}
b = wrap(a)
b.c = "test1"
b.d.e = "test2"
b.f.g.h = "test3"
b.f.i = "test4"
b.k["l.m.n"] = "test5"
expected = {
"c": "test1",
"d": {
"e": "test2"
},
"f": {
"g": {
"h": "test3"
},
"i": "test4"
},
"k": {
"l": {"m": {"n": "test5"}}
}
}
self.assertEqual(a, expected)
def test_assign2(self):
a = {}
b = wrap(a)
b_c = b.c
b.c.d = "test1"
b_c.e = "test2"
expected = {
"c": {
"d": "test1",
"e": "test2"
}
}
self.assertEqual(a, expected)
def test_assign3(self):
# IMPOTENT ASSIGNMENTS DO NOTHING
a = {}
b = wrap(a)
b.c = None
expected = {}
self.assertEqual(a, expected)
b.c.d = None
expected = {}
self.assertEqual(a, expected)
b["c.d"] = None
expected = {}
self.assertEqual(a, expected)
b.c.d.e = None
expected = {}
self.assertEqual(a, expected)
b.c["d.e"] = None
expected = {}
self.assertEqual(a, expected)
def test_assign4(self):
# IMPOTENT ASSIGNMENTS DO NOTHING
a = {"c": {"d": {}}}
b = wrap(a)
b.c.d = None
expected = {"c": {}}
self.assertEqual(a, expected)
a = {"c": {"d": {}}}
b = wrap(a)
b.c = None
expected = {}
self.assertEqual(a, expected)
def test_assign5(self):
a = {}
b = wrap(a)
b.c["d\.e"].f = 2
expected = {"c": {"d.e": {"f": 2}}}
self.assertEqual(a, expected)
def test_assign6(self):
a = {}
b = wrap(a)
b["c.d.e\.f"] = 1
b["c.d.e\.g"] = 2
expected = {"c": {"d": {"e.f": 1, "e.g": 2}}}
self.assertEqual(a, expected)
def test_assign7(self):
a = {}
b = wrap(a)
b["c.d.e\.f"] = 1
b["c.d.g\.h"] = 2
expected = {"c": {"d": {"e.f": 1, "g.h": 2}}}
self.assertEqual(a, expected)
def test_assign8(self):
a = {}
b = wrap(a)
b["a"][literal_field(literal_field("b.html"))]["z"] = 3
expected = {"a": {
"b\\.html": {"z": 3}
}}
self.assertEqual(a, expected)
def test_assign9(self):
a = {}
b = wrap(a)
b["a"]["."] = 1
expected = {"a": 1}
self.assertEqual(a, expected)
def test_setitem_and_deep(self):
a = {}
b = wrap(a)
b.c["d"].e.f = 3
expected = {"c": {"d": {"e": {"f": 3}}}}
self.assertEqual(a, expected)
def test_assign_and_use1(self):
a = wrap({})
agg = a.b
agg.c = []
agg.c.append("test value")
self.assertEqual(a, {"b": {"c": ["test value"]}})
self.assertEqual(a.b, {"c": ["test value"]})
self.assertEqual(a.b.c, ["test value"])
def test_assign_and_use2(self):
a = wrap({})
agg = a.b.c
agg += []
agg.append("test value")
self.assertEqual(a, {"b": {"c": ["test value"]}})
self.assertEqual(a.b, {"c": ["test value"]})
self.assertEqual(a.b.c, ["test value"])
def test_assign_none(self):
a = {}
A = wrap(a)
A[None] = "test"
self.assertEqual(a, {})
def test_increment(self):
a = {}
b = wrap(a)
b.c1.d += 1
b.c2.e += "e"
b.c3.f += ["f"]
b["c\\.a"].d += 1
self.assertEqual(a, {"c1": {"d": 1}, "c2": {"e": "e"}, "c3": {"f": ["f"]}, "c.a": {"d": 1}})
b.c1.d += 2
b.c2.e += "f"
b.c3.f += ["g"]
b["c\\.a"].d += 3
self.assertEqual(a, {"c1": {"d": 3}, "c2": {"e": "ef"}, "c3": {"f": ["f", "g"]}, "c.a": {"d": 4}})
def test_slicing(self):
def diff(record, index, records):
"""
WINDOW FUNCTIONS TAKE THE CURRENT record, THE index THAT RECORD HAS
IN THE WINDOW, AND THE (SORTED) LIST OF ALL records
"""
# COMPARE CURRENT VALUE TO MAX OF PAST 5, BUT NOT THE VERY LAST ONE
try:
return record - MAX(records[index - 6:index - 1:])
except Exception as e:
return None
data1_list = [1, 2, 3, 4, 5, 6, 7, 8, 9]
result1 = [diff(r, i, data1_list) for i, r in enumerate(data1_list)]
assert result1 == [-7, None, None, None, None, None, 2, 2, 2] # WHAT IS EXPECTED, BUT NOT WHAT WE WANT
data2_list = wrap(data1_list)
result2 = [diff(r, i, data2_list) for i, r in enumerate(data2_list)]
assert result2 == [None, None, 2, 2, 2, 2, 2, 2, 2]
def test_delete1(self):
a = wrap({"b": {"c": 1}})
del a.b.c
self.assertEqual({"b": {}}, a)
self.assertEqual(a, {"b": {}})
a = wrap({"b": {"c": 1}})
a.b.c=None
self.assertEqual({"b": {}}, a)
self.assertEqual(a, {"b": {}})
def test_delete2(self):
a = wrap({"b": {"c": 1, "d": 2}})
del a.b.c
self.assertEqual({"b": {"d": 2}}, a)
self.assertEqual(a, {"b": {"d": 2}})
a = wrap({"b": {"c": 1, "d": 2}})
a.b.c=None
self.assertEqual({"b": {"d": 2}}, a)
self.assertEqual(a, {"b": {"d": 2}})
def test_wrap(self):
d = {}
dd = wrap(d)
self.assertIs(unwrap(dd), d)
def test_object_wrap(self):
d = SampleData()
dd = datawrap(d)
self.assertEqual(dd["a"], 20)
self.assertEqual(dd, {"a": 20, "b": 30})
self.assertIs(unwrap(dd), dd)
def test_object_wrap_w_deep_path(self):
d = SampleData()
d.a = Data(c=3)
dd = datawrap(d)
self.assertEqual(dd["a.c"], 3)
self.assertEqual(dd, {"a": {"c":3}, "b": 30})
def test_deep_select(self):
d = wrap([{"a": {"b": 1}}, {"a": {"b": 2}}])
test = d.a.b
self.assertEqual(test, [1, 2])
def test_deep_select_list(self):
d = wrap({"a": {"b": [{"c": 1}, {"c": 2}]}})
test = d["a.b.c"]
self.assertEqual(test, [1, 2])
def test_set_default(self):
a = {"x": {"y": 1}}
b = {"x": {"z": 2}}
c = {}
d = set_default(c, a, b)
self.assertTrue(unwrap(d) is c, "expecting first parameter to be returned")
self.assertEqual(d.x.y, 1, "expecting d to have attributes of a")
self.assertEqual(d.x.z, 2, "expecting d to have attributes of b")
self.assertEqual(wrap(a).x.z, None, "a should not have been altered")
def test_Dict_of_Dict(self):
value = {"a": 1}
wrapped = Data(Data(value))
self.assertTrue(value is unwrap(wrapped), "expecting identical object")
def test_leaves_of_mappings(self):
a = wrap({"a": _TestMapping()})
a.a.a = {"a": 1}
a.a.b = {"b": 2}
leaves = wrap(dict(a.leaves()))
self.assertEqual(a.a.a['a'], leaves["a\.a\.a"], "expecting 1")
self.assertEqual(a.a.b['b'], leaves["a\.b\.b"], "expecting 2")
def test_null_set_index(self):
temp = Null
# expecting no crash
temp[0] = 1
temp[1] = None
def test_deep_null_assignment(self):
temp = wrap({"a": 0})
e = temp.e
e.s.t = 1
e.s.s = 2
self.assertEqual(temp, {"a": 0, "e": {"s": {"s": 2, "t": 1}}}, "expecting identical")
def test_null_inequalities(self):
self.assertEqual(Null < 1, None)
self.assertEqual(Null <= 1, None)
self.assertEqual(Null != 1, None)
self.assertEqual(Null == 1, None)
self.assertEqual(Null >= 1, None)
self.assertEqual(Null > 1, None)
self.assertEqual(1 < Null, None)
self.assertEqual(1 <= Null, None)
self.assertEqual(1 != Null, None)
self.assertEqual(1 == Null, None)
self.assertEqual(1 >= Null, None)
self.assertEqual(1 > Null, None)
self.assertEqual(Null < Null, None)
self.assertEqual(Null <= Null, None)
self.assertEqual(Null != Null, False)
self.assertEqual(Null == Null, True)
self.assertEqual(Null >= Null, None)
self.assertEqual(Null > Null, None)
self.assertEqual(Null < None, None)
self.assertEqual(Null <= None, None)
self.assertEqual(Null != None, False)
self.assertEqual(Null == None, True)
self.assertEqual(Null >= None, None)
self.assertEqual(Null > None, None)
self.assertEqual(None < Null, None)
self.assertEqual(None <= Null, None)
self.assertEqual(None != Null, False)
self.assertEqual(None == Null, True)
self.assertEqual(None >= Null, None)
self.assertEqual(None > Null, None)
def test_escape_dot(self):
self.assertAlmostEqual(literal_field("."), "\\.")
self.assertAlmostEqual(literal_field("\\."), "\\\\.")
self.assertAlmostEqual(literal_field("\\\\."), "\\\\\\.")
self.assertAlmostEqual(literal_field("a.b"), "a\.b")
self.assertAlmostEqual(literal_field("a\\.html"), "a\\\\.html")
def test_set_default_unicode_and_list(self):
a = {"a": "test"}
b = {"a": [1, 2]}
self.assertAlmostEqual(set_default(a, b), {"a": ["test", 1, 2]}, "expecting string, not list, nor some hybrid")
def test_deepcopy(self):
self.assertIs(deepcopy(Null), Null)
self.assertEqual(deepcopy(Data()), {})
self.assertEqual(deepcopy(Data(a=Null)), {})
def test_null_type(self):
self.assertIs(Null.__class__, NullType)
self.assertTrue(isinstance(Null, NullType))
def test_null_assign(self):
output = Null
output.changeset.files = None
def test_string_assign(self):
def test():
a = wrap({"a": "world"})
a["a.html"] = "value"
self.assertRaises(Exception, test, "expecting error")
def test_string_assign_null(self):
a = wrap({"a": "world"})
a["a.html"] = None
def test_empty_object_is_not_null(self):
self.assertFalse(wrap({}) == None, "expect empty objects to not compare well with None")
def test_add_null_to_list(self):
expected = wrap(["test", "list"])
test = expected + None
self.assertEqual(test, expected, "expecting adding None to list does not change list")
class _TestMapping(object):
def __init__(self):
self.a = None
self.b = None
class _UserDict:
"""
COPY OF UserDict
"""
def __init__(self, **kwargs):
self.data = {}
def __getitem__(self, key):
if key in self.data:
return self.data[key]
if hasattr(self.__class__, "__missing__"):
return self.__class__.__missing__(self, key)
raise KeyError(key)
def keys(self):
return self.data.keys()
class SampleData(object):
def __init__(self):
self.a = 20
self.b = 30
def __str__(self):
return str(self.a)+str(self.b)

355
vendor/tests/test_dots/test_dot_from_addict.py поставляемый
Просмотреть файл

@ -1,355 +0,0 @@
# encoding: utf-8
#
# DEC 2016 - Altered tests to work with differing definitions
#
# SNAGGED FROM https://github.com/mewwts/addict/blob/62e8481a2a5c8520259809fc306698489975c9f8/test_addict.py WITH THE FOLLOWING LICENCE
#
# The MIT License (MIT)
#
# Copyright (c) 2014 Mats Julian Olsen
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
from __future__ import unicode_literals
from collections import Mapping
from copy import deepcopy
import mo_json
from mo_testing.fuzzytestcase import FuzzyTestCase
from mo_dots import Data, set_default, unwrap
TEST_VAL = [1, 2, 3]
TEST_DICT = {'a': {'b': {'c': TEST_VAL}}}
class AddictTests(FuzzyTestCase):
def test_set_one_level_item(self):
some_dict = {'a': TEST_VAL}
prop = Data()
prop['a'] = TEST_VAL
self.assertEqual(prop, some_dict)
def test_set_two_level_items(self):
some_dict = {'a': {'b': TEST_VAL}}
prop = Data()
prop['a']['b'] = TEST_VAL
self.assertEqual(prop, some_dict)
def test_set_three_level_items(self):
prop = Data()
prop['a']['b']['c'] = TEST_VAL
self.assertEqual(prop, TEST_DICT)
def test_set_one_level_property(self):
prop = Data()
prop.a = TEST_VAL
self.assertEqual(prop, {'a': TEST_VAL})
def test_set_two_level_properties(self):
prop = Data()
prop.a.b = TEST_VAL
self.assertEqual(prop, {'a': {'b': TEST_VAL}})
def test_set_three_level_properties(self):
prop = Data()
prop.a.b.c = TEST_VAL
self.assertEqual(prop, TEST_DICT)
def test_init_with_dict(self):
self.assertEqual(TEST_DICT, Data(TEST_DICT))
def test_init_with_kws(self):
prop = Data(a=2, b={'a': 2}, c=[{'a': 2}])
self.assertEqual(prop, {'a': 2, 'b': {'a': 2}, 'c': [{'a': 2}]})
def test_init_with_list(self):
prop = Data([('0', 1), ('1', 2), ('2', 3)])
self.assertEqual(prop, {'0': 1, '1': 2, '2': 3})
def test_init_raises(self):
def init():
Data(5)
def init2():
Data('a')
self.assertRaises(Exception, init)
self.assertRaises(Exception, init2)
def test_init_with_empty_stuff(self):
a = Data({})
b = Data([])
self.assertEqual(a, {})
self.assertEqual(b, {})
def test_init_with_list_of_dicts(self):
a = Data({'a': [{'b': 2}]})
self.assertIsInstance(a.a[0], Data)
self.assertEqual(a.a[0].b, 2)
def test_getitem(self):
prop = Data(TEST_DICT)
self.assertEqual(prop['a']['b']['c'], TEST_VAL)
def test_empty_getitem(self):
prop = Data()
prop.a.b.c
self.assertEqual(prop, {})
def test_getattr(self):
prop = Data(TEST_DICT)
self.assertEqual(prop.a.b.c, TEST_VAL)
def test_isinstance(self):
self.assertTrue(isinstance(Data(), Mapping))
def test_str(self):
prop = Data(TEST_DICT)
self.assertEqual(str(prop), str(TEST_DICT))
def test_json(self):
some_dict = TEST_DICT
some_json = mo_json.value2json(some_dict)
prop = Data()
prop.a.b.c = TEST_VAL
prop_json = mo_json.value2json(prop)
self.assertEqual(some_json, prop_json)
def test_delitem(self):
prop = Data({'a': 2})
del prop['a']
self.assertEqual(prop, {})
def test_delitem_nested(self):
prop = Data(deepcopy(TEST_DICT))
del prop['a']['b']['c']
self.assertEqual(prop, {'a': {'b': {}}})
def test_delattr(self):
prop = Data({'a': 2})
del prop.a
self.assertEqual(prop, {})
def test_delattr_nested(self):
prop = Data(deepcopy(TEST_DICT))
del prop.a.b.c
self.assertEqual(prop, {'a': {'b': {}}})
def test_delitem_delattr(self):
prop = Data(deepcopy(TEST_DICT))
del prop.a['b']
self.assertEqual(prop, {'a': {}})
def test_set_prop_invalid(self):
prop = Data()
prop.keys = 2
prop.items = 3
self.assertEqual(prop, {'keys': 2, 'items': 3})
def test_dir(self):
key = 'a'
prop = Data({key: 1})
dir_prop = dir(prop)
dir_dict = dir(Data)
for d in dir_dict:
self.assertTrue(d in dir_prop, d)
self.assertTrue('__methods__' not in dir_prop)
self.assertTrue('__members__' not in dir_prop)
def test_dir_with_members(self):
prop = Data({'__members__': 1})
dir(prop)
self.assertTrue('__members__' in prop.keys())
def test_unwrap(self):
nested = {'a': [{'a': 0}, 2], 'b': {}, 'c': 2}
prop = Data(nested)
regular = unwrap(prop)
self.assertEqual(regular, prop)
self.assertEqual(regular, nested)
self.assertNotIsInstance(regular, Data)
def get_attr():
regular.a = 2
self.assertRaises(AttributeError, get_attr)
def get_attr_deep():
regular['a'][0].a = 1
self.assertRaises(AttributeError, get_attr_deep)
def test_to_dict_with_tuple(self):
nested = {'a': ({'a': 0}, {2: 0})}
prop = Data(nested)
regular = unwrap(prop)
self.assertEqual(regular, prop)
self.assertEqual(regular, nested)
self.assertIsInstance(regular['a'], tuple)
self.assertNotIsInstance(regular['a'][0], Data)
def test_update(self):
old = Data()
old.child.a = 'a'
old.child.b = 'b'
old.foo = 'c'
new = Data()
new.child.b = 'b2'
new.child.c = 'c'
new.foo.bar = True
old = set_default({}, new, old)
reference = {'foo': {'bar': True},
'child': {'a': 'a', 'c': 'c', 'b': 'b2'}}
self.assertEqual(old, reference)
def test_update_with_lists(self):
org = Data()
org.a = [1, 2, {'a': 'superman'}]
someother = Data()
someother.b = [{'b': 123}]
org.update(someother)
correct = {'a': [1, 2, {'a': 'superman'}],
'b': [{'b': 123}]}
org.update(someother)
self.assertEqual(org, correct)
self.assertIsInstance(org.b[0], Mapping)
def test_update_with_kws(self):
org = Data(one=1, two=2)
someother = Data(one=3)
someother.update(one=1, two=2)
self.assertEqual(org, someother)
def test_update_with_args_and_kwargs(self):
expected = {'a': 1, 'b': 2}
org = Data()
org.update({'a': 3, 'b': 2}, a=1)
self.assertEqual(org, expected)
def test_update_with_multiple_args(self):
org = Data()
def update():
org.update({'a': 2}, {'a': 1})
self.assertRaises(TypeError, update)
def test_hook_in_constructor(self):
a_dict = Data(TEST_DICT)
self.assertIsInstance(a_dict['a'], Data)
def test_copy(self):
class MyMutableObject(object):
def __init__(self):
self.attribute = None
foo = MyMutableObject()
foo.attribute = True
a = Data()
a.child.immutable = 42
a.child.mutable = foo
b = a.copy()
# immutable object should not change
b.child.immutable = 21
self.assertEqual(a.child.immutable, 21)
# mutable object should change
b.child.mutable.attribute = False
self.assertEqual(a.child.mutable.attribute, b.child.mutable.attribute)
# changing child of b should not affect a
b.child = "new stuff"
self.assertTrue(isinstance(a.child, Data))
def test_deepcopy(self):
class MyMutableObject(object):
def __init__(self):
self.attribute = None
foo = MyMutableObject()
foo.attribute = True
a = Data()
a.child.immutable = 42
a.child.mutable = foo
b = deepcopy(a)
# immutable object should not change
b.child.immutable = 21
self.assertEqual(a.child.immutable, 42)
# mutable object should not change
b.child.mutable.attribute = False
self.assertTrue(a.child.mutable.attribute)
# changing child of b should not affect a
b.child = "new stuff"
self.assertTrue(isinstance(a.child, Data))
def test_add_on_empty_dict(self):
d = Data()
d.x.y += 1
self.assertEqual(d.x.y, 1)
def test_add_on_non_empty_dict(self):
d = Data()
d.x.y = 'defined'
def run():
d.x += 1
self.assertRaises(TypeError, run)
def test_add_on_non_empty_value(self):
d = Data()
d.x.y = 1
d.x.y += 1
self.assertEqual(d.x.y, 2)
def test_add_on_unsupported_type(self):
d = Data()
d.x.y = 'str'
def test():
d.x.y += 1
self.assertRaises(TypeError, test)
def test_init_from_zip(self):
keys = ['a']
values = [42]
items = zip(keys, values)
d = Data(items)
self.assertEqual(d.a, 42)

23
vendor/tests/test_dots/test_fields.py поставляемый
Просмотреть файл

@ -1,23 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from mo_testing.fuzzytestcase import FuzzyTestCase
from mo_dots import relative_field
class TestFields(FuzzyTestCase):
def test_relative(self):
self.assertEqual(relative_field("testing", "testing"), ".")

115
vendor/tests/test_es_query.py поставляемый
Просмотреть файл

@ -1,115 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import unicode_literals
import unittest
from traceback import extract_tb
from mo_future import text_type
import jx_elasticsearch
from mo_files import File
from mo_json import json2value
from mo_logs import Log
from mo_logs import strings
from mo_logs.exceptions import Except, ERROR
from pyLibrary import convert
class TestFromES(unittest.TestCase):
# THE COMPLICATION WITH THIS TEST IS KNOWING IF
# THE NESTED TERMS ARE andED TOGETHER ON EACH
# NESTED DOCUMENT, OR *ANY* OF THE NESTED DOCUMENTS
# "and" IS AMBIGUOUS, AND THE CONTEXT DB JOIN (OR ES "nested")
# IS REQUIRED FOR DISAMBIGUATION.
# USUALLY I WOULD SIMPLY FORCE THE QUERY TO APPLY TO THE NESTED
# DOCUMENTS ONLY. RETURNING THE PARENT DOCUMENT IS WHAT'S
# AMBIGUOUS
def setUp(self):
self.esq=FromESTester("private_bugs")
def not_done_test1(self):
esquery = self.esq.query({
"from": "private_bugs",
"select": "*",
"where": {"and": [
{"range": {"expires_on": {"gte": 1393804800000}}},
{"range": {"modified_ts": {"lte": 1394074529000}}},
{"term": {"changes.field_name": "assigned_to"}},
{"term": {"changes.new_value": "klahnakoski"}}
]},
"limit": 10
})
expecting = {}
assert convert.value2json(esquery, pretty=True) == convert.value2json(expecting, pretty=True)
class FromESTester(object):
def __init__(self, index):
self.es = FakeES({
"host":"example.com",
"index":"index"
})
self.esq = jx_elasticsearch.new_instance(self.es)
def query(self, query):
try:
with self.esq:
self.esq.query(query)
return None
except Exception as e:
f = Except(ERROR, text_type(e), trace=extract_tb(1))
try:
details = str(f)
query = json2value(strings.between(details, ">>>>", "<<<<"))
return query
except Exception as g:
Log.error("problem", f)
class FakeES(object):
def __init__(self, settings):
self.settings = settings
pass
def search(self, query):
Log.error("<<<<\n{{query}}\n>>>>", {"query": convert.value2json(query)})
def get_schema(self):
return json2value(File("tests/resources/bug_version.json").read()).mappings.bug_version
# 4 - select None/single/list(1)/list(2)
# 4 - select aggregates/setop/aggop/count(no value)
# 5 - deep select: gparent/parent/self/child/gchild
# 3 - edges simple/deep(1)/deep(2)
# 4 - 0, 1, 2, 3 edges
# 4 - 0, 1, 2, 3 group by
# n - aggregates min/sum/count/max/etc...
# 3 - from memory/es/database sources
# where
# sort
# having
# window
#data
# 0, 1, 2, 3 properties
# 0, 1, 2, 3 children
# 0, 1, 2, 3 depth
# numeric/string/boolean values

0
vendor/tests/test_files/__init__.py поставляемый
Просмотреть файл

69
vendor/tests/test_files/speedtest_file.py поставляемый
Просмотреть файл

@ -1,69 +0,0 @@
import codecs
import io
from mo_logs import Log
from mo_times.timer import Timer
def test_simple(filename):
with Timer("simple time"):
with codecs.open(filename, "r", encoding="utf-8") as f:
for line in f:
id = int(line.split("\t")[0])
if id % 10000 == 0:
Log.note("{{id}}", id=id)
def test_buffered(filename):
with Timer("buffered time"):
with codecs.open(filename, "r", encoding="utf-8", buffering=2 ** 25) as f:
for line in f:
id = int(line.split("\t")[0])
if id % 10000 == 0:
Log.note("{{id}}", id=id)
def test_io(filename):
with Timer("io time"):
with io.open(filename, "r", buffering=2 ** 25) as f:
for line in f:
line = line.decode("utf-8")
id = int(line.split("\t")[0])
if id % 10000 == 0:
Log.note("{{id}}", id=id)
def test_binary(filename, buffering=2 ** 14):
with Timer("binary time (buffering=={{buffering}})", {"buffering": buffering}):
remainder = ""
with io.open(filename, "rb") as f:
while True:
block = f.read(buffering)
if block == "":
if remainder == "":
return None
return remainder
lines = (remainder + block).split("\n")
for line in lines[:-1]:
line = line.decode("utf-8")
id = int(line.split("\t")[0])
if id % 10000 == 0:
Log.note("{{id}}", id=id)
remainder = lines[-1]
def test_simple_binary(filename):
with Timer("simple binary time"):
with io.open(filename, "rb") as f:
for line in f:
line = line.decode("utf-8")
id = int(line.split("\t")[0])
if id % 10000 == 0:
Log.note("{{id}}", id=id)
test_file = "C:/Users/klahnakoski/git/Datazilla2ElasticSearch/results/recent_old.tab"
test_simple_binary(test_file)
test_binary(test_file, 2 ** 14)
test_binary(test_file, 2 ** 25)
test_io(test_file)
test_simple(test_file)
test_buffered(test_file)

53
vendor/tests/test_files/test_names.py поставляемый
Просмотреть файл

@ -1,53 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
import os
from mo_files import File
from mo_testing.fuzzytestcase import FuzzyTestCase
par = os.sep + ".."
class TestNames(FuzzyTestCase):
def test_relative_self(self):
f = File(".")
self.assertEqual(f.parent.filename, "..")
self.assertEqual(f.parent.parent.filename, ".."+par)
self.assertEqual(f.parent.parent.parent.filename, ".."+par+par)
def test_relative_self2(self):
f = File("")
self.assertEqual(f.parent.filename, "..")
self.assertEqual(f.parent.parent.filename, ".."+par)
self.assertEqual(f.parent.parent.parent.filename, ".."+par+par)
def test_relative_name(self):
f = File("test.txt")
self.assertEqual(f.parent.filename, "")
self.assertEqual(f.parent.parent.filename, "..")
self.assertEqual(f.parent.parent.parent.filename, ".."+par)
def test_relative_path(self):
f = File("a/test.txt")
self.assertEqual(f.parent.filename, "a")
self.assertEqual(f.parent.parent.filename, "")
self.assertEqual(f.parent.parent.parent.filename, "..")
def test_grandparent(self):
f = File.new_instance("tests/temp", "../..")
self.assertEqual(f.filename, ".")

25
vendor/tests/test_git.py поставляемый
Просмотреть файл

@ -1,25 +0,0 @@
# encoding: utf-8
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from mo_logs.strings import is_hex
from mo_testing.fuzzytestcase import FuzzyTestCase
from pyLibrary.env.git import get_git_revision
from pyLibrary.env.git import get_remote_revision
class TestGit(FuzzyTestCase):
def test_get_revision(self):
rev = get_git_revision()
self.assertTrue(is_hex(rev))
self.assertEqual(len(rev), 40)
def test_get_remote_revision(self):
rev = get_remote_revision('https://github.com/klahnakoski/pyLibrary.git', 'master')
self.assertTrue(is_hex(rev))
self.assertEqual(len(rev), 40)

0
vendor/tests/test_json/__init__.py поставляемый
Просмотреть файл

4
vendor/tests/test_json/resources/profile_speedtest_json.bat поставляемый
Просмотреть файл

@ -1,4 +0,0 @@
SET PYTHONPATH=.
pypy -m cProfile tests\speedtest_json.py
python -m cProfile tests\speedtest_json.py

4
vendor/tests/test_json/resources/speedtest_json.bat поставляемый
Просмотреть файл

@ -1,4 +0,0 @@
SET PYTHONPATH=.
pypy tests\speedtest_json.py
python tests\speedtest_json.py

144
vendor/tests/test_json/speedtest_json.py поставляемый
Просмотреть файл

@ -1,144 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
import json
import platform
import time
from decimal import Decimal
try:
import ujson
except Exception:
pass
from pyLibrary import convert
from mo_json import scrub
from mo_json.encoder import cPythonJSONEncoder, json_encoder
from mo_logs import Log
from mo_dots import wrap, unwrap
TARGET_RUNTIME = 10
EMPTY = (wrap({}), 200000)
UNICODE = (wrap(json._default_decoder.decode('{"key1": "\u0105\u0107\u017c", "key2": "\u0105\u0107\u017c"}')), 10000)
SIMPLE = (wrap({
'key1': 0,
'key2': True,
'key3': 'value',
'key4': 3.141592654,
'key5': 'string',
"key6": Decimal("0.33"),
"key7": [[], []]
}), 100000)
NESTED = (wrap({
'key1': 0,
'key2': SIMPLE[0].copy(),
'key3': 'value',
'key4': None,
'key5': SIMPLE[0].copy(),
'key6': ["test", u"test2", 99],
'key7': {1, 2.5, 3, 4},
u'key': u'\u0105\u0107\u017c'
}), 100000)
HUGE = ([NESTED[0]] * 1000, 100)
cases = [
'EMPTY',
'UNICODE',
'SIMPLE',
'NESTED',
'HUGE'
]
def test_json(results, description, method, n):
output = []
for case in cases:
try:
data, count = globals()[case]
if "scrub" in description:
#SCRUB BEFORE SENDING TO C ROUTINE (NOT FAIR, BUT WE GET TO SEE HOW FAST ENCODING GOES)
data = unwrap(scrub(data))
try:
example = method(data)
if case == "HUGE":
example = "<too big to show>"
except Exception as e:
Log.warning("json encoding failure", cause=e)
example = "<CRASH>"
t0 = time.time()
try:
for i in range(n):
for i in range(count):
output.append(method(data))
duration = time.time() - t0
except Exception:
duration = time.time() - t0
summary = {
"description": description,
"interpreter": platform.python_implementation(),
"time": duration,
"type": case,
"num": n,
"count": count,
"length": len(output),
"result": example
}
Log.note("{{interpreter}}: {{description}} {{type}} x {{num}} x {{count}} = {{time}} result={{result}}", **summary)
results.append(summary)
except Exception as e:
Log.warning("problem with encoding: {{message}}", {"message": e.message}, e)
class EnhancedJSONEncoder(json.JSONEncoder):
"""
NEEDED TO HANDLE MORE DIVERSE SET OF TYPES
PREVENTS NATIVE C VERSION FROM RUNNING
"""
def __init__(self):
json.JSONEncoder.__init__(self, sort_keys=True)
def default(self, obj):
if obj == None:
return None
elif isinstance(obj, set):
return list(obj)
elif isinstance(obj, Decimal):
return float(obj)
return json.JSONEncoder.default(self, unwrap(obj))
def main(num):
try:
Log.start()
results = []
test_json(results, "jsons.encoder", json_encoder, num)
test_json(results, "jsons.encoder (again)", json_encoder, num)
test_json(results, "scrub before json.dumps", cPythonJSONEncoder().encode, num)
test_json(results, "override JSONEncoder.default()", EnhancedJSONEncoder().encode, num)
test_json(results, "default json.dumps", json.dumps, num) # WILL CRASH, CAN NOT HANDLE DIVERSITY OF TYPES
try:
test_json(results, "scrubbed ujson", ujson.dumps, num)
except Exception:
pass
Log.note("\n{{summary}}", summary=convert.list2tab(results))
finally:
Log.stop()
if __name__ == "__main__":
main(1)

189
vendor/tests/test_json/test_json.py поставляемый
Просмотреть файл

@ -1,189 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import unicode_literals
import datetime
import unittest
from mo_dots import Data, wrap
from mo_future import text_type
from mo_json import json2value, value2json
from mo_logs import Log
from pyLibrary import convert
import mo_json
from mo_json.encoder import pretty_json, cPythonJSONEncoder
from mo_times.dates import Date
class TestJSON(unittest.TestCase):
def test_date(self):
output = value2json({"test": datetime.date(2013, 11, 13)})
Log.note("JSON = {{json}}", json= output)
def test_unicode1(self):
output = value2json({"comment": u"Open all links in the current tab, except the pages opened from external apps — open these ones in new windows"})
assert output == u'{"comment":"Open all links in the current tab, except the pages opened from external apps — open these ones in new windows"}'
if not isinstance(output, text_type):
Log.error("expecting unicode json")
def test_unicode2(self):
output = value2json({"comment": "testing accented char àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"})
assert output == u'{"comment":"testing accented char àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"}'
if not isinstance(output, text_type):
Log.error("expecting unicode json")
def test_unicode3(self):
output = value2json({"comment": u"testing accented char ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙"})
assert output == u'{"comment":"testing accented char ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙"}'
if not isinstance(output, text_type):
Log.error("expecting unicode json")
def test_double1(self):
test = {"value": 5.2025595183536973e-07}
output = value2json(test)
if output != u'{"value":5.202559518353697e-07}':
Log.error("expecting correct value")
def test_double2(self):
test = {"value": 52}
output = value2json(test)
if output != u'{"value":52}':
Log.error("expecting correct value")
def test_double3(self):
test = {"value": .52}
output = value2json(test)
if output != u'{"value":0.52}':
Log.error("expecting correct value")
def test_long1(self):
test = json2value("272757895493505930073807329622695606794392")
expecting = 272757895493505930073807329622695606794392
self.assertEqual(test, expecting)
def test_generator(self):
test = {"value": (x for x in [])}
output = value2json(test)
if output != u'{"value":[]}':
Log.error("expecting correct value")
def test_bad_key(self):
test = {24: "value"}
self.assertRaises(Exception, value2json, *[test])
def test_bad_long_json(self):
test = value2json({"values": [i for i in range(1000)]})
test = test[:1000] + "|" + test[1000:]
expected = u"Can not decode JSON at:\n\t..., 216, 217, 218, 219|, 220, 221, 222, 22...\n\t ^\n"
# expected = u'Can not decode JSON at:\n\t...9,270,271,272,273,27|4,275,276,277,278,2...\n\t ^\n'
try:
output = json2value(test)
Log.error("Expecting error")
except Exception as e:
if "Can not decode JSON" in e:
return # GOOD ENOUGH
if e.message != expected:
Log.error("Expecting good error message", cause=e)
def test_whitespace_prefix(self):
hex = "00 00 00 00 7B 22 74 68 72 65 61 64 22 3A 20 22 4D 61 69 6E 54 68 72 65 61 64 22 2C 20 22 6C 65 76 65 6C 22 3A 20 22 49 4E 46 4F 22 2C 20 22 70 69 64 22 3A 20 31 32 39 33 2C 20 22 63 6F 6D 70 6F 6E 65 6E 74 22 3A 20 22 77 70 74 73 65 72 76 65 22 2C 20 22 73 6F 75 72 63 65 22 3A 20 22 77 65 62 2D 70 6C 61 74 66 6F 72 6D 2D 74 65 73 74 73 22 2C 20 22 74 69 6D 65 22 3A 20 31 34 32 34 31 39 35 30 31 33 35 39 33 2C 20 22 61 63 74 69 6F 6E 22 3A 20 22 6C 6F 67 22 2C 20 22 6D 65 73 73 61 67 65 22 3A 20 22 53 74 61 72 74 69 6E 67 20 68 74 74 70 20 73 65 72 76 65 72 20 6F 6E 20 31 32 37 2E 30 2E 30 2E 31 3A 38 34 34 33 22 7D 0A"
json = convert.utf82unicode(convert.hex2bytes("".join(hex.split(" "))))
self.assertRaises(Exception, json2value, *[json])
def test_default_python(self):
test = {"add": Data(start="".join([" ", u"â€"]))}
output = value2json(test)
expecting = u'{"add":{"start":" â€"}}'
self.assertEqual(expecting, output, "expecting correct json")
def test_false(self):
test = value2json(wrap({"value": False}))
expecting = u'{"value":false}'
self.assertEqual(test, expecting, "expecting False to serialize as 'false'")
def test_empty_dict(self):
test = value2json(wrap({"match_all": wrap({})}))
expecting = u'{"match_all":{}}'
self.assertEqual(test, expecting, "expecting empty dict to serialize")
def test_empty_list1(self):
test = value2json(wrap({"a": []}))
expecting = u'{"a":[]}'
self.assertEqual(test, expecting, "expecting empty list to serialize")
def test_empty_list2(self):
test = value2json(wrap({"a": [], "b": 1}))
expecting = u'{"a":[],"b":1}'
self.assertEqual(test, expecting, "expecting empty list to serialize")
def test_deep_empty_dict(self):
test = value2json(wrap({"query": {"match_all": {}}, "size": 20000}))
expecting = u'{"query":{"match_all":{}},"size":20000}'
self.assertEqual(test, expecting, "expecting empty dict to serialize")
def test_pretty_json(self):
j = wrap({"not": {"match_all": wrap({})}})
test = pretty_json(j)
expecting = u'{"not": {"match_all": {}}}'
self.assertEqual(test, expecting, "expecting empty dict to serialize")
def test_Date(self):
test = Date(1430983248.0)
output = value2json(test)
expecting = '1430983248'
self.assertEqual(output, expecting, "expecting integer")
def test_float(self):
test = float(10.0)
output = value2json(test)
expecting = '10'
self.assertEqual(output, expecting, "expecting integer")
def test_nan(self):
test = float("nan")
output = value2json(test)
expecting = cPythonJSONEncoder().encode(mo_json.scrub(test))
self.assertEqual(output, expecting, "expecting " + expecting)
def test_inf(self):
test = float("+inf")
output = value2json(test)
expecting = cPythonJSONEncoder().encode(mo_json.scrub(test))
self.assertEqual(output, expecting, "expecting " + expecting)
def test_minus_inf(self):
test = float("-inf")
output = value2json(test)
expecting = cPythonJSONEncoder().encode(mo_json.scrub(test))
self.assertEqual(output, expecting, "expecting " + expecting)
def test_string_stripper(self):
test = {"hello": " world"}
mo_json.FIND_LOOPS = True
self.assertEqual(value2json(test), '{"hello":" world"}')
def test_json_is_unicode(self):
self.assertIsInstance(value2json({}), text_type)
def test_json_encode_slash(self):
self.assertEqual(value2json("/"), '"/"')
if __name__ == '__main__':
try:
Log.start()
unittest.main()
finally:
Log.stop()

344
vendor/tests/test_json/test_json_stream.py поставляемый
Просмотреть файл

@ -1,344 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import absolute_import
from __future__ import division
from __future__ import unicode_literals
from io import BytesIO
from mo_future import text_type
from mo_dots import Null
from mo_testing.fuzzytestcase import FuzzyTestCase
from mo_json import stream
class TestJsonStream(FuzzyTestCase):
def test_select_from_list(self):
json = slow_stream('{"1":[{"a":"b"}]}')
result = list(stream.parse(json, "1", ["1.a"]))
expected = [{"1":{"a": "b"}}]
self.assertEqual(result, expected)
def test_select_nothing_from_many_list(self):
json = slow_stream('{"1":[{"a":"b"}, {"a":"c"}]}')
result = list(stream.parse(json, "1"))
expected = [
{},
{}
]
self.assertEqual(result, expected)
def test_select_from_many_list(self):
json = slow_stream('{"1":[{"a":"b"}, {"a":"c"}]}')
result = list(stream.parse(json, "1", ["1.a"]))
expected = [
{"1": {"a": "b"}},
{"1": {"a": "c"}}
]
self.assertEqual(result, expected)
def test_select_from_diverse_list(self):
json = slow_stream('{"1":["test", {"a":"c"}]}')
result = list(stream.parse(json, "1", ["1.a"]))
expected = [
{"1": {}},
{"1": {"a": "c"}}
]
self.assertEqual(result[0]["1"], None)
self.assertEqual(result, expected)
def test_select_from_deep_many_list(self):
# 0123456789012345678901234567890123
json = slow_stream('{"1":{"2":[{"a":"b"}, {"a":"c"}]}}')
result = list(stream.parse(json, "1.2", ["1.2.a"]))
expected = [
{"1": {"2": {"a": "b"}}},
{"1": {"2": {"a": "c"}}}
]
self.assertEqual(result, expected)
def test_post_properties_error(self):
json = slow_stream('{"0":"v", "1":[{"a":"b"}, {"a":"c"}], "2":[{"a":"d"}, {"a":"e"}]}')
def test():
result = list(stream.parse(json, "1", ["0", "1.a", "2"]))
self.assertRaises(Exception, test)
def test_select_objects(self):
json = slow_stream('{"b":[{"a":1, "p":{"b":2, "c":{"d":3}}}, {"a":4, "p":{"b":5, "c":{"d":6}}}]}')
result = list(stream.parse(json, "b", ["b.a", "b.p.c"]))
expected = [
{"b": {"a": 1, "p": {"c": {"d": 3}}}},
{"b": {"a": 4, "p": {"c": {"d": 6}}}}
]
self.assertEqual(result, expected)
def test_select_all(self):
json = slow_stream('{"b":[{"a":1, "p":{"b":2, "c":{"d":3}}}, {"a":4, "p":{"b":5, "c":{"d":6}}}]}')
result = list(stream.parse(json, "b", ["b"]))
expected = [
{"b": {"a": 1, "p": {"b": 2, "c": {"d": 3}}}},
{"b": {"a": 4, "p": {"b": 5, "c": {"d": 6}}}}
]
self.assertEqual(result, expected)
def test_big_baddy(self):
source = """
{
"builds": [
{
"builder_id": 367155,
"buildnumber": 460,
"endtime": 1444699317,
"id": 77269739,
"master_id": 161,
"properties": {
"appName": "Firefox",
"appVersion": "44.0a1",
"basedir": "/c/builds/moz2_slave/m-in-w64-pgo-00000000000000000",
"branch": "mozilla-inbound",
"buildername": "WINNT 6.1 x86-64 mozilla-inbound pgo-build",
"buildid": "20151012133004",
"buildnumber": 460,
"builduid": "2794c8ed62f642aeae5cd3f6cd72bdfd",
"got_revision": "001f7d3139ce",
"jsshellUrl": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/jsshell-win64.zip",
"log_url": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/mozilla-inbound-win64-pgo-bm84-build1-build460.txt.gz",
"master": "http://buildbot-master84.bb.releng.scl3.mozilla.com:8001/",
"packageFilename": "firefox-44.0a1.en-US.win64.zip",
"packageUrl": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/firefox-44.0a1.en-US.win64.zip",
"platform": "win64",
"product": "firefox",
"project": "",
"repo_path": "integration/mozilla-inbound",
"repository": "",
"request_ids": [
83875568
],
"request_times": {
"83875568": 1444681804
},
"revision": "001f7d3139ce06e63075cb46bc4c6cbb607e4be4",
"scheduler": "mozilla-inbound periodic",
"script_repo_revision": "production",
"script_repo_url": "https://hg.mozilla.org/build/mozharness",
"slavename": "b-2008-ix-0099",
"sourcestamp": "001f7d3139ce06e63075cb46bc4c6cbb607e4be4",
"stage_platform": "win64-pgo",
"symbolsUrl": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/firefox-44.0a1.en-US.win64.crashreporter-symbols.zip",
"testPackagesUrl": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/test_packages.json",
"testsUrl": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/firefox-44.0a1.en-US.win64.web-platform.tests.zip",
"toolsdir": "/c/builds/moz2_slave/m-in-w64-pgo-00000000000000000/scripts",
"uploadFiles": "[u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\install\\\\sea\\\\firefox-44.0a1.en-US.win64.installer.exe', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\win64\\\\xpi\\\\firefox-44.0a1.en-US.langpack.xpi', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\mozharness.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.common.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.cppunittest.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.xpcshell.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.mochitest.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.talos.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.reftest.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.web-platform.tests.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.crashreporter-symbols.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.txt', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.json', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.mozinfo.json', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\test_packages.json', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\jsshell-win64.zip', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\host\\\\bin\\\\mar.exe', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\host\\\\bin\\\\mbsdiff.exe', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.checksums', u'c:\\\\builds\\\\moz2_slave\\\\m-in-w64-pgo-00000000000000000\\\\build\\\\src\\\\obj-firefox\\\\dist\\\\firefox-44.0a1.en-US.win64.checksums.asc']"
},
"reason": "The Nightly scheduler named 'mozilla-inbound periodic' triggered this build",
"request_ids": [
83875568
],
"requesttime": 1444681804,
"result": 0,
"slave_id": 8812,
"starttime": 1444681806
}
]}
"""
json = slow_stream(source)
expected = [{"builds": {
"requesttime": 1444681804,
"starttime": 1444681806,
"endtime": 1444699317,
"reason": "The Nightly scheduler named 'mozilla-inbound periodic' triggered this build",
"properties": {
"request_times": {
"83875568": 1444681804
},
"slavename": "b-2008-ix-0099",
"log_url": "http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win64-pgo/1444681804/mozilla-inbound-win64-pgo-bm84-build1-build460.txt.gz",
"buildername": "WINNT 6.1 x86-64 mozilla-inbound pgo-build"
}
}}]
result = list(stream.parse(
json,
"builds",
[
"builds.starttime",
"builds.endtime",
"builds.requesttime",
"builds.reason",
"builds.properties.request_times",
"builds.properties.slavename",
"builds.properties.log_url",
"builds.properties.buildername"
]
))
self.assertEqual(result, expected)
def test_constants(self):
# 01234567890123456789012345678901234567890123456789012345678901234567890123456789
json = slow_stream(u'[true, false, null, 42, 3.14, "hello world", "àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"]')
result = list(stream.parse(json, None, ["."]))
expected = [True, False, None, 42, 3.14, u"hello world", u"àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"]
self.assertEqual(result, expected)
def test_object_items(self):
json = slow_stream('{"a": 1, "b": 2, "c": 3}')
result = list(stream.parse(json, {"items": "."}, expected_vars={"name", "value"}))
expected = [
{"name": "a", "value": 1},
{"name": "b", "value": 2},
{"name": "c", "value": 3}
]
self.assertEqual(result, expected)
def test_nested_primitives(self):
json = slow_stream('{"u": "a", "t": [1, 2, 3]}')
result = list(stream.parse(json, "t", expected_vars={"t", "u"}))
expected = [
{"u": "a", "t": 1},
{"u": "a", "t": 2},
{"u": "a", "t": 3}
]
self.assertEqual(result, expected)
def test_select_no_items(self):
json = slow_stream('{"a": 1, "b": 2, "c": 3}')
result = list(stream.parse(json, {"items": "."}, expected_vars={}))
expected = [
{},
{},
{}
]
self.assertEqual(result, expected)
def test_array_object_items(self):
json = slow_stream('[{"a": 1}, {"b": 2}, {"c": 3}]')
result = list(stream.parse(json, {"items": "."}, expected_vars={"name", "value"}))
expected = [
{"name": "a", "value": 1},
{"name": "b", "value": 2},
{"name": "c", "value": 3}
]
self.assertEqual(result, expected)
def test_nested_items(self):
json = slow_stream('{"u": "a", "t": [{"a": 1}, {"b": 2}, {"c": 3}]}')
result = list(stream.parse(json, {"items": "t"}, expected_vars={"u", "t.name", "t.value"}))
expected = [
{"u": "a", "t": {"name": "a", "value": 1}},
{"u": "a", "t": {"name": "b", "value": 2}},
{"u": "a", "t": {"name": "c", "value": 3}}
]
self.assertEqual(result, expected)
def test_empty(self):
json = slow_stream('{"u": "a", "t": []}')
result = list(stream.parse(json, "t", expected_vars={"u", "t"}))
expected = [
{"u": "a", "t": Null}
]
self.assertEqual(result, expected)
def test_miss_item_in_list(self):
json = slow_stream('{"u": "a", "t": ["k", null, "m"]}')
result = list(stream.parse(json, "t", expected_vars={"u", "t"}))
expected = [
{"u": "a", "t": "k"},
{"u": "a", "t": Null},
{"u": "a", "t": "m"}
]
self.assertEqual(result, expected)
def test_ignore_elements_of_list(self):
json = slow_stream('{"u": "a", "t": ["k", null, "m"]}')
result = list(stream.parse(json, "t", expected_vars={"u"}))
expected = [
{"u": "a"},
{"u": "a"},
{"u": "a"}
]
self.assertEqual(result, expected)
def test_bad_item_in_list(self):
json = slow_stream('{"u": "a", "t": ["k", None, "m"]}')
def output():
list(stream.parse(json, "t", expected_vars={"u", "t"}))
self.assertRaises(Exception, output)
def test_object_instead_of_list(self):
json = slow_stream('{"u": "a", "t": "k"}')
result = list(stream.parse(json, "t", expected_vars={"u", "t"}))
expected = [
{"u": "a", "t": "k"}
]
self.assertEqual(result, expected)
def test_simple(self):
json = slow_stream('{"u": "a"}')
result = list(stream.parse(json, ".", expected_vars={"."}))
expected = [
{"u": "a"}
]
self.assertEqual(result, expected)
def test_missing_array(self):
json = slow_stream('{"u": "a"}')
result = list(stream.parse(json, "t", expected_vars={"u"}))
expected = [
{"u": "a"}
]
self.assertEqual(result, expected)
def test_not_used_array(self):
json = slow_stream('{"u": "a", "t": ["k", null, "m"]}')
def output():
list(stream.parse(json, "t", expected_vars={"."}))
self.assertRaises(Exception, output)
def test_nested_items_w_error(self):
json = slow_stream('{"u": "a", "t": [{"a": 1}, {"b": 2}, {"c": 3}], "v":3}')
def test():
result = list(stream.parse(json, {"items": "t"}, expected_vars={"u", "t.name", "v"}))
self.assertRaises(Exception, test)
def test_values_are_arrays(self):
json = slow_stream('{"AUTHORS": ["mozilla.org", "Licensing"], "CLOBBER": ["Core", "Build Config"]}')
result = list(stream.parse(json, {"items": "."}, expected_vars={"name", "value"}))
expected = [
{"name": "AUTHORS", "value": ["mozilla.org", "Licensing"]},
{"name": "CLOBBER", "value": ["Core", "Build Config"]}
]
self.assertEqual(result, expected)
def slow_stream(bytes):
if isinstance(bytes, text_type):
bytes = bytes.encode("utf8")
r = BytesIO(bytes).read
def output():
return r(1)
return output

190
vendor/tests/test_json/test_pypy_json.py поставляемый
Просмотреть файл

@ -1,190 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
from __future__ import unicode_literals
import datetime
import unittest
from mo_dots import Data, wrap
from mo_future import text_type
from mo_json import json2value
from mo_logs import Log
from pyLibrary import convert
import mo_json
from mo_json.encoder import pypy_json_encode, cPythonJSONEncoder, pretty_json
from mo_times.dates import Date
def value2json(value):
return pypy_json_encode(value)
class TestPyPyJSON(unittest.TestCase):
def test_date(self):
output = value2json({"test": datetime.date(2013, 11, 13)})
Log.note("JSON = {{json}}", json= output)
def test_unicode1(self):
output = value2json({"comment": u"Open all links in the current tab, except the pages opened from external apps — open these ones in new windows"})
assert output == u'{"comment":"Open all links in the current tab, except the pages opened from external apps — open these ones in new windows"}'
if not isinstance(output, text_type):
Log.error("expecting unicode json")
def test_unicode2(self):
output = value2json({"comment": "testing accented char àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"})
assert output == u'{"comment":"testing accented char àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"}'
if not isinstance(output, text_type):
Log.error("expecting text_type json")
def test_unicode3(self):
output = value2json({"comment": u"testing accented char ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙"})
assert output == u'{"comment":"testing accented char ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙"}'
if not isinstance(output, text_type):
Log.error("expecting unicode json")
def test_double1(self):
test = {"value":5.2025595183536973e-07}
output = value2json(test)
if output != u'{"value":5.202559518353697e-7}':
Log.error("expecting correct value")
def test_double2(self):
test = {"value": 52}
output = value2json(test)
if output != u'{"value":52}':
Log.error("expecting correct value")
def test_double3(self):
test = {"value": .52}
output = value2json(test)
if output != u'{"value":0.52}':
Log.error("expecting correct value")
def test_generator(self):
test = {"value": (x for x in [])}
output = value2json(test)
if output != u'{"value":[]}':
Log.error("expecting correct value")
def test_bad_key(self):
test = {24: "value"}
self.assertRaises(Exception, value2json, *[test])
def test_bad_long_json(self):
test = value2json({"values": [i for i in range(1000)]})
test = test[:1000] + "|" + test[1000:]
expected = u"Can not decode JSON at:\n\t..., 216, 217, 218, 219|, 220, 221, 222, 22...\n\t ^\n"
# expected = u'Can not decode JSON at:\n\t...9,270,271,272,273,27|4,275,276,277,278,2...\n\t ^\n'
try:
output = json2value(test)
Log.error("Expecting error")
except Exception as e:
if "Can not decode JSON" in e:
return # GOOD ENOUGH
if e.message != expected:
Log.error("Expecting good error message", cause=e)
def test_whitespace_prefix(self):
hex = "00 00 00 00 7B 22 74 68 72 65 61 64 22 3A 20 22 4D 61 69 6E 54 68 72 65 61 64 22 2C 20 22 6C 65 76 65 6C 22 3A 20 22 49 4E 46 4F 22 2C 20 22 70 69 64 22 3A 20 31 32 39 33 2C 20 22 63 6F 6D 70 6F 6E 65 6E 74 22 3A 20 22 77 70 74 73 65 72 76 65 22 2C 20 22 73 6F 75 72 63 65 22 3A 20 22 77 65 62 2D 70 6C 61 74 66 6F 72 6D 2D 74 65 73 74 73 22 2C 20 22 74 69 6D 65 22 3A 20 31 34 32 34 31 39 35 30 31 33 35 39 33 2C 20 22 61 63 74 69 6F 6E 22 3A 20 22 6C 6F 67 22 2C 20 22 6D 65 73 73 61 67 65 22 3A 20 22 53 74 61 72 74 69 6E 67 20 68 74 74 70 20 73 65 72 76 65 72 20 6F 6E 20 31 32 37 2E 30 2E 30 2E 31 3A 38 34 34 33 22 7D 0A"
json = convert.utf82unicode(convert.hex2bytes("".join(hex.split(" "))))
self.assertRaises(Exception, json2value, *[json])
def test_default_python(self):
test = {"add": Data(start="".join([" ", "â€"]))}
output = value2json(test)
expecting = u'{"add":{"start":" â€"}}'
self.assertEqual(expecting, output, "expecting correct json")
def test_false(self):
test = value2json(wrap({"value": False}))
expecting = u'{"value":false}'
self.assertEqual(test, expecting, "expecting False to serialize as 'false'")
def test_empty_dict(self):
test = value2json(wrap({"match_all": wrap({})}))
expecting = u'{"match_all":{}}'
self.assertEqual(test, expecting, "expecting empty dict to serialize")
def test_empty_list1(self):
test = value2json(wrap({"a": []}))
expecting = u'{"a":[]}'
self.assertEqual(test, expecting, "expecting empty list to serialize")
def test_empty_list2(self):
test = value2json(wrap({"a": [], "b": 1}))
expecting = u'{"a":[],"b":1}'
self.assertEqual(test, expecting, "expecting empty list to serialize")
def test_deep_empty_dict(self):
test = value2json(wrap({"query": {"match_all": {}}, "size": 20000}))
expecting = u'{"query":{"match_all":{}},"size":20000}'
self.assertEqual(test, expecting, "expecting empty dict to serialize")
def test_pretty_json(self):
j = wrap({"not": {"match_all": wrap({})}})
test = pretty_json(j)
expecting = u'{"not": {"match_all": {}}}'
self.assertEqual(test, expecting, "expecting empty dict to serialize")
def test_Date(self):
test = Date(1430983248.0)
output = value2json(test)
expecting = '1430983248'
self.assertEqual(output, expecting, "expecting integer")
def test_float(self):
test = float(10.0)
output = value2json(test)
expecting = '10'
self.assertEqual(output, expecting, "expecting integer")
def test_nan(self):
test = float("nan")
output = value2json(test)
expecting = cPythonJSONEncoder().encode(mo_json.scrub(test))
self.assertEqual(output, expecting, "expecting " + expecting)
def test_inf(self):
test = float("+inf")
output = value2json(test)
expecting = cPythonJSONEncoder().encode(mo_json.scrub(test))
self.assertEqual(output, expecting, "expecting " + expecting)
def test_minus_inf(self):
test = float("-inf")
output = value2json(test)
expecting = cPythonJSONEncoder().encode(mo_json.scrub(test))
self.assertEqual(output, expecting, "expecting " + expecting)
def test_string_stripper(self):
test = {"hello": " world"}
mo_json.FIND_LOOPS = True
self.assertEqual(value2json(test), '{"hello":" world"}')
def test_json_is_unicode(self):
self.assertIsInstance(value2json({}), text_type)
def test_json_encode_slash(self):
self.assertEqual(value2json("/"), '"/"')
if __name__ == '__main__':
try:
Log.start()
unittest.main()
finally:
Log.stop()

160
vendor/tests/test_json/test_typed_json.py поставляемый
Просмотреть файл

@ -1,160 +0,0 @@
# encoding: utf-8
#
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Author: Kyle Lahnakoski (kyle@lahnakoski.com)
#
import datetime
import unittest
from mo_dots import wrap
from mo_json.typed_encoder import EXISTS_TYPE, NUMBER_TYPE, STRING_TYPE, BOOLEAN_TYPE, NESTED_TYPE
from mo_logs.strings import quote
from pyLibrary.env.typed_inserter import TypedInserter
def typed_encode(value):
return TypedInserter().typed_encode({"value": value})['json']
class TestJSON(unittest.TestCase):
def test_date(self):
value = {"test": datetime.date(2013, 11, 13)}
test1 = typed_encode(value)
expected = u'{"test":{' + quote(NUMBER_TYPE) + u':1384318800},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_unicode1(self):
value = {"comment": u"Open all links in the current tab, except the pages opened from external apps — open these ones in new windows"}
test1 = typed_encode(value)
expected = u'{"comment":{' + quote(STRING_TYPE) + u':"Open all links in the current tab, except the pages opened from external apps — open these ones in new windows"},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_unicode2(self):
value = {"comment": "testing accented char àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"}
test1 = typed_encode(value)
expected = u'{"comment":{' + quote(STRING_TYPE) + u':"testing accented char àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ"},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_unicode3(self):
value = {"comment": u"testing accented char ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙"}
test1 = typed_encode(value)
expected = u'{"comment":{' + quote(STRING_TYPE) + u':"testing accented char ŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙"},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_double(self):
value = {"value": 5.2025595183536973e-07}
test1 = typed_encode(value)
expected = u'{"value":{' + quote(NUMBER_TYPE) + u':5.202559518353697e-7},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_empty_list(self):
value = {"value": []}
test1 = typed_encode(value)
expected = u'{"value":{' + quote(NESTED_TYPE) + u':[]},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_nested(self):
value = {"a": {}, "b": {}}
test1 = typed_encode(value)
expected = u'{"a":{' + quote(EXISTS_TYPE) + u':0},"b":{' + quote(EXISTS_TYPE) + u':0},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_list_of_objects(self):
value = {"a": [{}, "b"]}
test1 = typed_encode(value)
expected = u'{"a":{"'+NESTED_TYPE+u'":[{' + quote(EXISTS_TYPE) + u':0},{' + quote(STRING_TYPE) + u':"b"}],' + quote(EXISTS_TYPE) + u':2},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_empty_list_value(self):
value = []
test1 = typed_encode(value)
expected = u'{'+quote(NESTED_TYPE)+':[]}'
self.assertEqual(test1, expected)
def test_list_value(self):
value = [42]
test1 = typed_encode(value)
expected = u'{' + quote(NUMBER_TYPE) + u':42}'
self.assertEqual(test1, expected)
def test_list(self):
value = {"value": [23, 42]}
test1 = typed_encode(value)
expected = u'{"value":{' + quote(NUMBER_TYPE) + u':[23,42]},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_number_value(self):
value = 42
test1 = typed_encode(value)
expected = '{' + quote(NUMBER_TYPE) + u':42}'
self.assertEqual(test1, expected)
def test_empty_string_value(self):
value = u""
test1 = typed_encode(value)
expected = '{' + quote(STRING_TYPE) + u':""}'
self.assertEqual(test1, expected)
def test_string_value(self):
value = u"42"
test1 = typed_encode(value)
expected = '{' + quote(STRING_TYPE) + u':"42"}'
self.assertEqual(test1, expected)
def test_escaped_string_value(self):
value = "\""
test1 = typed_encode(value)
expected = '{' + quote(STRING_TYPE) + u':"\\""}'
self.assertEqual(test1, expected)
def test_bad_key(self):
test = {24: "value"}
self.assertRaises(Exception, typed_encode, *[test])
def test_false(self):
value = False
test1 = typed_encode(value)
expected = '{' + quote(BOOLEAN_TYPE) + u':false}'
self.assertEqual(test1, expected)
def test_true(self):
value = True
test1 = typed_encode(value)
expected = '{' + quote(BOOLEAN_TYPE) + u':true}'
self.assertEqual(test1, expected)
def test_null(self):
def encode_null():
value = None
typed_encode(value)
self.assertRaises(Exception, encode_null)
def test_empty_dict(self):
value = wrap({"match_all": wrap({})})
test1 = typed_encode(value)
expected = u'{"match_all":{' + quote(EXISTS_TYPE) + u':0},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_complex_object(self):
value = wrap({"s": 0, "r": 5})
test1 = typed_encode(value)
expected = u'{"r":{' + quote(NUMBER_TYPE) + u':5},"s":{' + quote(NUMBER_TYPE) + u':0},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_empty_list1(self):
value = wrap({"a": []})
test1 = typed_encode(value)
expected = u'{"a":{' + quote(NESTED_TYPE) + u':[]},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)
def test_empty_list2(self):
value = wrap({"a": [], "b": 1})
test1 = typed_encode(value)
expected = u'{"a":{' + quote(NESTED_TYPE) + ':[]},"b":{' + quote(NUMBER_TYPE) + u':1},' + quote(EXISTS_TYPE) + u':1}'
self.assertEqual(test1, expected)

Некоторые файлы не были показаны из-за слишком большого количества измененных файлов Показать больше