Merge pull request #5 from pkgw/next-round

Next round of improvements
This commit is contained in:
Peter Williams 2019-08-03 15:12:21 -04:00 коммит произвёл GitHub
Родитель 135591fe78 1678e6b7c8
Коммит a5b13ff19d
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
9 изменённых файлов: 346 добавлений и 151 удалений

Просмотреть файл

@ -1,4 +1,4 @@
# 0.1.0dev0 (unreleased)
# 0.1.0.dev0 (unreleased)
- No changes yet

172
README.md
Просмотреть файл

@ -6,12 +6,11 @@
<!--pypi-begin-->
[toasty] is a Python library that helps you create [TOAST] “tile pyramids”
from [FITS] astronomical image files. These multi-resolution all-sky maps can
be viewed in software such as the [AAS] [WorldWide Telescope].
from astronomical image data. These multi-resolution all-sky maps can be
viewed in software such as the [AAS] [WorldWide Telescope].
[toasty]: https://toasty.readthedocs.io/
[TOAST]: https://doi.org/10.3847/1538-4365/aaf79e
[FITS]: https://en.wikipedia.org/wiki/FITS
[AAS]: https://aas.org/
[WorldWide Telescope]: http://www.worldwidetelescope.org/
@ -30,7 +29,37 @@ The easiest way to install [toasty] is through [pip]:
pip install toasty
```
[pip]: https://pypi.org/project/pip/
[pip]: https://pip.pypa.io/
For more information, please see the full [toasty installation instructions].
[toasty installation instructions]: https://toasty.readthedocs.io/en/latest/installation.html
## Documentation, Examples, etc.
For documentation and examples, go to:
https://toasty.readthedocs.io/
## Contributions
Contributions to [toasty] are welcome! See
[the WorldWide Telescope contributors guide] for applicable information. We
use a standard workflow with issues and pull requests. All participants in
[toasty] and the WWT communities must abide by the [WWT Code of Conduct].
[the WorldWide Telescope contributors guide]: https://worldwidetelescope.github.io/contributing/
[WWT Code of Conduct]: https://worldwidetelescope.github.io/code-of-conduct/
## Release History
Releases of [toasty] are logged in the file [CHANGES.md](./CHANGES.md), as
well as release listings maintained by
[GitHub](https://github.com/WorldWideTelescope/toasty/releases) and
[PyPI](https://pypi.org/project/toasty/#history).
## Dependencies
@ -53,137 +82,12 @@ pip install toasty
[pytest]: https://docs.pytest.org/
### Usage
## Legalities
```python
from toasty import toast
toast(data_sampler, depth, base_dir, wtml_file=None, merge=True, base_level_only=False,
ra_range=None, dec_range=None, toast_tile=None, restart=False, top_layer=0)
```
where:
* **data_sampler** function or string
- A function of (lon, lat) that samples a datasetat the input 2D coordinate arrays
- A string giving a base toast directory that contains the base level of toasted tiles, using this option, only the merge step takes place, the given directory must contain a "depth" directory for the given depth parameter
* **depth** int
- The depth of the tile pyramid to create (4^d tiles are created at a depth of d)
* **base_dir** str
- The path to create the files at
* **wtml_file** str (optional)
- The path to write a WTML file to. If not present, no file will be written
* **merge** bool or callable (default True)
How to treat lower resolution tiles.
- If True, tiles above the lowest level (highest resolution) will be computed by averaging and downsampling the 4 subtiles.
- If False, sampler will be called explicitly for all tiles
- If a callable object, this object will be passed the 4x oversampled image to downsample
* **base_level_only** bool (default False)
If True only the bottem level of tiles will be created.
In this case merge will be set to True, but no merging will happen, and only the highest resolution layer of images will be created.
* **ra_range** array (optional)
* **dec_range** array (optional)
To toast only a portion of the sky give min and max ras and decs ([minRA,maxRA],[minDec,maxDec]) in degrees.
If these keywords are used base_level_only will be automatically set to true, regardless of its given value.
* **toast_tile** array\[n,x,y\] (optional)
If this keyword is used the output will be all the subtiles of toast_tile at the given depth (base_level_only will be automatically set to true, regardless of its given value.
* **top_layer** int (optional)
If merging this indicates the uppermost layer to be created.
[toasty] is copyright Chris Beaumont, Clara Brasseur, and the AAS WorldWide
Telescope Team. It is licensed under the MIT License.
## Acknowledgments
Toasty provides a few basic sampler functions:
* **healpix_sampler** for sampling from healpix arrays
* **cartesian_sampler** for sampling from cartesian-projections
* **normalizer** for applying an intensity normalization after sampling
### Examples
To toast an all-sky, Cartesian projection, 8 byte image:
```python
from toasty import toast, cartesian_sampler
from skimage.io import imread
data = imread('allsky.png')
sampler = cartesian_sampler(data)
toastDirectory = 'toast'
depth = 8 # approximately 0.165"/pixel at highest resolution
# Getting the full toast tile set for the entire sky
toast(sampler, depth, toastDirectory)
# Toasting a specific region of the sky defined by ra/dec bounds
raRange = [208.8,212.2]
decRange = [52.5,56.8]
toast(sampler, depth, outtoastDirectoryput, ra_range=raRange, dec_range=decRange)
# Toasting a specific region of the sky defined by a higher level toast tile
tile=[4,5,9]
toast(sampler, depth, toastDirectory, toast_tile=tile)
# Creating only the bottom layer of toast tiles
toast(sampler, depth, toastDirectory,base_level_only=True)
# Merging from a previously created toast layer up to a specified top layer
topLayer = 4
toast(toastDirectory, depth, toastDirectory, top_layer=topLayer)
```
To apply a log-stretch to an all sky FITS image:
```python
from toasty import toast, cartesian_sampler, normalizer
from astropy.io import fits
data = fits.open('allsky.fits')[0].data
vmin, vmax = 100, 65535
scaling = 'log'
contrast = 1
sampler = normalizer(cartesian_sampler(data), vmin, vmax
scaling, bias, contrast)
output = 'toast'
depth = 8
toast(sampler, depth, output)
```
To perform a custom transformation
```python
from toasty import toast
from astropy.io import fits
data = fits.open('allsky.fits')[0].data
def sampler(x, y):
"""
x and y are arrays, giving the RA/Dec centers
(in radians) for each pixel to extract
"""
... code to extract a tile from `data` here ...
output = 'toast'
depth = 8
toast(sampler, depth, output)
```
See ``toasty.tile`` for documentation on these functions.
### Using with WorldWide Telescope
To quickly preview a toast directory named `test`, navigate to the directory
where `test` exists and run
```
python -m toasty.viewer test
```
This will start a web server, probably at [http://0.0.0.0:8000](http://0.0.0:8000) (check the output for the actual address). Open this URL in a browser to get a quick look at the data.
For more information about using WorldWide Telescope with custom image data,
see [the official documentation](http://www.worldwidetelescope.org/Docs/worldwidetelescopedatafilesreference.html). The function `toasty.gen_wtml` can generate the wtml information for images generated with toasty.
For an example of tiles generated with Toasty, see [The ADS All Sky Survey](http://adsass.org/wwt). The code used to generate these images is [here](https://github.com/ChrisBeaumont/adsass/blob/master/toast/toast.py).
For an example of tiles generated using the Toasty functionality specific to this fork see [The STScI AstroView Panstarrs Survey](https://mast.stsci.edu/portal/Mashup/Clients/AstroView/AstroView.html?debug&avSurveyType=PANSTARRS). The code used to generate these images is [here](https://github.com/ceb8/toastPanstarrs).
Work on [toasty] is funded through the [AAS] [WorldWide Telescope] project.

Просмотреть файл

@ -9,15 +9,16 @@ These are notes for the `toasty` developers about how to create a new release.
5. For the final commit, update the version number in `setup.py` and
`docs/conf.py`, and add a proper version and date to `CHANGES.md`. Commit
with message `Release version ${version}`.
6. Create a pull request on GitHub for the new release.
6. Push to GitHub and create a pull request for the new release called
"Release PR for version $version".
7. Get it so that it passes CI, creating fixup commits as necessary.
8. When it's really really ready, `git clean -fxd && python setup.py sdist
bdist_wheel --universal && twine upload dist/*.tar.gz dist/*.whl`. If
`twine` finds problems, make any final changes and retry.
8. When it's really really ready, `git clean -fxd && python setup.py sdist &&
twine upload dist/*.tar.gz`. If `twine` finds problems, make any final
changes and retry.
9. If needed, do a `git rebase -i` to make the version-bump commit the last
one again.
10. `git tag v${version}`
11. Update the version number to `${cur_major}.${next_minor}.0dev0` and add a
11. Update the version number to `${cur_major}.${next_minor}.0.dev0` and add a
new separator in `CHANGES.md` along the lines of `${version} (unreleased)`.
Commit with a message of `Back to development.`
12. `git push` (with `-f` if history was rewritten) to the PR branch. This had

Просмотреть файл

@ -4,8 +4,8 @@ project = 'toasty'
author = 'Chris Beaumont and the AAS WorldWide Telescope Team'
copyright = '2014-2019, ' + author
release = '0.1.0dev0' # the full version string; also update ../setup.py
version = '0.1.0dev0' # the "short" version
release = '0.1.0.dev0' # the full version string; also update ../setup.py
version = '0.1.0.dev0' # the "short" version
extensions = [
'sphinx.ext.autodoc',

137
docs/examples.rst Normal file
Просмотреть файл

@ -0,0 +1,137 @@
=============================
Examples of How to Use Toasty
=============================
Here well summarize some of the ways that you can use toasty_.
.. _toasty: https://toasty.readthedocs.io/
Toasting a Cartesian, all-sky PNG image
=======================================
This script processes an existing all-sky PNG image that uses a Cartesian
projection, using scikit-image_ to load the data::
from toasty import toast, cartesian_sampler
from skimage.io import imread
data = imread('allsky.png')
sampler = cartesian_sampler(data)
output_directory = 'toast'
depth = 8 # approximately 0.165"/pixel at highest resolution
# Make the full toast tile set for the entire sky:
toast(sampler, depth, output_directory)
.. _scikit-image: https://scikit-image.org/
Toasting subsets of the sky
===========================
You dont have to build the full pyramid for the full sky all at once::
# Toast a specific region of the sky defined by RA/Dec bounds
toast(sampler, depth, output_directory,
ra_range = [208.8, 212.2], # degrees
dec_range = [52.5, 56.8], # degrees
)
# Toast a specific region of the sky defined by a higher-level TOAST tile
toast(sampler, depth, output_directory,
toast_tile = [4, 5, 9],
)
# Create only the bottom layer of toast tiles
toast(sampler, depth, output_directory,
base_level_only = True,
)
# Merge from a previously created toast layer up to a specified layer
toast(sampler, depth, output_directory,
top_layer = 4,
)
Controlling how data are turned into RGB
========================================
Here we apply a log-stretch to an all sky FITS image::
from toasty import toast, cartesian_sampler, normalizer
from astropy.io import fits
data = fits.open('allsky.fits')[0].data
vmin, vmax = 100, 65535
scaling = 'log'
contrast = 1
sampler = normalizer(
cartesian_sampler(data),
vmin, vmax
scaling, bias, contrast
)
output_directory = 'toast'
depth = 8
toast(sampler, depth, output_directory)
Non-Cartesian coordinate transformations
========================================
A custom “sampler” can be used to tell toasty_ what image values
correspond to what locations on the sky::
from toasty import toast
def sampler(x, y):
"""
x and y are arrays, giving the RA/Dec centers
(in radians) for each pixel to extract
"""
... code to produce pixel values goes here ...
output_directory = 'toast'
depth = 8
toast(sampler, depth, output_directory)
See :meth:`toasty.tile` for documentation on sampler functions.
Previewing toasts with AAS WorldWide Telescope
==============================================
To quickly preview a toast directory named ``mytoast``, run the command::
python -m toasty.viewer mytoast
This will start a web server, probably at ``http://0.0.0.0:8000``. (Check the
programs output for the actual address). Open this URL in a browser to get a
quick look at the data.
Example toasty outputs: ADS All-Sky Survey
==========================================
For an example of tiles generated with Toasty, see `the ADS All Sky Survey
<http://adsass.org/wwt>`_. The code used to generate these images is in `the
file toast.py`_ in the `adsass/wwt-frontend`_ repository on GitHub. This tile
pyramid was created by Chris Beaumont.
.. _the file toast.py: https://github.com/adsass/wwt-frontend/blob/master/toast/toast.py
.. _adsass/wwt-frontend: https://github.com/adsass/wwt-frontend/
Example toasty outputs: PanSTARRS in STScI AstroView
====================================================
Another example is a toasting of the “3π” survey of the PanSTARRS_ project,
viewable `here in STScIs AstroView`_ interface. The code used to generate
these images is in the GitHub repository `ceb8/toastPanstarrs`_. This tile
pyramid was created by Clara Brasseur.
.. _PanSTARRS: https://panstarrs.stsci.edu/
.. _here in STScIs AstroView: https://mast.stsci.edu/portal/Mashup/Clients/AstroView/AstroView.html?debug&avSurveyType=PANSTARRS
.. _ceb8/toastPanstarrs: https://github.com/ceb8/toastPanstarrs

Просмотреть файл

@ -2,10 +2,15 @@
toasty: Creating TOAST tile pyramids in Python
==============================================
The ``toasty`` package helps you create `TOAST
<https://arxiv.org/abs/1812.03926>`_ tile pyramids that can be used in
software such as the `AAS <https://aas.org>`_ `WorldWide Telescope
<http://worldwidetelescope.org/home>`_.
toasty_ is a Python library that helps you create TOAST_ “tile pyramids”
from astronomical image data. These multi-resolution all-sky maps can
be viewed in software such as the AAS_ `WorldWide Telescope`_.
.. _toasty: https://toasty.readthedocs.io/
.. _TOAST: https://arxiv.org/abs/1812.03926
.. _AAS: https://aas.org/
.. _WorldWide Telescope: http://www.worldwidetelescope.org/home
User guide
==========
@ -13,19 +18,22 @@ User guide
.. toctree::
:maxdepth: 1
overview
installation
examples
api
Getting help
============
If you run into any issues when using ``toasty``, please open an issue `on its
If you run into any issues when using toasty_, please open an issue `on its
GitHub repository <https://github.com/WorldWideTelescope/toasty/issues>`_.
Acknowledgments
===============
Work on ``toasty`` is funded through the American Astronomical Society
(`AAS`_) `WorldWide Telescope`_ project. The first version of ``toasty`` was
Work on toasty_ is funded through the American Astronomical Society
(`AAS`_) `WorldWide Telescope`_ project. The first version of toasty_ was
written by Chris Beaumont.

44
docs/installation.rst Normal file
Просмотреть файл

@ -0,0 +1,44 @@
=================
Installing toasty
=================
Installing toasty with pip
==========================
You can also install the latest release of toasty using pip_::
pip install toasty
.. _pip: https://pip.pypa.io/en/stable/
Dependencies
============
If you install toasty using pip_ as described above, any required dependencies
will get installed automatically. The `README in the Git repository`_ lists the
current dependencies if you would like to see an explict list.
.. _README in the Git repository: https://github.com/WorldWideTelescope/toasty/#readme
Installing the developer version
================================
If you want to use the very latest developer version, you should clone `this
repository <https://github.com/WorldWideTelescope/toasty/>`_ and manually
install the package in “editable” mode::
git clone https://github.com/WorldWideTelescope/toasty.git
cd toasty
pip install -e .
You will need to have Numpy_ and Cython_ installed on order to build the extension
module that toasty uses to speed up its core operations.
.. _Numpy: https://cython.org/
.. _Cython: https://numpy.org/
You can run the test suite with the command::
pytest toasty

101
docs/overview.rst Normal file
Просмотреть файл

@ -0,0 +1,101 @@
===========================
Overview: What is Toasting?
===========================
So, toasty_ is for toasting. What does that even mean?
.. _toasty: https://toasty.readthedocs.io/
This documentation will *not* go into detail because there are plans to
provide comprehensive information in a separate document that is not specific
to this particular library.
Alas, that documentation has not been written yet. In the meantime, the best
detailed reference is `McGlynn et al. 2019`_, which explains and justifies the
Tesselated Octahedral Adaptive Spherical Transformation (TOAST) projection
that toasty_ is concerned with. See also the `WorldWide Telescope Projection
Reference`_, although that document is a bit out of date. (Which shouldnt
matter, in principle, but its current Web expression may be a bit busted.) The
following text aims to give a quick overview of the concepts that these
documents work out in detail.
.. _McGlynn et al. 2019: https://ui.adsabs.harvard.edu/abs/2019ApJS..240...22M/abstract
.. _WorldWide Telescope Projection Reference: https://worldwidetelescope.gitbook.io/projection-reference/
The problem(s)
==============
Say that you are writing a piece of software that aim to let users
interactively explore a map of the Earth, the sky, or another planet. For the
purposes of this discussion, the key thing is that these are all spherical
entities. The user may want to zoom out and view large swathes of these
spheres at once, or they may want to zoom way in and see extremely fine
detail. The user will probably be accessing the maps over a network, and the
maps may be far too large to transmit in over the network in their entirety.
There are two problems to solve here. First, we need some way to store image
data covering an entire sphere. There is no unique best way to map the curved
surface of a sphere into a two-dimensional representation; in particular, many
common projections perform poorly at the poles.
Second, once we have some image data that we can map onto a sphere in some
satisfactory way, we need a way to transmit pieces of it to the user
incrementally. If they zoom out, it won't be practical to send them a huge
chunk of the full-resolution map.
The solution(s)
===============
Our solution to the above problems is to construct “tile pyramids” using the
TOAST spherical projection.
The TOAST projection does exactly what we need for the first problem: it maps
the entire sphere onto a 2D image that can easily be represented digitally.
While TOAST is not perfect in every way, it performs well at the poles and
maintains approximately uniform resolution at all locations on the sphere. If
youre familiar with this topic, you may know that the HEALPix_ projection
also operates in this problem space. We wont go into details here, but
suffice it to say that TOAST has some nice technical properties that make it
the preferred choice for software such as the AAS_ `WorldWide Telescope`_. In
particular, TOAST maps the sphere onto a square image, rather than the jagged
shape required by HEALPix.
.. _HEALPix: https://healpix.jpl.nasa.gov/
.. _AAS: https://aas.org/
.. _WorldWide Telescope: http://www.worldwidetelescope.org/home
A high-resolution full-sphere TOAST map may weigh in at a million pixels *on a
side*, or a trillion pixels total — far too large to manipulate in a
user-facing scenario. Creating a “tile pyramid” of the image makes it possible
to actually do useful things with a full-scale TOAST map. First, the
high-resolution map is broken into a set of square sub-images, “tiles,” each
256 pixels on a side. By the requirements of the TOAST standard, the number of
tiles on each side must be a power of 2. Then, lower-resolution maps are
created by downsampling the high-resolution map in 2×2 pixel blocks, yielding
derived images that are half as large as their parents along each axis. (Each
downsampled image therefore has 1/4 as many pixels as its parent.) By
construction, each of these lower-resolution maps can also be broken into
256×256 tiles. This process continues until a “level 0” map is created
consisting of a single tile that represents the full sphere. With a full
sphere having a solid angle of 4π steradians, each pixel in this level-0 map
covers about 0.0002 steradians or 0.6 square degrees. As a user zooms in or
out, or pans around the sphere, their computer is sent the tiles needed to
produce a visually acceptable view. Each individual tile file is small enough
that the data transfer remains tractable.
The role of toasty
==================
The toasty_ module helps create these TOAST tile pyramids from astronomical
image data. There are essentially three problems that it tackles:
1. Transforming image data from their native projection to the TOAST one.
2. Mapping scalar data values into RGB color images for user display.
3. Downsampling the high-resolution TOAST map all the way down to the level-0
map.
The process of creating TOAST tile pyramids from some input data is
colloquially referred to as “toasting.”

Просмотреть файл

@ -34,8 +34,8 @@ project homepage].
setup_args = dict(
name = 'toasty',
version = '0.1.0dev0', # also update docs/conf.py
description = 'Generate TOAST image tile pyramids from FITS files',
version = '0.1.0.dev0', # also update docs/conf.py
description = 'Generate TOAST image tile pyramids from existing image data',
long_description = get_long_desc(),
long_description_content_type = 'text/markdown',
url = 'https://toasty.readthedocs.io/',