WorldWideTelescope
/
toasty
зеркало из https://github.com/WorldWideTelescope/toasty.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Merge pull request #14 from pkgw/next 

Checkpoint some infrastructure work
This commit is contained in:
Peter Williams 2020-08-19 10:08:22 -04:00 коммит произвёл GitHub
Родитель 629a91119a 4dd91b77d6
Коммит fa0880375c
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
19 изменённых файлов: 987 добавлений и 466 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

2
docs/api.rst
Просмотреть файл

@ -9,7 +9,7 @@ API Documentation
:no-inheritance-diagram:
:no-inherited-members:
.. automodapi:: toasty.io
.. automodapi:: toasty.image
:no-inheritance-diagram:
:no-inherited-members:

49
docs/api/toasty.image.Image.rst Normal file
Просмотреть файл

@ -0,0 +1,49 @@
Image
=====
.. currentmodule:: toasty.image
.. autoclass:: Image
:show-inheritance:
.. rubric:: Attributes Summary
.. autosummary::
~Image.dtype
~Image.height
~Image.mode
~Image.shape
~Image.width
.. rubric:: Methods Summary
.. autosummary::
~Image.asarray
~Image.aspil
~Image.clear
~Image.fill_into_maskable_buffer
~Image.from_array
~Image.from_pil
~Image.make_thumbnail_bitmap
~Image.save_default
.. rubric:: Attributes Documentation
.. autoattribute:: dtype
.. autoattribute:: height
.. autoattribute:: mode
.. autoattribute:: shape
.. autoattribute:: width
.. rubric:: Methods Documentation
.. automethod:: asarray
.. automethod:: aspil
.. automethod:: clear
.. automethod:: fill_into_maskable_buffer
.. automethod:: from_array
.. automethod:: from_pil
.. automethod:: make_thumbnail_bitmap
.. automethod:: save_default

39
docs/api/toasty.image.ImageLoader.rst Normal file
Просмотреть файл

@ -0,0 +1,39 @@
ImageLoader
===========
.. currentmodule:: toasty.image
.. autoclass:: ImageLoader
:show-inheritance:
.. rubric:: Attributes Summary
.. autosummary::
~ImageLoader.colorspace_processing
~ImageLoader.desired_mode
~ImageLoader.psd_single_layer
.. rubric:: Methods Summary
.. autosummary::
~ImageLoader.add_arguments
~ImageLoader.create_from_args
~ImageLoader.load_path
~ImageLoader.load_pil
~ImageLoader.load_stream
.. rubric:: Attributes Documentation
.. autoattribute:: colorspace_processing
.. autoattribute:: desired_mode
.. autoattribute:: psd_single_layer
.. rubric:: Methods Documentation
.. automethod:: add_arguments
.. automethod:: create_from_args
.. automethod:: load_path
.. automethod:: load_pil
.. automethod:: load_stream

35
docs/api/toasty.image.ImageMode.rst Normal file
Просмотреть файл

@ -0,0 +1,35 @@
ImageMode
=========
.. currentmodule:: toasty.image
.. autoclass:: ImageMode
:show-inheritance:
.. rubric:: Attributes Summary
.. autosummary::
~ImageMode.F32
~ImageMode.RGB
~ImageMode.RGBA
.. rubric:: Methods Summary
.. autosummary::
~ImageMode.get_default_save_extension
~ImageMode.make_maskable_buffer
~ImageMode.try_as_pil
.. rubric:: Attributes Documentation
.. autoattribute:: F32
.. autoattribute:: RGB
.. autoattribute:: RGBA
.. rubric:: Methods Documentation
.. automethod:: get_default_save_extension
.. automethod:: make_maskable_buffer
.. automethod:: try_as_pil

6
docs/api/toasty.io.read_image.rst
Просмотреть файл

@ -1,6 +0,0 @@
read_image
==========
.. currentmodule:: toasty.io
.. autofunction:: read_image

6
docs/api/toasty.io.read_image_as_pil.rst
Просмотреть файл

@ -1,6 +0,0 @@
read_image_as_pil
=================
.. currentmodule:: toasty.io
.. autofunction:: read_image_as_pil

6
docs/api/toasty.io.save_png.rst
Просмотреть файл

@ -1,6 +0,0 @@
save_png
========
.. currentmodule:: toasty.io
.. autofunction:: save_png

16
docs/api/toasty.pyramid.PyramidIO.rst
Просмотреть файл

@ -11,17 +11,17 @@ PyramidIO
.. autosummary::
~PyramidIO.get_path_scheme
~PyramidIO.read_image
~PyramidIO.read_numpy
~PyramidIO.open_metadata_for_read
~PyramidIO.open_metadata_for_write
~PyramidIO.read_toasty_image
~PyramidIO.tile_path
~PyramidIO.write_image
~PyramidIO.write_numpy
~PyramidIO.write_toasty_image
.. rubric:: Methods Documentation
.. automethod:: get_path_scheme
.. automethod:: read_image
.. automethod:: read_numpy
.. automethod:: open_metadata_for_read
.. automethod:: open_metadata_for_write
.. automethod:: read_toasty_image
.. automethod:: tile_path
.. automethod:: write_image
.. automethod:: write_numpy
.. automethod:: write_toasty_image

6
docs/api/toasty.toast.SamplingToastDataSource.rst
Просмотреть файл

@ -10,10 +10,8 @@ SamplingToastDataSource
.. autosummary::
~SamplingToastDataSource.sample_data_layer
~SamplingToastDataSource.sample_image_layer
~SamplingToastDataSource.sample_layer
.. rubric:: Methods Documentation
.. automethod:: sample_data_layer
.. automethod:: sample_image_layer
.. automethod:: sample_layer

64
toasty/cli.py
Просмотреть файл

@ -65,6 +65,7 @@ def cascade_getparser(parser):
def cascade_impl(settings):
from .image import ImageMode
from .merge import averaging_merger, cascade_images
from .pyramid import PyramidIO
@ -74,7 +75,7 @@ def cascade_impl(settings):
if start is None:
die('currently, you must specify the start layer with the --start option')
cascade_images(pio, start, averaging_merger)
cascade_images(pio, ImageMode.RGBA, start, averaging_merger)
# "healpix_sample_data_tiles" subcommand
@ -100,19 +101,23 @@ def healpix_sample_data_tiles_getparser(parser):
def healpix_sample_data_tiles_impl(settings):
from .image import ImageMode
from .pyramid import PyramidIO
from .samplers import healpix_fits_file_sampler
from .toast import SamplingToastDataSource
pio = PyramidIO(settings.outdir)
sampler = healpix_fits_file_sampler(settings.fitspath)
ds = SamplingToastDataSource(sampler)
ds.sample_data_layer(pio, settings.depth)
ds = SamplingToastDataSource(ImageMode.F32, sampler)
ds.sample_layer(pio, settings.depth)
# "image_sample_tiles" subcommand
def image_sample_tiles_getparser(parser):
from .image import ImageLoader
ImageLoader.add_arguments(parser)
parser.add_argument(
'--outdir',
metavar = 'PATH',
@ -139,21 +144,21 @@ def image_sample_tiles_getparser(parser):
def image_sample_tiles_impl(settings):
from .io import read_image
from .image import ImageLoader
from .pyramid import PyramidIO
from .toast import SamplingToastDataSource
img = ImageLoader.create_from_args(settings).load_path(settings.imgpath)
pio = PyramidIO(settings.outdir)
data = read_image(settings.imgpath)
if settings.projection == 'plate-carree':
from .samplers import plate_carree_sampler
sampler = plate_carree_sampler(data)
sampler = plate_carree_sampler(img.asarray())
else:
die('the image projection type {!r} is not recognized'.format(settings.projection))
ds = SamplingToastDataSource(sampler)
ds.sample_image_layer(pio, settings.depth)
ds = SamplingToastDataSource(img.mode, sampler)
ds.sample_layer(pio, settings.depth)
# "multi_tan_make_data_tiles" subcommand
@ -329,6 +334,9 @@ def pipeline_reindex_impl(settings):
# "study_sample_image_tiles" subcommand
def study_sample_image_tiles_getparser(parser):
from .image import ImageLoader
ImageLoader.add_arguments(parser)
parser.add_argument(
'--outdir',
metavar = 'PATH',
@ -346,20 +354,17 @@ def study_sample_image_tiles_impl(settings):
import numpy as np
import PIL.Image
from wwt_data_formats.imageset import ImageSet
from .io import read_image_as_pil
from .image import ImageLoader
from .pyramid import PyramidIO
from .study import make_thumbnail_bitmap, tile_study_image
from .study import tile_study_image
# Prevent max image size aborts:
PIL.Image.MAX_IMAGE_PIXELS = None
# Load image.
# Load image and prep tiling
img = ImageLoader.create_from_args(settings).load_path(settings.imgpath)
pio = PyramidIO(settings.outdir)
img = read_image_as_pil(settings.imgpath)
tiling = tile_study_image(np.asarray(img), pio)
tiling = tile_study_image(img, pio)
# Thumbnail.
thumb = make_thumbnail_bitmap(img)
thumb = img.make_thumbnail_bitmap()
thumb.save(os.path.join(settings.outdir, 'thumb.jpg'), format='JPEG')
# Write out a stub WTML file. The only information this will actually
@ -373,10 +378,19 @@ def study_sample_image_tiles_impl(settings):
imgset.url = pio.get_path_scheme() + '.png'
stub_wtml(imgset, os.path.join(settings.outdir, 'index_rel.wtml'))
# Helpful hint:
print(f'Successfully tiled input "{settings.imgpath}" at level {imgset.tile_levels}.')
print('To create parent tiles, consider running:')
print()
print(f' toasty cascade --start {imgset.tile_levels} {settings.outdir}')
# "wwtl_sample_image_tiles" subcommand
def wwtl_sample_image_tiles_getparser(parser):
from .image import ImageLoader
ImageLoader.add_arguments(parser)
parser.add_argument(
'--outdir',
metavar = 'PATH',
@ -391,6 +405,7 @@ def wwtl_sample_image_tiles_getparser(parser):
def wwtl_sample_image_tiles_impl(settings):
# TODO: implement WWTL loading as an Image mode.
from io import BytesIO
import numpy as np
import PIL.Image
@ -399,12 +414,9 @@ def wwtl_sample_image_tiles_impl(settings):
from wwt_data_formats.layers import ImageSetLayer, LayerContainerReader
from wwt_data_formats.place import Place
from .io import read_image_as_pil
from .image import ImageLoader
from .pyramid import PyramidIO
from .study import make_thumbnail_bitmap, tile_study_image
# Prevent max image size aborts:
PIL.Image.MAX_IMAGE_PIXELS = None
from .study import tile_study_image
# Load WWTL and see if it matches expectations
lc = LayerContainerReader.from_file(settings.wwtl_path)
@ -421,15 +433,16 @@ def wwtl_sample_image_tiles_impl(settings):
die('WWTL imageset layer must have "SkyImage" projection type')
# Looks OK. Read and parse the image.
loader = ImageLoader.create_from_args(settings)
img_data = lc.read_layer_file(layer, layer.extension)
img = PIL.Image.open(BytesIO(img_data))
img = loader.load_stream(BytesIO(img_data))
# Tile it!
pio = PyramidIO(settings.outdir)
tiling = tile_study_image(np.asarray(img), pio)
tiling = tile_study_image(img, pio)
# Thumbnail.
thumb = make_thumbnail_bitmap(img)
thumb = img.make_thumbnail_bitmap()
thumb.save(os.path.join(settings.outdir, 'thumb.jpg'), format='JPEG')
# Write a WTML file. We reuse the existing imageset as much as possible,
@ -442,6 +455,7 @@ def wwtl_sample_image_tiles_impl(settings):
if not imgset.name:
imgset.name = 'Toasty'
imgset.file_type = '.png'
imgset.thumbnail_url = 'thumb.jpg'
imgset.url = pio.get_path_scheme() + '.png'

542
toasty/image.py Normal file
Просмотреть файл

@ -0,0 +1,542 @@
# -*- mode: python; coding: utf-8 -*-
# Copyright 2020 the AAS WorldWide Telescope project
# Licensed under the MIT License.
"""
Low-level loading and handling of images.
Here, images are defined as 2D data buffers stored in memory. Images might be
small, if dealing with an individual tile, or extremely large, if loading up a
large study for tiling.
"""
from __future__ import absolute_import, division, print_function
__all__ = '''
Image
ImageLoader
ImageMode
'''.split()
from enum import Enum
from PIL import Image as pil_image
import numpy as np
class ImageMode(Enum):
"""
Allowed image "modes", describing their pixel data formats.
These align with PIL modes when possible, but we expect to need to support
modes that aren't present in PIL (namely, float64). There are also various
obscure PIL modes that we do not support.
"""
RGB = 'RGB'
RGBA = 'RGBA'
F32 = 'F'
def get_default_save_extension(self):
"""
Get the file extension to be used in this mode's "default" save format.
Returns
-------
The extension, without a period; either "png" or "npy"
"""
# For RGB, could use JPG? But would complexify lots of things downstream.
if self in (ImageMode.RGB, ImageMode.RGBA):
return 'png'
elif self == ImageMode.F32:
return 'npy'
else:
raise Exception('unhandled mode in get_default_save_extension')
def make_maskable_buffer(self, buf_height, buf_width):
"""
Return a new, uninitialized buffer of the specified shape, with a mode
compatible with this one but able to accept undefined values.
Parameters
----------
buf_height : int
The height of the new buffer
buf_width : int
The width of the new buffer
Returns
-------
An uninitialized :class:`Image` instance.
Notes
-----
"Maskable" means that the buffer can accommodate undefined values.
If the image is RGB or RGBA, that means that the buffer will have an
alpha channel. If the image is scientific, that means that the buffer
will be able to accept NaNs.
"""
mask_mode = self
if self in (ImageMode.RGB, ImageMode.RGBA):
arr = np.empty((buf_height, buf_width, 4), dtype=np.uint8)
mask_mode = ImageMode.RGBA
elif self == ImageMode.F32:
arr = np.empty((buf_height, buf_width), dtype=np.float32)
else:
raise Exception('unhandled mode in make_maskable_buffer()')
return Image.from_array(mask_mode, arr)
def try_as_pil(self):
"""
Attempt to convert this mode into a PIL image mode string.
Returns
-------
A PIL image mode string, or None if there is no exact counterpart.
"""
return self.value
class ImageLoader(object):
"""
A class defining how to load an image.
This is implemented as its own class since there can be some options
involved, and we want to provide a centralized place for handling them all.
TODO: support FITS, Numpy, etc.
"""
colorspace_processing = 'srgb'
desired_mode = None
psd_single_layer = None
@classmethod
def add_arguments(cls, parser):
"""
Add standard image-loading options to an argparse parser object.
Parameters
----------
parser : :class:`argparse.ArgumentParser`
The argument parser to modify
Returns
-------
The :class:`ImageLoader` class (for chainability).
Notes
-----
If you are writing a command-line interface that takes a single image as
an input, use this function to wire in to standardized image-loading
infrastructure and options.
"""
parser.add_argument(
'--colorspace-processing',
metavar = 'MODE',
default = 'srgb',
help = 'What kind of RGB colorspace processing to perform: '
'"none", "srgb" to convert to sRGB (the default)',
)
# not exposing desired_mode -- shouldn't be something the for the user to deal with
parser.add_argument(
'--psd-single-layer',
type = int,
metavar = 'NUMBER',
help = 'If loading a Photoshop image, the (0-based) layer number to load -- saves memory',
)
return cls
@classmethod
def create_from_args(cls, settings):
"""
Process standard image-loading options to create an :class:`ImageLoader`.
Parameters
----------
settings : :class:`argparse.Namespace`
Settings from processing command-line arguments
Returns
-------
A new :class:`ImageLoader` initialized with the settings.
"""
loader = cls()
loader.psd_single_layer = settings.psd_single_layer
loader.colorspace_processing = settings.colorspace_processing
return loader
def load_pil(self, pil_img):
"""
Load an already opened PIL image.
Parameters
----------
pil_img : :class:`PIL.Image.Image`
The image.
Returns
-------
A new :class:`Image`.
Notes
-----
This function should be used instead of :meth:`Image.from_pil` because
may postprocess the image in various ways, depending on the loader
configuration.
"""
# Make sure that we end up in the right color space. From experience, some
# EPO images have funky colorspaces and we need to convert to sRGB to get
# the tiled versions to appear correctly.
if self.colorspace_processing != 'none' and 'icc_profile' in pil_img.info:
assert self.colorspace_processing == 'srgb' # more modes, one day?
from io import BytesIO
from PIL import ImageCms
in_prof = ImageCms.getOpenProfile(BytesIO(pil_img.info['icc_profile']))
out_prof = ImageCms.createProfile('sRGB')
xform = ImageCms.buildTransform(in_prof, out_prof, pil_img.mode, pil_img.mode)
ImageCms.applyTransform(pil_img, xform, inPlace=True)
# Make sure that we end up with the right mode, if requested.
if self.desired_mode is not None:
desired_pil = self.desired_mode.try_as_pil()
if desired_pil is None:
raise Exception('cannot convert PIL image to desired mode {}'.format(self.desired_mode))
if pil_img.mode != desired_pil:
pil_img = pil_img.convert(desired_pil)
return Image.from_pil(pil_img)
def load_stream(self, stream):
"""
Load an image into memory from a file-like stream.
Parameters
----------
stream : file-like
The data to load. Reads should yield bytes.
Returns
-------
A new :class:`Image`.
"""
# TODO: one day, we'll support FITS files and whatnot and we'll have a
# mode where we get a Numpy array but not a PIL image. For now, just
# pass it off to PIL and hope for the best.
# Prevent PIL decompression-bomb aborts. Not thread-safe, of course.
old_max = pil_image.MAX_IMAGE_PIXELS
try:
pil_image.MAX_IMAGE_PIXELS = None
pilimg = pil_image.open(stream)
finally:
pil_image.MAX_IMAGE_PIXELS = old_max
# Now pass it off to generic PIL handling ...
return self.load_pil(pilimg)
def load_path(self, path):
"""
Load an image into memory from a filesystem path.
Parameters
----------
path : str
The filesystem path to load.
Returns
-------
A new :class:`Image`.
"""
# Special handling for Photoshop files, used for some very large mosaics
# with transparency (e.g. the PHAT M31/M33 images). TODO: it would be
# better to sniff the PSD filetype instead of just looking at
# extensions. But, lazy.
if path.endswith('.psd') or path.endswith('.psb'):
try:
from psd_tools import PSDImage
except ImportError:
pass
else:
psd = PSDImage.open(path)
# If the Photoshop image is a single layer, we can save a lot of
# memory by not using the composite() function. This has helped
# me process very large Photoshop files.
if self.psd_single_layer is not None:
pilimg = psd[self.psd_single_layer].topil()
else:
pilimg = psd.composite()
return self.load_pil(pilimg)
# (One day, maybe we'll do more kinds of sniffing.) No special handling
# came into play; just open the file and auto-detect.
with open(path, 'rb') as f:
return self.load_stream(f)
class Image(object):
"""A 2D data array stored in memory.
This class primarily exists to help us abstract between the cases where we
have "bitmap" RGB(A) images and "science" floating-point images.
"""
_mode = None
_pil = None
_array = None
@classmethod
def from_pil(cls, pil_img):
"""
Create a new Image from a PIL image.
Parameters
----------
pil_img : :class:`PIL.Image.Image`
The source image.
Returns
-------
A new :class:`Image` wrapping the PIL image.
"""
# Make sure that the image data are actually loaded from disk. Pillow
# lazy-loads such that sometimes `np.asarray(img)` ends up failing
# mysteriously if the image is loaded from a file handle that is closed
# promptly.
pil_img.load()
inst = cls()
inst._pil = pil_img
try:
inst._mode = ImageMode(pil_img.mode)
except ValueError:
raise Exception('image mode {} is not supported'.format(pil_img.mode))
return inst
@classmethod
def from_array(cls, mode, array):
"""Create a new Image from an array-like data variable.
Parameters
----------
mode : :class:`ImageMode`
The image mode.
array : array-like object
The source data.
Returns
-------
A new :class:`Image` wrapping the data.
Notes
-----
The array will be converted to be at least two-dimensional. The data
array shape should match the requirements of the mode.
"""
array = np.atleast_2d(array)
array_ok = False
if mode == ImageMode.F32:
array_ok = (array.ndim == 2 and array.dtype == np.dtype(np.float32))
elif mode == ImageMode.RGB:
array_ok = (array.ndim == 3 and array.shape[2] == 3 and array.dtype == np.dtype(np.uint8))
elif mode == ImageMode.RGBA:
array_ok = (array.ndim == 3 and array.shape[2] == 4 and array.dtype == np.dtype(np.uint8))
else:
raise ValueError('unhandled image mode {} in from_array()'.format(mode))
if not array_ok:
raise ValueError('expected array compatible with mode {}, but got shape/dtype = {}/{}'
.format(mode, array.shape, array.dtype))
inst = cls()
inst._mode = mode
inst._array = array
return inst
def asarray(self):
"""Obtain the image data as a Numpy array.
Returns
-------
If the image is an RGB(A) bitmap, the array will have shape ``(height, width, planes)``
and a dtype of ``uint8``, where ``planes`` is either 3
or 4 depending on whether the image has an alpha channel. If the image
is science data, it will have shape ``(height, width)`` and have a
floating-point dtype.
"""
if self._pil is not None:
return np.asarray(self._pil)
return self._array
def aspil(self):
"""Obtain the image data as :class:`PIL.Image.Image`.
Returns
-------
If the image was loaded as a PIL image, the underlying object will be
returned. Otherwise the data array will be converted into a PIL image,
which requires that the array have an RGB(A) format with a shape of
``(height, width, planes)``, where ``planes`` is 3 or 4, and a dtype of
``uint8``.
"""
if self._pil is not None:
return self._pil
return pil_image.fromarray(self._array)
@property
def mode(self):
return self._mode
@property
def dtype(self):
# TODO: can this be more efficient? Does it need to be?
return self.asarray().dtype
@property
def shape(self):
if self._array is not None:
return self._array.shape
return (self._pil.height, self._pil.width, len(self._pil.getbands()))
@property
def width(self):
return self.shape[1]
@property
def height(self):
return self.shape[0]
def fill_into_maskable_buffer(self, buffer, iy_idx, ix_idx, by_idx, bx_idx):
"""
Fill a maskable buffer with a rectangle of data from this image.
Parameters
----------
buffer : :class:`Image`
The destination buffer image, created with :meth:`ImageMode.make_maskable_buffer`.
iy_idx : slice or other indexer
The indexer into the Y axis of the source image (self).
ix_idx : slice or other indexer
The indexer into the X axis of the source image (self).
by_idx : slice or other indexer
The indexer into the Y axis of the destination *buffer*.
bx_idx : slice or other indexer
The indexer into the X axis of the destination *buffer*.
Notes
-----
This highly specialized function is used to tile images efficiently. No
bounds checking is performed. The rectangles defined by the indexers in
the source and destination are assumed to agree in size. The regions of
the buffer not filled by source data are masked, namely: either filled
with alpha=0 or with NaN, depending on the image mode.
"""
i = self.asarray()
b = buffer.asarray()
if self.mode == ImageMode.RGB:
b.fill(0)
b[by_idx,bx_idx,:3] = i[iy_idx,ix_idx]
b[by_idx,bx_idx,3] = 255
elif self.mode == ImageMode.RGBA:
b.fill(0)
b[by_idx,bx_idx] = i[iy_idx,ix_idx]
elif self.mode == ImageMode.F32:
b.fill(np.nan)
b[by_idx,bx_idx] = i[iy_idx,ix_idx]
else:
raise Exception('unhandled mode in fill_into_maskable_buffer')
def save_default(self, path_or_stream):
"""
Save this image to a filesystem path or stream
Parameters
----------
path_or_stream : path-like object or file-like object
The destination into which the data should be written. If file-like,
the stream should accept bytes.
"""
if self.mode in (ImageMode.RGB, ImageMode.RGBA):
self.aspil().save(path_or_stream, format='PNG')
elif self.mode == ImageMode.F32:
np.save(path_or_stream, self.asarray())
else:
raise Exception('unhandled mode in save_default')
def make_thumbnail_bitmap(self):
"""Create a thumbnail bitmap from the image.
Returns
-------
An RGB :class:`PIL.Image.Image` representing a thumbnail of the input
image. WWT thumbnails are 96 pixels wide and 45 pixels tall and should
be saved in JPEG format.
"""
if self.mode == ImageMode.F32:
raise Exception('cannot thumbnail-ify non-RGB Image')
THUMB_SHAPE = (96, 45)
THUMB_ASPECT = THUMB_SHAPE[0] / THUMB_SHAPE[1]
if self.width / self.height > THUMB_ASPECT:
# The image is wider than desired; we'll need to crop off the sides.
target_width = int(round(self.height * THUMB_ASPECT))
dx = (self.width - target_width) // 2
crop_box = (dx, 0, dx + target_width, self.height)
else:
# The image is taller than desired; crop off top and bottom.
target_height = int(round(self.width / THUMB_ASPECT))
dy = (self.height - target_height) // 2
crop_box = (0, dy, self.width, dy + target_height)
thumb = self.aspil().crop(crop_box)
thumb.thumbnail(THUMB_SHAPE)
# Depending on the source image, the mode might be RGBA, which can't
# be JPEG-ified.
thumb = thumb.convert('RGB')
return thumb
def clear(self):
"""
Fill the image with whatever "empty" value is most appropriate for its mode.
Notes
-----
If the mode is RGB or RGBA, the buffer is filled with zeros. If the mode is
floating-point, the buffer is filled with NaNs.
"""
if self._mode in (ImageMode.RGB, ImageMode.RGBA):
self.asarray().fill(0)
elif self._mode == ImageMode.F32:
self.asarray().fill(np.nan)
else:
raise Exception('unhandled mode in clear()')

81
toasty/io.py
Просмотреть файл

@ -1,81 +0,0 @@
# -*- mode: python; coding: utf-8 -*-
# Copyright 2013-2020 Chris Beaumont and the AAS WorldWide Telescope project
# Licensed under the MIT License.
from __future__ import absolute_import, division, print_function
__all__ = '''
read_image_as_pil
read_image
save_png
'''.split()
from PIL import Image
import numpy as np
def save_png(pth, array):
"""
Save an array as a PNG image
Parameters
----------
pth : str
Path to write to
array : array-like
Image to save
"""
Image.fromarray(array).save(pth)
def read_image_as_pil(path):
"""Load a bitmap image into a PIL Image.
The loading supports whatever image formats PIL does. As a special-case
hack, if the input path has extension ``.psd`` or ``.psb``, the
``psd_tools`` module will be used if available.
Parameters
----------
path : str
The path of the image to read
Returns
-------
img : :class:`PIL.Image.Image`
The image data.
"""
if path.endswith('.psd') or path.endswith('.psb'):
try:
from psd_tools import PSDImage
except ImportError:
pass
else:
psd = PSDImage.open(path)
return psd.composite()
return Image.open(path)
def read_image(path):
"""Load a bitmap image into a Numpy array.
The loading is generally done using PIL (the Python Imaging Library, usually the
"pillow" implementation these days) so it supports whatever image formats
PIL does. As a special-case hack, if the input path has extension ``.psd`` or ``.psb``,
the ``psd_tools`` module will be used if available.
Parameters
----------
path : str
The path of the image to read
Returns
-------
data : :class:`numpy.ndarray`
The image data. The array will have shape ``(height, width, planes)``, where
the first two axes are the image shape and the third is the number of color planes:
3 for RGB or potentially 4 for RGBA. The data type will be ``uint8``.
"""
return np.asarray(read_image_as_pil(path))

39
toasty/merge.py
Просмотреть файл

@ -1,5 +1,5 @@
# -*- mode: python; coding: utf-8 -*-
# Copyright 2019 the AAS WorldWide Telescope project
# Copyright 2019-2020 the AAS WorldWide Telescope project
# Licensed under the MIT License.
"""General tools for merging and downsampling tiles
@ -30,6 +30,7 @@ cascade_images
import numpy as np
from . import pyramid
from .image import Image
def averaging_merger(data):
@ -38,7 +39,7 @@ def averaging_merger(data):
Parameters
----------
data : array
See the Merger Protocol specification.
See the Merger Protocol specification.
Returns
-------
@ -49,7 +50,7 @@ def averaging_merger(data):
return np.nanmean(data.reshape(s), axis=(1, 3)).astype(data.dtype)
def cascade_images(pio, start, merger):
def cascade_images(pio, mode, start, merger):
"""Downsample image tiles all the way to the top of the pyramid.
This function will walk the tiles in the tile pyramid, merging child tile
@ -58,14 +59,16 @@ def cascade_images(pio, start, merger):
Parameters
----------
pio : :class:`toasty.pyramid.PyramidIO`
An object managing I/O on the tiles in the pyramid.
An object managing I/O on the tiles in the pyramid.
mode : :class:`toasty.image.ImageMode`
The image mode (i.e., RGB or scientific) to process.
start : nonnegative integer
The depth at which to start the cascade process. It is assumed that
the tiles *at this depth* are already populated by some other means.
This function will create new tiles at shallower depths.
The depth at which to start the cascade process. It is assumed that
the tiles *at this depth* are already populated by some other means.
This function will create new tiles at shallower depths.
merger : a merger function
The method used to create a parent tile from its child tiles. This
is a callable that follows the Merger Protocol.
The method used to create a parent tile from its child tiles. This
is a callable that follows the Merger Protocol.
"""
buf = None
@ -84,23 +87,23 @@ def cascade_images(pio, start, merger):
# processed.
children = pyramid.pos_children(pos)
img0 = pio.read_image(children[0], default='none')
img1 = pio.read_image(children[1], default='none')
img2 = pio.read_image(children[2], default='none')
img3 = pio.read_image(children[3], default='none')
img0 = pio.read_toasty_image(children[0], mode, default='none')
img1 = pio.read_toasty_image(children[1], mode, default='none')
img2 = pio.read_toasty_image(children[2], mode, default='none')
img3 = pio.read_toasty_image(children[3], mode, default='none')
if img0 is None and img1 is None and img2 is None and img3 is None:
continue # No data here; ignore
if buf is not None:
buf.fill(0)
buf.clear()
for slidx, subimg in zip(SLICES, (img0, img1, img2, img3)):
if subimg is not None:
if buf is None:
buf = np.zeros((512, 512) + subimg.shape[2:], dtype=np.uint8)
buf = mode.make_maskable_buffer(512, 512)
buf[slidx] = subimg
buf.asarray()[slidx] = subimg.asarray()
merged = merger(buf)
pio.write_image(pos, merged)
merged = Image.from_array(mode, merger(buf.asarray()))
pio.write_toasty_image(pos, merged)

10
toasty/pipeline.py
Просмотреть файл

@ -658,6 +658,7 @@ class BitmapInputImage(InputImage):
return self._bitmap
def _process_image_data(self, imgset, outdir):
from .image import Image, ImageMode
self._ensure_bitmap()
needs_tiling = self._bitmap.width > 2048 or self._bitmap.height > 2048
@ -674,19 +675,18 @@ class BitmapInputImage(InputImage):
# Create the base layer
pio = PyramidIO(outdir, scheme='LXY')
img_data = np.asarray(self._bitmap)
tiling = tile_study_image(img_data, pio)
image = Image.from_pil(self._bitmap)
tiling = tile_study_image(image, pio)
tiling.apply_to_imageset(imgset)
# Cascade to create the coarser tiles
cascade_images(pio, imgset.tile_levels, averaging_merger)
cascade_images(pio, ImageMode.RGBA, imgset.tile_levels, averaging_merger)
imgset.url = pio.get_path_scheme() + '.png'
imgset.file_type = '.png'
# Deal with the thumbnail
from .study import make_thumbnail_bitmap
thumb = make_thumbnail_bitmap(self._bitmap)
thumb = Image.from_pil(self._bitmap).make_thumbnail_bitmap()
thumb.save(os.path.join(outdir, 'thumb.jpg'), format='JPEG')
imgset.thumbnail_url = 'thumb.jpg'

143
toasty/pyramid.py
Просмотреть файл

@ -28,6 +28,8 @@ from collections import namedtuple
import numpy as np
import os.path
from .image import ImageLoader
Pos = namedtuple('Pos', 'n x y')
@ -56,9 +58,9 @@ def is_subtile(deeper_pos, shallower_pos):
Parameters
----------
deeper_pos : Pos
A tile position.
A tile position.
shallower_pos : Pos
A tile position that is shallower than *deeper_pos*.
A tile position that is shallower than *deeper_pos*.
Returns
-------
@ -80,16 +82,16 @@ def pos_parent(pos):
Parameters
----------
pos : Pos
A tile position.
A tile position.
Returns
-------
parent : Pos
The tile position that is the parent of *pos*.
The tile position that is the parent of *pos*.
x_index : integer, 0 or 1
The horizontal index of the child inside its parent.
The horizontal index of the child inside its parent.
y_index : integer, 0 or 1
The vertical index of the child inside its parent.
The vertical index of the child inside its parent.
"""
if pos.n < 1:
@ -109,7 +111,7 @@ def pos_children(pos):
Parameters
----------
pos : :class:`Pos`
A tile position.
A tile position.
Returns
-------
@ -152,12 +154,12 @@ def generate_pos(depth):
Parameters
----------
depth : int
The tile depth to recurse to.
The tile depth to recurse to.
Yields
------
pos : :class:`Pos`
An individual position to process.
An individual position to process.
"""
for item in _postfix_pos(Pos(0, 0, 0), depth):
@ -185,9 +187,9 @@ class PyramidIO(object):
Parameters
----------
pos : Pos
The tile to get a path for.
The tile to get a path for.
extension : str, default: "png"
The file extension to use in the path.
The file extension to use in the path.
Returns
-------
@ -245,117 +247,88 @@ class PyramidIO(object):
"""
return self._scheme
def read_image(self, pos, extension='png', default='none'):
"""Read an image file for the specified tile position.
def read_toasty_image(self, pos, mode, default='none'):
"""
Read a toasty Image for the specified tile position.
Parameters
----------
pos : :class:`Pos`
The tile position to read.
extension : str, defaults to "png"
The file extension to use when constructing the path to read.
The tile position to read.
mode : :class:`toasty.image.ImageMode`
The image data mode to read. This will affect the file extension probed
and the mode of the returned image.
default : str, defaults to "none"
What to do if the specified tile file does not exist. If this is
"none", ``None`` will be returned instead of an array. If this is
"zeros3", an array of zeros with shape ``(256, 256, 3)`` and dtype
``np.uint8`` will be returned. If it is "zeros4", a similar array
of shape ``(256, 256, 4)`` will be returned. Otherwise,
:exc:`ValueError` will be raised.
Returns
-------
The image data as a numpy array, or one of the values as specified
based on the parameter *default*. For a typical PNG image, the
returned array will have shape ``(256, 256, 4)`` and dtype
``np.uint8``.
What to do if the specified tile file does not exist. If this is
"none", ``None`` will be returned instead of an image. If this is
"masked", an all-masked image will be returned, using
:meth:`~toasty.image.ImageMode.make_maskable_buffer`.
Otherwise, :exc:`ValueError` will be raised.
"""
from .io import read_image
p = self.tile_path(pos, mode.get_default_save_extension())
loader = ImageLoader()
loader.desired_mode = mode
try:
return read_image(self.tile_path(pos, extension))
img = loader.load_path(p)
except IOError as e:
if e.errno != 2:
raise # not EEXIST
if default == 'none':
return None
elif default == 'zeros3':
return np.zeros((256, 256, 3), dtype=np.uint8)
elif default == 'zeros4':
return np.zeros((256, 256, 4), dtype=np.uint8)
elif default == 'masked':
return mode.make_maskable_buffer(256, 256)
else:
raise ValueError('unexpected value for "default": {!r}'.format(default))
def write_image(self, pos, data, extension='png'):
"""Write an image file for the specified tile position.
assert img.mode == mode
return img
The conversion of the array into an image is handled by the
:func:`toasty.io.save_png` function — see its documentation for
specifics. Generally, *data* should be an array of shape ``(256, 256,
3)`` and dtype ``np.uint8``.
def write_toasty_image(self, pos, image):
"""Write a toasty Image for the specified tile position.
Parameters
----------
pos : :class:`Pos`
The tile position to write.
data : array-like
The image data to write.
extension : str, defaults to "png"
The file extension to use when constructing the path to write.
The tile position to write.
image : :class:`toasty.image.Image`
The image to write.
"""
from .io import save_png
save_png(self.tile_path(pos, extension), data)
p = self.tile_path(pos, image.mode.get_default_save_extension())
image.save_default(p)
def read_numpy(self, pos, extension='npy', default='nan'):
"""Read a Numpy file for the specified tile position.
def open_metadata_for_read(self, basename):
"""
Open a metadata file in read mode.
Parameters
----------
pos : :class:`Pos`
The tile position to read.
extension : str, defaults to "npy"
The file extension to use when constructing the path to read.
default : str, defaults to "nan"
What to do if the specified tile file does not exist. If this is
"none", ``None`` will be returned instead of an array. If this is
"nan", an array of NaNs with shape ``(256, 256)`` and dtype
``np.double`` will be returned. Otherwise, :exc:`ValueError` will be
raised.
basename : str
The basename of the metadata file
Returns
-------
The saved numpy array, or one of the values as specified based on the
parameter *default*.
A readable and closeable file-like object returning bytes.
"""
try:
return np.load(self.tile_path(pos, extension))
except IOError as e:
if e.errno != 2:
raise # not EEXIST
return open(os.path.join(self._base_dir, basename), 'rb')
if default == 'none':
return None
elif default == 'nan':
arr = np.empty((256, 256), dtype=np.double)
arr.fill(np.nan)
return arr
else:
raise ValueError('unexpected value for "default": {!r}'.format(default))
def write_numpy(self, pos, data, extension='npy'):
"""Write a numpy file for the specified tile position.
def open_metadata_for_write(self, basename):
"""
Open a metadata file in write mode.
Parameters
----------
pos : :class:`Pos`
The tile position to write.
data : array-like
The numpy data to write.
extension : str, defaults to "npy"
The file extension to use when constructing the path to write.
basename : str
The basename of the metadata file
Returns
-------
A writable and closeable file-like object accepting bytes.
"""
np.save(self.tile_path(pos, extension), data)
return open(os.path.join(self._base_dir, basename), 'wb')

50
toasty/samplers.py
Просмотреть файл

@ -1,8 +1,9 @@
# -*- mode: python; coding: utf-8 -*-
# Copyright 2013-2019 Chris Beaumont and the AAS WorldWide Telescope project
# Copyright 2013-2020 Chris Beaumont and the AAS WorldWide Telescope project
# Licensed under the MIT License.
"""“Sampler” functions that fetch image data as a function of sky coordinates.
"""
“Sampler” functions that fetch image data as a function of sky coordinates.
The Sampler Protocol
--------------------
@ -36,16 +37,15 @@ def healpix_sampler(data, nest=False, coord='C', interpolation='nearest'):
Parameters
----------
data : array
The HEALPix data
The HEALPix data
nest : bool (default: False)
Whether the data is ordered in the nested HEALPix style
Whether the data is ordered in the nested HEALPix style
coord : 'C' | 'G'
Whether the image is in Celestial (C) or Galactic (G) coordinates
Whether the image is in Celestial (C) or Galactic (G) coordinates
interpolation : 'nearest' | 'bilinear'
What interpolation scheme to use.
What interpolation scheme to use.
WARNING: bilinear uses healpy's get_interp_val,
which seems prone to segfaults
WARNING: bilinear uses healpy's get_interp_val, which seems prone to segfaults
Returns
-------
@ -104,15 +104,14 @@ def healpix_fits_file_sampler(path, extension=None, interpolation='nearest'):
Parameters
----------
path : string
The path to the FITS file.
The path to the FITS file.
extension : integer or None (default: None)
Which extension in the FITS file to read. If not specified, the first
extension with PIXTYPE = "HEALPIX" will be used.
Which extension in the FITS file to read. If not specified, the first
extension with PIXTYPE = "HEALPIX" will be used.
interpolation : 'nearest' | 'bilinear'
What interpolation scheme to use.
What interpolation scheme to use.
WARNING: bilinear uses healpy's get_interp_val,
which seems prone to segfaults
WARNING: bilinear uses healpy's get_interp_val, which seems prone to segfaults
Returns
-------
@ -128,8 +127,13 @@ def healpix_fits_file_sampler(path, extension=None, interpolation='nearest'):
extension = _find_healpix_extension_index(f)
data, hdr = f[extension].data, f[extension].header
# grab the first healpix parameter
# grab the first healpix parameter and convert to native endianness if
# needed.
data = data[data.dtype.names[0]]
if data.dtype.byteorder not in '=|':
data = data.byteswap().newbyteorder()
nest = hdr.get('ORDERING') == 'NESTED'
coord = hdr.get('COORDSYS', 'C')
@ -151,7 +155,7 @@ def plate_carree_sampler(data):
Parameters
----------
data : array-like, at least 2D
The map to sample in plate carrée projection.
The map to sample in plate carrée projection.
Returns
-------
@ -187,18 +191,18 @@ def normalizer(sampler, vmin, vmax, scaling='linear', bias=0.5, contrast=1):
Parameters
----------
sampler : function
An input sampler function with call signature ``vec2pix(lon, lat) -> data``.
An input sampler function with call signature ``vec2pix(lon, lat) -> data``.
vmin : float
The data value to assign to 0 (black).
The data value to assign to 0 (black).
vmin : float
The data value to assign to 255 (white).
The data value to assign to 255 (white).
bias : float between 0-1, default: 0.5
Where to assign middle-grey, relative to (vmin, vmax).
Where to assign middle-grey, relative to (vmin, vmax).
contrast : float, default: 1
How quickly to ramp from black to white. The default of 1
ramps over a data range of (vmax - vmin)
How quickly to ramp from black to white. The default of 1
ramps over a data range of (vmax - vmin)
scaling : 'linear' | 'log' | 'arcsinh' | 'sqrt' | 'power'
The type of intensity scaling to apply
The type of intensity scaling to apply
Returns
-------

108
toasty/study.py
Просмотреть файл

@ -10,7 +10,6 @@ from __future__ import absolute_import, division, print_function
__all__ = '''
StudyTiling
make_thumbnail_bitmap
tile_study_image
'''.split()
@ -86,6 +85,11 @@ class StudyTiling(object):
self._img_gy0 = (self._p2n - self._height) // 2
def n_deepest_layer_tiles(self):
"""Return the number of tiles in the highest-resolution layer."""
return 4**self._tile_levels
def apply_to_imageset(self, imgset):
"""Fill the specific ``wwt_data_formats.imageset.ImageSet`` object
with parameters defined by this tiling,
@ -232,15 +236,47 @@ class StudyTiling(object):
)
def tile_study_image(img_data, pio):
def tile_image(self, image, pio):
"""Tile an in-memory image as a study.
Parameters
----------
image : :class:`toasty.image.Image`
In-memory image data. The image's dimensions must match the ones
for which this tiling was computed.
pio : :class:`toasty.pyramid.PyramidIO`
A handle for doing I/O on the tile pyramid
Returns
-------
Self.
"""
if image.height != self._height:
raise ValueError('height of image to be sampled does not match tiling')
if image.width != self._width:
raise ValueError('width of image to be sampled does not match tiling')
buffer = image.mode.make_maskable_buffer(256, 256)
for pos, width, height, image_x, image_y, tile_x, tile_y in self.generate_populated_positions():
iy_idx = slice(image_y, image_y + height)
ix_idx = slice(image_x, image_x + width)
by_idx = slice(tile_y, tile_y + height)
bx_idx = slice(tile_x, tile_x + width)
image.fill_into_maskable_buffer(buffer, iy_idx, ix_idx, by_idx, bx_idx)
pio.write_toasty_image(pos, buffer)
return self
def tile_study_image(image, pio):
"""Tile an image as a study, loading the whole thing into memory.
Parameters
----------
img_data : array-like
An array of image data, of shape ``(height, width, nchan)``,
where nchan is 3 (RGB) or 4 (RGBA). The dtype should be compatible
with :class:`np.uint8`.
image : :class:`toasty.image.Image`
The image to tile.
pio : :class:`toasty.pyramid.PyramidIO`
A handle for doing I/O on the tile pyramid
@ -249,62 +285,6 @@ def tile_study_image(img_data, pio):
A :class:`StudyTiling` defining the tiling of the image.
"""
img_data = np.asarray(img_data)
tiling = StudyTiling(img_data.shape[1], img_data.shape[0])
buffer = np.empty((256, 256, 4), dtype=np.uint8)
if img_data.shape[2] == 3:
has_alpha = False
elif img_data.shape[2] == 4:
has_alpha = True
else:
raise ValueError('unexpected number of image channels; shape %r' % (img_data.shape,))
for pos, width, height, image_x, image_y, tile_x, tile_y in tiling.generate_populated_positions():
buffer.fill(0)
if has_alpha:
buffer[tile_y:tile_y+height,tile_x:tile_x+width] = \
img_data[image_y:image_y+height,image_x:image_x+width]
else:
buffer[tile_y:tile_y+height,tile_x:tile_x+width,:3] = \
img_data[image_y:image_y+height,image_x:image_x+width]
buffer[tile_y:tile_y+height,tile_x:tile_x+width,3] = 255
pio.write_image(pos, buffer)
tiling = StudyTiling(image.width, image.height)
tiling.tile_image(image, pio)
return tiling
def make_thumbnail_bitmap(bitmap):
"""Create a thumbnail bitmap from a :class:`PIL.Image`.
Parameters
----------
bitmap : :class:`PIL.Image`
The image to thumbnail.
Returns
-------
A :class:`PIL.Image` representing a thumbnail of the input image. WWT
thumbnails are 96 pixels wide and 45 pixels tall and should be saved in
JPEG format.
"""
THUMB_SHAPE = (96, 45)
THUMB_ASPECT = THUMB_SHAPE[0] / THUMB_SHAPE[1]
if bitmap.width / bitmap.height > THUMB_ASPECT:
# The image is wider than desired; we'll need to crop off the sides.
target_width = int(round(bitmap.height * THUMB_ASPECT))
dx = (bitmap.width - target_width) // 2
crop_box = (dx, 0, dx + target_width, bitmap.height)
else:
# The image is taller than desired; crop off top and bottom.
target_height = int(round(bitmap.width / THUMB_ASPECT))
dy = (bitmap.height - target_height) // 2
crop_box = (0, dy, bitmap.width, dy + target_height)
thumb = bitmap.crop(crop_box)
thumb.thumbnail(THUMB_SHAPE)
return thumb

10
toasty/tests/test_toast.py
Просмотреть файл

@ -1,5 +1,5 @@
# -*- mode: python; coding: utf-8 -*-
# Copyright 2013-2019 Chris Beaumont and the AAS WorldWide Telescope project
# Copyright 2013-2020 Chris Beaumont and the AAS WorldWide Telescope project
# Licensed under the MIT License.
from __future__ import absolute_import, division, print_function
@ -21,9 +21,9 @@ except ImportError:
from .. import toast
from .._libtoasty import mid
from ..io import read_image, save_png
from ..image import ImageMode
from ..samplers import plate_carree_sampler, healpix_fits_file_sampler
from ..toast import generate_images, gen_wtml, SamplingToastDataSource
from ..toast import generate_images, gen_wtml, read_image, save_png, SamplingToastDataSource
def mock_sampler(x, y):
@ -224,6 +224,6 @@ class TestSamplingToastDataSource(object):
image_test(b, a, 'Failed for %s' % subpth)
def test_default(self):
stds = SamplingToastDataSource(self.sampler)
stds.sample_image_layer(self.pio, 1)
stds = SamplingToastDataSource(ImageMode.RGB, self.sampler)
stds.sample_layer(self.pio, 1)
self.verify_toast()

241
toasty/toast.py
Просмотреть файл

@ -1,9 +1,12 @@
# -*- mode: python; coding: utf-8 -*-
# Copyright 2013-2019 Chris Beaumont and the AAS WorldWide Telescope project
# Copyright 2013-2020 Chris Beaumont and the AAS WorldWide Telescope project
# Licensed under the MIT License.
"""Computations for the TOAST projection scheme and tile pyramid format.
TODO this all needs to be ported to modern Toasty infrastructure and
wwt_data_formats.
"""
from __future__ import absolute_import, division, print_function
@ -25,17 +28,17 @@ import logging
import numpy as np
from ._libtoasty import subsample, mid
from .io import save_png, read_image
from .image import Image
from .norm import normalize
from .pyramid import Pos, depth2tiles, is_subtile, pos_parent
level1 = [
[np.radians(c) for c in row]
for row in [
[(0, -90), (90, 0), (0, 90), (180, 0)],
[(90, 0), (0, -90), (0, 0), (0, 90)],
[(0, 90), (0, 0), (0, -90), (270, 0)],
[(180, 0), (0, 90), (270, 0), (0, -90)],
[(0, -90), (90, 0), (0, 90), (180, 0)],
[(90, 0), (0, -90), (0, 0), (0, 90)],
[(0, 90), (0, 0), (0, -90), (270, 0)],
[(180, 0), (0, 90), (270, 0), (0, -90)],
]
]
@ -74,7 +77,7 @@ def toast_tile_area(tile):
Parameters
----------
tile : :class:`Tile`
A TOAST tile.
A TOAST tile.
Returns
-------
@ -107,7 +110,7 @@ def minmax_tile_filter(ra_range, dec_range):
Parameters
----------
ra_range, dec_range: (array)
The ra and dec ranges to be toasted (in the form [min,max]).
The ra and dec ranges to be toasted (in the form [min,max]).
"""
def is_overlap(tile):
@ -135,8 +138,8 @@ def nxy_tile_filter(layer,tx,ty):
Parameters
----------
layer,tx,ty: (int)
Layer and x,y coordinates, for a tile that will serve at the "super-tile"
such that all subtiles will be toasted/merged.
Layer and x,y coordinates, for a tile that will serve at the "super-tile"
such that all subtiles will be toasted/merged.
"""
@ -160,15 +163,15 @@ def _postfix_corner(tile, depth, bottom_only, tile_filter):
Parameters
----------
tile : Tile
Parameters of the current tile.
Parameters of the current tile.
depth : int
The depth to descend to.
The depth to descend to.
bottom_only : bool
If True, only yield tiles at max_depth.
If True, only yield tiles at max_depth.
tile_filter : callable
A function with signature ``tile_filter(tile) -> bool`` that determines
which tiles will be yielded; tiles for which it returns ``False`` will
be skipped.
A function with signature ``tile_filter(tile) -> bool`` that determines
which tiles will be yielded; tiles for which it returns ``False`` will
be skipped.
"""
n = tile[0].n
@ -216,18 +219,18 @@ def generate_tiles(depth, bottom_only=True, tile_filter=None):
Parameters
----------
depth : int
The tile depth to recurse to.
The tile depth to recurse to.
bottom_only : bool
If True, then only the lowest tiles will be yielded.
If True, then only the lowest tiles will be yielded.
tile_filter : callable or None (the default)
If not None, a filter function applied to the process;
only tiles for which ``tile_filter(tile)`` returns True
will be yielded
If not None, a filter function applied to the process;
only tiles for which ``tile_filter(tile)`` returns True
will be yielded
Yields
------
tile : Tile
An individual tile to process. Tiles are yield deepest-first.
An individual tile to process. Tiles are yield deepest-first.
The ``n = 0`` depth is not included.
@ -247,6 +250,18 @@ def generate_tiles(depth, bottom_only=True, tile_filter=None):
yield item
# This is where we start needing to revamp all of the I/O and pyramid-management stuff:
import PIL.Image
def save_png(pth, array):
PIL.Image.fromarray(array).save(pth)
def read_image(path):
return np.asarray(PIL.Image.open(path))
def generate_images(
data_sampler,
depth,
@ -261,41 +276,41 @@ def generate_images(
Parameters
----------
data_sampler : func or string
- A function that takes two 2D numpy arrays of (lon, lat) as input,
and returns an image of the original dataset sampled
at these locations; see :mod:`toasty.samplers`.
- A string giving a base toast directory that contains the
base level of toasted tiles, using this option, only the
merge step takes place, the given directory must contain
a "depth" directory for the given depth parameter
- A function that takes two 2D numpy arrays of (lon, lat) as input,
and returns an image of the original dataset sampled
at these locations; see :mod:`toasty.samplers`.
- A string giving a base toast directory that contains the
base level of toasted tiles, using this option, only the
merge step takes place, the given directory must contain
a "depth" directory for the given depth parameter
depth : int
The maximum depth to tile to. A depth of N creates
4^N pngs at the deepest level
The maximum depth to tile to. A depth of N creates
4^N pngs at the deepest level
merge : bool or callable (default True)
How to treat lower resolution tiles.
How to treat lower resolution tiles.
- If True, tiles above the lowest level (highest resolution)
will be computed by averaging and downsampling the 4 subtiles.
- If False, sampler will be called explicitly for all tiles
- If a callable object, this object will be passed the
4x oversampled image to downsample
- If True, tiles above the lowest level (highest resolution)
will be computed by averaging and downsampling the 4 subtiles.
- If False, sampler will be called explicitly for all tiles
- If a callable object, this object will be passed the
4x oversampled image to downsample
base_level_only : bool (default False)
If True only the bottem level of tiles will be created.
In this case merge will be set to True, but no merging will happen,
and only the highest resolution layer of images will be created.
If True only the bottem level of tiles will be created.
In this case merge will be set to True, but no merging will happen,
and only the highest resolution layer of images will be created.
tile_filter: callable (optional)
A function that takes a tile and determines if it is in toasting range.
If not given default_tile_filter will be used which simply returns True.
A function that takes a tile and determines if it is in toasting range.
If not given default_tile_filter will be used which simply returns True.
top: int (optional)
The topmost layer of toast tiles to create (only relevant if
base_level_only is False), default is 0.
The topmost layer of toast tiles to create (only relevant if
base_level_only is False), default is 0.
Yields
------
(pth, tile) : str, ndarray
pth is the relative path where the tile image should be saved
pth is the relative path where the tile image should be saved
"""
if tile_filter is None:
tile_filter = lambda t: True
@ -414,6 +429,7 @@ def _default_merge(mosaic):
mosaic[1::2, 1::2] / 4.).astype(mosaic.dtype)
# XXX TODO: this should be superseded by use of wwt_data_formats
def gen_wtml(base_dir, depth, **kwargs):
"""
Create a minimal WTML record for a pyramid generated by toasty
@ -421,27 +437,27 @@ def gen_wtml(base_dir, depth, **kwargs):
Parameters
----------
base_dir : str
The base path to a toast pyramid, as you wish for it to appear
in the WTML file (i.e., this should be a path visible to a server)
The base path to a toast pyramid, as you wish for it to appear
in the WTML file (i.e., this should be a path visible to a server)
depth : int
The maximum depth of the pyramid
The maximum depth of the pyramid
**kwargs
Keyword arguments may be used to set parameters that appear in the
generated WTML file. Keywords that are honored are:
Keyword arguments may be used to set parameters that appear in the
generated WTML file. Keywords that are honored are:
- FolderName
- BandPass
- Name
- Credits
- CreditsUrl
- ThumbnailUrl
- FolderName
- BandPass
- Name
- Credits
- CreditsUrl
- ThumbnailUrl
Unhandled keywords are silently ignored.
Unhandled keywords are silently ignored.
Returns
-------
wtml : str
A WTML record
A WTML record
"""
kwargs.setdefault('FolderName', 'Toasty')
kwargs.setdefault('BandPass', 'Visible')
@ -476,38 +492,38 @@ def toast(data_sampler, depth, base_dir,
Parameters
----------
data_sampler : func or string
- A function of (lon, lat) that samples a dataset
at the input 2D coordinate arrays
- A string giving a base toast directory that contains the
base level of toasted tiles, using this option, only the
merge step takes place, the given directory must contain
a "depth" directory for the given depth parameter
- A function of (lon, lat) that samples a dataset
at the input 2D coordinate arrays
- A string giving a base toast directory that contains the
base level of toasted tiles, using this option, only the
merge step takes place, the given directory must contain
a "depth" directory for the given depth parameter
depth : int
The maximum depth to generate tiles for.
4^n tiles are generated at each depth n
The maximum depth to generate tiles for.
4^n tiles are generated at each depth n
base_dir : str
The path to create the files at
The path to create the files at
wtml_file : str (optional)
The path to write a WTML file to. If not present,
no file will be written
The path to write a WTML file to. If not present,
no file will be written
merge : bool or callable (default True)
How to treat lower resolution tiles.
How to treat lower resolution tiles.
- If True, tiles above the lowest level (highest resolution)
will be computed by averaging and downsampling the 4 subtiles.
- If False, sampler will be called explicitly for all tiles
- If a callable object, this object will be passed the
4x oversampled image to downsample
- If True, tiles above the lowest level (highest resolution)
will be computed by averaging and downsampling the 4 subtiles.
- If False, sampler will be called explicitly for all tiles
- If a callable object, this object will be passed the
4x oversampled image to downsample
base_level_only : bool (default False)
If True only the bottem level of tiles will be created.
In this case merge will be set to True, but no merging will happen,
and only the highest resolution layer of images will be created.
If True only the bottem level of tiles will be created.
In this case merge will be set to True, but no merging will happen,
and only the highest resolution layer of images will be created.
tile_filter : callable or None (the default)
An optional function ``accept_tile(tile) -> bool`` that filters tiles;
only tiles for which the fuction returns ``True`` will be
processed.
An optional function ``accept_tile(tile) -> bool`` that filters tiles;
only tiles for which the fuction returns ``True`` will be
processed.
top_layer: int (optional)
If merging this indicates the uppermost layer to be created.
If merging this indicates the uppermost layer to be created.
"""
if wtml_file is not None:
@ -541,30 +557,29 @@ def toast(data_sampler, depth, base_dir,
class SamplingToastDataSource(object):
"""Generate tiles for a TOAST projection from a "sampler" function."""
_mode = None
"The :class:`toasty.image.ImageMode` of this data source."
_sampler = None
"The sampler callable that will produce data for tiling."
def __init__(self, sampler):
def __init__(self, mode, sampler):
self._mode = mode
self._sampler = sampler
def sample_data_layer(self, pio, depth):
"""Generate a data layer of the TOAST tile pyramid through direct sampling.
def sample_layer(self, pio, depth):
"""Generate a layer of the TOAST tile pyramid through direct sampling.
Parameters
----------
pio : :class:`toasty.pyramid.PyramidIO`
A :class:`~toasty.pyramid.PyramidIO` instance to manage the I/O with
the tiles in the tile pyramid.
A :class:`~toasty.pyramid.PyramidIO` instance to manage the I/O with
the tiles in the tile pyramid.
depth : int
The depth of the layer of the TOAST tile pyramid to generate. The
number of tiles in each layer is ``4**depth``. Each tile is 256×256
TOAST pixels, so the resolution of the pixelization at which the
data will be sampled is a refinement level of ``2**(depth + 8)``.
Notes
-----
This function will create Numpy save files, which will then have to be
converted to PNG files through some kind of colormapping process.
The depth of the layer of the TOAST tile pyramid to generate. The
number of tiles in each layer is ``4**depth``. Each tile is 256×256
TOAST pixels, so the resolution of the pixelization at which the
data will be sampled is a refinement level of ``2**(depth + 8)``.
"""
for tile in generate_tiles(depth, bottom_only=True):
@ -577,36 +592,4 @@ class SamplingToastDataSource(object):
tile.increasing,
)
sampled_data = self._sampler(lon, lat)
pio.write_numpy(tile.pos, sampled_data)
def sample_image_layer(self, pio, depth):
"""Generate an image layer of the TOAST tile pyramid through direct sampling.
Parameters
----------
pio : :class:`toasty.pyramid.PyramidIO`
A :class:`~toasty.pyramid.PyramidIO` instance to manage the I/O with
the tiles in the tile pyramid.
depth : int
The depth of the layer of the TOAST tile pyramid to generate. The
number of tiles in each layer is ``4**depth``. Each tile is 256×256
TOAST pixels, so the resolution of the pixelization at which the
data will be sampled is a refinement level of ``2**(depth + 8)``.
Notes
-----
The sampler must produce data that can be converted to a PNG image via
the :func:`toasty.io.save_png` function.
"""
for tile in generate_tiles(depth, bottom_only=True):
lon, lat = subsample(
tile.corners[0],
tile.corners[1],
tile.corners[2],
tile.corners[3],
256,
tile.increasing,
)
sampled_data = self._sampler(lon, lat)
pio.write_image(tile.pos, sampled_data)
pio.write_toasty_image(tile.pos, Image.from_array(self._mode, sampled_data))