Граф коммитов

641 Коммитов

Автор SHA1 Сообщение Дата
Peter Williams 5ce6af73ba toasty/toast.py: add generate_tiles_filtered
This will allow us to be a lot more efficient when doing chunked TOAST samplings.
2021-10-05 15:43:07 -04:00
Peter Williams 298c98746f toasty/toast.py: add toast_tile_get_coords() 2021-10-05 15:38:52 -04:00
Peter Williams 811a64eac7 toasty/toast.py: add create_single_tile()
With a slight reordering of the built-in level 1 tiles so that we can recurse simply.
2021-10-05 15:37:32 -04:00
Peter Williams 7ea2894975 toasty/pyramid.py: don't create every directory when removing lockfiles 2021-10-05 15:34:17 -04:00
Peter Williams 3bab4f779f
Merge pull request #62 from imbasimba/robustify-fits-support
Ignore BinTableHDU's and HDU's lacking shape
2021-09-30 02:41:28 +00:00
Henrik Norman a64a727703 Ignore BinTableHDU's and HDU's lacking shape 2021-09-29 18:43:43 +02:00
Peter Williams 3d23dd7ec7
Merge pull request #61 from pkgw/fits-study
Fix up tile-study with FITS files
2021-09-17 19:34:46 +00:00
Peter Williams 2a339443b8 ci/azure-build-and-test.yml: ensure we can test multi-WCS in the CI 2021-09-17 15:21:42 -04:00
Peter Williams 489d1732da toasty/tests/test_multi_tan.py: work around macOS platform differences 2021-09-17 15:12:42 -04:00
Peter Williams 301e6dda9a toasty/tests/test_multi_tan.py: add a multi-WCS test too
Inspired by the error I just found in multi-tan, let's add a test here
that processes the wcs512 sample file with the multi-wcs framework and
checks the barycenters. It works!
2021-09-17 14:37:29 -04:00
Peter Williams b1c2634d2d toasty/tests/test_multi_tan.py: update and expand
We add a test for tile-study of a FITS file here, even though there's a tile_study.py,
since this has the reference data for what the proper tiling looks like.
2021-09-17 14:31:51 -04:00
Peter Williams 8ca72e2819 toasty/multi_tan.py: correct sub-tiling width/height calculations 2021-09-17 14:31:09 -04:00
Peter Williams 01e3b02df5 docs: update API docs 2021-09-17 13:41:40 -04:00
Peter Williams 4715aaa822 toasty/study.py: correct study tiling into FITS
If we're tiling into FITS format, we have to emulate the funkiness in
the multi-WCS and multi-TAN processors: the data buffers for the
individual tiles need to be flipped vertically. So, copy-paste the code
to do that.
2021-09-17 13:38:09 -04:00
Peter Williams 2bc6ddbd81 toasty/builder.py: provide better study tiling API
It's helpful to split the study tiling operation into two steps,
preparation and execution, because (1) you can only correctly apply WCS
information after preparation and (2) it's nice to check that the WCS is
all set up correctly before spending the time on the execution step. Here
we add some new APIs to Builder and use them in tile-study

We also improve tile-study to ensure that its image has negative
(JPEG-like) parity, which is required for WWT studies.
2021-09-17 13:35:32 -04:00
Peter Williams bf509529fe
Merge pull request #60 from pkgw/misc-fixes
Miscellaneous fixes
2021-09-09 23:32:02 -04:00
Peter Williams 2aec180d02 README.md: updates 2021-09-09 23:25:23 -04:00
Peter Williams 7c61165ce8 setup.py: require astropy for docs build
This should fix the ReadTheDocs docs.
2021-09-09 23:23:08 -04:00
Peter Williams 92b9f45670
Merge pull request #59 from pkgw/check-avm
Add check-avm and AVM mode to tile-study
2021-09-09 23:08:52 -04:00
Peter Williams 29444f6b43 Add another AVM test variation 2021-09-09 22:53:05 -04:00
Peter Williams d85adb56bb Workaround AVM test different on macOS 2021-09-09 22:43:34 -04:00
Peter Williams 02be9d4377 Robustify the AVM test infrastructure a bit 2021-09-09 22:31:51 -04:00
Peter Williams 174aa9e749 Add `--avm` option to `toasty tile-study`
In an ideal world, we'd just automatically detect AVM and Do The Right
Thing, but not a lot of people are necessarily going to have pyavm
installed and we don't necessarily want to require it.

Part of this is making the decision to just unilaterally invert the
parity of the WCS that pyavm hands us based on the AVM data. From the
files that I've seen in the wild, this looks to be the correct thing to
do.
2021-09-09 22:19:03 -04:00
Peter Williams 2ffd0a995d toasty/multi_wcs.py: block warnings when deserializing WCS
This was accidentally removed in the rework to handle very large files.
2021-09-09 21:48:01 -04:00
Peter Williams 8c3fe9b233 Add `toasty check-avm`
A quick utility command to check whether an image contains AVM metadata, and
optionally dump out those data.
2021-09-08 17:08:13 -04:00
Peter Williams 2a238a7d44
Merge pull request #58 from pkgw/issue-56
toasty/builder.py: fix WWTL tiling
2021-09-08 14:05:03 -04:00
Peter Williams cd0b3f4cef toasty/builder.py: fix WWTL tiling
With some of the Builder rearrangements, we weren't properly
reinitializing the data structures anymore.

Closes #56.
2021-09-08 09:47:41 -04:00
Peter Williams 37b5626607
Merge pull request #57 from pkgw/fits-maxmin
Various FITS processing improvements
2021-09-07 11:15:41 -04:00
Peter Williams 00ff16048e toasty/multi_wcs.py: reproject images in chunks
The reproject package is pretty memory-hungry, and for large images I
find that it can get killed for memory overuse pretty easily. Here we
mitigate that by reprojecting large images in horizontal bands, AKA
"chunks". This can reduce the memory consumption significantly, and for
small images the behavior is equivalent to what we had before.
2021-09-03 17:42:46 -04:00
Peter Williams eb04f35e2a toasty/collection.py: add a mechanism for manual bad-data values 2021-09-03 12:18:32 -04:00
Peter Williams 4085733c80 toasty/image.py: prevent more annoying warnings with all-NaN data 2021-09-03 12:17:07 -04:00
Peter Williams 125e037027 toasty/collection.py: have SimpleFitsCollection work with naxis != 2
Some FITS files have additional axes of size 1. We can work with them just fine, but we need
to strip out the non-celestial axes to make the rest of our machinery happy.
2021-09-01 16:57:39 -04:00
Peter Williams b8a92c9a4f toasty/image.py: write out DATAMAX/DATAMIN with each FITS tile
Since tiles aren't big, we know it's not too slow to do the min/max operations.
2021-09-01 16:56:32 -04:00
Peter Williams 342155bd0b
Merge pull request #55 from astrodavid10/master
add tile-allsky --projection=plate-carree-panorama
2021-08-25 10:15:43 -04:00
A. David Weigel b83069c8a0
Update tile-allsky.rst 2021-08-25 00:10:19 -05:00
A. David Weigel 617ab57aae
Update builder.py 2021-08-25 00:04:59 -05:00
A. David Weigel 567a18b24a
Update cli.py 2021-08-25 00:04:16 -05:00
Peter Williams d667e3e525
Merge pull request #54 from pkgw/add-wwtl-name
tile-wwtl: add a --name argument as supported in other similar commands
2021-08-19 16:36:11 -04:00
Peter Williams b5eaf8010d tile-wwtl: add a --name argument as supported in other similar commands
Closes #53.
2021-08-19 16:14:40 -04:00
Peter Williams 39123d6df5 ci/azure-deployment.yml: fix Python publication
There's a recent change that causes a failure when the -u shell option
is turned on.
2021-08-06 15:10:07 -04:00
Peter Williams 055969ca41
Merge pull request #52 from pkgw/multi-tan
Generate proper tiled FITS datasets
2021-08-06 11:32:02 -04:00
Peter Williams b391922a66 Attempt to fix up docs CI build 2021-08-06 10:45:02 -04:00
Peter Williams 2445a858d6 toasty/multi_tan.py: add a simple parallel implementation
On my HSC test case, this makes things about 60% faster at 16x parallelism -- not
exactly linear scaling, but not bad either.
2021-08-05 19:29:26 -04:00
Peter Williams 3431bebd91 toasty/collection.py: add RubinDirectoryCollection for my LSST simulation data 2021-08-05 19:06:18 -04:00
Peter Williams 61d274d08e toasty/multi_wcs.py: update to use collections and proper parity-awareness 2021-08-05 15:25:11 -04:00
Peter Williams 408c92750f toasty/image.py: bugfix for missing PCi_i
Looks like the WCS header export will omit the on-diagonal terms if they're
unity. No big deal.
2021-08-05 13:51:59 -04:00
Peter Williams adf1fe07db toasty/multi_tan.py: completely revamp
Haven't touched this code in a while, but WWT can now display tiled FITS
so we're about to use it a lot more! The new code has to be careful
about the parities of FITS vs. tilings but I believe that we're doing
that all correctly now. We now also emit WTML using the standard
framework and otherwise modernize this code.

No parallel implementation yet, just trying to tidy up the serial
approach first.
2021-08-04 21:32:48 -04:00
Peter Williams eb3b95cc55 toasty/builder.py: standardize initialization of the imageset URL and file_type 2021-08-04 21:05:19 -04:00
Peter Williams 10679ca33f toasty/image.py: add parity-related operations
And also the ImageDescription class that was referenced in the
new "collection" module a few commits ago. I don't love the code
duplication here.
2021-08-04 17:08:23 -04:00
Peter Williams 7be88f3147 toasty/merge.py: prepare for FITS cascading
To render tiled FITS, things are a bit awkward because the individual
FITS files are rendered by WWT assuming a bottoms-up (positive parity)
data layout, but the overall tiling coordinate system is still top-down.
If you work through the geometry, the cascade process can be nearly
identical for both this case and the case where the tiles are also
top-down: all you need to change is to reorder the vertical stacking in
the cascade process. So, do that. In order for this to work, we need to
add some infrastructure to "know" the expected vertical data layout of
the pyramid tiles.
2021-08-04 16:55:30 -04:00