aa4ed8f719
Merge pull request #94 from pkgw/fix-macosx-reporting
...
toasty/pyramid.py: avoid crash on macOS/Py3.7
2022-09-07 17:57:06 +00:00
6e2ba04cc1
toasty/pyramid.py: avoid crash on macOS/Py3.7
...
Discovered in conda-forge build process. Apparently Queue.qsize() just
doesn't work!
2022-09-07 13:47:50 -04:00
46a32c2d7f
Merge pull request #93 from pkgw/concept-doi
...
ci/zenodo.json5: fill in the Zenodo concept record ID
2022-09-07 01:39:47 +00:00
aef1b52d25
ci/zenodo.json5: fill in the Zenodo concept record ID
...
This information will allow us to tie together the series of Toasty
releases in Zenodo's version-DOI scheme.
2022-09-06 16:53:22 -04:00
9107ee90e7
Merge pull request #92 from pkgw/work-around-dnf
...
ci/azure-build-and-test.yml: turn off docs linkcheck so that builds can fully pass
2022-09-06 19:32:38 +00:00
fe0ee22ed1
ci/azure-build-and-test.yml: turn off docs linkcheck so that builds can fully pass
2022-09-06 15:23:30 -04:00
cc1b6f01f0
Merge pull request #90 from pkgw/pyramid-leaves
...
Add Pyramid.visit_leaves()
2022-09-06 15:18:09 +00:00
9f2fb70eb2
toasty/pyramid.py: use makedirs=False in Pyramid.read_image()
...
For DECaPS2 pixel-counting, I/O throughput is becoming a big deal. We
shouldn't makedirs when reading from pyramids in general. Also add some
other performance notes -- making sure to specify the default PyramidIO
format is important for large pyramids.
2022-09-04 10:12:55 -04:00
84110233fc
docs/api: update
2022-09-03 09:26:35 -04:00
757ccda80f
toasty/toast.py: switch sampling to use Pyramid.visit_leaves()
2022-09-03 09:22:31 -04:00
05a5089662
toasty/pyramid.py: add Pyramid.visit_leaves()
2022-09-03 09:06:36 -04:00
c1d5d03a18
Merge pull request #89 from pkgw/pyramid
...
Add a new `Pyramid` class for dealing with pyramids more robustly
2022-09-03 12:29:05 +00:00
f58ef474f0
toasty/merge.py: fix up docs here
2022-09-02 21:59:46 -04:00
71113b3364
toasty/pyramid.py: add Pyramid.count_leaf_tiles()
2022-09-02 21:40:15 -04:00
b88276a90b
toasty/pyramid.py: tidy the reduction iterator a little bit
2022-09-02 21:32:14 -04:00
6a18814be1
toasty/pyramid.py: write up new docs
2022-09-02 21:30:05 -04:00
f752b58e17
toasty/samplers.py: avoid Numpy warning
2022-09-02 20:24:15 -04:00
5227f995d1
toasty/pyramid.py: short-circuit walk if nothing to do
2022-09-02 20:22:19 -04:00
71abaa8485
Some tidying
2022-09-02 20:22:19 -04:00
98cd82b6ad
toasty/merge.py: migrate to new Pyramid API
...
This file now becomes much shorter.
2022-09-02 20:22:19 -04:00
3194f523a1
toast/pyramid.py: add new Pyramid object
...
This encapsulates the pyramid-walking needed by the `merge` module in a
more uniform and flexible interface. Documentation to be added after I
do more work to validate the implementation.
2022-09-02 20:22:19 -04:00
323e792770
toasty/toast.py: apply filter to level-1 tiles
...
This is just more accurate, and I'm preeeety sure that the lat/lon
tile filter can handle the tile coordinates ...
2022-09-02 20:18:04 -04:00
cd662ec2b6
Merge pull request #88 from pkgw/no-empty
...
Tweaks based on DECaPS2 processing
2022-09-03 00:17:12 +00:00
d9bd2eb136
toasty/progress.py: write API docs
2022-09-02 20:04:50 -04:00
f4e12fb881
toasty/progress.py: use TTY-style output if we're in Jupyter
2022-09-02 20:03:13 -04:00
2ee4ff4f1d
Reformat various files with "black"
2022-09-02 19:45:53 -04:00
029857c668
Port uses of tqdm to the progress_bar wrapper
2022-09-02 19:45:33 -04:00
a66eb197c7
toasty/progress.py: add this
2022-09-02 19:31:33 -04:00
214abd651b
Do not write completely masked images
2022-09-01 22:19:19 -04:00
13ea3eaa65
Merge pull request #87 from pkgw/remote-view
...
Add tunneled "view" capability
2022-08-31 21:31:01 +00:00
7b8e89b74a
toasty/cli.py: robustify tunneled view against funny filenames
2022-08-31 16:31:14 -04:00
e1a61905e1
Update the docs for the "view" command
2022-08-31 16:25:27 -04:00
feac7570f1
setup.py: slight tidying
2022-08-31 16:25:27 -04:00
c1a801c0cd
Tunneled view functionality requires wwt_data_formats >=0.15
2022-08-31 16:25:27 -04:00
bf47f50425
toasty/cli.py: checkpoint work on tunneled "view" functionality
2022-08-31 16:25:27 -04:00
698787082c
toasty/cli.py: add --tile-only mode to "view"
2022-08-31 16:25:27 -04:00
9c49c5c846
Merge pull request #86 from imbasimba/toast-from-array-wcs
...
WCS sampling & FITS to TOAST FITS
2022-08-31 20:23:34 +00:00
e9a8774280
Cythonize the `latlon_tile_filter` function
...
In processing a FITS-to-TOAST dataset going down to level 14, with about
24 million tiles to process, I discovered that the tile filter function
was a non-trivial bottleneck -- it took about 10 minutes just to count
the tiles that needed to be processed! Some profiling showed that most
of the time was actually in the Numpy min/max functions. Here we
Cythonize the function and avoid the Numpy functions. My measurements
indicate a speedup of a factor of around 20.
2022-08-31 15:46:23 -04:00
f5b85adae2
toasty/samplers.py: no longer need to rename function arguments
2022-08-31 15:46:23 -04:00
8c07b23c6c
toasty/samplers.py: use some precomputed pi constants
2022-08-31 15:46:23 -04:00
3a7781912a
toasty/merge.py: make tiles_to_process a set, not a list
...
The "initiate readiness state" algorithm repeatedly tests whether
positions are in the tiles_to_process set. If large numbers of tiles
(tens of millions) are being processed, the algorithm basically fails
because such tests have O(n) scaling with lists. With a set, things
become tractable.
There may well be further room for improvement here, but this one change
makes an enormous difference for big problems.
2022-08-31 15:46:23 -04:00
0e6ceaced2
toasty/fits_tiler.py: wire in add_place_for_toast option
2022-08-29 17:26:31 -04:00
19ef824067
toasty/collection.py: improve DASCH hack/fix
...
As noted in the comments, my DASCH sample file that crosses the pole has
WCS headers that seem to have another problem beyond their specification
of the TPV polynomial distortions. I need to set LONPOLE explicitly to
get the correct orientation on the sky.
2022-08-29 11:39:21 -04:00
d0d18f44b4
toasty/fits_tiler.py: fix method autodetection logic
2022-08-29 11:33:46 -04:00
56d828b03b
Updated TOAST test with new out_dir rules
2022-08-25 14:16:10 +02:00
d3f950e586
Setting distinct out dir for TOAST and TAN tiled FITS - to not override the other if it exists
2022-08-25 09:31:51 +02:00
9fe718451f
tile_toast can now handle a collection of FITS files
2022-08-24 18:30:08 +02:00
07832f7710
toasty/fits_tiler.py: check if hips_{pixel_cut,data_range} are present
...
In one test image I have, hipsgen produces a "BAD PIXEL CUT" warning and
doesn't emit this information. (I think it's because the file has an
integer data type.) Avoid the crash if this is the case.
2022-08-24 08:07:23 -04:00
8c45b1c45c
Fixed supposed typo in redifine_lon: max -> min for hi index
2022-08-24 11:38:52 +02:00
4b95075d9c
tile_toast now properly adheres to cli_progress parameter
2022-08-24 08:53:26 +02:00
Peter Williams
Henrik Norman