* Upgrade stac-fastapi / pgstac
Upgrades to an unreleased version of stac-fastapi to capture a pgstac
upgrade for item search improvements and cql2-text support. Also
upgrades fastapi version to match dependency requirements in stac-fastapi.
* Fix tests for new behavior of `limit`
When searching for ids, limit is no longer respected. You will get all
ids you requested. Two tests needed to be modified to account for this
behavior, to ensure that a paginated request was constructed, avoid
using ids and instead use a wide datetime to ensure a large response set
that can be limited.
The tiler_href env var value was updated to an incorrect value. This
fixes the appropriate setting, and adds a test to ensure that the setting
is applied as part of the generated tiler links.
TTL for table config and redis cached values is reduced to 1 second in
local development environment to prevent stale values returning during
development. In PCT environments, the table config TTL is also reduced
to 1 second to support faster iteration on config in the test
environment. The PCT redis TTL was lowered to 1 minute to still provide some
efficiency during test, but to lower the time to wait for new
items/collections to show up that were previously cached, as that a main
use case of the environment. Default values remain the same, so
staging/prod are not affected, but can now be set at a target, rather
than default, value.
* Fix ip exception table name in terraform
CI failed to deploy due to reference to undeclared resource.
* Make storage and cache top level helm config
Rather than duplicating the config for shared the resources of table
storage and redis cache, treat them as top level config similar to
postgres. Also fixes references to stac namespace from published tiler
helm chart.
* Add redis to docker-compose services
* Refactor get_request_ip into utils
* Rename TABLE_TTL -> TTL, will use it with caching
* Add redis caching and rate limiting to pccommon
* Add rate limit configuration
* Caching and rate limiting on STAC API
* Add redis to terraform
* Add redis config to helm charts
* Connect to redis for tests
* Add test for rate limit, but skip as it is nondeterministic
* Update CHANGELOG
* Implement backpressure
* Add backpressure to Helm chart
* Use decorators for rate_limit and back_pressure
* Add IP exception table that avoids rate limiting
* Get the IP from the last in the list, not first
If using an X-Forwarded-For to get the IP,
get the last one as that will be the IP coming from
the last proxy. The first IP can be anything set on
the header of the request. Since we take the
X-Azure-ClientIP header value first, this would
have not been used, but changing for best practice.
* And IP exception table to Helm chart
* Allow traffic from planetarycomputer-test
* Add ip-exception config to tiler docker-compose
Co-authored-by: Matt McFarland <mmcfarland@microsoft.com>
* Add correct PCT SAS token href for PCT tiler signing
Needed both the APIM url and the /token endpoint set in order for PCT
tiler to sign and read assets correctly.
* Consistently use quotes for jinja variables
We used a mix, and both are valid. However, my editor routinely tries to
add spaces between unquoted double braces using the JSON linter leading
to unintended syntax errors in the rendered Values file.
* Fix classmap legend serialization
ORJSONResponse requires that dict keys are strings when serializing to
JSON. The legend classmaps use numeric keys for titiler. Switching to
the default JSONResponse allows non-string keys. Adds tests to
prevent regressions.
* Add documentation for collection config classes
* Add additional dependencies to pccommon
* Fix mypy error in pctiler
Brining in the type stubs for cachetools caused
mypy to complain about unknown types for the key
function
* Refactor scripts to test pccommon
Also run flake8 on pccommon,
which wasn't happening
* Linting fixups
* Add tables classes
* Refactor collection config in pccommon
Also refactor CommonConfig to use pydantic settings.
Create a table setup for collection configuration
and container configuration.
Use cachetools to cache the configuration.
* Add Azurite setup
Encode collection configuration and container
configuration (which was hardcoded) as JSON.
This can be used to populate the initial
table structure in deployment as well, after
which this test data will diverge from production
settings.
* Update codebase to use refactored configuration
* Set azurite settings in docker-compose
Also account for environment prefix for DEBUG
that change with refactor to use BaseSettings in
CommonConfig
* Move to using only pytest for consistency
* Test get render config for naip
* Refactor config code layout
Enable configuration of TTL
* Run azurite setup in scripts/setup
Also fix setup_azurite
* Add mosaicInfo and queriables to collection config
* Remove usage of requirements.txt
This was being used inconsistently.
* Add script for local package install
* Fetch queryables from storage tables
* Use orjson in pccommon
* Use ORJSONResponse
* Remove unused endpoint prefixes
* Add mosiac/info endpoint
* Add method to fetch all rows
* Add CLI for loading and dumping config data
* Variable for k8s version; update dev
* Allow AKS to pull from ACR
* Storage Tables in terraform
* Add config table env vars to helm charts
* Update ingress apiVersion
* Make note in deploy README about updating tables
* Update CHANGELOG
* Linting/formatting
* Remove unused __init__ override
This was left over from a previous
implementation, should have been cleaned up.
* Remove unused vars in dev terraform
* Allow cli to dump configs by id
Co-authored-by: Matt McFarland <mmcfarland@microsoft.com>
* Implement queryables through proxy to online json files
* lint
* Guard against 404
* Add logic for dynamic calculation of collection queryable intersection
* Add tests and appease mypy
* Update changelog
* Improve queryable test specificity
* Use cache and guard against errors
* Add test for /queryable 404 and use proper cache key
* Use Extensions conformance classes.
Move to using only the default conformance classes and
extension-provided conformance classes.
Refactor extensions to be located in pcstac.config,
so that tests can share the same extensions.
* Guard against incorrect or duplicate c-classes
Do this by checking that any conformance class coming
from api.stacspec.org has the correct STAC API version.
* Fix STAC API version advertised by docs.
* Update api version to 1.2
The only place this is used is in the API documentation.
* Update CHANGELOG
* gnatsgo render config
* Override response limit for GET requests
Restores PC default limit of 250 items in response of GET searches and
collection item path queries.
* Update changelog
* Keep /items endpoint at default limit of 10
* Change gnatsgo collection ID to gnatsgo-rasters
* Add ability to scope test run
* Add test for search and /items page limit
* Update gnatsgo asset name
Co-authored-by: Rob Emanuele <rdemanuele@gmail.com>
* Fix describedby collection link
Use trailing slash to keep dataset path-part.
* Remove superfluous trace log
The service logs all requests with collection information, the trace is
duplicative.
* Fix titiler param bug
* Set PC_SDK_SAS_URL per environment
Previously, outside of development environments, the PC signing SDK
would use the production SAS URL in all environments. Instead, have the
default stack point to its own deployed SAS service to fully test out
integrated changes.
Additionally, change the default dev environment SAS URL to staging, as
it will most likely match any forthcoming releases to test against.
* Update changelog
* Add dedicated health check endpoint for pctiler
Also sets the liveness probe to use this new endpoint. Uses the same URL
path as pcstac, this will help isolate requests in the logs.
* Use constants for logging service name
* Fix local volume mount
The directory is copied in the Dockerfile, so it was loading on the
container, however this typo prevented reloading of pccommon when
changes were made without rebuilding the image.
* Consolidate tracing functionality
Rather than maintaining parity between two request tracing
implementations, create a common trace that can be used by both
projects.
* Prevent request tracing on health check endpoints
* Parse collection/item ids from search for logging
* Allow ACR overrides in dev deployments
Defaults to publish images for staging deploys
* Rename for Python convention
* Include request ip in trace
* Remove unused config
* Use request sensitive middleware for trace logs
Traditional middleware will corrupt usage of starlette request objects
by downstream route functions. Use a middleware class crafted for
accessing the request body without interfering with further processing.
* Lint: formatting
Auto-formatter
* Upgrade to unreleased stac-fastapi
Bring in upstream changes to stac-fastapi to include unreleased support
for cql2-json.
* Update get_collection arguments
* lint
* Update for cql2 and dynamic extensions
* CHANGELOG
Co-authored-by: Nathan Zimmerman <npzimmerman@gmail.com>
* Use exception logging to capture failures in app insights
* Format
* move global exception middleware to pccommon
* format
* fix return type of middleware
* remove unused
* Update changelog
* wip -- log encountered exceptions
* include extra metadata with exception
* remove metrics reporting
* configure liveness paths
* remove erroneous middlewares
* format
* remove more erroneous middleware
* Remove unused metrics module
* remove unused
* Remove pctiler unused
* Remove more unused
* Add special cd exception for this branch
* run on pull requests instead 🤞🏻
* correct tiler liveness env variable name
* bail on LIVENESS_PATH checks
* Format (_really_ thought I did that already)
* Remove pr trigger from cd
* Remove unused import
Co-authored-by: Rob Emanuele <rdemanuele@gmail.com>
Co-authored-by: Matt McFarland <mmcfarland@microsoft.com>
* Add helm chart publication
* Remove old cipublish path
* switch from "1" to "true"
GitHub sets CI to "true" rather than 1:
https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables
* wip -- attempt the run
* don't condition deploy step for now
* switch branch back to main
youc an use $default-branch in templates, but not
actual workflows:
https://github.blog/changelog/2020-07-22-github-actions-better-support-for-alternative-default-branch-names/
* switch branch back to main
youc an use $default-branch in templates, but not
actual workflows:
https://github.blog/changelog/2020-07-22-github-actions-better-support-for-alternative-default-branch-names/
* set azure env variables
* parse and echo from json
* don't set tenant id?
https://github.com/hashicorp/terraform-provider-azuread/issues/343#issuecomment-721455149
??
* oops i don't think these are getting passed through
* add override compose
this compose file un-sets the env variables that we don't have because
we don't have a .env file in ci, so I _think_ it should inherit them
from the environment
* add back mqe resource group
why was this deleted
am i being trolled
* Add path debugging
* check what tf keys we got
* remove api management from ingress 🤞🏻
* all container registries through github
* remove debug prints from jinja
* re-remove the mqe resource group
because it was deliberately missing, not accidentally
* skip tests for a sec
* debug echo cluster name and rg
* require az account env variables
* reuse tf env variable names
* don't point to `latest` tag
* Add workflow for publishing helm charts to GH Pages.
* Rename charts, separate published, reset versions
* Use -dev suffix for development release
* Publish dev charts only on main
* Rename mqe -> stac, dqe -> tiler in codebase
* Update README
Rename MQE to STAC API and DQE to Tiler; also editorally make things more terse
* Remove testdata, move loadtestdata to tests
* Remove out of date and generic docs for tiler
* Use cipublish to publish images
* Rename python packages to be prefixed with 'pc'
* Remove stac-vrt
* Remove unused model and method.
Also formatting
* Remove scripts/env
* Upgrade to stac-fastapi 2.2.0
* Delete unusued doc images
* Update deployment code with renames.
* Update cert-manager, other deployment fixes
* Test on cibuild, remove GA test branch trigger
* Fixup PR template
Co-authored-by: Nathan Zimmerman <npzimmerman@gmail.com>
Co-authored-by: James Santucci <james.santucci@gmail.com>
* order docs
* rewrap all the things
* move headers out a level
* add all the tables of contents
* fill in PQE_METADATA_URL with prod url
* fill in PQE_DATA_URL with prod url
* fill in docs URLs
* correct some statements in deployment doc
* re-initialize changelog in chan-compatible format
* correct branch name again
* Remove general planetary computer docs
Tom Augspurger
Matt McFarland
Rob Emanuele
Vincent Sarago
Nathan Zimmerman
Pete Gadomski
Mark Mathis
James Santucci
Microsoft Open Source