* init
* update
* update flake8 config
* code format changes
* fix long lines
* fix imports
* function app changes
* code format changes
* test
* remove readme
* remove redundant packages and revert python version
* revert python version
* remove redundant type hints
* use settings class inherited from baseSettings
* change kql
* use placeholder for timer schedule
* change test parameter
* update assertions and logger
* remove import
* remove dash in table name
* update test id
* format
* add no-integration flag
* add dependencies
* test
* change trigger to run every hour
* use azure clients as context manager
* add context managers in test
* role assignment for function app
* change LAW name
* change role
* change provider's name
* change name of LAW
* better readability
* better readability
* format
* add logging
* UPDATE function settings
* suppress mypy warning
* update env variables for function app
* typo
---------
Co-authored-by: elay <yileihu@microsoft.com>
* Set root-path from uvicorn
Sets the root path in uvicorn to match that set via FastAPI. This is
required with updates to starlette, which were included previously.
Fixes https://github.com/microsoft/PlanetaryComputer/issues/360
* New black formatting
* Add map token endpoint to tiler service
The tiler service will generate a token for use against an azure maps
instance, using the identity of the tiler (when deployed) or the local
developer credentials (in local development).
A test has been added that requires a local identity, and this has been
skipped in CI, which does not have access to those kind of credentials.
This endpoint will be used by the Data Catalog app to avoid distributing
an azure maps key within that application.
* Remove unneeded role assignment
* Remove unused variables
* Consolidate dev dependencies into requirements-dev.txt
* Add simple azure storage support to pccommon
* Set up azurite container for dev output
* Add pcfuncs codebase
This includes the "animation" and "image" functions,
as well as the "funclib" library that contains
reusable code for fetching rasters from the
PC data API.
* Add pcfuncs to dev servers and processes
* Add Azure Functions to deployment
* Move funcs information to main README
* Fix titiler link
* Avoid flake8 on setup.py
* Delete unused reprojection utility method/type
* Remove debug log information
* Don't restrict concurrency setting
* Upgrade uvicorn
* Pin version of Azurite due to bug in upsert
A recent release of Azurite caused tests with upserts to fail due to
https://github.com/Azure/Azurite/issues/1565.
* Consolidate how dev depenencies are installed
Previous to this change pcstac and pctiler had
two methods for installing dev dependencies.
This caused different behavior between the two
test projects. For instance, one project auto
upgraded the pytest asyncio plugin, which
under a new version started printing warnings.
This users the setup.py method for installing dev
dependencies consistently and pins some test dependencies
* Use asyncio_mode = auto to quell warnings
* Fix dependencies in docker-compose
* Use hostname in caching key.
Previous to this change, requests coming in from
different hostnames were causing the cache to return
results for different hostnames if that service was
accessible through both - this is the case after a release,
when planetarycomputer-staging.msft.com and planetarycomputer.msft.com
both point to the same stack. This was causing cached
results to be returned with links for staging through
requests to the production endpoints. Added the hostname
to the cache key will ensure cached results are scoped
to the specific hostname.
* Update CHANGELOG
This commit moves the environment of the
tiler and stac services into an env file.
This can be overridden by setting the
PC_STAC_ENV_FILE and PC_TILER_ENV_FILE
in order to more easily switch things like
database connections.
* Allow GDAL to rety certain HTTP failures
* Use uvicorn for stac api
This matches the change for pctiler in #67
* Clean up env vars
Some env vars were used only in gunicorn, which was removed previously.
Other helm values were nested under the wrong keys.
* Specify min/max connection size for pcstac at 1
This overrides the stac-fastapi default of 10, and ensures a single
instance only opens a single database connection.
* Add CQL_TEXT conformace class
* Updates from ./scripts/format
TTL for table config and redis cached values is reduced to 1 second in
local development environment to prevent stale values returning during
development. In PCT environments, the table config TTL is also reduced
to 1 second to support faster iteration on config in the test
environment. The PCT redis TTL was lowered to 1 minute to still provide some
efficiency during test, but to lower the time to wait for new
items/collections to show up that were previously cached, as that a main
use case of the environment. Default values remain the same, so
staging/prod are not affected, but can now be set at a target, rather
than default, value.
* Add redis to docker-compose services
* Refactor get_request_ip into utils
* Rename TABLE_TTL -> TTL, will use it with caching
* Add redis caching and rate limiting to pccommon
* Add rate limit configuration
* Caching and rate limiting on STAC API
* Add redis to terraform
* Add redis config to helm charts
* Connect to redis for tests
* Add test for rate limit, but skip as it is nondeterministic
* Update CHANGELOG
* Implement backpressure
* Add backpressure to Helm chart
* Use decorators for rate_limit and back_pressure
* Add IP exception table that avoids rate limiting
* Get the IP from the last in the list, not first
If using an X-Forwarded-For to get the IP,
get the last one as that will be the IP coming from
the last proxy. The first IP can be anything set on
the header of the request. Since we take the
X-Azure-ClientIP header value first, this would
have not been used, but changing for best practice.
* And IP exception table to Helm chart
* Allow traffic from planetarycomputer-test
* Add ip-exception config to tiler docker-compose
Co-authored-by: Matt McFarland <mmcfarland@microsoft.com>
* Add additional dependencies to pccommon
* Fix mypy error in pctiler
Brining in the type stubs for cachetools caused
mypy to complain about unknown types for the key
function
* Refactor scripts to test pccommon
Also run flake8 on pccommon,
which wasn't happening
* Linting fixups
* Add tables classes
* Refactor collection config in pccommon
Also refactor CommonConfig to use pydantic settings.
Create a table setup for collection configuration
and container configuration.
Use cachetools to cache the configuration.
* Add Azurite setup
Encode collection configuration and container
configuration (which was hardcoded) as JSON.
This can be used to populate the initial
table structure in deployment as well, after
which this test data will diverge from production
settings.
* Update codebase to use refactored configuration
* Set azurite settings in docker-compose
Also account for environment prefix for DEBUG
that change with refactor to use BaseSettings in
CommonConfig
* Move to using only pytest for consistency
* Test get render config for naip
* Refactor config code layout
Enable configuration of TTL
* Run azurite setup in scripts/setup
Also fix setup_azurite
* Add mosaicInfo and queriables to collection config
* Remove usage of requirements.txt
This was being used inconsistently.
* Add script for local package install
* Fetch queryables from storage tables
* Use orjson in pccommon
* Use ORJSONResponse
* Remove unused endpoint prefixes
* Add mosiac/info endpoint
* Add method to fetch all rows
* Add CLI for loading and dumping config data
* Variable for k8s version; update dev
* Allow AKS to pull from ACR
* Storage Tables in terraform
* Add config table env vars to helm charts
* Update ingress apiVersion
* Make note in deploy README about updating tables
* Update CHANGELOG
* Linting/formatting
* Remove unused __init__ override
This was left over from a previous
implementation, should have been cleaned up.
* Remove unused vars in dev terraform
* Allow cli to dump configs by id
Co-authored-by: Matt McFarland <mmcfarland@microsoft.com>
* Fix describedby collection link
Use trailing slash to keep dataset path-part.
* Remove superfluous trace log
The service logs all requests with collection information, the trace is
duplicative.
* Fix titiler param bug
* Set PC_SDK_SAS_URL per environment
Previously, outside of development environments, the PC signing SDK
would use the production SAS URL in all environments. Instead, have the
default stack point to its own deployed SAS service to fully test out
integrated changes.
Additionally, change the default dev environment SAS URL to staging, as
it will most likely match any forthcoming releases to test against.
* Update changelog
* Add dedicated health check endpoint for pctiler
Also sets the liveness probe to use this new endpoint. Uses the same URL
path as pcstac, this will help isolate requests in the logs.
* Use constants for logging service name
* Fix local volume mount
The directory is copied in the Dockerfile, so it was loading on the
container, however this typo prevented reloading of pccommon when
changes were made without rebuilding the image.
* Consolidate tracing functionality
Rather than maintaining parity between two request tracing
implementations, create a common trace that can be used by both
projects.
* Prevent request tracing on health check endpoints
* Parse collection/item ids from search for logging
* Allow ACR overrides in dev deployments
Defaults to publish images for staging deploys
* Rename for Python convention
* Include request ip in trace
* Remove unused config
* Use request sensitive middleware for trace logs
Traditional middleware will corrupt usage of starlette request objects
by downstream route functions. Use a middleware class crafted for
accessing the request body without interfering with further processing.
* Lint: formatting
Auto-formatter
* Add helm chart publication
* Remove old cipublish path
* switch from "1" to "true"
GitHub sets CI to "true" rather than 1:
https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables
* wip -- attempt the run
* don't condition deploy step for now
* switch branch back to main
youc an use $default-branch in templates, but not
actual workflows:
https://github.blog/changelog/2020-07-22-github-actions-better-support-for-alternative-default-branch-names/
* switch branch back to main
youc an use $default-branch in templates, but not
actual workflows:
https://github.blog/changelog/2020-07-22-github-actions-better-support-for-alternative-default-branch-names/
* set azure env variables
* parse and echo from json
* don't set tenant id?
https://github.com/hashicorp/terraform-provider-azuread/issues/343#issuecomment-721455149
??
* oops i don't think these are getting passed through
* add override compose
this compose file un-sets the env variables that we don't have because
we don't have a .env file in ci, so I _think_ it should inherit them
from the environment
* add back mqe resource group
why was this deleted
am i being trolled
* Add path debugging
* check what tf keys we got
* remove api management from ingress 🤞🏻
* all container registries through github
* remove debug prints from jinja
* re-remove the mqe resource group
because it was deliberately missing, not accidentally
* skip tests for a sec
* debug echo cluster name and rg
* require az account env variables
* reuse tf env variable names
* don't point to `latest` tag
* Add workflow for publishing helm charts to GH Pages.
* Rename charts, separate published, reset versions
* Use -dev suffix for development release
* Publish dev charts only on main
* Rename mqe -> stac, dqe -> tiler in codebase
* Update README
Rename MQE to STAC API and DQE to Tiler; also editorally make things more terse
* Remove testdata, move loadtestdata to tests
* Remove out of date and generic docs for tiler
* Use cipublish to publish images
* Rename python packages to be prefixed with 'pc'
* Remove stac-vrt
* Remove unused model and method.
Also formatting
* Remove scripts/env
* Upgrade to stac-fastapi 2.2.0
* Delete unusued doc images
* Update deployment code with renames.
* Update cert-manager, other deployment fixes
* Test on cibuild, remove GA test branch trigger
* Fixup PR template
Co-authored-by: Nathan Zimmerman <npzimmerman@gmail.com>
Co-authored-by: James Santucci <james.santucci@gmail.com>
Matt McFarland
elayrocks
Gustavo Hidalgo
Rob Emanuele
Vincent Sarago