Added convenience method for getting container client / filesystem (#49)

* Added convenience method for getting filesystem

.github/workflows/continuous-integration.yml

@@ -12,21 +12,14 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.8", "3.9"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: Cache dependencies
-        uses: actions/cache@v2
-        with:
-          path: ~/.cache/pip
-          key: pip-${{ hashFiles('requirements-dev.txt') }}
-          restore-keys: pip-
-
       - name: Execute linters and test suites
         run: ./scripts/cibuild

CHANGELOG.md

@@ -3,6 +3,8 @@
 ## New Features
 
 * `sign` now automatically retries failed HTTP requests.
+* Added a convenience method `planetary_computer.get_container_client` for getting an authenticated ``azure.storage.blob.ContainerClient``.
+* Added a convenience method `planetary_computer.get_adlfs_filesystem` for getting an authenticated ``adlfs.AzureBlobFileSystem``.
 
 # 0.4.7
 

README.md

@@ -19,32 +19,31 @@ planetarycomputer configure
 Alternatively, a subscription key may be provided by specifying it in the `PC_SDK_SUBSCRIPTION_KEY` environment variable. A subcription key is not required for interacting with the service, however having one in place allows for less restricted rate limiting.
 
 
-## Development
-
-The following steps may be followed in order to develop locally:
-
-```bash
-## Create and activate venv
-python3 -m venv env
-source env/bin/activate
-
-## Install requirements
-python3 -m pip install -r requirements-dev.txt
-
-## Install locally
-pip install -e .
-
-## Format code
-./scripts/format
-
-## Run tests
-./scripts/test
-```
-
-
 ## Usage
 
-This library currently assists with signing Azure Blob Storage URLs. The `sign` function operates directly on an HREF string, as well as several [PySTAC](https://github.com/stac-utils/pystac) objects: `Asset`, `Item`, and `ItemCollection`. In addition, the `sign` function accepts a [STAC API Client](https://github.com/stac-utils/pystac-client) `ItemSearch`, which performs a search and returns the resulting `ItemCollection` with all assets signed. The following example demonstrates these use cases:
+This library assists with signing Azure Blob Storage URLs. The `sign` function operates directly on an HREF string, as well as several [PySTAC](https://github.com/stac-utils/pystac) objects: `Asset`, `Item`, and `ItemCollection`. In addition, the `sign` function accepts a [STAC API Client](https://pystac-client.readthedocs.io/en/stable/) `ItemSearch`, which performs a search and returns the resulting `ItemCollection` with all assets signed.
+
+### Automatic signing
+
+If you're using pystac-client we recommend you use its feature to [automatically sign results](https://pystac-client.readthedocs.io/en/stable/usage.html#automatically-modifying-results) with ``planetary_computer.sign_inplace``:
+
+```python
+import planetary_computer
+import pystac_client
+
+from pystac_client import Client
+import planetary_computer, requests
+api = Client.open(
+   'https://planetarycomputer.microsoft.com/api/stac/v1',
+   modifier=planetary_computer.sign_inplace,
+)
+```
+
+Now all the results you get from that client will be signed.
+
+### Manual signing
+
+Alternatively, you can manually call ``planetary_computer.sign`` on your results.
 
 ```python
 from pystac import Asset, Item, ItemCollection
@@ -80,6 +79,35 @@ search = ItemSearch(
 signed_item_collection = pc.sign(search)
 ```
 
+### Convenience methods
+
+You'll occasionally need to interact with the Blob Storage container directly, rather than
+using STAC items. We include two convenience methods for this:
+
+* `planetary_computer.get_container_client`: Get an [`azure.storage.blob.ContainerClient`](https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.containerclient?view=azure-python)
+* `planetary_computer.get_adlfs_fliesystem`: Get an [`adlfs.AzureBlobFilesystem`](https://github.com/fsspec/adlfs)
+
+## Development
+
+The following steps may be followed in order to develop locally:
+
+```bash
+## Create and activate venv
+python3 -m venv env
+source env/bin/activate
+
+## Install requirements
+python3 -m pip install -r requirements-dev.txt
+
+## Install locally
+pip install -e .
+
+## Format code
+./scripts/format
+
+## Run tests
+./scripts/test
+```
 
 ## Contributing
 

planetary_computer/__init__.py

@@ -11,13 +11,17 @@ from planetary_computer.sas import (
     sign_item_collection,
 )
 from planetary_computer.settings import set_subscription_key
+from planetary_computer._adlfs import get_adlfs_filesystem, get_container_client
 
 from planetary_computer.version import __version__
 
 __all__ = [
+    "get_adlfs_filesystem",
+    "get_container_client",
     "set_subscription_key",
     "sign_asset",
     "sign_assets",
+    "sign_inplace",
     "sign_item_collection",
     "sign_item",
     "sign_url",

planetary_computer/_adlfs.py

@@ -0,0 +1,53 @@
+import typing
+
+import azure.storage.blob
+from planetary_computer.sas import get_token
+
+if typing.TYPE_CHECKING:
+    import adlfs
+
+
+def get_container_client(
+    account_name: str, container_name: str
+) -> azure.storage.blob.ContainerClient:
+    """
+    Get a :class:`azure.storage.blob.ContainerClient` with credentials.
+
+    Args:
+        account_name (str): The storage account name.
+        container_name (str): The storage container name.
+    Returns:
+        The :class:`azure.storage.blob.ContainerClient` with the short-lived SAS token
+        set as the credential.
+    """
+    token = get_token(account_name, container_name).token
+    return azure.storage.blob.ContainerClient(
+        f"https://{account_name}.blob.core.windows.net",
+        container_name,
+        credential=token,
+    )
+
+
+def get_adlfs_filesystem(
+    account_name: str, container_name: str
+) -> "adlfs.AzureBlobFileSystem":
+    """
+    Get an :class:`adlfs.AzureBlobFileSystem` with credentials.
+
+    Args:
+        account_name (str): The storage account name.
+        container_name (str): The storage container name.
+    Returns:
+        The :class:`adlfs.AzureBlobFileSystem` with the short-lived SAS token
+        set as the credential.
+    """
+    try:
+        import adlfs
+    except ImportError as e:
+        raise ImportError(
+            "'planetary_computer.get_adlfs_filesystem' requires "
+            "the optional dependency 'adlfs'."
+        ) from e
+    token = get_token(account_name, container_name).token
+    fs = adlfs.AzureBlobFileSystem(account_name, credential=token)
+    return fs

planetary_computer/sas.py

@@ -14,6 +14,7 @@ from pystac import Asset, Item, ItemCollection, STACObjectType, Collection
 from pystac.utils import datetime_to_str
 from pystac.serialization.identify import identify_stac_object_type
 from pystac_client import ItemSearch
+import pystac_client
 import urllib3.util.retry
 
 from planetary_computer.settings import Settings
@@ -336,7 +337,11 @@ def _search_and_sign(search: ItemSearch, copy: bool = True) -> ItemCollection:
         a "msft:expiry" property is added to the Item properties indicating the
         earliest expiry time for any assets that were signed.
     """
-    return sign(search.get_all_items())
+    if pystac_client.__version__ >= "0.5.0":
+        items = search.item_collection()
+    else:
+        items = search.get_all_items()
+    return sign(items)
 
 
 @sign.register(Collection)

requirements-dev.txt

@@ -3,4 +3,5 @@ flake8==4.0.1
 ipdb==0.13.9
 mypy==0.961
 types-requests==2.28.1
-setuptools==63.1.0
+setuptools==63.1.0
+pytest

scripts/cibuild

@@ -20,7 +20,7 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
         # Install/upgrade dependencies
         python -m pip install --upgrade pip
         pip install -r requirements-dev.txt
-        pip install -e .
+        pip install -e .[adlfs]
 
         ./scripts/test
     fi

scripts/test

@@ -28,6 +28,6 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
         flake8 planetary_computer tests
 
         echo "Running unit tests..."
-        python -m unittest discover tests
+        pytest -vs tests
     fi
 fi

setup.cfg

@@ -19,6 +19,9 @@ install_requires =
     pytz>=2020.5
     requests>=2.25.1
 
+[options.extras_require]
+adlfs = adlfs
+
 [options.entry_points]
 console_scripts =
     planetarycomputer = planetary_computer.scripts.cli:app

tests/test_adlfs.py

@@ -0,0 +1,25 @@
+import sys
+from typing import Any
+
+import azure.storage.blob
+import pytest
+
+import planetary_computer
+
+
+def test_get_adlfs_filesystem_raises(monkeypatch: Any) -> None:
+    monkeypatch.setitem(sys.modules, "adlfs", None)
+    with pytest.raises(ImportError):
+        planetary_computer.get_adlfs_filesystem("nrel", "oedi")
+
+
+def test_get_adlfs_filesystem() -> None:
+    fs = planetary_computer.get_adlfs_filesystem("nrel", "oedi")
+    assert fs.account_url == "https://nrel.blob.core.windows.net"
+    assert fs.credential is not None
+
+
+def test_get_container_client() -> None:
+    cc = planetary_computer.get_container_client("nrel", "oedi")
+    assert cc.primary_endpoint.startswith("https://nrel.blob.core.windows.net/oedi")
+    assert isinstance(cc, azure.storage.blob.ContainerClient)