azure-kusto-parquet-conv/arrow-rs/arrow-pyarrow-integration-testing

Arrow + PyArrow integration testing

This is a Rust crate that tests compatibility between Rust's Arrow implementation and PyArrow.

Note that this crate uses two languages and an external ABI:

• Rust
• Python
• C ABI privately exposed by Pyarrow.
• PyCapsule ABI publicly exposed by pyarrow

Basic idea

Pyarrow exposes a C ABI to convert arrow arrays from and to its C implementation, see here.

This package uses the equivalent struct in Rust (arrow::array::ArrowArray), and verifies that we can use pyarrow's interface to move pointers from and to Rust.

Relevant literature

• Arrow's CDataInterface
• Arrow PyCapsule Interface
• Rust's FFI
• Pyarrow private binds
• PyO3

How to develop

# prepare development environment (used to build wheel / install in development)
python -m venv venv
venv/bin/pip install maturin toml pytest pytz pyarrow>=5.0

Whenever rust code changes (your changes or via git pull):

source venv/bin/activate
maturin develop
pytest -v .