Software


Pangeo

Pangeo is a community effort for big data in the geosciences. As part of this effort, we are bringing together the a collection of Python tools, including Xarray, Dask, and Jupyter to facilitate highly scalable data analytics on large geoscientific datasets. I am one of the co-founders of the Pangeo project and sit on the Pangeo Steering Council.

Xarray

Xarray is the fusion of Python’s Pandas, Numpy, and netCDF4 packages. It offers unparalleled computational ability on labeled N-dimensional arrays. I have been contributing to the Xarray project since May 2014.

Dask

Dask is Python a library for scalable computing with dynamic task scheduling. As part of my work with Pangeo and Xarray, I have been contributing to the Dask ecosystem. In particular, I have worked extensively on Dask deployment utilities like Dask-jobqueue and Dask-Kubernetes.

Zarr

Zarr is a data format and for chunked, compressed, multidimensional arrays. It is designed for working with large scientific datasets and supports multiple backend storage interfaces (cloud storage, https, posix, etc.). I am on the Zarr core team and have mostly focused on optimizations for working with Zarr from Xarray and Dask.