Joris Van den Bossche

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing, and is becoming the de facto standard for tabular data. This talk will give an overview of the recent developments both in Apache Arrow itself as how it is being adopted in the PyData ecosystem (and beyond) and can improve your day-to-day data analytics workflows.

The Apache Arrow (https://arrow.apache.org/) project specifies a standardized language-independent columnar memory format for tabular data. It enables shared computational libraries, zero-copy shared memory, efficient (inter-process) communication without serialization overhead, etc. Nowadays, Apache Arrow is supported by many programming languages and projects, and is becoming the de facto standard for tabular data. But what does that mean in practice? There is a growing set of tools in the Python bindings, PyArrow, and a growing number of projects that use (Py)Arrow to accelerate data interchange and actual data processing. This talk will give an overview of the recent developments both in Apache Arrow itself as how it is being adopted in the PyData ecosystem (and beyond) and can improve your day-to-day data analytics workflows.

Pandas has reached a 2.0 milestone in 2023. But what does that mean? And what is coming after 2.0? This talk will give an overview of what happened in the latest releases of pandas and highlight some topics and major new features the pandas project is working on.

The pandas 2.0 release is targeted for the first quarter of 2023. This is a major milestone for the pandas project, and this talk will start with an overview of this release. Pandas 2.0 includes some new (experimental) features, but mostly means enforcing deprecations that have been accumulated in the 1.x series, along with some necessary breaking changes. But that doesn’t mean there are no interesting features to talk about! The main part of the presentation will showcase some new features, both already released as opt-in features or to come in future releases. Support for non-nanosecond resolution datetimes, allowing time spans ranging over a billion of years. Improved support for nullable data types, including easy opt-in options for I/O functions. Experimental integration with pyarrow to back columns of a DataFrame (beyond the string dtype). A major change that is under way is a change to the copy and view semantics of operations in pandas (related to the well-known (or hated) SettingWithCopyWarning). This is already available as an experimental opt-in to test and use the new behaviour, and will probably be a highlight of pandas 3.0.

Other people: Patrick Hoefler

This is part of the maintainers track. In this session, we want to share some updates on the DataFrame ecosystem: the DataFrame interchange protocol (https://data-apis.org/dataframe-protocol/latest/purpose_and_scope.html) and Arrow C Data interface (https://arrow.apache.org/docs/format/CDataInterface.html), and the integration of those interoperability protocols with different libraries. Further, we want to have an open conversation about challenges and requirements related to DataFrame interoperability and supporting multiple DataFrame libraries in projects.

In this session, we want to share some updates on the DataFrame ecosystem: the DataFrame interchange protocol (https://data-apis.org/dataframe-protocol/latest/purpose_and_scope.html) and Arrow C Data interface (https://arrow.apache.org/docs/format/CDataInterface.html), and the integration of those interoperability protocols with different libraries. Further, we want to have an open conversation about challenges and requirements related to DataFrame interoperability and supporting multiple DataFrame libraries in projects.

This tutorial is an introduction to geospatial data analysis, with a focus on tabular vector data using GeoPandas. It will show how GeoPandas and related libraries can improve your workflow (importing GIS data, visualizing, joining and preparing for analysis, exploring spatial relationships, …).

This tutorial is an introduction to geospatial data analysis in Python, with a focus on tabular vector data using GeoPandas. The content focuses on introducing the participants to the different libraries to work with geospatial data and will cover munging geo-data and exploring relations over space. This includes importing data in different formats (e.g. shapefile, GeoJSON), visualizing, combining and tidying them up for analysis, and will use libraries such as pandas, geopandas, shapely, pyproj, matplotlib, folium, … Please find the material and installation instructions at https://github.com/jorisvandenbossche/geopandas-tutorial

This workshop introduces the Dask-GeoPandas library and walks you through its key components, allowing you to take a GeoPandas workflow and run it in parallel, out-of-core and even distributed on a remote cluster.

The geospatial Python ecosystem provides a nice set of tools for working with vector data, including Shapely for geometry operations and GeoPandas to work with tabular data (and many other packages for IO, visualization, domain specific processing, …). One of the limitations of those core tools is a sub-optimal performance and limited scaling possibilities. The PyData ecosystem is increasingly embracing Dask as a tool of choice when the scale of the task goes beyond capabilities of Pandas or Numpy. Over the last years, effort has been put in improving the performance through vectorized interfaces to GEOS, the underlying C library of Shapely. In turn, that enables releasing the GIL and makes the Dask - GeoPandas combination more interesting. Since GeoPandas is an extension to the pandas DataFrame, the same way how Dask scales pandas can be applied on GeoPandas as well. Initial effort to build a bridge between Dask and GeoPandas is currently taking the shape of the dask-geopandas library. This workshop provides an introduction to dask-geopandas and walks you through the key aspects of the library. We will touch the specificity of vector geospatial data when it comes to parallelisation, cover use cases where dask-geopandas provides major benefits as well as those, where it currently struggles. You will learn how to turn your GeoPandas workflow to Dask-GeoPandas one and what are the rules of thumb when doing so. Prerequisites: * basic familiarity with GeoPandas Level: * intermediate

Other people: Martin Fleischmann

GeoPandas is one of the core packages in the Python ecosystem to work with geospatial vector data. This talk will give an overview of recent developments in GeoPandas and the broader ecosystem.

GeoPandas is one of the core packages in the Python ecosystem to work with geospatial vector data. By combining the power of several open source geo tools (GEOS/Shapely, GDAL/fiona, PROJ/pyproj) and extending the pandas data analysis library to work with geographic objects, it is designed to make working with geospatial data in Python easier. This talk will give an overview of recent developments in the GeoPandas community, both in the project itself as in the broader ecosystem of packages on which GeoPandas depends or that extend GeoPandas. We will highlight some changes and new features in recent GeoPandas versions, such as the new interactive explore() visualisation method, improvements in joining based on proximity, better IO options for PostGIS and Apache Parquet and Feather files, and others. But some of the important improvements coming to GeoPandas are happening in other packages. The Shapely 2.0 release is nearing completion, and will provide fast vectorized versions of all its geospatial functionalities. This will help to substantially improve the performance of GeoPandas. In the area of reading and writing traditional GIS files using GDAL, the pyogrio package is being developed to provide a speed-up on that front. Another new project is dask-geopandas, which is merging the geospatial capabilities of GeoPandas with the scalability of Dask. This way, we can achieve parallel and distributed geospatial operations.

Other people: Martin Fleischmann

The goal of GeoPandas is to make working with geospatial vector data in python easier. GeoPandas (https://github.com/geopandas/geopandas) extends the pandas data analysis library to work with geographic objects and spatial operations.

Pandas is a package for handling and analysing tabular data, and one of the drivers of the popularity of Python for data science. GeoPandas combines the capabilities of pandas and shapely (python interface to the GEOS librabry), providing geospatial operations in pandas and a high-level interface to multiple geometries to shapely. It combines the power of whole ecosystem of geo tools by building upon the capabilities of many other libraries including fiona (reading/writing data with GDAL), pyproj (projections), rtree (spatial index), ... Further, by working together with Dask, it can also be used to perform geospatial analyses in parallel on multiple cores or distributed across a cluster. GeoPandas enables you to easily do operations in python that would otherwise require a spatial database such as PostGIS.

The growing Python data science ecosystem, including the foundational packages Numpy and Pandas, provides powerful tools for data analysis that are widely used in a variety of applications. Typically, these libraries were designed for data that fits in memory and for computations that run on a single core.

Dask is a Python library for parallel and distributed computing, using blocked algorithms and task scheduling. By leveraging the existing Python data ecosystem, Dask enables to compute on arrays and dataframes that are larger than memory, while exploiting parallelism or distributed computing power, but in a familiar interface (mirroring Numpy arrays and Pandas dataframes).

Slides: https://jorisvandenbossche.github.io/talks/2017FOSDEMdask/