Eclipse Aerospace brings two decades of flight data into Databricks

Eclipse Aerospace has modernized the ingestion of nearly 20 years of proprietary flight data with Databricks, giving service centers and engineers faster access to flight histories. The approach replaces a multi-stage decoding pipeline with a Spark-native path built around PySpark custom data sources and the company’s existing Rust decoder.
- Eclipse Aerospace stores more than 3TB of flight data spanning nearly 20 years.
- The data covers more than a million flight hours and uses proprietary formats.
- A Rust decoder now runs through a PySpark custom data source inside Spark.
- Databricks reports twice the performance of a similar PyArrow-based pure-Python source.
- Databricks reports nearly 20x the performance of a native Python implementation without Arrow.
A proprietary format created an ingestion bottleneck
Eclipse Aerospace stores more than 3TB of flight data in Databricks, covering more than a million flight hours. The data comes from near-real-time satellite telemetry, bulk uploads of complete flight histories and external sources, but all of the telemetry uses proprietary file formats that require custom decoders.
Before PySpark custom data sources became available in Databricks Runtime 15.4 LTS, the company had three main options. It could load files as binary data and split them with Python user-defined functions, use a Scala decoder on dedicated compute, or decode the files into an intermediate format before loading them into Databricks.
According to Databricks, the intermediate approach used a Rust decoder on AWS Lambda to create Avro files. That added latency, duplicated schema logic, reduced visibility in Spark monitoring and required the company to store both the original binary files and their Avro copies.
Rust decoding now runs inside Spark
The new design wraps Eclipse Aerospace’s existing Rust decoder in a Python package built with PyO3 and Maturin. The decoder itself remains separate from Spark, while a thin PySpark layer handles options, partitioning and schema definition.
Databricks says the design uses Apache Arrow as the interface between Rust and Python. Rust creates an Arrow RecordBatch, which is passed to Python as a pyarrow.RecordBatch through the Arrow C Data Interface. This avoids a serialization step and does not create individual Python objects for every row.
Because decoding now takes place inside Spark, the ingestion process can use the Spark UI, job metrics and downstream Delta workflows. Databricks Serverless also allocates and manages on-demand compute, so Eclipse Aerospace does not need to keep dedicated infrastructure running for rapid data access.
Performance and platform implications
Databricks reports that the Rust-backed Python package delivered twice the benchmarked performance of a similar PyArrow-based pure-Python data source. Compared with a native Python implementation without Arrow, it delivered nearly 20x the performance, according to the company.
PySpark custom data sources are based on the Python Data Source API introduced in Apache Spark 4.0. Databricks says the API is generally available from Databricks Runtime 15.4 LTS and is supported on Serverless environment 2 and later. Since the API is implemented in Python, users do not need to build a separate JVM connector or JAR.
For Eclipse Aerospace, the result is a simpler route from specialized flight data to analytics, with less duplicated storage and fewer operational handoffs. Databricks says the same approach can help teams work with other formats that mainstream analytics engines cannot access directly, including HDF5 for scientific data and ARINC 429 for aerospace data.
Databricks: Eclipse Aerospace simplifies flight-data ingestion with Databricks

