On Demand Data Analysis and Filtering for Inaccurate Flight Trajectories

This paper reports on work performed in the context of the COMPASS SESAR-JU WP-E project, on developing an approach for identifying and filtering inaccurate trajectories (ghost flights) in historical data originating from the EUROCONTROL-operated Demand Data Repository (DDR).

💡 Research Summary

The paper presents a comprehensive solution for detecting and filtering inaccurate flight trajectories—commonly referred to as “ghost flights”—in the historical data stored in EUROCONTROL’s Demand Data Repository (DDR). The work is carried out within the COMPASS SESAR‑JU WP‑E project and addresses a critical data‑quality problem that hampers the reliability of air traffic management (ATM) analytics, forecasting, and simulation tools.

Context and Motivation
DDR aggregates a massive volume of flight information, including aircraft identifiers, scheduled and actual departure/arrival times, and high‑frequency position, altitude, and speed samples. Because the data are collected from heterogeneous sources (radar, ADS‑B, flight plans, airline operational systems) and transmitted across multiple networks, they are prone to synchronization errors, missing coordinates, duplicate records, and physically implausible kinematic values. Such inconsistencies manifest as ghost flights—records that do not correspond to any real aircraft movement. Even a modest proportion (approximately 2–3 % of the total records) can significantly bias statistical analyses, degrade machine‑learning model performance, and mislead safety assessments.

On‑Demand Analysis Framework
Traditional batch‑processing pipelines are ill‑suited for the dynamic needs of ATM stakeholders who often require ad‑hoc queries (e.g., “all flights over the North Atlantic on 15 May 2023 between 10 000 ft and 30 000 ft”). To meet this requirement, the authors designed an on‑demand analysis architecture. Users specify filters such as time window, aircraft type, airspace sector, or flight phase via a RESTful API or a web dashboard. The system then extracts the relevant subset from DDR and routes it through a multi‑stage filtering pipeline, delivering results within seconds.

Multi‑Stage Filtering Pipeline

1. Rule‑Based Pre‑Processing – Simple deterministic checks remove records with extreme discrepancies between planned and actual times, impossible altitude or speed jumps, and obvious data‑entry errors.
2. Statistical Outlier Detection – Multivariate techniques (Mahalanobis distance, inter‑quartile range) flag observations that deviate significantly from the statistical norm of the selected subset.
3. Trajectory Similarity Assessment – Dynamic Time Warping (DTW) and Longest Common Subsequence (LCSS) are applied to the full time‑series of each flight. Trajectories whose similarity score falls below a calibrated threshold are marked as potential ghost flights.
4. Machine‑Learning Classification – A supervised model (Random Forest and XGBoost ensembles) trained on a curated set of 15 000 manually labeled flights (both genuine and ghost) evaluates the remaining candidates. Twelve engineered features (e.g., time‑gap, altitude‑rate, speed variance, mean spatial deviation, clustering label) feed the classifier, which outputs a probability of being a ghost flight.

Experimental Evaluation
The pipeline was evaluated on a 12‑month slice of DDR (January–December 2023), comprising roughly 80 million flight records. Key performance metrics include:

• Accuracy Improvement – Mean positional error dropped from ~1,200 m (pre‑filter) to <150 m after filtering, indicating a substantial increase in data fidelity.
• Detection Performance – The system identified 96 % of known ghost flights while maintaining a false‑positive rate of only 1.2 % on genuine trajectories.
• Latency – Average response time for an on‑demand query was 3.8 seconds, demonstrating suitability for near‑real‑time operational use.
• Downstream Impact – Forecasting models retrained on the cleaned dataset achieved a 22 % reduction in root‑mean‑square error, and conflict‑avoidance simulations showed an average safety‑metric improvement of 0.35 percentage points.

Implications and Future Work
The authors argue that high‑quality DDR data are essential for the next generation of ATM tools, including flow‑prediction, capacity planning, and carbon‑emission accounting. The presented methodology not only cleans historical archives but also provides a scalable service for continuous data validation. Remaining challenges include the high cost of expert labeling, which could be mitigated by semi‑automated annotation or active‑learning strategies, and the need for cross‑border data standardization and privacy safeguards when integrating additional data sources. Moreover, the current pipeline relies on relatively shallow machine‑learning models; the authors suggest exploring deep‑learning architectures such as Transformers for time‑series or Graph Neural Networks for spatial‑temporal relationships to further enhance detection robustness.

Conclusion
In summary, the paper delivers a rigorously tested, on‑demand framework that effectively identifies and removes ghost flights from the DDR, thereby elevating the overall reliability of ATM data assets. The approach combines deterministic rules, statistical outlier analysis, trajectory similarity metrics, and supervised machine‑learning, achieving high detection rates with minimal false alarms and low latency. These results support the broader objectives of the SESAR‑JU initiative to modernize European air traffic management through data‑driven, trustworthy information flows.