Automated aircraft defect risk analysis utilising maintenance and pilot reports

Abstract

This paper demonstrates Natural Language Processing (NLP) on a dataset of tens-of-thousands of maintenance and pilot reports for a fleet of aircraft, which includes the defect description text and the scheduled departure delay in minutes or if the flight was cancelled. The NLP framework is developed by fine-tuning pretrained Large Language Models (LLMs) for predicting the risk level that a defect report poses to fleet availability. The model is trained on a portion of the reports, utilising the delay duration or cancellation status to determine a low or high impact based on time threshold values. This LLM approach goes beyond using hard coded rules based on individual words in reports and enables more nuanced contextual analysis of all report text that May not be apparent at an individual report level. Results for fine-tuned pretrained LLMs demonstrate an overall test accuracy of 72%, with a greater accuracy of 90% when filtering for higher confidence predictions of low or high impact reports. The impact and likelihood of reoccurrence based on dates determines the overall risk of aircraft parts, location and condition across the fleet. Subsequently, the fine-tuned model is applied to Federal Aviation Administration (FAA) Service Difficulty Reporting System (SDRS) data, presenting an alternative analytical view to conventional reliability reporting of the risks to fleet aircraft. Furthermore, it is demonstrated that fine-tuned LLMs can distinguish reports for part failures with greater accuracy than legacy NLP processes, which could save hours in reliability analysis. Overall, these NLP approaches could enable aircraft maintainers to make decisions on how to manage resources, vary maintenance schedules and optimise sustainment programs with a predictive risk model that can be applied to new incoming maintenance and pilot reports.