Informative frame classification of endoscopic videos using convolutional neural networks and hidden Markov models

The goal of endoscopic analysis is to find abnormal lesions and determine further therapy from the obtained information. For example, in case of Barrett’s esophagus, the objective of endoscopy is to timely detect dysplastic lesions, before endoscopic resection is no longer possible. However, the procedure produces a variety of non-informative frames and lesions can be missed due to poor video quality. Especially when analyzing entire endoscopic videos made by non-expert endoscopists, informative frame classification is crucial to e.g. video quality grading. This analysis involves classification problems such as polyp detection or dysplasia detection in Barrett’s Esophagus. This work concentrates on the design of an automated indication of informativeness of video frames. We propose an algorithm consisting of state-of-the-art deep learning techniques, to initialize frame-based classification, followed by a hidden Markov model to incorporate temporal information and control consistent decision making. Results from the performed experiments show that the proposed model improves on the state-of-the-art with an F1-score of 91%, and a substantial increase in sensitivity of 10%, thereby indicating improved labeling consistency. Additionally, the algorithm is capable of processing 261 frames per second, which is multiple times faster compared to other informative frame classification algorithms, thus enabling real-time computation.