Informative frame classification of endoscopic videos using convolutional neural networks and hidden Markov models

Joost van der Putten, Jeroen de Groof, Fons van der Sommen, Maarten Struyvenberg, Svitlana Zinger, Wouter Curvers, Erik Schoon, Jacques Bergman, Peter H.N. de With

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

12 Citations (Scopus)


The goal of endoscopic analysis is to find abnormal lesions and determine further therapy from the obtained information. For example, in case of Barrett’s esophagus, the objective of endoscopy is to timely detect dysplastic lesions, before endoscopic resection is no longer possible. However, the procedure produces a variety of non-informative frames and lesions can be missed due to poor video quality. Especially when analyzing entire endoscopic videos made by non-expert endoscopists, informative frame classification is crucial to e.g. video quality grading. This analysis involves classification problems such as polyp detection or dysplasia detection in Barrett’s Esophagus. This work concentrates on the design of an automated indication of informativeness of video frames. We propose an algorithm consisting of state-of-the-art deep learning techniques, to initialize frame-based classification, followed by a hidden Markov model to incorporate temporal information and control consistent decision making. Results from the performed experiments show that the proposed model improves on the state-of-the-art with an F1-score of 91%, and a substantial increase in sensitivity of 10%, thereby indicating improved labeling consistency. Additionally, the algorithm is capable of processing 261 frames per second, which is multiple times faster compared to other informative frame classification algorithms, thus enabling real-time computation.
Original languageEnglish
Title of host publication2019 IEEE International Conference on Image Processing (ICIP)
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers
Number of pages5
ISBN (Electronic)978-1-5386-6249-6
Publication statusPublished - 26 Aug 2019
Event26th IEEE International Conference on Image Processing (ICIP 2019) - Taipei, Taiwan, Taipei, Taiwan
Duration: 22 Sept 201925 Sept 2019


Conference26th IEEE International Conference on Image Processing (ICIP 2019)
Abbreviated titleICIP 2019


  • Deep learning
  • Endoscopy
  • Hidden Markov Models
  • Informative frame classification


Dive into the research topics of 'Informative frame classification of endoscopic videos using convolutional neural networks and hidden Markov models'. Together they form a unique fingerprint.

Cite this