Automatic segmentation of OCT retinal boundaries using recurrent neural networks and graph search

Thumbnail Image
File version

Version of Record (VoR)

Kugelman, Jason
Alonso-Caneiro, David
Read, Scott A
Vincent, Stephen J
Collins, Michael J
Griffith University Author(s)
Primary Supervisor
Other Supervisors
File type(s)

The manual segmentation of individual retinal layers within optical coherence tomography (OCT) images is a time-consuming task and is prone to errors. The investigation into automatic segmentation methods that are both efficient and accurate has seen a variety of methods proposed. In particular, recent machine learning approaches have focused on the use of convolutional neural networks (CNNs). Traditionally applied to sequential data, recurrent neural networks (RNNs) have recently demonstrated success in the area of image analysis, primarily due to their usefulness to extract temporal features from sequences of images or volumetric data. However, their potential use in OCT retinal layer segmentation has not previously been reported, and their direct application for extracting spatial features from individual 2D images has been limited. This paper proposes the use of a recurrent neural network trained as a patch-based image classifier (retinal boundary classifier) with a graph search (RNN-GS) to segment seven retinal layer boundaries in OCT images from healthy children and three retinal layer boundaries in OCT images from patients with age-related macular degeneration (AMD). The optimal architecture configuration to maximize classification performance is explored. The results demonstrate that a RNN is a viable alternative to a CNN for image classification tasks in the case where the images exhibit a clear sequential structure. Compared to a CNN, the RNN showed a slightly superior average generalization classification accuracy. Secondly, in terms of segmentation, the RNN-GS performed competitively against a previously proposed CNN based method (CNN-GS) with respect to both accuracy and consistency. These findings apply to both normal and AMD data. Overall, the RNN-GS method yielded superior mean absolute errors in terms of the boundary position with an average error of 0.53 pixels (normal) and 1.17 pixels (AMD). The methodology and results described in this paper may assist the future investigation of techniques within the area of OCT retinal segmentation and highlight the potential of RNN methods for OCT image analysis.

Journal Title

Biomedical Optics Express

Conference Title
Book Title




Thesis Type
Degree Program
Publisher link
Patent number
Grant identifier(s)
Rights Statement
Rights Statement

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement. Users may use, reuse, and build upon the article, or use the article for text or data mining, so long as such uses are for non-commercial purposes and appropriate attribution is maintained. All other rights are reserved.

Item Access Status
Access the data
Related item(s)

Atomic, molecular and optical physics

Materials engineering

Science & Technology

Life Sciences & Biomedicine

Physical Sciences

Biochemical Research Methods


Persistent link to this record

Kugelman, J; Alonso-Caneiro, D; Read, SA; Vincent, SJ; Collins, MJ, Automatic segmentation of OCT retinal boundaries using recurrent neural networks and graph search, Biomedical Optics Express, 2018, 9 (11), pp. 5759-5777