Registration of Multimodal Fluorescein Images

1. Introduction

The diagnosis and quantification of retinal diseases relies on the interpretation of color images and fluorescein angiography image sequences by qualified experts. The fluorescein angiography image sequence depicts the circulation of the sodium fluorescein dye in the retinal vessels and interpreting critical phases in the circulation is a key element for diagnosis. Grey level of the angiograms vary substantially during the circulation of the dye and image sequence as depicted in Figure 1, and the geometric registration of the angiograms requires the use of strong geometric invariants. Similarly registering the color image with the acquired angiograms for facilitating the expert’s analysis faces similar problems as the grey level are not consistent across modalities. Figure 1 shows various examples of multimodal retinal images. We present a fully automatic method for registering color and fluorescein angiograms. The method consists of three main steps: First, seed positions of Y-feature are computed using a PCA-based analysis of directional filters responses. Second, an articulated model of the Y-feature is fitted to the image features using a gradient descent method. Third, the extracted Y-features are matched by maximizing the mutual information, and images are globally registered using an affine model.

(a)	(b)
(c)	(d)
Figure 1. Multimodal Retinal Images

2. Estimating Initial Y-feature Positions

In this section, we present a method for locating candidate Y-features in the image. Y-features are characterized by regions in the image where three vessels converge. The position of the Y-feature has at most 3 strong responses from various directional filter output. The PCA based analysis of filter outputs allows locating the position of the Y-feature in the image. In Figure 2 we show the extracted seed points using the PCA analysis of the directional filters.

(a)	(b)
Figure 2: Initial Y-feature position in two images of different modalities. Only the best 100 features are selected.

3. Fitting Articulated Y-feature Model

We propose a Y-feature extraction method based on the fitting of an articulated model. The articulated model is fitted by maximizing the local intensities inside the template and gradient information on the boundary of the template. The considered articulated model for the Y-features has 8 DOF, which include the center position, three angles, and three widths for each branch. The length of each branch is fixed.

Figure 3. Articulated model of a Y-feature

In the figure and video below, we illustrate fitting process on bright and dark vessels in the case of color images and angiograms.

(a)	(b)	(c)	(d)
(e)	(f)	(g)	(h)

Figure 4. Fitting a Y-feature model to the image data. (a)(e)Initial estimation of a bright or dark vessel around the circular boundary. The red cross shows local peaks, and the green cross shows the local valleys characterizing respectively bright and dark vessels. The yellow and white points correspond respectively to the seed point, and the estimated center point. The green and red crosses indicate respectively local valleys, and peaks in the grey level distribution. (b)(f) Initial Y-feature model of bright or dark vessel. (c)(g) Y-feature model after fitting. (d)(h) Detected Y-features. Every valid Y-feature is classified into bright or dark vessels.

Video 1. Fitting Process (download)

4. Matching and Global Registration

We propose to match extracted Y-feature across modalities and through different phases of the circulation using the local maximization of the mutual information. A RANSAC (Random Sample Consensus) method is used for pairwise registration of images. This however does not provide robust registration of the image sequence, and a global registration is required. We propose achieving a optimal global registration of multimodal frames using all pairs’ shortest path. Frames with less matching points are discarded automatically.

(a) (b)

Figure 5. Global registration of frames across modalities using a minimum spanning tree. (a) Pairwise registration of consecutive images (b) Global registration constructed from the minimum spanning tree.

(a)	(b)
(c)	(d)
Figure 6. (a) (b) Fitted and validated Y-feature. Squares indicate non-validated Y-features. (c),(d) Matching pairs of Y-features across modalities

Video 2. Matching Process (download)

Figure 7. A view of registered images

Figure 8. Shortest path from the reference frame to other frames.

Figure 9. Mosaic image of registered angiograms

Video 3. Mosaic Result (short, 411KB)

Video 4. Mosaic Result (long, 42MB)

References

[1] Tae Eun Choe, Isaac Cohen, "Registration of Multimodal Fluorescein Images Sequence of the Retina, " ICCV (International Conference on Computer Vision) 2005, Volume I, pp. 106-113. October 2005

[2] Tae Eun Choe, Isaac Cohen, Munwai Lee, Gerard Medioni, "Optimal Global Mosaic Generation from Retinal Images," ICPR 2006, to appear