Content-Based Retrieval of Digital Video

Abstract

In the next few years consumers will have access to large amounts of video and image data either created by themselves with digital video and still cameras or by having access to other image and video content electronically. Existing personal computer hardware and software has not been designed to manage large quantities of multimedia content. As a result, research in the area of content-based video retrieval (CBVR) has been underway for the last fifteen years. This research aims to improve CBVR by providing an accurate and reliable shape-colour representation and by providing a new 3D user interface called DomeWorld for the efficient browsing of large video databases. Existing feature extraction techniques designed for use in large databases are typically simple techniques as they must conform to the limited processing and storage constraints that are exhibited by large scale databases. Conversely, more complex feature extraction techniques provide higher level descriptions of the underlying data but are time consuming and require large amounts of storage making them less useful for large databases. In this thesis a technique for medium to high level shape representation is presented that exhibits efficient storage and query performance. The technique uses a very accurate contour detection system that incorporates a new asymmetry edge detector which is shown to perform better than other contour detection techniques combined with a new summarisation technique to efficiently store contours. In addition, contours are represented by histograms further reducing space requirements and increasing query performance. A new type of histogram is introduced called the fuzzy histogram and is applied to content-based retrieval systems for the first time. Fuzzy histograms improve the ranking of query results over non-fuzzy techniques especially in low bin-count histogram configurations. The fuzzy contour histogram approach is compared with an exhaustive contour comparison technique and is found to provide equivalent or better results. A number of colour distribution representation techniques were investigated for integration with the contour histogram and the fuzzy HSV histogram was found to provide the best performance. When the colour and contour histograms were integrated less overall bins were required as each histogram compensates for the other’s weaknesses. The result is that only a quarter of the bins were required than either colour or contour histogram alone further reducing query times and storage requirements. This research also improves the user experience with a new user interface called DomeWorld that uses three-dimensional translucent domes. Existing user interfaces are either designed for image databases, for browsing videos, or for browsing large non-multimedia data sets. DomeWorld is designed to be able to browse both image and video databases through a number of innovative techniques including hierarchical clustering, radial space-filling layout of nodes, three-dimensional presentation, and translucent domes that allow the hierarchical nature of the data to be viewed whilst also seeing the relationship between child nodes a number of levels deep. A taxonomy of existing image, video, and large data set user interfaces is presented and the proposed user interface is evaluated within the framework. It is found that video database user interfaces have four requirements: context and detail, gisting, clustering, and integration of video and images. None of the 27 evaluated user interfaces satisfy all four requirements. The DomeWorld user interface is designed to satisfy all of the requirements and presents a step forward in CBVR user interaction. This thesis investigates two important areas of CBVR, structural indexing and user interaction, and presents techniques which advance the field. These two areas will become very important in the future when users must access and manage large collections of image and video content.