Recently lip image analysis has received much attention because the visual information extracted has been shown to provide significant improvement for speech recognition and speaker authentication especially in noisy environment. Lip image segmentation plays an important role in lip image analysis. This chapter will describe different lip image segmentation techniques, with emphasis on segmenting color lip images. In addition to provide a review of different approaches, we will describe in detail the state-of-art classification-based techniques recently proposed by our group for color lip segmentation: the "spatial fuzzy c-mean clustering (SFCM)" and the "fuzzy c-means with shape function (FCMS)". These methods integrate the color information along with different kinds of spatial information into a fuzzy clustering structure and demonstrate superiority in segmenting color lip images with natural low contrast in comparison with many traditional image segmentation techniques.