Lip contour extraction was a useful technique for obtaining a mouth shape in an image, and was one of the most important techniques for human-machine interface applications, such as lip reading and speech recognition. In this paper, a new method to extract lip contour from video was propose based on the fact that the lip color and skin-color was varied in the different color spaces. We first extracted frames from digital video first; then we classified face into lip area and non-lip area by the Support Vector Machine. At last we obtained some parameters from the lip area to reconstruct the lip contour. The experiment results proved that the proposed method was accurate and robust.