Ball tracking is a fundamental task in sports video analytics. However, it is challenging because the ball is small and moves fast. We propose a general framework consists of a vision based ball detector and a motion based tracker to accurately track the ball in broadcast videos. Unlike usual frame by frame based detection followed by tracking, our ball detector itself is video based, which exploits both spatial and temporal information. Furthermore, instead of working in the raw image space, our ball detector works in the feature space through transfer learning. It contains few trainable parameters, and is easy to train. Our tracker complements the detector. It can find the most likely ball track based on detections by using a Viterbi based data association technique. It uses multiple motion models to associate a new detection to existing tracks even when the ball changes its direction. It can also tolerate temporary ball occlusion by introducing “null” nodes (representing miss detections) in the Viterbi algorithm and by back tracking. Our algorithm is evaluated using broadcast videos downloaded from tennis TV channels. Our experiments show that our method achieves 99.22% accuracy and is better than any existing ball tracking method. Besides, due to the modular design and the fact that our detector can be easily re-trained to detect other objects, our algorithm can be generalised to track other small and fast moving objects.