A reliable crack detection system is essential for automatic safety inspection of infrastructures such as roads and bridges. Queensland's population is increasing, putting greater pressure on our already aging civil infrastructure. To get the most out of government investment in our State's thousands of bridges, we need to find quicker, cheaper and more reliable ways to assess and maintain them. In this paper, we propose a novel method for crack detection via salient structure extraction from textured background. This method contains two key steps. In the first step, we extract strong edges and distinguish them from strong textures in a local neighborhood via a relative total variation approach. In the second step, the spatial distribution of texture features are calculated so as to detect cracks as salient structures that are not widely spread across the whole image. The outputs from these two steps are fused to calculate the final structure saliency map which is then binarised to generate the crack masks. This method was evaluated on a crack dataset with images collected from the Internet. Comparison with several alternative approaches shows the superior performance of our method.