Leaf image patterns have been actively researched for plant species recognition. However, as a very challenging fine-grained pattern identification issue, cultivar recognition in which the leaf image patterns usually have very subtle difference among cultivars has not yet received considerable attention in computer vision community. In this paper, a novel leaf image descriptor, named local angle co-occurrence histograms, is proposed for addressing this issue. It is a kind of co-occurrence descriptors that encoding both shape and texture features which make them more informative than the existing individual descriptors and co-occurrence features. A feature fusion scheme is proposed to integrate the handcrafted descriptors with deep learning features for further boosting the retrieval performance. The experimental results on the challenging soybean cultivar recognition and peanut cultivar recognition both indicate the superiority of the proposed method over the state-of-the-art methods on leaf image pattern characterization and validate the effectiveness of the proposed method for fine-grained leaf image retrieval.