Leaf image patterns have been actively researched for plant species recognition. However, as a very challenging fine-grained pattern identification issue, cultivar recognition in which the leaf image patterns usually have very subtle difference among cultivars has not yet received considerable attention in computer vision and pattern recognition community. In this paper, a novel symmetric geometric configuration, named Symmetric Binary Tree (SBT) which has multiple symmetric branch pairs and can change in size, is designed to mine the multiple scale co-occurrence texture patterns. The resulting SBT descriptors encode both shape and texture features which make them more informative than the existing individual descriptors and co-occurrence features. A novel feature fusion scheme, named K-NN Based Handcrafted and Deep Features Fusion (KNN-HDFF) that encodes the neighbouring information of distance measure, is proposed for further boosting the retrieval performance. Extensive experiments conducted on the challenging soybean cultivar leaf image dataset and peanut cultivar leaf image dataset consistently indicate the superiority of the proposed method over the state-of-the-art methods on fine-grained leaf image retrieval. We also conduct extensive experiments of feature fusions using the proposed KNN-HDFF on the benchmark datasets and the experimental results prove its potential for improving the performance of cultivar identification which also indicates that fusing handcrafted and deep features may be the direction to address the challenging fine-grained image recognition problem.