Categorization objects at a sub-ordinate level inevitably poses a significant challenge, i.e. , inter-class difference is very subtle and only exists in a few key parts. Therefore, how to localize these key parts for discriminative visual categorization without requiring expensive pixel-level annotations becomes a core question. To that end, this paper introduces a novel asymmetry tolerant part segmentation network (ATP-Net). ATP-Net simultaneously learns to segment parts and identify objects in an end-to-end manner using only image-level category labels. Given the intrinsic asymmetry property of part alignment, a desirable learning of part segmentation should be capable of incorporating such property. Despite the efforts towards regularizing weakly supervised part segmentation, none of them consider this vital and intrinsic property, i.e. , the spatial asymmetry of part alignment. Our work, for the first time, proposes to explicitly characterize the spatial asymmetry of part alignment for visual tasks. We propose a novel asymmetry loss function to guide the part segmentation by encoding the spatial asymmetry of part alignment, i.e. , restricting the upper bound of how asymmetric those self-similar parts are to each other in the network learning. Via a comprehensive ablation study, we verify the effectiveness of the proposed ATP-Net in driving the network learning towards semantically meaningful part segmentation and discriminative visual categorization. Consistently superior/competitive performance are reported on 12 datasets covering crop cultivar classification, plant disease classification, bird/butterfly species classification, large-scale natural image classification, attribute recognition and landmark localization.