Computer vision techniques can be applied to detect structural defects of different concrete structures. In this aspect, deep transfer learning algorithms play a key role in terms of automated crack identification. To the best of the authors’ knowledge, selecting appropriate models and tuning them for the classification of crack images, especially in adverse conditions, is a topic that has been neglected until now. Henceforth, to test the robustness and stability of deep transfer learning networks, eight popular pre-trained convolutional neural networks (CNN) models with different network architecture complexities were tasked with image classification challenges. This refinement was created by (i) varying a key hyperparameter used for tuning and (ii) feeding the networks with two variants of adverse conditions in image data. This chapter provides evidence of the optimal batch sizes that should be used and the best repurposed small and large deep learning networks that can achieve outstanding crack classification capabilities with compromised image data. According to the results, GoogleNet and Xception networks have been challenged and confirmed as high-performing networks on average, particularly when used with reasonable batch sizes.