Language is an essential aspect of communication. We can understand and communicate each other’s feelings through language. However, certain members of our society cannot talk or usually listen, leaving them with only sign language as a means of communication. Although researchers put a lot of time and effort into deciphering sign languages, most of their efforts have been focused on sign digits, and some are limited to simple samples. To address these prevalent concerns in earlier research, we created a new dataset of Bangla alphabets consisting of 2340 samples with different backgrounds. We also proposed a custom CNN architecture and compared its performance with other state-of-the-art models like ResNet, EfficientNet InceptionV3, and VGG19. All state-of-the-art models were trained and evaluated with custom dataset weights and ImageNet weights, and the best results were compared to our custom CNN. Our custom CNN did better than all the state-of-the-art models on our dataset with 92% accuracy.