Image segmentation is one of the major challenges in real-world computer vision applications. Context-embedded network models proposed for image segmentation have outperformed context-free models. However, optimized values of many parameters need to consider for such a complex network. The manual parameter selection process is ineffective and produces suboptimal performance for such a model. Therefore, we propose a context-based genetically optimized network model for image segmentation in this paper. Genetic algorithms enhance the performance of the deep network model by determining the best parameter values. The proposed three-level deep network is adaptable to image context by extracting visual and context-rich features and optimally integrating them to obtain final pixel labels for scene images. The genetic algorithm ensures optimal parameter values in all three levels to obtain a globally optimized network model to achieve the best segmentation results.