Evolutionary optimization methods have been utilized to optimize a wide range of models, including many complex neural network models. Manual parameter selection requires substantial trial and error and specialist domain knowledge of the inherent structure, which does not guarantee the best outcomes. We propose a three-layered novel architecture for semantic segmentation and optimize it using two distinct evolutionary algorithm-based optimization processes namely genetic algorithm and particle swarm optimization. To fully optimize an end-to-end image segmentation framework, the network design is tested using various combinations of a few parameters. An automatic search is conducted for the optimal parameter values to maximize the performance of the image segmentation framework. Evolutionary Algorithm (EA)-based optimization of the three-layered semantic segmentation network optimizes parameter values holistically, which produces the best performance. We evaluated our proposed architecture and optimization on two benchmark datasets. The evaluation results show that the proposed optimization can achieve better accuracy than the existing approaches.