Envelope analysis is a commonly used technique in fault diagnosis of rolling element bearings. The selection of a suitable frequency band for demodulation in envelope analysis has traditionally relied on the expertise of diagnosis technicians. The manual selection does not always give the best possible results in revealing the defect frequencies. To overcome this problem, a new demodulation band optimization approach is proposed which is based on a real-coded genetic algorithm with a novel fitness function and crossover selection process. The fitness function uses the ratio between fault frequency peaks and the maximum peak not corresponding to defects in the envelope spectrum. The crossover selection process uses the triangle series method to divide the probability of individuals in the population based on the fitness score obtained. The proposed method is assessed using vibration signals from two different rotor-bearing systems, i.e., a bearing testrig with seeded defects and the Case Western Reserve University bearing dataset. For all the cases, the method can find the optimized demodulation bands successfully for bearing fault detection. The method is further benchmarked with a well-established fast kurtogram approach which proves the effectiveness and superior capability of the developed algorithm, though the computational complexity needs improvement in future work.