This work proposes a framework for the physics-based simulation and high-fidelity visualization of fire following earthquake (FFE) considering building seismic damage. The seismic damage of regional buildings is simulated using multiple degree-of-freedom building models in conjunction with nonlinear time-history analysis. In parallel, a high-fidelity visualization is presented to simulate fire spread and smoke effects. A case study of downtown Taiyuan with 44,152 buildings is performed. The results show that the influence of different ground motions and building seismic resistances on fire ignition and fire spread can be taken into account and that the FFE scene can be displayed realistically.