Effective seismic damage simulation is an important task in improving earthquake resistance and safety of dense urban areas. There exist two significant technical challenges for realizing such a simulation: accurate prediction and realistic display. A high-fidelity structural model is proposed herein to accurately predict the seismic damage that was inflicted on a large number of buildings in an urban area via time-history analysis (THA), with which the local damage to different building stories is also explicitly obtained. The accuracy and efficiency of the proposed model are validated by a refined finite element (FE) analysis of a typical building. A physics engine-based algorithm is also proposed that realistically displays building collapse, thus overcoming the limitations of the high-fidelity structural model. Furthermore, a visualization system integrating the proposed model and collapse simulation is developed so as to completely display the seismic damage in detail. Finally, the simulated seismic damage of a real medium-sized Chinese city is evaluated to demonstrate the advantages of the proposed techniques, which can provide critically important reference information for urban disaster prevention and mitigation.