Under the combined effects of in-plane deformation and out-of-plane acceleration during an earthquake, cracking occurs in the exterior infill walls of buildings causing debris to fall. The falling debris ineluctably blocks roads and hinders fast pedestrian evacuations. To identify the failure criterion for exterior infill walls, specialized test apparatus was designed, and pseudo-static experiments of infill walls were conducted in this work. The influences of different height-to-thickness ratios, drift ratios and out-of-plane accelerations on falling debris were taken into consideration in the proposed apparatus design. The experiment indicated that the infill blocks fell sooner and the falling area was larger for a thinner infill wall with a larger tilting angle. Based on the experimental results, a prediction model was proposed to evaluate the percentage of falling area of infill walls. Using the proposed failure criterion, a pedestrian evacuation simulation was conducted on the Tsinghua University campus as a case study. Moreover, the debris distribution and evacuation results under different earthquake intensities were also analyzed. The results of the case study revealed that a considerable area of roads located in densely built-up areas was covered by debris, resulting in a more than 30% increase in time for people in the nearby buildings to evacuate. Compared to the scenario without falling debris, the evacuation time with debris was 5% longer when subjected to MCE (Maximal Considered Earthquake). The proposed failure criterion for masonry infill walls can be employed in debris distribution calculation, which offers technical support for urban planning and evacuation drills.