The ability to manipulate and measure the time-frequency structure of quantum light is useful for information processing and metrology. Measuring this structure is also important when developing quantum light sources with high modal purity that can interfere with other independent sources. Here, we present and experimentally demonstrate a scheme based on intensity interferometry to measure the joint spectral mode of photon pairs produced by spontaneous parametric down-conversion. We observe correlations in the spectral phase of the photons due to chirp in the pump. We show that our scheme can be combined with stimulated emission tomography to quickly measure their mode using bright classical light. Our scheme does not require phase stability, nonlinearities, or spectral shaping and thus is an experimentally simple way of measuring the modal structure of quantum light.