Systems in the dispersive regime of cavity quantum electrodynamics (QED) are approaching the limits of validity of the dispersive approximation. We present a model which takes into account nonlinear corrections to the dressing of the atom by the field. We find that in the presence of pure dephasing, photons populating the cavity act as a heat bath on the atom, inducing incoherent relaxation and excitation. These effects are shown to reduce the achievable signal-to-noise ratio in cavity QED realizations where the atom is measured indirectly through cavity transmission and in particular in circuit QED.