The Heisenberg limit traditionally provides a lower bound on the phase uncertainty scaling as 1/ N , where N is the mean number of photons in the probe. However, this limit has a number of loopholes which potentially might be exploited to achieve measurements with even greater accuracy. Here we close these loopholes by proving a completely rigorous form of the Heisenberg limit for the average error over all phase shifts, applicable to any estimate of a completely unknown phase shift. Our result gives a completely general, constraint-free, and nonasymptotic statement of the Heisenberg limit. It holds for all phase estimation schemes, including multiple passes, nonlinear phase shifts, multimode probes, and arbitrary measurements.