Numerous studies have proven that natural particle-packed granular materials, such as soil and rock, are consistent with the grain-size fractal rule. The majority of existing studies have regarded these materials as ideal fractal structures, while few have viewed them as particle-packed materials to study the pore structure. In this study, theoretical analysis, the discrete element method, and digital image processing were used to explore the general rules of the pore structures of grain-size fractal granular materials. The relationship between the porosity and grain-size fractal dimension was determined based on bi-dispersed packing and the geometric packing theory. The pore structure of the grain-size fractal granular material was proven to differ from the ideal fractal structure, such as the Menger sponge. The empirical relationships among the box-counting dimension, lacunarity, succolarity, grain-size fractal dimension, and porosity were provided. A new segmentation method for the pore structure was proposed. Moreover, a general function of the pore size distribution was developed based on the segmentation results, which was verified by the soil-water characteristic curves from the experimental database.