The facile and scalable fabrication of ultrafine (<5 nm) nanoparticles (NPs) as effective catalysts is the key for enhancing the kinetics of most hydrogen storage materials (HSMs). The direct fabrication of ultrafine NPs in HSMs is obviously a challenge because of the inevitable NPs agglomeration during the thermo-reduction. Herein, we report a mechanochemical-force-driven procedure for the one-step preparation of Ni NPs (2–3 nm) in a MgH2 matrix, which capitalizes on the in situ bottom-up reduction of Ni-MOF-74 in the presence of MgH2 as a reducing and sacrificing agent at room temperature. Both theoretical calculations and experimental investigations show that ultrafine Ni NPs are much more effective on catalytic hydrogenation/dehydrogenation in Mg due to the size effect. These findings may facilitate the fabrication of other catalyzed HSMs using different MOFs as catalyst precursors.