A simple and efficient method for synthesizing complex graphene-inspired BNCO nanoflakes by plasma-enhanced hot filament chemical vapour deposition using B4C as a precursor and N2/H2 reactive gases is reported. The results of the field emission scanning electron microscopy, X-ray diffractometer, micro-Raman spectroscopy, Fourier transform infrared spectroscopy and X-ray photoelectron spectroscopy have evidenced that the BNCO nanostructures are composed of graphene-like hexagonal boron nitride nanoflakes with carbon and oxygen admixtures. The photoluminescence properties of the BNCO nanoflakes were studied in a Ramalog system using a He–Cd laser, and the results have demonstrated that the BNCO nanoflakes can generate strong green photoluminescence in the range of 515–569 nm, which was attributed to the aggregation of carbon atoms in the BNCO nanoflakes. The growth mechanism of the BNCO nanoflakes was also studied to show that the NH3+ ions generated in plasma play an important role in the formation of the BNCO nanoflakes. The results propose a novel and efficient method for the synthesis of BN-based nanomaterials using B4C precursors, and contribute to the design of the functional BN-based nanomaterials for various applications including optoelectronics, nanoelectronics, medical equipment, and wear-resistant materials for acceleration channels of electric propulsion thrusters.