In order to identify new anticancer compounds from nature, a prefractionated library derived from Australian endemic plants was generated and screened against the prostate cancer cell line LNCaP using a metabolic assay. Fractions from the seeds, leaves, and wood of Anopterus macleayanus showed cytotoxic activity and were subsequently investigated using a combination of bioassay-guided fractionation and mass-directed isolation. This led to the identification of four new diterpenoid alkaloids, 6α-acetoxyanopterine (1), 4′-hydroxy-6α-acetoxyanopterine (2), 4′-hydroxyanopterine (3), and 11α-benzoylanopterine (4), along with four known compounds, anopterine (5), 7β-hydroxyanopterine (6), 7β,4′-dihydroxyanopterine (7), and 7β-hydroxy-11α-benzoylanopterine (8); all compounds were purified as their trifluoroacetate salt. The chemical structures of 1–8 were elucidated after analysis of 1D/2D NMR and MS data. Compounds 1–8 were evaluated for cytotoxic activity against a panel of human prostate cancer cells (LNCaP, C4-2B, and DuCaP) and nonmalignant cell lines (BPH-1 and WPMY-1), using a live-cell imaging system and a metabolic assay. All compounds showed potent cytotoxicity with IC50 values of <400 nM; compound 1 was the most active natural product from this series, with an IC50 value of 3.1 nM toward the LNCaP cell line. The live-cell imaging assay on 1–8 showed a concentration- and time-dependent effect on the cell morphology and proliferation of LNCaP cells.