Background Umbilical catheters are commonly inserted in newborns in the neonatal intensive care unit (NICU) yet are associated with serious adverse events (AEs) such as malposition, migration, infection, thrombosis, hepatic complications, cardiac effusion, and cardiac tamponade. There is a need to determine the incidence and risk factors for AEs to inform safe practice. Objectives The objective of this study was to determine the incidence and risk factors for AEs (all-cause and individual types) associated with umbilical venous catheters (UVCs) and umbilical arterial catheters (UACs) in the NICU. Methods A retrospective cohort study was conducted in an Australian level-VI NICU over a 3-year period. Any newborn who had both a UVC and UAC insertion attempt was included. Results There were 236 neonates who had 494 catheters (245 UVCs and 249 UACs). Of these, 71% of UVCs (95% confidence interval [CI]: 65.6–76.9%; incidence rate: 181.1–237.3 per 1000 catheter days) and 43.8% of UACs (95% CI: 38–50.5%; incidence rate: 102.0–146.3 per 1000 catheter days) were associated with an AE. The most common AE was malposition on first X-ray for UVCs (60.1%, 95% CI: 55.1–67.3) and UACs (32.6%, 95% CI: 26.8–39.6). A dwell time of ≥7 days was a significant predictor of UAC failure (incidence risk ratio: 1.5, 95% CI: 1.1–2.1, p = 0.006) and migration of the UVC (incidence risk ratio: 3.5, 95% CI: 1.0–11.5, p = 0.043). Conclusion Adverse events related to insertion occurred in a relatively high percentage of umbilical catheters placed. Increased dwell time remains a significant risk factor for catheter migration and overall failure. Practice change and consideration of risk factors for both individual and overall AE risk are necessary to reduce complications.