Managers and policy makers depend on empirical research to guide and support biosecurity measures that mitigate introduced species' impacts. Research contributing to this knowledge base generally uses null hypothesis significance testing to determine the significance of data patterns. However, reliance on traditional statistical significance testing methods, combined with small effect and sample size and large variability inherent to many impact studies, may obscure effects on native species, communities or ecosystems. This may result in false certainty of no impact. We investigated potential Type II error rates and effect sizes for 31 non-significant empirical evaluations of impact for introduced algal and crustacean species. We found low power consistently led to acceptance of Type II errors at rates 5.6-19 times greater than Type I errors (despite moderate to large effect sizes). Our results suggest that introduced species for which impact studies have statistically non-significant outcomes (often interpreted as "no impact") may potentially have large impacts that are missed due to small sample or effect sizes and/or high variation. This alarming willingness to "miss" impacts has severe implications for conservation efforts, including under-managing species' impacts and discounting the costs of Type II errors.