Cascading human pressures and environmental change are affecting the natural dynamics of animal populations. Forecasting population abundances from time-series data provides an important avenue for testing competing ecological theories and for supporting conservation planning and sustainable use, yet changing system dynamics may lead to erroneous predictions. Predictions from a model fitted and tested on historical system dynamics may become irrelevant if system dynamics change. Here we describe methods to test predictability in rapidly changing systems where model parameters are likely to be non-stationary. We presented two ways to split time series into training and test datasets so that training data were (1) contemporary to the testing data (‘modern split’) and (2) not contemporary to the testing data (‘legacy split’). As a case study, we compare the predictability of four temperate reef species in a global warming hotspot. The case study and simulation tests confirmed low predictability in the legacy split when compared to the modern split. We found that the legacy split had errors that could be more than four times larger for a species that had a rapid collapse in abundance and non-stationary population dynamics. As expected for the species with rapid collapse, the legacy split estimated much lower predictability than the modern split. Our approach is applicable to any time-series forecasting method and a large range of species and systems, including fisheries and threatened species population modelling, where rapidly changing environments present threats to both the species and management efficacy. Accumulated lessons from across species and systems should shed light on critical generalities that precede broader ecosystem change.