Limited data are available on physical activity tracking among adults in low- and middle-income countries. Using a longitudinal design, we assessed trends and correlates of physical activity among Sri Lankan adults. Individuals selected through age-stratified random sampling, were screened initially in 2007 (n = 2986) and reevaluated in 2014 (n = 2148). On both occasions, structured interviews and clinical measurements were completed. Approximately 40% of the participants engaged in recommended levels of physical activity both at baseline and follow-up. One-fifth reported increased physical activity at follow-up, a similar proportion reported being persistently inactive or a reduction in physical activity. In the adjusted analysis, being persistently active was associated with male sex, a lower educational level and income, being free of any chronic disease conditions, better self-rated health, and sitting time <8 hours. Our findings support public health interventions to help maintain recommended physical activity levels over time, particularly for subgroups at high-risk of physical inactivity.