Agriculture applications rely on accurate land monitoring, especially paddy areas, for timely food security control and support actions. However, traditional monitoring requires field works or surveys performed by experts, which is costly, slow, and sparse. Agriculture monitoring systems are looking for sustainable land use monitoring solutions, starting with remote sensing on satellite data for cheap and timely paddy mapping. The aim of this study is to develop an autonomous and intelligent system built on top of imagery data streams, which is available from low-Earth orbiting satellites, to differentiate crop areas from non-crop areas. However, such agriculture mapping framework poses unique challenges for satellite image processing, including the seasonal nature of crop, the complexity of spectral channels, and adversarial conditions such as cloud and solar radiance. In this paper, we propose a novel multi-temporal high-spatial resolution classification method with an advanced spatio-temporal–spectral deep neural network to locate paddy fields at the pixel level for a whole year long and for each temporal instance. Our method is built and tested on the case study of Landsat 8 data due to its high spatial resolution. Empirical evaluations on real imagery datasets of different landscapes from 2016 to 2018 show the superior of our mapping model against the baselines with over 0.93 F1-score, the importance of each model design, the robustness against seasonal effects, and the visual mapping results.