An EEG-based Brain-Computer Interface (BCI) is a system that enables a user to communicate with and intuitively control external devices solely using the user's intentions. Current EEG-based BCI research usually involves a subject-specific adaptation step before a BCI system is ready to be employed by a new user. However, the subject-independent scenario, in which a well-trained model can be directly applied to new users without pre-calibration, is particularly desirable yet rarely explored. Considering this critical gap, our focus in this paper is the subject-independent scenario of EEG-based human intention recognition. We present a Graph-based Hierarchical Attention Model (G-HAM) that utilizes the graph structure to represent the spatial information of EEG sensors and the hierarchical attention mechanism to focus on both the most discriminative temporal periods and EEG nodes. Extensive experiments on a large EEG dataset containing 105 subjects indicate that our model is capable of exploiting the underlying invariant EEG patterns across different subjects and generalizing the patterns to new subjects with better performance than a series of state-of-the-art and baseline approaches.