It is now well recognized that the recipients’ evaluations need to be given serious consideration when theorizing impoliteness. Yet despite the importance placed on evaluations by recipients, the role of the recipient in interaction has been reduced through theorizing within the field to the ascribing of (perceived) intentions or interpreting of (perceived) social norms and expectations. We suggest, in this paper, that this under-theorizes the role of the recipient vis-à-vis evaluations of impoliteness. Building on an account of (im)politeness as social practice (Haugh 2013b, 2015; Kádár and Haugh 2013), we argue that evaluations of impoliteness inevitably involve those recipients construing the speaker’s action as a particular kind of social action, and holding them accountable for that particular kind of social action with respect to particular dimension(s) of the moral order (Haugh 2013a, 2015). The accountability of social action is underpinned, in part, by the presumed agency of participants. Agency involves the socially mediated capacity to act that is afforded through 1. knowing one has the ability to act, 2. knowing that these actions may affect others (and self), and 3. knowing that one will thus be held accountable for those actions (Ahearn 2001; Duranti 2004; Mitchell forthcoming). We argue that a focus on agency in theorizing impoliteness allows for the ways in which recipients do not just simply invoke social norms or (in some cases at least) perceived speaker intentions in evaluating talk or conduct as impolite, but may also exercise their own agency in construing the speaker’s actions as a particular kind of action, and thus as offensive or not. It is concluded that the agency exercised by recipients with respect to the degree to which they hold speakers accountable for impolite or offensive stances needs to be examined more carefully in theorizing about (im)politeness more generally.