Hate crimes undermine tolerance and social inclusion by conveying an “outsider” status of the victim and other group members to the broader community. Yet, limited research considers whether non-victims recognize hate crime incidents when they occur. Using census and survey data for 4,000 residents living in 145 communities, we ask whether local residents “see” hate crime when it happens in their neighborhood and whether the neighborhood context influences the association between residents’ perceptions of hate crime and self-reported hate victimization. We find that residents’ perceptions are positively related to victim self-reports; however, this relationship weakens in ethnically diverse and disadvantaged areas. This suggests that residents’ perceptions of hate crime may be more dependent upon the community context than non-hate crimes.