Humans can detect facial expressions of both simple, basic emotions and expressions reflecting more complex states of mind. The latter includes emotional expressions that regulate social interactions ("social expressions" such as looking hostile or friendly) and expressions that reflect the inner thought state of others ("cognitive expressions" such as looking pensive). To explore the neural substrate of this skill, we examined performance on a test of detection of such complex expressions in patients with lesions of the temporal lobe (n = 54) or frontal lobe (n = 31). Of the temporal group, 18 had unilateral focal lesions of the amygdala and of the frontal group, 14 patients had unilateral lesions of the ventromedial prefrontal cortex-two regions held to be pivotal in mediating social cognitive skills. Damage to either the left or right amygdala was associated with impairment in the recognition of both social and cognitive expressions, despite an intact ability to extract information relating to invariant physical attributes. Lesions to all of the right prefrontal cortex-not just the ventromedial portions-led to a specific deficit in recognizing complex social expressions with a negative valence. The deficit in the group with right prefrontal cortical damage may contribute to the disturbances in social behavior associated with such lesions. The results also suggest that the amygdala has a role in processing a wide range of emotional expressions.