Semantic cognition, which encompasses all conceptually based behavior, is dependent on the successful interaction of two key components: conceptual representations and regulatory control. Qualitatively distinct disorders of semantic knowledge follow damage to the different parts of this system. Previous studies have shown that patients with multimodal semantic impairment following CVA--a condition referred to as semantic aphasia (SA)--perform poorly on a range of conceptual tasks due to a failure of executive control following prefrontal and/or temporo-parietal infarction [Jefferies, E., & Lambon Ralph, M. A. Semantic impairment in stroke aphasia versus semantic dementia: A case-series comparison. Brain, 129, 2132-2147, 2006]. Although a deficit of core semantic control would be expected to impair all modalities in parallel, most research exploring this condition has focused on tasks in the verbal domain. In a novel exploration of semantic control in the nonverbal domain, therefore, we assessed eight patients with SA on two experiments that examined object use knowledge under different levels of task constraint. Patients exhibited three key characteristics of semantic deregulation: (a) difficulty using conceptual knowledge flexibly to support the noncanonical uses of everyday objects; (b) poor inhibition of semantically related distractor items; and (c) improved object use with the provision of more tightly constraining task conditions following verbal and pictorial cues. Our findings are consistent with the notion that a neural network incorporating the left inferior prefrontal and temporo-parietal areas (damaged in SA) underpins regulation of semantic activation across both verbal and nonverbal modalities.