Group-level voxelwise statistical analyses of lesion-deficit relationships have been used to implicate brain structures critical for specific aspects of human cognition and behavior. Current approaches fail to account for the role of fiber tract disruptions in causing deficit, and confound cortical damage with damage to fibers of passage. Here, we develop a framework, Generalized Lesion-Symptom Mapping (GLSM), to integrate fiber tract information from DTI-based tractographic atlases in tractwise and voxelwise lesion-deficit analyses. First, we used the geniculo-calcarine system as a validation testbed. Using logistic regressions we predicted right homonymous visual field deficits in 149 subjects with focal brain damage based on lesion location, with and without incorporating fiber tract information. A probabilistic fiber tract atlas [Wakana S, Jiang H, Nagae-Poetscher LM, Van Zijl PC, Mori S. Fiber tract-based atlas of human white matter anatomy. Radiology 2004;230:77-87] coregistered to our reference brain was used to estimate disconnection to the optic radiations and adjacent fiber tracts. When tract information was not incorporated, lesions in multiple sectors of the temporal lobe were associated with visual field defects. When the optic radiations were incorporated, this artifactual association was eliminated and the calcarine cortex was correctly isolated. Among the incorporated tracts, only the optic radiations significantly predicted visual field defects. Second, we applied the approach to impairments of higher visuoperceptual functions in 111 subjects who were administered the Hooper Visual Organization Test. We included all six association fiber tracts available in the atlas, plus the optic radiations. Tract inclusion narrowed the cortical sectors associated with impaired performance in a manner consistent with recent fMRI findings. The left cingulum and inferior longitudinal fasciculus, significantly predicted impairments. The results demonstrate the viability, validity and value of incorporating fiber tract information in lesion-deficit analyses. The enhanced analysis framework opens a new avenue for studying neural systems, with the potential to facilitate identification of both cortical sectors and fiber tracts critical for cognitive functioning.