Skip to main content

Advertisement

Springer Nature Link
Log in

Identifying and Quantifying Multisensory Integration: A Tutorial Review

  • Review
  • Published:
Brain Topography Aims and scope Submit manuscript

Abstract

We process information from the world through multiple senses, and the brain must decide what information belongs together and what information should be segregated. One challenge in studying such multisensory integration is how to quantify the multisensory interactions, a challenge that is amplified by the host of methods that are now used to measure neural, behavioral, and perceptual responses. Many of the measures that have been developed to quantify multisensory integration (and which have been derived from single unit analyses), have been applied to these different measures without much consideration for the nature of the process being studied. Here, we provide a review focused on the means with which experimenters quantify multisensory processes and integration across a range of commonly used experimental methodologies. We emphasize the most commonly employed measures, including single- and multiunit responses, local field potentials, functional magnetic resonance imaging, and electroencephalography, along with behavioral measures of detection, accuracy, and response times. In each section, we will discuss the different metrics commonly used to quantify multisensory interactions, including the rationale for their use, their advantages, and the drawbacks and caveats associated with them. Also discussed are possible alternatives to the most commonly used metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Ahissar M, Ahissar E, Bergman H, Vaadia E (1992) Encoding of sound-source location and movement: activity of single neurons and interactions between adjacent neurons in the monkey auditory cortex. J Neurophysiol 67(1):203–215

    PubMed  CAS  Google Scholar 

  • Allman BL, Meredith MA (2007) Multisensory processing in “unimodal” neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. J Neurophysiol 98(1):545–549

    PubMed  Google Scholar 

  • Allman BL, Keniston LP, Meredith MA (2008) Subthreshold auditory inputs to extrastriate visual neurons are responsive to parametric changes in stimulus quality: sensory-specific versus non-specific coding. Brain Res 1242:95–101. doi:10.1016/j.brainres.2008.03.086

    PubMed  CAS  PubMed Central  Google Scholar 

  • Allman BL, Keniston LP, Meredith MA (2009) Not just for bimodal neurons anymore: the contribution of unimodal neurons to cortical multisensory processing. Brain Topogr 21(3–4):157–167. doi:10.1007/s10548-009-0088-3

    PubMed  PubMed Central  Google Scholar 

  • Altieri N, Townsend JT (2011) An assessment of behavioral dynamic information processing measures in audiovisual speech perception. Front Psychol 2:238. doi:10.3389/fpsyg.2011.00238

    PubMed  PubMed Central  Google Scholar 

  • Altieri N, Stevenson RA, Wallace MT, Wenger MJ (2013) Learning to associate auditory and visual stimuli: behavioral and neural mechanisms. Brain Topogr. doi:10.1007/s10548-013-0333-7

    PubMed  Google Scholar 

  • Andersen RA, Musallam S, Pesaran B (2004a) Selecting the signals for a brain-machine interface. Curr Opin Neurobiol 14(6):720–726. doi:10.1016/j.conb.2004.10.005

    PubMed  CAS  Google Scholar 

  • Andersen TS, Tiippana K, Sams M (2004b) Factors influencing audiovisual fission and fusion illusions. Brain Res Cogn Brain Res 21(3):301–308. doi:10.1016/j.cogbrainres.2004.06.004

    PubMed  Google Scholar 

  • Angelaki DE, Gu Y, DeAngelis GC (2011) Visual and vestibular cue integration for heading perception in extrastriate visual cortex. J Physiol 589(4):825–833

    PubMed  CAS  PubMed Central  Google Scholar 

  • Asano E, Juhasz C, Shah A, Muzik O, Chugani DC, Shah J, Sood S, Chugani HT (2005) Origin and propagation of epileptic spasms delineated on electrocorticography. Epilepsia 46(7):1086–1097. doi:10.1111/j.1528-1167.2005.05205.x

    PubMed  PubMed Central  Google Scholar 

  • Ashby FG (1982) Testing the assumptions of exponential, additive reaction time models. Mem Cogn 10:125–134

    CAS  Google Scholar 

  • Ashby FG, Townsend JT (1986) Varieties of perceptual independence. Psychol Rev 93(2):154–179

    PubMed  CAS  Google Scholar 

  • Barth DS, Goldberg N, Brett B, Di S (1995) The spatiotemporal organization of auditory, visual, and auditory-visual evoked potentials in rat cortex. Brain Res 678(1–2):177–190

    PubMed  Google Scholar 

  • Baum SH, Martin RC, Hamilton AC, Beauchamp MS (2012) Multisensory speech perception without the left superior temporal sulcus. NeuroImage. doi:10.1016/j.neuroimage.2012.05.034

    PubMed  PubMed Central  Google Scholar 

  • Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3(2):93–113

    PubMed  PubMed Central  Google Scholar 

  • Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7(11):1190–1192

    PubMed  CAS  Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823

    PubMed  CAS  Google Scholar 

  • Beauchamp MS, Nath AR, Pasalar S (2010) fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J Neurosci 30(7):2414–2417. doi:10.1523/JNEUROSCI.4865-09.2010

    PubMed  CAS  PubMed Central  Google Scholar 

  • Bell AH, Corneil BD, Munoz DP, Meredith MA (2003) Engagement of visual fixation suppresses sensory responsiveness and multisensory integration in the primate superior colliculus. Eur J Neurosci 18(10):2867–2873

    PubMed  CAS  Google Scholar 

  • Bell AH, Fecteau JH, Munoz DP (2004) Using auditory and visual stimuli to investigate the behavioral and neuronal consequences of reflexive covert orienting. J Neurophysiol 91(5):2172–2184

    PubMed  Google Scholar 

  • Bell AH, Meredith MA, Van Opstal AJ, Munoz DP (2005) Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements. J Neurophysiol 93(6):3659–3673

    PubMed  Google Scholar 

  • Berens P, Keliris GA, Ecker AS, Logothetis NK, Tolias AS (2008a) Comparing the feature selectivity of the gamma-band of the local field potential and the underlying spiking activity in primate visual cortex. Front Syst Neurosci 2:2. doi:10.3389/neuro.06.002.2008

    PubMed  PubMed Central  Google Scholar 

  • Berens P, Keliris GA, Ecker AS, Logothetis NK, Tolias AS (2008b) Feature selectivity of the gamma-band of the local field potential in primate primary visual cortex. Front Neurosci 2(2):199–207. doi:10.3389/neuro.01.037.2008

    PubMed  PubMed Central  Google Scholar 

  • Berens P, Logothetis NK, Tolias AS (2013) Local field potentials, BOLD and spiking activity–relationships and physiological mechanisms. Nature 2:67

    Google Scholar 

  • Berman AL (1961) Interaction of cortical responses to somatic and auditory stimuli in anterior ectosylvian gyrus of cat. J Neurophysiol 24:608–620

    PubMed  CAS  Google Scholar 

  • Bertelson P (1998) Starting from the ventriloquist: The perception of multimodal events. In: Sabourin M, Craik FIM, Robert M (eds) Advances in psychological science, vol 2., Biological and cognitive aspectsPsychological Press, Hove, pp 419–439

    Google Scholar 

  • Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29(6):578–584

    PubMed  CAS  Google Scholar 

  • Besle J, Fort A, Giard M (2004) Interest and Validity of the additive model in electrophysiological studies of multisensory interactions. Cogn Process 5:189–192

    Google Scholar 

  • Besle J, Fischer C, Bidet-Caulet A, Lecaignard F, Bertrand O, Giard MH (2008) Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans. J Neurosci 28(52):14301–14310. doi:10.1523/JNEUROSCI.2875-08.2008

    PubMed  CAS  Google Scholar 

  • Besle J, Bertrand O, Giard MH (2009) Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 258:143–151. doi:10.1016/j.heares.2009.06.016

    PubMed  Google Scholar 

  • Bizley JK, King AJ (2008) Visual-auditory spatial processing in auditory cortical neurons. Brain Res 1242:24–36. doi:10.1016/j.brainres.2008.02.087

    PubMed  CAS  Google Scholar 

  • Bolognini N, Convento S, Rossetti A, Merabet LB (2013) Multisensory processing after a brain damage: clues on post-injury crossmodal plasticity from neuropsychology. Neurosci Biobehav Rev 37(3):269–278

    PubMed  Google Scholar 

  • Boyden ES (2011) Optogenetics: using light to control the brain. Cerebrum 2011:16

    PubMed  PubMed Central  Google Scholar 

  • Boynton GM, Engel SA, Glover GH, Heeger DJ (1996) Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci 16(13):4207–4221

    PubMed  CAS  Google Scholar 

  • Bradley DC, Chang GC, Andersen RA (1998) Encoding of three-dimensional structure-from-motion by primate area MT neurons. Nature 392(6677):714–717. doi:10.1038/33688

    PubMed  CAS  Google Scholar 

  • Brecht M, Singer W, Engel AK (1999) Patterns of synchronization in the superior colliculus of anesthetized cats. J Neurosci 19(9):3567–3579

    PubMed  CAS  Google Scholar 

  • Britten KH, Shadlen MN, Newsome WT, Movshon JA (1992) The analysis of visual motion: a comparison of neuronal and psychophysical performance. J Neurosci 12(12):4745–4765

    PubMed  CAS  Google Scholar 

  • Britten KH, Newsome WT, Shadlen MN, Celebrini S, Movshon JA (1996) A relationship between behavioral choice and the visual responses of neurons in macaque MT. Visual Neurosci 13(1):87–100

    CAS  Google Scholar 

  • Burnett LR, Stein BE, Chaponis D, Wallace MT (2004) Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124(3):535–547. doi:10.1016/j.neuroscience.2003.12.026

    PubMed  CAS  Google Scholar 

  • Burnett LR, Stein BE, Perrault TJ Jr, Wallace MT (2007) Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration. Exp Brain Res Experimentelle Hirnforschung 179(2):325–338. doi:10.1007/s00221-006-0789-8

    Google Scholar 

  • Burton H, Sinclair RJ, Hong SY, Pruett JR Jr, Whang KC (1997) Tactile-spatial and cross-modal attention effects in the second somatosensory and 7b cortical areas of rhesus monkeys. Somatosens Mot Res 14(4):237–267

    PubMed  CAS  Google Scholar 

  • Buzsaki G, Draguhn A (2004) Neuronal oscillations in cortical networks. Science 304(5679):1926–1929. doi:10.1126/science.1099745

    PubMed  CAS  Google Scholar 

  • Calvert GA, Thesen T (2004) Multisensory integration: methodological approaches and emerging principles in the human brain. Journal of physiology, Paris 98(1–3):191–205

    PubMed  Google Scholar 

  • Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. NeuroImage 14(2):427–438

    PubMed  CAS  Google Scholar 

  • Calvert GA, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT, Cambridge

    Google Scholar 

  • Cappe C, Murray MM, Barone P, Rouiller EM (2010) Multisensory facilitation of behavior in monkeys: effects of stimulus intensity. J Cogn Neurosci 22(12):2850–2863. doi:10.1162/jocn.2010.21423

    PubMed  Google Scholar 

  • Cappe C, Thelen T, Romei V, Thut G, Murray MM (2012) Looming signals reveal synergistic principles of multisensory integration. J Neurosci 32(4):1171–1182. doi:10.1523/JNEUROSCI.5517-11.2012

    PubMed  CAS  Google Scholar 

  • Carriere BN, Royal DW, Perrault TJ, Morrison SP, Vaughan JW, Stein BE, Wallace MT (2007) Visual deprivation alters the development of cortical multisensory integration. J Neurophysiol 98(5):2858–2867. doi:10.1152/jn.00587.2007

    PubMed  Google Scholar 

  • Churan J, Ilg UJ (2001) Processing of second-order motion stimuli in primate middle temporal area and medial superior temporal area. J Opt Soc Am A Opt Image Sci Vis 18(9):2297–2306

    PubMed  CAS  Google Scholar 

  • Colonius H, Diederich A, Steenken R (2009) Time-window-of-integration (TWIN) model for saccadic reaction time: effect of auditory masker level on visual-auditory spatial interaction in elevation. Brain Topogr 21(3–4):177–184. doi:10.1007/s10548-009-0091-8

    PubMed  Google Scholar 

  • Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. The Journal of the Acoustical Society of America 119(6):4065–4073

    PubMed  PubMed Central  Google Scholar 

  • Corneil BD, Van Wanrooij M, Munoz DP, Van Opstal AJ (2002) Auditory-visual interactions subserving goal-directed saccades in a complex scene. J Neurophysiol 88(1):438–454

    PubMed  CAS  Google Scholar 

  • Dahmen JC, Keating P, Nodal FR, Schulz AL, King AJ (2010) Adaptation to stimulus statistics in the perception and neural representation of auditory space. Neuron 66(6):937–948. doi:10.1016/j.neuron.2010.05.018

    PubMed  CAS  PubMed Central  Google Scholar 

  • Dale AM, Buckner RL (1997) Selective averaging of rapidly presented individual trials using fMRI. Hum Brain Mapp 5:329–340

    PubMed  CAS  Google Scholar 

  • David SV, Hayden BY, Mazer JA, Gallant JL (2008) Attention to stimulus features shifts spectral tuning of V4 neurons during natural vision. Neuron 59(3):509–521. doi:10.1016/j.neuron.2008.07.001

    PubMed  CAS  PubMed Central  Google Scholar 

  • de Lafuente V, Romo R (2005) Neuronal correlates of subjective sensory experience. Nat Neurosci 8(12):1698–1703. doi:10.1038/nn1587

    PubMed  Google Scholar 

  • de Peralta Menendez RG, Andino SG, Lantz G, Michel CM, Landis T (2001) Noninvasive localization of electromagnetic epileptic activity. I. Method descriptions and simulations. Brain Topogr 14(2):131–137

  • de Peralta Grave, Menendez R, Murray MM, Michel CM, Martuzzi R, Gonzalez Andino SL (2004) Electrical neuroimaging based on biophysical constraints. NeuroImage 21(2):527–539. doi:10.1016/j.neuroimage.2003.09.051

    Google Scholar 

  • Deisseroth K (2011) Optogenetics. Nat Methods 8(1):26–29. doi:10.1038/nmeth.f.324

    PubMed  CAS  Google Scholar 

  • Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404

    PubMed  Google Scholar 

  • Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    PubMed  CAS  Google Scholar 

  • Dodd JV, Krug K, Cumming BG, Parker AJ (2001) Perceptually bistable three-dimensional figures evoke high choice probabilities in cortical area MT. J Neurosci 21(13):4809–4821

    PubMed  CAS  Google Scholar 

  • Donders FC (1868) Over de Snelheid van Psychische Processen. Onderzoekingen Gedaan in het Psychologisch Laboratorium der Utrechetsche Hoogeschool

  • Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57(1):11–23

    PubMed  CAS  PubMed Central  Google Scholar 

  • Einevoll GT, Kayser C, Logothetis NK, Panzeri S (2013) Modelling and analysis of local field potentials for studying the function of cortical circuits. Nat Rev Neurosci 14(11):770–785

    PubMed  CAS  Google Scholar 

  • Fenno L, Yizhar O, Deisseroth K (2011) The development and application of optogenetics. Annu Rev Neurosci 34:389–412. doi:10.1146/annurev-neuro-061010-113817

    PubMed  CAS  Google Scholar 

  • Fetsch CR, DeAngelis GC, Angelaki DE (2013) Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci 14(6):429–442

    PubMed  CAS  Google Scholar 

  • Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res Experimentelle Hirnforschung 203(2):381–389. doi:10.1007/s00221-010-2240-4

    Google Scholar 

  • Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol 88(1):540–543

    PubMed  Google Scholar 

  • Frens MA, Van Opstal AJ (1998) Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Res Bull 46(3):211–224

    PubMed  CAS  Google Scholar 

  • Fries P (2005) A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends Cogn Sci 9(10):474–480. doi:10.1016/j.tics.2005.08.011

    PubMed  Google Scholar 

  • Fries P, Reynolds JH, Rorie AE, Desimone R (2001) Modulation of oscillatory neuronal synchronization by selective visual attention. Science 291(5508):1560–1563. doi:10.1126/science.291.5508.1560

    PubMed  CAS  Google Scholar 

  • Gauthier J, Joober R, Mottron L, Laurent S, Fuchs M, De Kimpe V, Rouleau GA (2003) Mutation screening of FOXP2 in individuals diagnosed with autistic disorder. Am J Med Genet A 118A(2):172–175. doi:10.1002/ajmg.a.10105

    PubMed  Google Scholar 

  • Gerstein G, Bedenbaugh P, Aertsen AM (1989) Neuronal assemblies. IEEE Trans Biomed Eng 36(1):4–14

    Google Scholar 

  • Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25(20):5004–5012. doi:10.1523/JNEUROSCI.0799-05.2005

    PubMed  CAS  Google Scholar 

  • Giard M, Besle J (2010) Methodological considerations: Electrophysiology of multisensory interactions in humans. In: Kaiser J, Naumer MJ (eds) Multisensory object perception in the primate brain. Springer, New York

    Google Scholar 

  • Glover GH (1999) Deconvolution of impulse response in event-related BOLD fMRI. NeuroImage 9(4):416–429

    PubMed  CAS  Google Scholar 

  • Goense JB, Logothetis NK (2008) Neurophysiology of the BOLD fMRI signal in awake monkeys. Curr Biol 18(9):631–640. doi:10.1016/j.cub.2008.03.054

    PubMed  CAS  Google Scholar 

  • Gomez-Ramirez M, Kelly SP, Molholm S, Sehatpour P, Schwartz TH, Foxe JJ (2011) Oscillatory sensory selection mechanisms during intersensory attention to rhythmic auditory and visual inputs: a human electrocorticographic investigation. J Neurosci 31(50):18556–18567

    PubMed  CAS  PubMed Central  Google Scholar 

  • Gondan M, Röder B (2006) A new method for detecting interactions between the senses in event-related potentials. Brain Res 1073–1074:389–397. doi:10.1016/j.brainres.2005.12.050

    PubMed  Google Scholar 

  • Gray CM, Maldonado PE, Wilson M, McNaughton B (1995) Tetrodes markedly improve the reliability and yield of multiple single-unit isolation from multi-unit recordings in cat striate cortex. J Neurosci Methods 63(1–2):43–54

    PubMed  CAS  Google Scholar 

  • Green DM, Swets JA (1966) Signal detection theory and psychophysics, vol 1974. Wiley, New York

  • Grill-Spector K, Malach R (2001) fMR-adaptation: a tool for studying the functional properties of human cortical neurons. Acta Psychol 107(1–3):293–321

    CAS  Google Scholar 

  • Guido W, Lu SM, Vaughan JW, Godwin DW, Sherman SM (1995) Receiver operating characteristic (ROC) analysis of neurons in the cat’s lateral geniculate nucleus during tonic and burst response mode. Vis Neurosci 12(4):723–741

    PubMed  CAS  Google Scholar 

  • Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA (2003) Visual localization ability influences cross-modal bias. J Cogn Neurosci 15(1):20–29

    PubMed  CAS  Google Scholar 

  • Hairston WD, Burdette JH, Flowers DL, Wood FB, Wallace MT (2005) Altered temporal profile of visual-auditory multisensory interactions in dyslexia. Exp Brain Res Experimentelle Hirnforschung 166(3–4):474–480. doi:10.1007/s00221-005-2387-6

    Google Scholar 

  • Hauthal N, Thorne JD, Debener S, Sandmann P (2013) Source localisation of visual evoked potentials in congenitally deaf individuals. Brain Topogr. doi:10.1007/s10548-013-0341-7

    PubMed  Google Scholar 

  • Heeger DJ, Ress D (2002) What does fMRI tell us about neuronal activity? Nat Rev 3(2):142–151. doi:10.1038/nrn730

    CAS  Google Scholar 

  • Henze DA, Borhegyi Z, Csicsvari J, Mamiya A, Harris KD, Buzsaki G (2000) Intracellular features predicted by extracellular recordings in the hippocampus in vivo. J Neurophysiol 84(1):390–400

    PubMed  CAS  Google Scholar 

  • Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293

    PubMed  CAS  Google Scholar 

  • Hillyard SA, Teder-Salejarvi WA, Munte TF (1998) Temporal dynamics of early perceptual processing. Curr Opin Neurobiol 8(2):202–210

    PubMed  CAS  Google Scholar 

  • Iriki A, Tanaka M, Iwamura Y (1996) Coding of modified body schema during tool use by macaque postcentral neurones. NeuroReport 7(14):2325–2330

    PubMed  CAS  Google Scholar 

  • James TW, Stevenson RA (2012) The use of fMRI to assess multisensory integration. In: Wallace MH, Murray MM (eds) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London

    Google Scholar 

  • James TW, Kim S, Stevenson RA (2009) Assessing multisensory interaction with additive factors and functional MRI. In: The International Society for Psychophysics, Galway, Ireland, 2009

  • James TW, Stevenson RA, Kim S (2012) Inverse effectiveness in multisensory processing. In: Stein BE (ed) The new handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  • Jenkins WM, Merzenich MM, Ochs MT, Allard T, Guic-Robles E (1990) Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. J Neurophysiol 63(1):82–104

    PubMed  CAS  Google Scholar 

  • Jermakowicz WJ, Casagrande VA (2007) Neural networks a century after Cajal. Brain Res Rev 55(2):264–284. doi:10.1016/j.brainresrev.2007.06.003

    PubMed  PubMed Central  Google Scholar 

  • Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85(2):506–522

    PubMed  CAS  Google Scholar 

  • Jiang W, Jiang H, Stein BE (2002) Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci 14(8):1240–1255

    PubMed  Google Scholar 

  • Kajikawa Y, Schroeder CE (2011) How local is the local field potential? Neuron 72(5):847–858. doi:10.1016/j.neuron.2011.09.029

    PubMed  CAS  PubMed Central  Google Scholar 

  • Katzner S, Nauhaus I, Benucci A, Bonin V, Ringach DL, Carandini M (2009) Local origin of field potentials in visual cortex. Neuron 61(1):35–41. doi:10.1016/j.neuron.2008.11.016

    PubMed  CAS  PubMed Central  Google Scholar 

  • Kayser C, Petkov CI, Augath M, Logothetis NK (2005) Integration of touch and sound in auditory cortex. Neuron 48(2):373–384

    PubMed  CAS  Google Scholar 

  • Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. J Neurosci 27(8):1824–1835. doi:10.1523/JNEUROSCI.4737-06.2007

    PubMed  CAS  Google Scholar 

  • Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18(7):1560–1574. doi:10.1093/cercor/bhm187

    PubMed  Google Scholar 

  • Kayser C, Petkov CI, Logothetis NK (2009) Multisensory interactions in primate auditory cortex: fMRI and electrophysiology. Hear Res 258(1–2):80–88. doi:10.1016/j.heares.2009.02.011

    PubMed  Google Scholar 

  • Kim S, James TW (2010) Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience. Hum Brain Mapp 31(5):678–693. doi:10.1002/hbm.20897

    PubMed  CAS  Google Scholar 

  • Kim S, Stevenson RA, James TW (2012) Visuo-haptic neuronal convergence demonstrated with an inversely effective pattern of BOLD activation. J Cogn Neurosci 24(4):830–842. doi:10.1162/jocn_a_00176

    PubMed  Google Scholar 

  • King AJ, Walker KM (2012) Integrating information from different senses in the auditory cortex. Biol Cybern 106(11–12):617–625. doi:10.1007/s00422-012-0502-x

    PubMed  Google Scholar 

  • Kinsella TJ, Trivette G, Rowland J, Sorace R, Miller R, Fraass B, Steinberg SM, Glatstein E, Sherins RJ (1989) Long-term follow-up of testicular function following radiation therapy for early-stage Hodgkin’s disease. J Clin Oncol 7(6):718–724

    PubMed  CAS  Google Scholar 

  • Koval MJ, Lomber SG, Everling S (2011) Prefrontal cortex deactivation in macaques alters activity in the superior colliculus and impairs voluntary control of saccades. J Neurosci 31(23):8659–8668. doi:10.1523/JNEUROSCI.1258-11.2011

    PubMed  CAS  Google Scholar 

  • Kreiman G, Hung CP, Kraskov A, Quiroga RQ, Poggio T, DiCarlo JJ (2006) Object selectivity of local field potentials and spikes in the macaque inferior temporal cortex. Neuron 49(3):433–445. doi:10.1016/j.neuron.2005.12.019

    PubMed  CAS  Google Scholar 

  • Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE (2005) An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J Neurophysiol 94(3):1904–1911. doi:10.1152/jn.00263.2005

    PubMed  Google Scholar 

  • Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53(2):279–292. doi:10.1016/j.neuron.2006.12.011

    PubMed  CAS  PubMed Central  Google Scholar 

  • Lakatos P, Karmos G, Mehta AD, Ulbert I, Schroeder CE (2008) Entrainment of neuronal oscillations as a mechanism of attentional selection. Science 320(5872):110–113. doi:10.1126/science.1154735

    PubMed  CAS  Google Scholar 

  • Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res Experimentelle Hirnforschung 166(3–4):289–297

    Google Scholar 

  • Lee C, Rohrer WH, Sparks DL (1988) Population coding of saccadic eye movements by neurons in the superior colliculus. Nature 332(6162):357–360. doi:10.1038/332357a0

    PubMed  CAS  Google Scholar 

  • Leopold DA, Logothetis NK (1996) Activity changes in early visual cortex reflect monkeys’ percepts during binocular rivalry. Nature 379(6565):549–553

    PubMed  CAS  Google Scholar 

  • Li X, Basso MA (2005) Competitive stimulus interactions within single response fields of superior colliculus neurons. J Neurosci 25(49):11357–11373

    PubMed  CAS  Google Scholar 

  • Liu J, Newsome WT (2005) Correlation between speed perception and neural activity in the middle temporal visual area. J Neurosci 25(3):711–722. doi:10.1523/JNEUROSCI.4034-04.2005

    PubMed  CAS  Google Scholar 

  • Liu J, Newsome WT (2006) Local field potential in cortical area MT: stimulus tuning and behavioral correlations. J Neurosci 26(30):7779–7790

    PubMed  CAS  Google Scholar 

  • Logothetis NK (2003) The underpinnings of the BOLD functional magnetic resonance imaging signal. J Neurosci 23(10):3963–3971

    PubMed  CAS  Google Scholar 

  • Logothetis NK (2008) What we can do and what we cannot do with fMRI. Nature 453(7197):869–878. doi:10.1038/nature06976

    PubMed  CAS  Google Scholar 

  • Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412(6843):150–157

    PubMed  CAS  Google Scholar 

  • Lomber SG, Malhotra S (2008) Double dissociation of ‘what’ and ‘where’ processing in auditory cortex. Nat Neurosci 11(5):609–616. doi:10.1038/nn.2108

    PubMed  CAS  Google Scholar 

  • Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453

    PubMed  Google Scholar 

  • Luce RD (1959) Individual choice behavior: a theoretical analysis. Courier Dover Publications, New York

    Google Scholar 

  • Macmillan NA, Creelman CD (2004) Detection theory: a user’s guide, 2nd edn. Psychology Press, USA

    Google Scholar 

  • Magri C, Schridde U, Murayama Y, Panzeri S, Logothetis NK (2012) The amplitude and timing of the BOLD signal reflects the relationship between local field potential power at different frequencies. J Neurosci 32(4):1395–1407. doi:10.1523/JNEUROSCI.3985-11.2012

    PubMed  CAS  Google Scholar 

  • Maier A, Logothetis NK, Leopold DA (2007) Context-dependent perceptual modulation of single neurons in primate visual cortex. Proc Natl Acad Sci USA 104(13):5620–5625. doi:10.1073/pnas.0608489104

    PubMed  CAS  PubMed Central  Google Scholar 

  • Malhotra S, Hall AJ, Lomber SG (2004) Cortical control of sound localization in the cat: unilateral cooling deactivation of 19 cerebral areas. J Neurophysiol 92(3):1625–1643. doi:10.1152/jn.01205.2003

    PubMed  Google Scholar 

  • Malhotra S, Stecker GC, Middlebrooks JC, Lomber SG (2008) Sound localization deficits during reversible deactivation of primary auditory cortex and/or the dorsal zone. J Neurophysiol 99(4):1628–1642. doi:10.1152/jn.01228.2007

    PubMed  Google Scholar 

  • Martuzzi R, Murray MM, Michel CM, Thiran J-P, Maeder PP, Clarke S, Meuli RA (2007) Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb Cortex 17(7):1672–1679

    PubMed  Google Scholar 

  • Massaro DW (1987) Speech Perception by Ear and Eye. In: Dodd B, Campbell BA (eds) Hearing by eye: the psychology of lip reading. Lawrence Erlbaum Associates Ltd., London, pp 53–84

    Google Scholar 

  • Massaro DW (2004) From Multisensory Integration to Talking Heads and Language Learning. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 153–176

    Google Scholar 

  • Matsuhashi M, Ikeda A, Ohara S, Matsumoto R, Yamamoto J, Takayama M, Satow T, Begum T, Usui K, Nagamine T, Mikuni N, Takahashi J, Miyamoto S, Fukuyama H, Shibasaki H (2004) Multisensory convergence at human temporo-parietal junction: epicortical recording of evoked responses. Clin Neurophysiol 115(5):1145–1160. doi:10.1016/j.clinph.2003.12.009

    PubMed  Google Scholar 

  • Mattis J, Tye KM, Ferenczi EA, Ramakrishnan C, O’Shea DJ, Prakash R, Gunaydin LA, Hyun M, Fenno LE, Gradinaru V, Yizhar O, Deisseroth K (2012) Principles for applying optogenetic tools derived from direct comparative analysis of microbial opsins. Nat Methods 9(2):159–172. doi:10.1038/nmeth.1808

    CAS  PubMed Central  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746–748

    PubMed  CAS  Google Scholar 

  • Mercier MR, Foxe JJ, Fiebelkorn IC, Butler JS, Schwartz TH, Molholm S (2013) Auditory-driven phase reset in visual cortex: human electrocorticography reveals mechanisms of early multisensory integration. NeuroImage 79:19–29

    PubMed  PubMed Central  Google Scholar 

  • Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221(4608):389–391

    PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1985) Descending efferents from the superior colliculus relay integrated multisensory information. Science 227(4687):657–659

    PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354

    PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662

    PubMed  CAS  Google Scholar 

  • Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7(10):3215–3229

    PubMed  CAS  Google Scholar 

  • Merzenich MM, Nelson RJ, Kaas JH, Stryker MP, Jenkins WM, Zook JM, Cynader MS, Schoppmann A (1987) Variability in hand surface representations in areas 3b and 1 in adult owl and squirrel monkeys. J Comp Neurol 258(2):281–296. doi:10.1002/cne.902580208

    PubMed  CAS  Google Scholar 

  • Michel CM, Murray MM (2012) Towards the utilization of EEG as a brain imaging tool. NeuroImage 61(2):371–385. doi:10.1016/j.neuroimage.2011.12.039

    PubMed  Google Scholar 

  • Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, Grave de Peralta R (2004) EEG source imaging. Clin Neurophysiol 115(10):2195–2222. doi:10.1016/j.clinph.2004.06.001

    PubMed  Google Scholar 

  • Michel CM, Koenig T, Brandeis D, Gianotti LR, Wackermann J (2009) Electrical neuroimaging. Cambridge University Press, Cambridge

    Google Scholar 

  • Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cognit Psychol 14(2):247–279

    PubMed  CAS  Google Scholar 

  • Miller J, Ulrich R, Lamarre Y (2001) Locus of the redundant-signals effect in bimodal divided attention: a neurophysiological analysis. Percept Psychophys 63(3):555–562

    PubMed  CAS  Google Scholar 

  • Mishra J, Martinez A, Hillyard SA (2008) Cortical processes underlying sound-induced flash fusion. Brain Res 1242:102–115. doi:10.1016/j.brainres.2008.05.023

    PubMed  CAS  PubMed Central  Google Scholar 

  • Mitzdorf U (1985) Current source-density method and application in cat cerebral cortex: investigation of evoked potentials and EEG phenomena. Physiol Rev 65(1):37–100

    PubMed  CAS  Google Scholar 

  • Mitzdorf U (1987) Properties of the evoked potential generators: current source-density analysis of visually evoked potentials in the cat cortex. Int J Neurosci 33(1–2):33–59

    PubMed  CAS  Google Scholar 

  • Montemurro MA, Rasch MJ, Murayama Y, Logothetis NK, Panzeri S (2008) Phase-of-firing coding of natural visual stimuli in primary visual cortex. Curr Biol 18(5):375–380. doi:10.1016/j.cub.2008.02.023

    PubMed  CAS  Google Scholar 

  • Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. NeuroImage 21(1):125–135

    PubMed  Google Scholar 

  • Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ (2005) Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15(7):963–974. doi:10.1093/cercor/bhh197

    PubMed  Google Scholar 

  • Murray MM, Brunet D, Michel CM (2008) Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr 20(4):249–264. doi:10.1007/s10548-008-0054-5

    PubMed  Google Scholar 

  • Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley RL (1998) Effects of localized auditory information on visual target detection performance using a helmet-mounted display. Hum Factors 40(3):452–460

    PubMed  CAS  Google Scholar 

  • Niwa M, Johnson JS, O’Connor KN, Sutter ML (2012) Active engagement improves primary auditory cortical neurons’ ability to discriminate temporal modulation. J Neurosci 32(27):9323–9334. doi:10.1523/JNEUROSCI.5832-11.2012

    PubMed  CAS  PubMed Central  Google Scholar 

  • O’Riordan MA (2004) Superior visual search in adults with autism. Autism 8(3):229–248. doi:10.1177/1362361304045219

    PubMed  Google Scholar 

  • O’Riordan M, Plaisted K (2001) Enhanced discrimination in autism. Q J Exp Psychol 54(4):961–979

    Google Scholar 

  • Perkel DH, Gerstein GL, Moore GP (1967) Neuronal spike trains and stochastic point processes II. Simultaneous spike trains. Biophys J 7(4):419–440. doi:10.1016/S0006-3495(67)86597-4

    PubMed  CAS  PubMed Central  Google Scholar 

  • Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90(6):4022–4026. doi:10.1152/jn.00494.2003

    PubMed  Google Scholar 

  • Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93(5):2575–2586

    PubMed  Google Scholar 

  • Pesaran B, Musallam S, Andersen RA (2006) Cognitive neural prosthetics. Curr Biol 16(3):R77–80. doi:10.1016/j.cub.2006.01.043

    PubMed  CAS  Google Scholar 

  • Pettersen KH, Hagen E, Einevoll GT (2008) Estimation of population firing rates and current source densities from laminar electrode recordings. J Comput Neurosci 24(3):291–313

    PubMed  Google Scholar 

  • Pieters JPM (1983) Sternberg’s additive factor method and underlying psychological processes: some theoretical consideration. Psychol Bull 93:411–426

    PubMed  CAS  Google Scholar 

  • Pockett S, Purdy SC, Brennan BJ, Holmes MD (2013) Auditory click stimuli evoke event-related potentials in the visual cortex. NeuroReport 24(15):837–840

    PubMed  Google Scholar 

  • Powers AR 3rd, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29(39):12265–12274. doi:10.1523/JNEUROSCI.3501-09.2009

    PubMed  CAS  PubMed Central  Google Scholar 

  • Powers AR 3rd, Hevey MA, Wallace MT (2012) Neural correlates of multisensory perceptual learning. J Neurosci 32(18):6263–6274. doi:10.1523/JNEUROSCI.6138-11.2012

    PubMed  CAS  PubMed Central  Google Scholar 

  • Price NS, Born RT (2010) Timescales of sensory- and decision-related activity in the middle temporal and medial superior temporal areas. J Neurosci 30(42):14036–14045. doi:10.1523/JNEUROSCI.2336-10.2010

    PubMed  CAS  PubMed Central  Google Scholar 

  • Puce A, Epling JA, Thompson JC, Carrick OK (2007) Neural responses elicited to face motion and vocalization pairings. Neuropsychologia 45(1):93–106. doi:10.1016/j.neuropsychologia.2006.04.017

    PubMed  PubMed Central  Google Scholar 

  • Raab DH (1962) Statistical facilitation of simple reaction times. Trans N Y Acad Sci 24:574–590

    PubMed  CAS  Google Scholar 

  • Radeau M (1994) Auditory-visual spatial interaction and modularity. Curr Psychol Cogn 13(1):3–51

    PubMed  CAS  Google Scholar 

  • Ramos-Estebanez C, Merabet LB, Machii K, Fregni F, Thut G, Wagner TA, Romei V, Amedi A, Pascual-Leone A (2007) Visual phosphene perception modulated by subthreshold crossmodal sensory stimulation. J Neurosci 27(15):4178–4181. doi:10.1523/JNEUROSCI.5468-06.2007

    PubMed  CAS  Google Scholar 

  • Rauch A, Rainer G, Logothetis NK (2008) The effect of a serotonin-induced dissociation between spiking and perisynaptic activity on BOLD functional MRI. Proc Natl Acad Sci USA 105(18):6759–6764. doi:10.1073/pnas.0800312105

    PubMed  CAS  PubMed Central  Google Scholar 

  • Recanzone GH, Guard DC, Phan ML (2000a) Frequency and intensity response properties of single neurons in the auditory cortex of the behaving macaque monkey. J Neurophysiol 83(4):2315–2331

    PubMed  CAS  Google Scholar 

  • Recanzone GH, Guard DC, Phan ML, Su TK (2000b) Correlation between the activity of single auditory cortical neurons and sound-localization behavior in the macaque monkey. J Neurophysiol 83(5):2723–2739

    PubMed  CAS  Google Scholar 

  • Romei V, Murray MM, Merabet LB, Thut G (2007) Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J Neurosci 27(43):11465–11472. doi:10.1523/JNEUROSCI.2827-07.2007

    PubMed  CAS  Google Scholar 

  • Romei V, Murray MM, Cappe C, Thut G (2009) Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds. Curr Biol 19(21):1799–1805. doi:10.1016/j.cub.2009.09.027

    PubMed  CAS  Google Scholar 

  • Romei V, Murray MM, Cappe C, Thut G (2013) The contributions of sensory dominance and attentional bias to cross-modal enhancement of visual cortex excitability. J Cogn Neurosci 25(7):1122–1135. doi:10.1162/jocn_a_00367

    PubMed  Google Scholar 

  • Romo R, Hernandez A, Zainos A, Salinas E (1998) Somatosensory discrimination based on cortical microstimulation. Nature 392(6674):387–390

    PubMed  CAS  Google Scholar 

  • Romo R, Hernandez A, Zainos A, Lemus L, Brody CD (2002) Neuronal correlates of decision-making in secondary somatosensory cortex. Nat Neurosci 5(11):1217–1225

    PubMed  CAS  Google Scholar 

  • Romo R, Hernandez A, Zainos A (2004) Neuronal correlates of a perceptual decision in ventral premotor cortex. Neuron 41(1):165–173

    PubMed  CAS  Google Scholar 

  • Royal DW, Carriere BN, Wallace MT (2009) Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Exp Brain Res 198(2–3):127–136. doi:10.1007/s00221-009-1772-y

    PubMed  PubMed Central  Google Scholar 

  • Sarko DK, Nidiffer AR, Powers AR, Ghose D, Fister MC, Hillock-Dunn A, Krueger J, Wallace MT (2012) Spatial and temporal features of multisensory processes: Bridging animal and human studies. In: Murray MM, Wallace MT (eds) Frontiers in the neural bases of multisensory processes. CRC Press, Boca Raton, pp 191–215

    Google Scholar 

  • Schweickert R (1978) A critical path generalization of the additive factor method: analysis of a Stroop task. J Math Psychol 18:105–139

    Google Scholar 

  • Shams L, Kamitani Y, Shimojo S (2000) Illusions. What you see is what you hear. Nature 408(6814):788

    PubMed  CAS  Google Scholar 

  • Shams L, Ma WJ, Beierholm U (2005) Sound-induced flash illusion as an optimal percept. NeuroReport 16(17):1923–1927

    PubMed  Google Scholar 

  • Singer W (1993) Neuronal representations, assemblies and temporal coherence. Prog Brain Res 95:461–474

    PubMed  CAS  Google Scholar 

  • Snyder LH, Dickinson AR, Calton JL (2006) Preparatory delay activity in the monkey parietal reach region predicts reach reaction times. J Neurosci 26(40):10091–10099. doi:10.1523/JNEUROSCI.0513-06.2006

    PubMed  CAS  Google Scholar 

  • Spence C, Driver J (2000) Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism. NeuroReport 11(9):2057–2061

    PubMed  CAS  Google Scholar 

  • Sperdin HF, Cappe C, Murray MM (2010) The behavioral relevance of multisensory neural response interactions. Front Neurosci 3:9

    Google Scholar 

  • Spierer L, Manuel AL, Bueti D, Murray MM (2013) Contributions of pitch and bandwidth to sound-induced enhancement of visual cortex excitability in humans. Cortex; a journal devoted to the study of the nervous system and behavior 49(10):2728–2734. doi:10.1016/j.cortex.2013.01.001

    PubMed  Google Scholar 

  • Stanford TR, Stein BE (2007) Superadditivity in multisensory integration: putting the computation in context. NeuroReport 18(8):787–792. doi:10.1097/WNR.0b013e3280c1e315

    PubMed  Google Scholar 

  • Stanford TR, Quessy S, Stein BE (2005) Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25(28):6499–6508

    PubMed  CAS  PubMed Central  Google Scholar 

  • Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266. doi:10.1038/nrn2331

    PubMed  CAS  Google Scholar 

  • Stein BE, Wallace MT (1996) Comparisons of cross-modality integration in midbrain and cortex. Prog Brain Res 112:289–299

    PubMed  CAS  Google Scholar 

  • Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448(2):355–358

    PubMed  CAS  Google Scholar 

  • Steinmetz PN, Roy A, Fitzgerald PJ, Hsiao SS, Johnson KO, Niebur E (2000) Attention modulates synchronized neuronal firing in primate somatosensory cortex. Nature 404(6774):187–190

    PubMed  CAS  Google Scholar 

  • Stekelenburg JJ, Vroomen J (2007) Neural correlates of multisensory integration of ecologically valid audiovisual events. J Cogn Neurosci 19(12):1964–1973. doi:10.1162/jocn.2007.19.12.1964

    PubMed  Google Scholar 

  • Sternberg S (1969a) The discovery of processing stages: extensions of Donders’ method. Acta Psychol 30:276–315

    Google Scholar 

  • Sternberg S (1969b) Memory-scanning: mental processes revealed by reaction-time experiments. Am Sci 57(4):421–457

    PubMed  CAS  Google Scholar 

  • Sternberg S (1975) Memory scanning: new findings and current controversies. Exp Pshychol 27:1–32

    Google Scholar 

  • Sternberg S (1998) Discovering mental processing stages: The method of additive factors. In: Scarborough D, Sternberg S (eds) An invitation to cognitive science, vol 4., Methods, models, and conceptual issuesMIT Press, Cambridge, pp 739–811

    Google Scholar 

  • Sternberg S (2001) Seperate modifiability, mental modules, and the use of pure and composite measures to reveal them. Acta Psychol 106:147–246

    CAS  Google Scholar 

  • Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. NeuroImage 44(3):1210–1223. doi:10.1016/j.neuroimage.2008.09.034

    PubMed  Google Scholar 

  • Stevenson RA, Wallace MT (2013) Multisensory temporal integration: task and stimulus dependencies. Exp Brain Res Experimentelle Hirnforschung Experimentation cerebrale 227(2):249–261. doi:10.1007/s00221-013-3507-3

    Google Scholar 

  • Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95

    PubMed  Google Scholar 

  • Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res Experimentelle Hirnforschung 198(2–3):183–194. doi:10.1007/s00221-009-1783-8

    Google Scholar 

  • Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. NeuroImage 49(4):3308–3318. doi:10.1016/j.neuroimage.2009.12.001

    PubMed  PubMed Central  Google Scholar 

  • Stevenson R, Bushmakin M, Kim S, Wallace M, Puce A, James T (2012a) Inverse effectiveness and multisensory interactions in visual event-related potentials with audiovisual speech. Brain Topogr 1:19. doi:10.1007/s10548-012-0220-7

    Google Scholar 

  • Stevenson RA, Bushmakin M, Kim S, Wallace MT, Puce A, James TW (2012b) Inverse effectiveness and multisensory interactions in visual event-related potentials with audiovisual speech. Brain Topogr 25(3):308–326. doi:10.1007/s10548-012-0220-7

    PubMed  PubMed Central  Google Scholar 

  • Stevenson RA, Zemtsov RK, Wallace MT (2012c) Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. J Exp Psychol Hum Percept Perform. doi:10.1037/a0027339

    PubMed  PubMed Central  Google Scholar 

  • Stevenson RA, Wilson MM, Powers AR, Wallace MT (2013) The effects of visual training on multisensory temporal processing. Exp Brain Res Experimentelle Hirnforschung Experimentation cerebrale 225(4):479–489. doi:10.1007/s00221-012-3387-y

    Google Scholar 

  • Stevenson RA, Siemann JK, Schneider BC, Eberly HE, Woynaroski TG, Camarata SM, Wallace MT (2014) Multisensory temporal integration in autism spectrum disorders. J Neurosci 34(3):691–697. doi:10.1523/jneurosci.3615-13.2014

    PubMed  CAS  PubMed Central  Google Scholar 

  • Takarae Y, Luna B, Minshew NJ, Sweeney JA (2008) Patterns of visual sensory and sensorimotor abnormalities in autism vary in relation to history of early language delay. J Int Neuropsychol Soc 14(6):980–989. doi:10.1017/S1355617708081277

    PubMed  PubMed Central  Google Scholar 

  • Tal N, Amedi A (2009) Multisensory visual-tactile object related network in humans: insights gained using a novel crossmodal adaptation approach. Exp Brain Res Experimentelle Hirnforschung Experimentation cerebrale 198(2–3):165–182. doi:10.1007/s00221-009-1949-4

    Google Scholar 

  • Talsma D, Woldorff MG (2005) Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J Cogn Neurosci 17(7):1098–1114. doi:10.1162/0898929054475172

    PubMed  Google Scholar 

  • Taylor DA (1976) Stage analysis of reaction time. Psychol Bull 83:161–191

    PubMed  CAS  Google Scholar 

  • Taylor K, Mandon S, Freiwald WA, Kreiter AK (2005) Coherent oscillatory activity in monkey area v4 predicts successful allocation of attention. Cereb Cortex 15(9):1424–1437. doi:10.1093/cercor/bhi023

    PubMed  CAS  Google Scholar 

  • Teder-Salejarvi WA, McDonald JJ, Di Russo F, Hillyard SA (2002) An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res Cogn Brain Res 14(1):106–114

    PubMed  CAS  Google Scholar 

  • Thelen A, Cappe C, Murray MM (2012) Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. NeuroImage 62(3):1478–1488. doi:10.1016/j.neuroimage.2012.05.027

    PubMed  Google Scholar 

  • Thiele A, Distler C, Hoffmann KP (1999) Decision-related activity in the macaque dorsal visual pathway. Eur J Neurosci 11(6):2044–2058

    PubMed  CAS  Google Scholar 

  • Thompson KG, Hanes DP, Bichot NP, Schall JD (1996) Perceptual and motor processing stages identified in the activity of macaque frontal eye field neurons during visual search. J Neurophysiol 76(6):4040–4055

    PubMed  CAS  Google Scholar 

  • Townsend JT (1984) Uncovering mental processes with factorial experiments. J Math Psychol 28(4):363–400

    Google Scholar 

  • Townsend JT, Ashby FG (1978) Methods of modeling capacity in simple processing systems. Cogn Theory 3:200–239

    Google Scholar 

  • Townsend JT, Ashby FG (1980) Decomposing the reaction time distribution: pure insertion and selective influence revisited. J Math Psychol 21:93–123

    Google Scholar 

  • Townsend JT, Nozawa G (1995) Spatio-temporal properties of elementary perception: an investigation of parrallel, serial, and coactive theories. J Math Psychol 39:321–359

    Google Scholar 

  • Townsend JT, Thomas RD (1994) Stochastic dependencies in parallel and serial models: effects on systems factorial interactions. J Math Psychol 38(1):1–34

    Google Scholar 

  • Uka T, Sasaki R, Kumano H (2012) Change in choice-related response modulation in area MT during learning of a depth-discrimination task is consistent with task learning. J Neurosci 32(40):13689–13700. doi:10.1523/JNEUROSCI.4406-10.2012

    PubMed  CAS  Google Scholar 

  • Usrey WM, Reid RC (1999) Synchronous activity in the visual system. Annu Rev Physiol 61:435–456. doi:10.1146/annurev.physiol.61.1.435

    PubMed  CAS  Google Scholar 

  • Van Opstal AJ, Munoz DP (2004) Auditory-visual interactions subserving primate gaze orienting. In: Calvert GA, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 373–394

    Google Scholar 

  • Van Wanrooij MM, Bell AH, Munoz DP, Van Opstal AJ (2009) The effect of spatial-temporal audiovisual disparities on saccades in a complex scene. Exp Brain Res. doi:10.1007/s00221-009-1815-4

    PubMed  PubMed Central  Google Scholar 

  • van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607. doi:10.1016/j.neuropsychologia.2006.01.001

    PubMed  Google Scholar 

  • Viswanathan A, Freeman RD (2007) Neurometabolic coupling in cerebral cortex reflects synaptic more than spiking activity. Nat Neurosci 10(10):1308–1312. doi:10.1038/nn1977

    PubMed  CAS  Google Scholar 

  • Vroomen J, Baart M (2009) Recalibration of phonetic categories by lipread speech: measuring aftereffects after a 24-hour delay. Lang Speech 52(Pt 2–3):341–350

    PubMed  Google Scholar 

  • Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72(4):871–884. doi:10.3758/APP.72.4.871

    PubMed  Google Scholar 

  • Vroomen J, Stekelenburg JJ (2010) Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. J Cogn Neurosci 22(7):1583–1596. doi:10.1162/jocn.2009.21308

    PubMed  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res Experimentelle Hirnforschung Experimentation cerebrale 91(3):484–488

    CAS  Google Scholar 

  • Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76(2):1246–1266

    PubMed  CAS  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80(2):1006–1010

    PubMed  CAS  Google Scholar 

  • Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79. doi:10.1186/1471-2202-9-79

    PubMed  PubMed Central  Google Scholar 

  • Wenger MJ, Townsend JT (2000) Basic response time tools for studying general processing capacity in attention, perception, and cognition. J Gen Psychol 127(1):67–99

    PubMed  CAS  Google Scholar 

  • Werner S, Noppeney U (2010) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex 20(8):1829–1842. doi:10.1093/cercor/bhp248

    PubMed  Google Scholar 

  • Wilkinson LK, Meredith MA, Stein BE (1996) The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res 112(1):1–10

    PubMed  CAS  Google Scholar 

  • Xing D, Yeh CI, Shapley RM (2009) Spatial spread of the local field potential and its laminar variation in visual cortex. J Neurosci 29(37):11540–11549. doi:10.1523/JNEUROSCI.2573-09.2009

    PubMed  CAS  PubMed Central  Google Scholar 

  • Yang T, Maunsell JH (2004) The effect of perceptual learning on neuronal responses in monkey visual area V4. J Neurosci 24(7):1617–1626

    PubMed  Google Scholar 

  • Yizhar O, Fenno L, Zhang F, Hegemann P, Diesseroth K (2011a) Microbial opsins: a family of single-component tools for optical control of neural activity. Cold Spring Harb Protoc 2011(3): top102

  • Yizhar O, Fenno LE, Davidson TJ, Mogri M, Deisseroth K (2011b) Optogenetics in neural systems. Neuron 71(1):9–34. doi:10.1016/j.neuron.2011.06.004

    PubMed  CAS  Google Scholar 

  • Zion Golumbic EM, Poeppel D, Schroeder CE (2012) Temporal context in speech processing and attentional stream selection: a behavioral and neural perspective. Brain Lang 122(3):151–161. doi:10.1016/j.bandl.2011.12.010

    PubMed  PubMed Central  Google Scholar 

  • Zion Golumbic E, Cogan GB, Schroeder CE, Poeppel D (2013a) Visual input enhances selective speech envelope tracking in auditory cortex at a “cocktail party”. J Neurosci 33(4):1417–1426. doi:10.1523/JNEUROSCI.3675-12.2013

    PubMed  Google Scholar 

  • Zion Golumbic EM, Ding N, Bickel S, Lakatos P, Schevon CA, McKhann GM, Goodman RR, Emerson R, Mehta AD, Simon JZ, Poeppel D, Schroeder CE (2013b) Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”. Neuron 77(5):980–991. doi:10.1016/j.neuron.2012.12.037

    PubMed  CAS  PubMed Central  Google Scholar 

Download references

Acknowledgments

Funding for this work was provided by a Banting Postdoctoral Fellowship, It’s only a matter of time: Neural networks underlying multisensory perceptual binding, National Institutes of Health grant F32 DC011993, Multisensory Integration and Temporal Processing in ASD, a grant from the Vanderbilt Institute for Clinical and Translational Research, VICTR VR5807.1, Development and Modulation of Multisensory Integration, National Institutes of Health grant R34 DC010927, Evaluation of Sensory Integration Treatment in ASD, National Institutes of Health grant 5T32 MH018921-24, From Brain and Behavioral Science to Intervention, a Vanderbilt Kennedy Center MARI/Hobbs Award, the Vanderbilt Brain Institute, the Vanderbilt University Kennedy Center, the Idaho IDeA Network of Biomedical Research Excellence, and National Institutes of Health grants P20 RR016454 and P20 GM103408.

Author information

Authors and Affiliations

  1. Department of Psychology, University of Toronto, Toronto, ON, Canada

    Ryan A. Stevenson

  2. Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, TN, USA

    Ryan A. Stevenson, Diana K. Sarko, Aaron R. Nidiffer & Mark T. Wallace

  3. Vanderbilt Brain Institute, Nashville, TN, USA

    Ryan A. Stevenson, Juliane Krueger Fister, LeAnne R. Kurela, Justin K. Siemann & Mark T. Wallace

  4. Vanderbilt Kennedy Center, Nashville, TN, USA

    Ryan A. Stevenson, Justin K. Siemann & Mark T. Wallace

  5. Department of Psychology, Vanderbilt University, Nashville, TN, USA

    Dipanwita Ghose & Mark T. Wallace

  6. Neuroscience Graduate Program, Vanderbilt University Medical Center, Nashville, TN, USA

    Juliane Krueger Fister, LeAnne R. Kurela & Justin K. Siemann

  7. Department of Anatomy, Cell Biology and Physiology, Edward Via College of Osteopathic Medicine, Spartanburg, SC, USA

    Diana K. Sarko

  8. Department of Communication Sciences and Disorders, Idaho State University, Pocatello, ID, USA

    Nicholas A. Altieri

  9. Silvio O. Conte Center for Neuroscience Research, Vanderbilt University Medical Center, Nashville, TN, USA

    Justin K. Siemann

  10. Clinical Neuroscience Scholars Program, Vanderbilt University Medical Center, Nashville, TN, USA

    Juliane Krueger Fister, LeAnne R. Kurela & Justin K. Siemann

  11. Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA

    Thomas W. James

  12. Department of Psychiatry, Vanderbilt University, Nashville, TN, USA

    Mark T. Wallace

  13. Rm 523, Sidney Smith Bldg, 100 Saint George St., Toronto, ON, Canada

    Ryan A. Stevenson

Authors
  1. Ryan A. Stevenson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dipanwita Ghose
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Juliane Krueger Fister
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Diana K. Sarko
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nicholas A. Altieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Aaron R. Nidiffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. LeAnne R. Kurela
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Justin K. Siemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Thomas W. James
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Mark T. Wallace
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryan A. Stevenson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stevenson, R.A., Ghose, D., Fister, J.K. et al. Identifying and Quantifying Multisensory Integration: A Tutorial Review. Brain Topogr 27, 707–730 (2014). https://doi.org/10.1007/s10548-014-0365-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10548-014-0365-7

Keywords