Decoding

J.NeuralEng.4(2007)264–275

JOURNALOFNEURALENGINEERING

doi:10.1088/1741-2560/4/3/012

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

GSchalk1,2,JKub´anek1,3,KJMiller4,5,NRAnderson6,ECLeuthardt7,8,JGOjemann7,DLimbrick8,DMoran6,LAGerhardt2andJRWolpaw1

12

BCIR&DProgr,WadsworthCtr,NYSDepartmentofHealth,Albany,NY,USAElec,Comp,andSystEngDepartment,RenssPolytInst,Troy,NY,USA3

DepartmentofCybern,CzechTechUniversity,Prague,CzechRepublic4

DepartmentofPhysics,UniversityofWashington,Seattle,WA,USA5

DepartmentofMedicine,UniversityofWashington,Seattle,WA,USA6

DepartmentofBiomedEng,WashingtonUniversity,St.Louis,MO,USA7

DepartmentofNeurosurgery,UniversityofWashSchoolofMed,Seattle,WA,USA8

DepartmentofNeurolSurg,Barnes-JewishHospital,St.Louis,MO,USAE-mail:schalk@wadsworth.org

Received27September2006

Acceptedforpublication30May2007Published22June2007

Onlineatstacks.iop.org/JNE/4/264

Abstract

Signalsfromthebraincouldprovideanon-muscularcommunicationandcontrolsystem,abrain–computerinterface(BCI),forpeoplewhoareseverelyparalyzed.AcommonBCI

researchstrategybeginsbydecodingkinematicparametersfrombrainsignalsrecordedduringactualarmmovement.Ithasbeenassumedthattheseparameterscanbederivedaccuratelyonlyfromsignalsrecordedbyintracorticalmicroelectrodes,butthelong-termstabilityofsuchelectrodesisuncertain.Thepresentstudydisprovesthiswidespreadassumptionbyshowinginhumansthatkinematicparameterscanalsobedecodedfromsignalsrecordedbysubduralelectrodesonthecorticalsurface(ECoG)withanaccuracycomparabletothatachievedinmonkeystudiesusingintracorticalmicroelectrodes.AnewECoGfeaturelabeledthelocalmotorpotential(LMP)providedthemostinformationaboutmovement.Furthermore,featuresdisplayedcosinetuningthathaspreviouslybeendescribedonlyforsignalsrecordedwithinthebrain.TheseresultssuggestthatECoGcouldbeamorestableandlessinvasivealternativetointracorticalelectrodesforBCIsystems,andcouldalsoproveusefulinstudiesofmotorfunction.

MThisarticlefeaturesonlinemultimediaenhancements

(Somefiguresinthisarticleareincolouronlyintheelectronicversion)

1.Introduction

Brain–computerinterfaces(BCIs)usebrainsignalstocommunicateauser’sintent[1].Becausethesesystemsdonotdependonperipheralnervesandmuscles,theycanbeusedbypeoplewithseveremotordisabilities.PracticalapplicationsofBCItechnologyarecurrentlyimpededbythelimitationsandrequirementsofnon-invasiveandinvasivemethods.

1741-2560/07/030264+12$30.00

Non-invasiveBCIsuseelectroencephalographicactivity(EEG)recordedfromthescalp[1].Whilenon-invasiveBCIscansupportmultidimensionalcontrol[2],theiruserequiresextensiveusertraining.InvasiveBCIsuseactivityfrommultipleneuronsrecordedwithinthebrain[3–6].SignalsrecordedwithincortexhavehigherfidelityandmightsupportBCIsystemsthatrequirelesstrainingthanEEG-basedsystems.However,clinicalimplementationsareimpeded

264

©2007IOPPublishingLtdPrintedintheUK

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

Figure1.Electrodelocationsinthefivesubjects.TheelectrodeswereprojectedontherighthemisphereforsubjectsAandE(see

asterisks).ThebraintemplateonthebottomrighthighlightsthelocationofthecentralsulcusandSylvianfissure,andalsooutlinesrelevantBrodmannareas.

Table1.Clinicalprofiles.Allsubjectswereliterateandfunctionallyindependent.SubjectAhadapriorleftanteriortemporallobectomy.SubjectBhadnotraumaticorstructurallesion.SubjectChadarightanteriorfrontaltraumaticinjury.SubjectDhadarightposteriortemporalarteriovenousmalformationembolized20yearsearlier.SubjectEhadaleftfrontaldysembryoplasticneuroepithelialtumor.

SubjectABCDE

Age2324384818

SexMFMMF

HandRRRRR

CognitivecapacityNormal(IQ88)Normal(IQ97)Borderline(IQ70)

Normal(IQ82;Rightsupquadrvisualdeficit)Normal(IQ86)

GridlocationLeftfrontaltemporalRightfrontaltemporalRightfrontalRighttemporalLeftfrontal

Seizurefocus

Lefttemporal

RightorbitofrontalandtemporalRightfrontal

RighttemporaloccipitalfocusLeftfrontal

mainlybytherisksofsurgicalimplantationandbythesubstantialproblemsinachievingandmaintainingstablelong-termrecordings[7,8].Whileafewrecentstudieshavebeguntoapplynon-invasiveandinvasiveBCItechnologiestotheneedsofseverelydisabledindividuals[9,10],theseissuesremaincrucialobstaclesthatcurrentlyprohibitwidespreadclinicaluseinhumans.

Inthecurrentabsenceoftechniquestoextracthigh-fidelitysignalsfromEEGandofmethodstorecordactivityfromwithinthebrainsafelyandoverlongperiods,theuseofelectrocorticographicactivity(ECoG)recordedfromthecorticalsurfacecouldbeapowerfulandpracticalalternative.ECoGhashigherspatialresolutionthanEEG(i.e.,tenthsofmillimetersversuscentimeters),broaderbandwidth(i.e.,0–500Hz[11]versus0–40Hz),higheramplitude(i.e.,50–100µVmaximumversus10–20µV),andfarlessvulnerabilitytoartifactssuchasEMG[12].Atthesametime,becauseECoGdoesnotrequireelectrodesthatpenetratecortex,itislikelytohavegreaterlong-termstability[13–17]andtoproducelesstissuedamage.

WepreviouslyshowedthatECoGsignalsassociatedwithimageryofarbitrarytaskscanprovideone-dimensionalBCIcontrolwithlittletraining[18].Itispossiblethatusingmoreintuitivetasks(suchasimaginedhandmovements)mightmoreefficientlyextendthiscontroltomultipledimensions.However,moststudiesthatdecodedhandmovementsfrombrainsignalshavebeeninmonkeys[19–21].Onlylimitedrelevantinformationisavailableinhumans[18,22,23].

Inthisstudy,wesetouttodetermineifitispossibletofaithfullydecodeinrealtimekinematicparametersfromECoGsignalsrecordedinhumans.Westudiedfivesubjectswhowereaskedtouseajoysticktomoveacursorsoas

totrackatargetthatmovedonacomputerscreen.TheprincipalresultsshowthatECoGsignalscanbeusedtoaccuratelydecodetwo-dimensionaljoystickkinematicsinhumans.Theyalsoshowthattheseresultsarewithintherangeofthoseachievedinstudiesusingintracorticalmicroelectroderecordingsinmonkeysthatalsoaimedtodecodetwo-dimensionalkinematicparameters.Furthermore,theyindicatethatanewbrainsignalcomponent,whichwelabelthelocalmotorpotential(LMP),holdssubstantialinformationaboutmovementdirection.Finally,ECoGfeaturescanalsoexhibitthesamekindofcosinetuningpreviouslydetectedonlywithintracorticalmicroelectrodesinmonkeys[21,24–32].TheseresultsprovidestrongevidencethatECoGcouldbeusedtoprovideaccuratemultidimensionalBCIcontrol,andalsosuggestthatECoGisapotentiallypowerfultoolforthestudyofbrainfunction.

2.Methods

2.1.Subjects

Thesubjectsinthisstudywerefivepatientswithintractableepilepsywhounderwenttemporaryplacementofsubduralelectrodearraystolocalizeseizurefocipriortosurgicalresection.Theyincludedthreemen(subjectsA,CandD)andtwowomen(subjectsBandE).(Seetable1foradditionalinformation.)Allgaveinformedconsent.ThestudywasapprovedbytheInstitutionalReviewBoardoftheUniversityofWashingtonSchoolofMedicine.Eachsubjecthada48-or64-electrodegridplacedoverthefronto-parietal-temporalregionincludingpartsofsensorimotorcortex(seefigure1fordetails).Thesegridsconsistedofflatelectrodeswitha

265

GSchalketal

diameterof4mm(2.3mmexposed)andaninter-electrodedistanceof1cm,andwereimplantedforabout1week.GridplacementsanddurationofECoGmonitoringwerebasedsolelyontherequirementsoftheclinicalevaluation,withoutanyconsiderationofthisstudy.Followingplacementofthesubduralgrid,eachsubjecthadpostoperativeanterior–posteriorandlateralradiographstoverifygridlocation.2.2.Experimentalparadigm

Duringthestudy,eachsubjectwasinasemi-recumbentpositioninahospitalbedabout1mfromavideoscreen.He/sheusedajoystickwiththehandcontralateraltotheimplantedelectrodearraytomoveawhitecursorintwodimensionstotrackagreentarget.Thetargetmovedcounter-clockwiseinacirclethatwaspositionedinthecenterofacomputerscreen.Thetargetencouragedthesubjects,whowereoftenimpairedbypost-operativerecovery,toengageincontinuousmovements.Substantialvariabilityinthesubjects’trackingtrajectoriesallowedustomakeadditionalinferencesaboutdifferentaspectsofthemovement(seesection4fordetails).Thediameterofthecirclewas61%(onesubject)or85%(allotherfoursubjects)ofthescreen’sheight.Onefullrevolutionofthetargettook6.3sforallsubjects.(Atthesamemovementspeed,atypicalcenter-outtask(i.e.,movingacursorfromthecenterofthescreentotheperipheryofthescreen)wouldhavehadamovementdurationoflessthan1.2s.)Toallowforofflineanalyses,thepositionofthecursorandthetargetwerestoredalongwiththedigitizedECoGsignals.Joystickpositionwasmappedtocursorvelocityandthejoystickproducedsignificantforcefeedbacktothesubjectsothatthistaskapproachedtheisometricforcetasksusedin[33–35].Subjectswereaskedtouseshoulderandproximalarmmovementsratherthanwristmovements.Theywerealsoaskedtomaintainaconstantposture,butneitherbody,head,norhandwererestrainedinanyway.2.3.Datacollection

Inallexperiments,werecordedECoGfromtheelectrodegridusingthegeneral-purposeBCIsystemBCI2000[36]connectedtoaNeuroscanSynamps2system.SimultaneousclinicalmonitoringwasachievedusingaconnectorthatsplitthecablescomingfromthesubjectintoonesetthatwasconnectedtotheclinicalmonitoringsystemandanothersetthatwasconnectedtotheBCI2000/Neuroscansystem.Thus,atnotimewasclinicalcareorclinicaldatacollectioncompromised.Allelectrodeswerereferencedtoaninactiveelectrode.Thesignalswereamplified,bandpassfilteredbetween0.15and200Hz,digitizedat1000Hz,andstoredinBCI2000.Theamountofdataobtainedvariedfromsubjecttosubject,anddependedonthesubject’sphysicalstateandavailability.Thedurationofthedatasetsaveraged443s(range130–830s).EachdatasetwasvisuallyinspectedandallchannelsthatdidnotclearlycontainECoGactivity(e.g.,suchaschannelsthatcontainedflatsignalsornoiseduetobrokenconnections)wereremovedpriortoanalysis,whichleft48–64channelsforouranalyses.266

2.4.3Dcorticalmapping

Weusedlateralskullradiographstoidentifythestereotacticcoordinatesofeachgridelectrodewithsoftware[37]thatduplicatedthemanualproceduredescribedin[38].WedefinedcorticalareasusingTalairach’sCo-PlanarStereotaxicAtlasoftheHumanBrain[39]andaTalairachtransformation(http://ric.uthscsa.edu/projects/talairachdaemon.html).Weobtainedatemplate3Dcorticalbrainmodel(subject-specificbrainmodelswerenotavailable)fromsourcecodeprovidedontheAFNISUMAwebsite(http://afni.nimh.nih.gov/afni/suma).Finally,weprojectedeachsubject’selectrodelocationsonthis3DbrainmodelandgeneratedactivationmapsusingacustomMatlabprogram.

2.5.Featureextractionandselection

Wefirstre-referencedthesignalfromeachelectrodeusingacommonaveragereference(CAR)montage.Todothis,

weobtainedtheCAR-filtered󰀁

s󰀁

h=sh−1󰀁signalsHhatchannelhusing

Hq=1sq.Hwasthetotalnumberofchannelsandshwastheoriginalsignalsampleataparticulartime.

Foreach333mstimeperiod(overlappingby166ms),wethenconvertedthetime-seriesECoGdataintothefrequencydomainwithanautoregressivemodel[40]oforder50.Usingthismodel,wecalculatedspectralamplitudesbetween0and200Hzin1Hzbins.Weaveragedthesespectralamplitudesinparticularfrequencyranges(8–12Hz,18–24Hz,35–42Hz,42–70Hz,70–100Hz,100–140Hz,140–190Hz)inthemu,betaandgammafrequencybands,similartothoseusedin[21].Inaddition,visualinspectionidentifiedspecificchannelsinwhichECoGvoltagelevelappearedtocorrelatewithkinematicparameters.Itthusseemedthattheselocationswereamplitude-modulatedinthetime-domain(i.e.,exhibitingthelocalmotorpotential(LMP))ratherthaninalowbandinthefrequency-domain.

Figure2showsanexampleECoGtimecourseforsubjectC(A),thespatialdistributionofchannelsthatexhibittheLMP(B),andthemagnifiedtimecourseofchannel35,aswellastheXpositionofthecursorandmovingtarget(C).ThecorrelationoftheECoGtimecoursewiththemovementparametersisevidentandclearlyfocusedonselectchannelsoverhandsensorimotorcortex.Themagnificationshownin(C)demonstratesanexampleofgoodcorrelationbetweenECoGtimecourse(blacktrace)withtheXpositionofthecursor(thickdarkgreentrace).ItalsoillustratesanexampleofmodesttrackingperformanceindicatedbythepoorconcurrencebetweentheXpositionofthecursor(thickdarkgreentrace)andtheXpositionofthemovingtarget(thinlightgreentrace)between45and60s.

Toaccountforthepossibilityofmovement-relatedLMPmodulation,weaddedtothefrequency-basedfeatureslistedabovethe333msrunningaverageoftherawunrectifiedsignal.Thisyieldedeightfeaturesperchannel,i.e.,atotalmaximumof8×64=512features.Finally,weappliedarunningaveragefilter(boxcarwindow,lengthwas9samples(9×166ms=1494ms))toeachofthesefeatures.

Toreducethislargenumberoffeatures,weemployedthecorrelation-basedfeatureselector(CFS)thatisimplementedin

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

(A)

212223242526272829

(B)1

8

57

NORMALIZED AMPLITUDE3031323334353637383940XcrsXtrkYcrsYtrk

64

(C)

30

20

TIME (s)

4560

04060

Figure2.ExampleECoGtimecourseforsubjectC.(A)TimecourseofECoGsignalsforchannels21–40andfortheXpositionofthecursor(Xcrs),theXpositionofthetrackingtarget(Xtrk),aswellastheYpositionofthecursor(Ycrs)andtarget(Ytrk).ChannelsthatexhibitatimecoursethatiscorrelatedwiththemovementparametersXcrsorYcrs(i.e.,atimecourseexhibitingtheLMP)areindicatedwithsymbols.(B)Electrodelocationsincludingchannelnumbers.SymbolsindicatethelocationsofchannelsthatshowtheLMP.

(C)MagnificationofECoGtimecourseofchannel35from30to60s,aswellastheXpositionofthecursor(thickdarkgreentrace)andtheXpositionofthemovingtarget(thinlightgreentrace).

theJava-basedWekapackage[41],whichranksfeaturesubsetsratherthanindividualfeatures.(Ittherebytakesintoaccountnotonlythecorrelationofanyoneparticularfeaturewiththevaluestobedecoded,butalsothecross-correlationbetweenfeatures.)Theuseofthisprocedurereducedthenumberoffeaturesto5–20(10onaverage)forthedifferentdatasets.2.6.Classification

UsingtheECoGfeaturesselectedbytheCFSprocedure,wethenderivedonelinearmodelforeachofthefourkinematicparametersofthesubject’scursor(i.e.,horizontalposition,verticalposition,horizontalvelocity,verticalvelocity).WeusedtheECoGfeaturestodecodeeachofthefourkinematicparametersimmediatelyfollowingtheperiodrepresentingthesefeatures(i.e.,causalprediction)sothatthesameprocedurecouldbeusedinrealtime.2.7.Evaluation

Theperformanceofthelinearmodelswasevaluatedusing5-foldcross-validation,i.e.,eachdatasetwasdividedintofiveparts,thelinearmodelsweredeterminedfrom4/5thofthedataset(trainingset)andtested(i.e.,thecoefficientsofthelinearmodelderivedusingtheregressionwereapplied)ontheremaining1/5th(testset).Thisprocedurewasthenrepeatedfivetimes—eachtime,adifferent1/5thofthedataset

wasusedasthetestset.(Thefeatureselectionprocedurewasalwaysappliedtothetrainingsetonly.)

Weevaluatedtheperformanceofeachofthefivemodelsbycross-correlatingthedecodedkinematicparameterswiththeactualvaluesforpositionandvelocity.Thisresultedinacorrelationcoefficientrforeachdataset,cross-validationfold,andeachofthefourkinematicparameters.2.8.Directionaltuning

Inadditionalanalyses,wedeterminedtherelationshipbetweeneachECoGfeaturefandthedirection(i.e.,angle)ofmovement.Todothis,weassigned,foreachcross-validationfoldofeachdataset,thefeaturesamplesftothecorrespondingmovementdirection,whichwediscretizedin20equidistantbinsfrom−180to+180◦.Dependingonthelengthofthedatasetandthejoystickmovementpatterns,eachofthesebinscontainedavariableamountoffeaturesamples(19onaverage).The20binsiandthedistributionoffeaturesamplesfiwithineachbindefinedatuningcurveforeachfeature,location,cross-validationfoldandsubject.Wethendeterminedwhethertheseobservedtuningcurveswereafunctionofmovementdirection(i.e.,theyweretuned)oracosinefunctionofmovementdirection(i.e.,theywerecosinetuned),similartotheapproachin[21].Thisprocedureisdescribedinshortbelow.

267

GSchalketal

Todeterminewhetheracurvewastuned,wecalculatedtheprobabilitythateachtuningcurvedifferedfromrandomlygeneratedtuningcurves.Todothis,wefirstcalculatedatuningindexmeasureSNRthatrelatedthevarianceofallfeaturevaluesσ2(f)totheaveragevarianceofthefeaturevalues

󰀃󰀂1󰀁2022

)

σ(f):SNR=1󰀁σ20(f.withineachbin20ii=1σ2(f)Then,weshuffledallfeaturevaluessuchthattheywere

assignedtorandomlychosenbins,calculatedthevalueofSNR,andrepeatedthisprocedure200times,whichresultedin200measuresofSNR.WemodeledthesemeasurementsusingaGaussiandistribution(i.e.,wecalculatedtheSNRmeanandstandarddeviation)1.WefinallydeterminedtheprobabilityptthatthevalueofSNRfortheobservedtuningcurvewasgeneratedbytheGaussianmodeldistributionofrandomlygeneratedSNRvalues.Atuningcurvewasconsideredtunedifptwassmallerthan0.001.

Foreachtuningcurvethatwasconsideredtuned,wealsodeterminedwhetheritwascosinetuned.Todothis,wecalculatedthemeanvaluefiofallfeatureswithineachbin,whichdefinedtheaverageobservedtuningcurve.Wethencalculatedthecorrelationcoefficientrbetweenthisaveragetuningcurveandacosinefunctionthatwasfitthroughtothiscurve.Similartoabove,wethendeterminedtheprobabilitypctthattheobservedtuningcurvewasgeneratedbyadistributionofrandomlygeneratedrvalues.Atuningcurvewasconsideredcosinetunedifpctwassmallerthan0.001.

20i=1

i

Table2.Decodingofkinematicparameters.Correlationcoefficients(r)betweentheactualanddecodedkinematic

parameters(horizontalpositionofthecursor(X),verticalposition(Y),horizontalvelocity(Vx)andverticalvelocity(Vy))andtheaverageacrosskinematicparameters(Avgr).Topgroup:

correlationcoefficientsaregiven,foreachparameteranddataset,fortheworstandthebestofthefivecross-validationfolds.Bottomgroup:medianvaluesofcorrelationbetweentheactualanddecodedkinematicparameters,calculatedacrossallfivecross-validationfolds.Theseresultsdemonstratethatgoodreconstructionofkinematicparameters(ondatathatwerenotusedtotrainthealgorithm)ispossibleusingECoGsignalsinhumans.SubjectABCDEABCDE

X

Y

Vx

0.18–0.480.03–0.730.04–0.350.30–0.720.09–0.610.420.590.100.580.32

Vy

0.39–0.660.18–0.520.45–0.850.58–0.680.11–0.660.590.320.670.660.49

0.490.470.500.620.35Avgr

0.49–0.610.20–0.490.19–0.60−0.13–0.720.50–0.810.18–0.800.40–0.640.28–0.780.14–0.520.04–0.480.580.420.710.570.37

0.380.550.510.680.22

movementparametersusingintracorticalimplantsinnon-humanprimates.Thistableshowsthatthecorrelationoftheactualwiththedecodedtrajectories,andthusthefidelityofthedecoding,reportedinthepresentstudy,iswithintherangeofthoseachievedbeforeonlyusingimplantedmicroelectrodes.3.2.RelativeimportanceofanatomicalareasandeCoGfeatures

WealsostudiedtherelativeimportanceofthedifferentanatomicalareasandECoGfeatures(i.e.,thesevenfrequency-basedfeaturesandtheLMP)thatwereimplicatedinthedecodingofcursorpositionandvelocity.Todothis,weanalyzedthedataasdescribedbeforeexceptthatwefirstnormalizedthefeatureswithrespecttotheirstandarddeviations2.Thisallowedtheweightsthatwerederivedbythelinearregressionandassociatedwithparticularfeaturesandlocationstobeusedasameasureofimportanceindecodingacertainkinematicparameter.WethenusedtheCFSfeatureselectionprocedureandlinearregressiontoproduceweightsforspecificfeaturesatparticularlocationsforthebestfourcross-validationfoldsineachsubject,andforeachofthefourkinematicparameters.Wefinallyremovedthebiasduetothenumberofelectrodesbynormalizingeachsetofweightsbythesumofallweights.

Todeterminetherelativeimportanceofdifferentanatomicalareas,wethensimplyaccumulatedtheweightsforeachlocation(sothatoneelectrodecouldbeassignedmultipleweightsfromdifferentfoldsand/orfeatures)andplottedtheresultsona3Dmodelofthecortex.(SubjectsAandEhadelectrodegridsonthelefthemisphere.Weprojectedtheelectrodelocationsfromthesesubjectstothe

Wedidnotutilizethisnormalizationbeforebecausewewereinterestedinderivingresultsthatcouldhavebeenachievedinrealtime.Weherecalculatedthestandarddeviationonthewholedataset,whichcannotbeperformedinrealtime.

2

3.Results

Tostudythefidelityofthetrajectorydecodingandthecharacteristicsoftheassociatedbrainsignals,wedeterminedtheaccuracyofdecodedcursorpositionandvelocity,comparedittopublishedresultsusingimplantedmicroelectrodes,andestablishedtheanatomicallocationandECoGfeaturesthatheldthemostinformation.WealsodeterminedtheanatomicallocationandECoGfeaturesthatwerecosinetunedtomovementdirection.Theresultsoftheseevaluationsaredescribedbelow.

3.1.Accuratedecodingofkinematicparameters

Table2showstheprincipalresultsofthisstudy,whicharegivenincorrelationcoefficientscalculatedbetweenactualanddecodedkinematicparameters.ThegenerallyhighcorrelationcoefficientsdemonstratethatitispossibletoinferaccurateinformationaboutjoystickkinematicparametersinrealtimeusingECoGsignalsinhumans(seefigure3andthemovieinthesupplementarymaterialatstacks.iop.org/JNE/4/264foractualanddecodedtrajectories).Becausethecalculationofdecodingparametersonlyinvolvesthetrainingdataset,butnotthetestdataset,similarresultscanbeexpectedinonlineexperiments.

Table3comparestheresultsofthepresentstudytothosepreviouslyreportedfordecodingoftwo-dimensional

WeassessedthenormalityofthesemeasurementsusingaKolmogorov–Smirnofftest.Thistestdeterminedthat90%ofalldistributionswereconsideredGaussianatthe0.05level.

1

268

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

X CURSOR POSITIONY CURSOR POSITION20

40

61

81

0

(A)

r = 0.61r = 0.49

0

X CURSOR POSITION(B)

041

TIME(s)

83124166

Y CURSOR POSITIONr = 0.60

TIME(s)

20

r = 0.72

TIME(s)

406181

041

TIME(s)

83124166

X CURSOR POSITION(C)

Y CURSOR POSITION0

r = 0.81r = 0.80

06

X CURSOR POSITION012

X CURSOR POSITION030

TIME(s)

6090120

Y CURSOR POSITION(E)

r = 0.52

TIME(s)

253750

Y CURSOR POSITION(D)

r = 0.64

TIME(s)

1319266

TIME(s)

131926

r = 0.78

012

r = 0.48

TIME(s)

253750

030

TIME(s)

6090120

Figure3.Actualanddecodedmovementtrajectories.Thisfigureshowsexamplesforactual(thinredtraces)anddecoded(thickgreentraces)XandYcursorpositionforallsubjects(forthebestcross-validationfoldforeachofXandYcursorposition),aswellasthe

respectivecorrelationcoefficientsr.Thehighcorrelationcoefficientsevidencethegenerallycloseconcurrencebetweenactualanddecodedcursorpositions.

Table3.Comparisontointracorticalstudies.Wecomparedtheresultsofthepresentstudytopublishedresultsusingtwo-dimensionaltasksandmicroelectroderecordinginmonkeys.(Weincludedonlythosereportsthatdescribedmethodsthatcouldhavebeenachievedinrealtime.)Averagecorrelationcoefficientsforpublishedpositionand/orvelocityvalues(PositionrandVelocityr,respectively)acrossallsubjectsareshown.Thecorrelationoftheactualwiththedecodedtrajectories,andthusthefidelityofthedecoding,reportedinthepresentstudy,iswithintherangeofthoseachievedusingimplantedmicroelectrodesinmonkeys.

Studyandsource

SchwartzandMoran(1999,p2713)

Carmenaetal(2003,figures1(F)and3(C))Paninskietal(2004,table1)Lebedevetal(2005,table2)

Averbecketal(2005,est.fromfigures8(A)and(B))Presentstudy(2006,table2)

Positionr–

0.33–0.630.47––0.50

Velocityr0.77

0.27–0.73–0.560.740.48

righthemispheretofacilitateinterpretation.)Theseresultsareshowninfigure4,whichshowsthetopographicaldistributionofweights(colorcodedwithredcorrespondingtothehighestweight)accumulatedforallfeaturesandsubjects.Table4reportstheseweightsbrokendownbyBrodmann’sareaandECoGfeature.

Theseweightsaregenerallyhighformotorandpre-motorcorticalareas(Brodmann’sarea4and6,respectively),butalsoforadditionalareassuchasdorsolateralprefrontalcortex(whichhasbeenimplicatedinotherguidedmotortasks[42])andthosethatdonothaveobviousmotorcontrolrelevance(suchastheactivationatthetipofthetemporalpole).Thehighweightsreportedintable4fortheLMPalsoindicatetheimportantcontributionoftheLMPtoourresults.

Wedrawfourconclusionsfromtheseanalyses.First,thecortexoffersopportunitiestoinferkinematicparametersoverwidespreadareasofcortex,notonlyoverclassicalsensorimotorareas.Thisnotionisconsistentwitharecentreviewonthistopic[43].Second,sensorycortexhadonlyamodestinfluence,whichsuggeststhatmovementdecoding

269

GSchalketal

Figure4.Anatomicalareasholdinginformationaboutmovementparameters.Colorsrepresenttheweightsaccumulatedforallfeaturesandsubjectsattherespectivelocations,andthusindicatetherelativeimportanceofthesesitesindecodingcursorpositionorvelocity(transparentcolorandredcorrespondtozeroandmaximumweight,respectively).Theweightsarenormalizedforeachmovement

parameter.Thedominantfocusoverhandandproximalarmareasofmotorcortex,indicatedbythehighestweightsgiventotheseareas,isevident.Inaddition,otherlocationsareinvolvedforwhichtheanatomicalrelevanceisnotclear.Thetotalareacoveredbytheelectrodegridsinthefivesubjectsisindicatedbytheblueoutline.

Table4.RelativeimportanceofanatomicalareasandECoGfeatures.ThefourtablescontainweightsassignedtosignalsinparticularBrodmann’sareasandECoGfeaturesforcursorpositionandvelocity.Thetwomostimportantareasandfeaturesaregiveninbold.Motorandpre-motorcortices,aswellastheLMPfeature,heldthehighestweights,andthusthemostinformationaboutkinematicparameters.

Horizontalcursorposition

Area123467891020212237383940424344454647SUM

8–120.02

0.14

0.200.10

0.06

0.110.010.200.260.060.04

0.08

0.08

0.23

0.03

0.210.070.18

18–24

35–420.10

0.06

0.380.180.17

42–70

70–100

100–1400.060.630.510.46

140–1900.990.601.440.10

LMP0.400.190.991.050.671.710.110.320.420.740.660.29

0.070.04

0.080.05

0.18

1.23

0.45

1.55

0.94

0.211.01

2.06

3.50

0.06

0.020.130.02

0.390.520.799.25

SUM0.552.280.943.202.490.772.180.220.010.621.440.060.891.190.380.220.670.080.661.18

Area123467891020212237383940424344454647SUM

8–120.050.380.250.03

18–240.19

35–420.210.040.180.070.06

Verticalcursorposition42–700.030.080.510.12

0.05

0.40

0.050.12

70–1000.030.160.01

100–1400.040.53

140–190

LMP0.030.532.361.532.510.311.02

SUM0.061.124.161.852.970.360.471.650.180.350.690.061.290.011.030.100.340.190.111.401.63

0.230.03

0.33

0.130.290.06

0.050.180.08

0.04

0.57

0.19

0.06

0.14

0.060.180.050.10

0.01

0.050.180.13

0.050.090.18

0.10

0.170.080.090.311.020.010.570.16

0.11

0.190.04

0.190.05

0.15

0.130.04

0.130.11

0.05

0.07

0.031.301.34

0.48

1.27

0.47

13.12

0.040.111.55

0.080.75

0.070.101.22

1.14

Horizontalcursorvelocity

Area12346789101121223337383940424344454647SUM

8–120.200.15

0.3318–240.05

35–420.250.240.140.280.08

42–700.050.020.080.210.03

70–1000.130.07

100–140

140–190

LMP0.171.101.072.382.190.400.83

SUM0.231.761.892.772.990.310.481.250.140.610.59

Area12346789101121223337383940424344454647SUM

8–120.05

0.04

0.060.210.040.040.070.08

0.07

0.13

0.060.110.05

0.120.040.080.79

0.200.070.040.690.05

0.030.10

18–24

35–420.120.080.30

Verticalcursorvelocity42–700.080.120.42

70–1000.18

100–1400.540.060.070.25

140–1900.061.421.171.120.19

LMP0.310.080.011.331.910.561.060.130.780.470.510.11

0.06

0.07

0.04

0.04

0.670.960.190.380.500.120.670.2811.01

SUM0.542.381.702.593.100.030.601.250.190.780.620.750.110.060.931.380.230.550.500.350.850.53

0.340.03

0.340.09

0.07

0.070.14

0.25

0.12

0.310.040.06

0.11

0.070.090.190.38

0.060.05

0.090.23

0.04

0.11

0.100.20

0.09

0.241.130.81

0.07

0.01

0.08

0.08

0.05

0.88

0.47

0.041.22

0.88

0.54

0.80

0.170.091.20

1.881.2214.020.29

1.560.101.320.070.340.080.252.061.22

0.090.30

0.04

0.131.27

0.81

0.090.39

1.00

4.06

270

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

isprimarilyrelatedtotheexecutionofmovementandnottosensoryfeedback.Third,eyemovementslikelydidnotplayasubstantialroleinmovementdecodinginourparadigm.Fourth,theevidentanatomicalrelevancesuggeststhatoursignalsarenotanartifactbutratherreflectphysiologicaleventsrelatedtomovementcontrol.3.3.Directionaltuning

TheprevioussectiondescribedtheimportanceofparticularbrainareasandECoGfeaturesforthedecodingofcursorpositionandvelocityusingalinearmodel.Theseanalysesfocusedondecodingstrategiesthatcouldbeutilizedinrealtime,whichisimportantforpotentialbrain–computerinterfacingapplications.Studiesintheprimateliteraturehavealsoinvestigatedtherelationshipofsignalfeatureswiththemovementdirection(i.e.,theangleofthemovement).Thesestudies(e.g.,[24])haveshownthatsignalfeatures(i.e.,firingratesofparticularneurons)thatarederivedfromelectrodesimplantedwithinthebraincanbetuned,i.e.,areafunctionofmovementdirection,orevencosinetuned,i.e.,areacosinefunctionofmovementdirection.

TostudythispossibilityforourECoGfeatures,weinvestigatedtheeffectofmovementdirectiononfeatureamplitudeusinganapproachsimilartothatemployedin[21].Inshort,wecalculatedtheamplitudesoftheeightfeaturesasafunctionofmovementdirection(i.e.,theangleofthemovementmeasuredin−180to+180◦).Thisproducedonetuningcurveforeachsubject,cross-validationfold,electrodelocationandfeature.Asdescribedinsection2,wethenderivedtheprobabilitiesthattheresultingtuningcurvesweretuned(pt)andcosinetuned(pct).Weselectedthosetuningcurvesfromthebestfourcross-validationfoldsthatweretunedatpt<0.001andcosinetunedatpct<0.001andderivedfromeachofthemanindexofcosinetuning(ict=−logpreviousanalysis,inwhichweusedtheCFS10(pct)).Asinthefeatureselectorandlinearregression,thepresentanalysisderivedmeasures(i.e.,cosinetuningindicesict)forparticularfeaturesatparticularlocations,cross-validationfoldsandsubject.Wethenaccumulatedthesecosinetuningmeasuresacrosscross-validationfoldsandsubjectsandagainprojectedtheelectrodelocationsforsubjectsAandEontotherighthemisphere.

Theresultsoftheseanalysesareshowninfigure5.Thisfigureshowsthecosinetuningindices,accumulatedforallsubjectsandfeatures(figure5(A))andforallsubjectsandeachfeature(figure5(B)).Theseaccumulatedcosinetuningindicesarecolorcoded(seecolorbars).Itisalsoshowninfigure5(C)threeexampleLMPtuningcurvesforthreesubjectsandforlocationsmarkedintheLMPinfigure5(B).ThehighvaluesofthecosinetuningindicesoverprimarilydifferentmotorcorticalareasandfortheLMP(thescaleoftheLMPinfigure5(B)isfrom0to20)againvalidatethedominantroleoftheselocationsandthisECoGfeature,respectively.Furthermore,thetopographiesofthetuningindicesshowsubstantialcorrelationforthedifferentfrequencyfeaturesbutaremarkedlydifferentfortheLMP.ThetuningindextopographyfortheLMPismorediffuse,butagainpeaksoverhandareaofmotorcortex.

4.Discussion

Inthisstudy,weshowedthatECoGsignalscanbeusedtoaccuratelydecodetwo-dimensionaljoysticktrajectoriesinhumans.Wealsocharacterizedanewbrainsignal,thelocalmotorpotential(LMP),thatholdssubstantialinformationaboutkinematicparameters.Furthermore,wedemonstratedthatECoGfeaturescanexhibitthesamekindofcosinetuningthathasbeenpreviouslydescribedforneuronalfiringratesandlocalfieldpotentials(LFPs)recordedusingintracorticalmicroelectrodes[21,44].TheseresultsindicatethatECoGprovidesinformationthatgreatlyexceedsinspecificitythatprovidedbyEEGandisinimportantrespectscomparabletothatprovidedbymicroelectrodesimplantedincortex.ThisstudyfurtherimpliesthatECoGhascharacteristicsthatmakeitattractivenotonlyforBCIresearch,butalsoforbasicneuroscienceinvestigationsofbrainfunction.4.1.Thelocalmotorpotential

ThesuccessfuldecodingofjoystickmovementachievedinthisstudydependedinlargepartontheLMPcomponent.Becausenopreviousreporthas,toourknowledge,describedtherelationshipofthisnewbrainsignalfeaturetokinematicparameters,wewereconcernedthatitmightbeartifactual.Ourresultsstronglyindicatethatthisisnotthecase.First,theinitialstepinanalysiswastheapplicationofacommonaveragereferencefilter.Thisfilter,whichimprovedperformancecomparedtowhenitwasnotapplied,removessignalswithlowspatialfrequenciessuchasthosethatwouldbeexpectedforanartifactcreatedbywiremovementorsomeotherexternalinfluence.Furthermore,analysesshowedthattheLMPwasmostoftenlocatedoveranatomicallyrelevantareas,andthatitcouldexhibitcosinetuningsimilartothatdescribedforsignalsrecordedbyintracorticalmicroelectrodes.

ItissurprisingthattheLMPhasapparentlynotbeenpreviouslydescribedinintracorticalorscalprecordings.ItispossiblethattheLMPcannotbedetectedonthescalp.Moreover,inourparadigmkinematicparameterschangedrelativelyslowly(i.e.,onefullcirclein6.3s=0.16Hz),whereasintracorticalstudiesinmonkeysoftenutilizedhigherspeeds(i.e.,around1–2Hz).Inthelattercases,associatedLMPcomponentscouldhavebeenmaskedbyotheractivity.Alternatively,theLMPcouldbeacontinuouscorrelatetoevokedLFPchangesthathavebeendescribedforothertasks,suchascenter-out[45]andreachingtasks[46].Finally,itispossiblethatinpreviousstudiestheLMPcomponentwasfilteredoutattheamplificationorpost-processingstage.Indeed,whenwere-analyzeddatafromourpreviousstudy[18]withoutapplyingahigh-passfilter3,wefoundthatLMPamplitudeinoneparticularlocationwasmodulatedbythedirectionofjoystickmovement,andLMPamplitudeinoneimmediatelyadjacentlocationwasmodulatedbyhandopening/closingandrest.Theselocationscloselymatchedthosethatshowedmodulationofthespectralamplitudeat

3

Inthepreviousstudy,wehadanalyzedsignalsfrom0to200Hz.However,wehadsubtractedthemeanfromeach280mswindowpriortoanalysis,whichactedasahigh-passfilter.

271

GSchalketal

(A)

(B)

(C)

Figure5.Cosinetuning.Thisfigureshowsthespatialdistributionofthecosinetuningindex,accumulatedforallsubjectsandfeatures(A),andforallsubjectsandeachfeature(B).Thecosinetuningindexiscolorcoded(seecolorbars).ThescaleoftheLMPfigurein(B)isfrom0to20.Thetuningcurvesin(C)arecalculatedfortheLMPatthelocationsmarkedintheLMPfigurein(B).

18Hz(i.e.,atraditionalfrequency-basedfeatureinthebetaband).Seethesupplementarymaterialsatstacks.iop.org/JNE/4/264fordetails.

Atthispoint,thephysiologicaloriginoftheLMPcomponentisamatterofspeculation.Itispossiblethatitreflectsfiringratemodulationofneuronslocatedimmediatelyunderneaththeelectrode.Inthiscase,theLMPmayberelatedtothedirectionally-specificratemodulationsobservedinsingle-unitstudiesusingcenter-outortrackingtasks272

[21,47–54].WhatcomplicatesthisinterpretationisthefactthatthetuningtopographiesfortheLMPandfrequency-basedfeaturesareclearlydifferent(seefigure5(B)),whichsuggestsdifferingoriginatingprocesses.Furthermore,itmaybedifficulttotheoreticallymodeltherelationshipbetweensingle-unitactivityandECoGactivityasresultingECoGactivitycouldbedominatedbythedegreeofsynchronousactivityofunderlyingcellsratherthansimplybythemagnitudeofcellactivity.Hence,determinationoftherelationship

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

betweenthesesourcesofbrainsignalactivitywilllikelyrequiresimultaneousrecordingsofsingle-unitandECoGactivity(e.g.,[55]),oratleastofsingle-unitandlocalfieldpotentialactivity(e.g.,[20,56,57]).4.2.Relevanceforbrain–computerinterfaces

Togetherwithresultsfrompreviousstudiesinmonkeys,thepresentresultssuggestthatECoG-basedBCIusecouldbemoreintuitive,i.e.,subjectscouldusemovement-relatedimageryratherthanimageryofarbitrarytasksformultidimensionalBCIcontrol.Thus,BCItrainingtimemightbereducedbyusingECoGandmovement-relatedimagery.Atthesametime,itiscurrentlynotclearwhatfactorsgoverntheneedforBCItrainingtime.Itispossiblethatthephysiologicalnatureofthebrainsignalisimportant.Inatypicalmu-orbeta-rhythmEEG-basedBCI,brainsignalsassociatedwithimaginedlimbmovementsarefirstidentified.Thesesignalsarethenusedtoprovideone-ortwo-dimensionalcontrol.Whiletheoriginsofthesescalp-recordedrhythmsarenotentirelyclear[58,59],theyarenotbelievedtobestronglycorrelatedwithmovementdirection.Thus,theirusefordirectionalmovementcontrolmightrequireconsiderableplasticityandusertraining.Incontrast,BCIsystemsusingimplantedmicroelectrodesmayrequirelessusertraining.Thesesystemstypicallyutilizesingle-unitactionpotentialsorlocalfieldpotentials(LFPs)derivedfromneuronsinmotorcortex.Neuronsarethenidentifiedthathavefiringrates/LPFsrelatedtoparametersofhandmovements[24,52,53,60–69].Theyarethencombinedtoproducemultidimensionalcontrolsignals.Whenmonkeysareprovidedfeedbackbasedonbrainsignalsratherthanactualhandmovements,theyinitiallycontinuetomovethehandbutthenlearntoproducethesamesignalswithouttheactualphysicalmovements[4].Itislikelythatthetransformationofbrainsignalsthattypicallyencodemovementdirectionintodirectionalnon-muscularcommandsdemandslesscorticalreorganizationandusertrainingthanthetransformationofbrainsignalsthatdonotnormallyencodedirection,suchasscalp-recordedmuandbetarhythms.Thus,theresultsofthepresentstudy,whichshowthatkinematicparameterscanbedecodedfromECoGsignals,suggestthatthepotentialtraining-timeadvantageofimplantedmicroelectroderecordingscouldalsobeachievedusingECoG.

ForclinicalapplicationsofBCItechnology,chronicimplantsofECoGelectrodeswouldberequired.Theliteraturesuggeststhatsubdural/epiduralelectrodesexhibitgoodlong-termstability[13–17].Inaddition,thereareseveraltheoreticalreasonswhyECoGelectrodeswillprobablynotbeaffectedbythesubstantialstabilityproblemsassociatedwithimplantedmicroelectrodes.TheareacoveredbyeachECoGelectrodeismuchlarger,andthusitsimpedancemuchlower,thanisthecaseforamicroelectrode.Moreover,sinceECoGelectrodesdonotpenetratecortex,thereactiveresponsesofthebraintypicalwithmicroelectrodesshouldbesubstantiallyreduced.Evenifscartissueweretoformunderneaththeelectrodes,theelectrodes’lowimpedanceshouldenableeffectivelongtermrecordings.Furthermore,

ECoGrecordingsrequireadramaticallylowerbandwidth(i.e.,500Hzsampling,andmuchlessifonlytheLMPisextracted)thansingle-neuronrecordingsusingmicroelectrodes(i.e.,10–50kHz).Theselowertechnicalneedstranslatetosubstantiallydecreasedprocessingandpowerrequirements.Lowerpowerrequirementsmeanlessheatdissipationandlongerbatterylife.Thesesubstantialtechnicaladvantageswillfacilitatethedesignofelectrode/telemittersystemsthatcouldbechronicallyimplantedandwouldnotrequireanypercutaneousconnection.Thiswouldgreatlyreducethelong-termriskofinfection.

Ourresultsdidnotdependonthecirculartrajectoryofthetargetandwerespecifictothejoystickmovements.Trackingperformance(i.e.,howwellthesubjectstrackedthetarget)wasmodest(r=0.56,r=0.57,r=0.50,r=0.50calculated,foreachofthekinematicparameters,betweenthecursorandthetarget,respectively)andnotpredictiveofdecodingperformance(p=0.84,p=0.88,p=0.62,p=0.91forthefourkinematicparameters,respectively).Thisisinconsistentwiththehypothesisthatdecodingperformanceisdependentonhowwellthesubjectstrackedthecirculartrajectoryofthetarget.Insummary,ourresultsdidnotdependonthecirculartrajectoryofthetargetandtherewassomedegreeofindependencebetweenthekinematicparameters.Atthesametime,thecurrentparadigmwassimplynotdesignedtoindependentlyexamineallfourkinematicparameters.Inconsequence,futurestudiescouldexpandonourinitialresultsandutilizedifferentmovementpatterns,directionsandspeeds,todeterminehowtheresultsinhumansusingECoGrelatetothebodyofknowledgethathasbeenestablishedforsignalsrecordedfromintracorticalmicroelectrodes.4.3.Currentexperimentallimitations

ThepresentresultsstronglyencouragefurtherinvestigationsusingECoG.Atthesametime,therewillultimatelybelimitstowhatcanbeachievedusingthecurrentlyusedpatientpopulation.Ourstudy,likepracticallyallhumanECoGstudiestodate,reliedonelectrodegridsimplantedforclinicalreasons.Thus,thegridsoftendonotcoverlocationsmostappropriateforourpurposeand,inaddition,aredifferentforeachpatient.Furthermore,thephysicalandcognitiveconditionandlevelofcooperationofeachpatientareimpairedand/orvariable.Finally,thepatients’posturecanbecontrolledformostlyonlyusinginstructions.Thisrelativelyuncontrolledexperimentalsituationisincontrasttothetypicallyrigorouslycontrolledanimalstudies.

Despitetheseissues,thepresentresults,inwhichweutilizedallavailabledataforeachsubject,comparefavorablytoresultsthathavebeenachievedinhighlycontrolledanimalstudies.Atthesametime,thepresentsituationsimplydoesnotpermitsystematicstudiesusingcontrolledexperiments,whichwillultimatelylimittheinformationthatcanbederived.Weexpectthatthepresentresults,andtheresultsofthestudiesthatwillfollow,willprovideampleevidenceoftheutilityoftheECoGplatformtosupportFDAapprovalofsubduralorepiduralimplantsinhumansforBCIpurposes.

273

GSchalketal

Acknowledgments

WethankDrElizabethWinterWolpawforhelpfulcommentsonthemanuscript.WealsothankDrsHolmesandMillerforpatientcareandtelemetrysupport.WefurtheracknowledgetheassistanceofDrsAshley,DowlingandEisenmaninthedepartmentofNeurologicalSurgery,BarnesJewishHospital,StLouis,inexperimentsleadinguptothepresentstudy.Finally,wearegratefulforthecommentsbytheanonymousreviewers,whichhelpedtogreatlyimprovethispaper.ThisworkwassupportedinpartbygrantsfromNIH(EB006356(GS),NS41272(JO),HD30146(JRW)andEB00856(JRW)),theMcDonnellCenterforHigherBrainFunction(JO)andtheJamesSMcDonnellFoundation(ECLandJRW).

References

[1]WolpawJR,BirbaumerN,McFarlandDJ,PfurtschellerG

andVaughanTM2002Brain–computerinterfacesforcommunicationandcontrolElectroenceph.Clin.Neurophysiol.113767–91

[2]WolpawJRandMcFarlandDJ2004Controlofa

two-dimensionalmovementsignalbyanoninvasive

brain–computerinterfaceinhumansProc.NatlAcad.Sci.USA10117849–54

[3]GeorgopoulosAP,SchwartzABandKettnerRE1986

NeuronalpopulationcodingofmovementdirectionScience2331416–9

[4]TaylorDM,TillerySIandSchwartzAB2002Directcortical

controlof3DneuroprostheticdevicesScience2961829–32[5]SerruyaMD,HatsopoulosNG,PaninskiL,FellowsMRand

DonoghueJP2002InstantneuralcontrolofamovementsignalNature416141–2

[6]LebedevMA,CarmenaJM,O’DohertyJE,

ZacksenhouseM,HenriquezCS,PrincipeJCandNicolelisMA2005Corticalensembleadaptationtorepresentvelocityofanartificialactuatorcontrolledbyabrain–machineinterfaceJ.Neurosci.254681–93[7]ShainW,SpataroL,DilgenJ,HaverstickK,RettererS,

IsaacsonM,SatzmanandTurnerJN2003ControllingcellularreactiveresponsesaroundneuralprostheticdevicesusingperipheralandlocalinterventionstrategiesIEEETrans.NeuralSyst.Rehabil.Eng.11186–8

[8]DonoghueJP,NurmikkoA,FriehsGandBlackM2004

DevelopmentofneuromotorprosthesesforhumansSuppl.Clin.Neurophysiol.57592–606[9]K¨ublerA,NijboerF,MellingerJ,VaughanTM,PawelzikH,

SchalkG,McFarlandDJ,BirbaumerNandWolpawJR2005Patientswithalscanusesensorimotorrhythmstooperateabrain–computerinterfaceNeurology641775–7

[10]HochbergLR,SerruyaMD,FriehsGM,MukandJA,

SalehM,CaplanAH,BrannerA,ChenD,PennRDandDonoghueJP2006NeuronalensemblecontrolofprostheticdevicesbyahumanwithtetraplegiaNature442164–71

[11]StabaRJ,WilsonCL,BraginA,FriedIandEngelJ2002

Quantitativeanalysisofhigh-frequencyoscillations

(80–500Hz)recordedinhumanepileptichippocampusandentorhinalcortexJ.Neurophysiol.881743–52

[12]FreemanWJ,HolmesMD,BurkeBCandVanhataloS2003

SpatialspectraofscalpEEGandEMGfromawakehumansClin.Neurophysiol.1141053–68

[13]LoebGE,WalkerAE,UematsuSandKonigsmarkBW1977

HistologicalreactiontovariousconductiveanddielectricfilmschronicallyimplantedinthesubduralspaceJ.Biomed.Mater.Res.11195–210

274

[14]BullaraLA,AgnewWF,YuenTG,JacquesSand

PudenzRH1979EvaluationofelectrodearraymaterialforneuralprosthesesNeurosurgery5681–6

[15]YuenTG,AgnewWFandBullaraLA1987Tissueresponse

topotentialneuroprostheticmaterialsimplantedsubdurallyBiomaterials8138–41

[16]PilcherWHandRusyniakWG1993Complicationsof

epilepsysurgeryNeurosurg.Clin.N.Am.4311–25

[17]MargalitEetal2003Visualandelectricalevokedresponse

recordedfromsubduralelectrodesimplantedabovethevisualcortexinnormaldogsundertwomethodsofanesthesiaJ.Neurosci.Methods123129–37

[18]LeuthardtEC,SchalkG,WolpawJR,OjemannJGand

MoranDW2004Abrain–computerinterfaceusingelectrocorticographicsignalsinhumansJ.Neural.Eng.163–71

[19]MehringC,RickertJ,VaadiaE,CardosadeOliveiraS,

AertsenAandRotterS2003InferenceofhandmovementsfromlocalfieldpotentialsinmonkeymotorcortexNat.Neurosci.61253–4

[20]AndersenRA,MusallamSandPesaranB2004Selectingthe

signalsforabrain–machineinterfaceCurr.Opin.Neurobiol.14720–6

[21]RickertJ,OliveiraSC,VaadiaE,AertsenA,RotterSand

MehringC2005EncodingofmovementdirectionindifferentfrequencyrangesofmotorcorticallocalfieldpotentialsJ.Neurosci.258815–24

[22]ToroC,CoxC,FriehsG,OjakangasC,MaxwellR,GatesJR,

GumnitRJandEbnerTJ19948–12Hzrhythmic

oscillationsinhumanmotorcortexduringtwo-dimensionalarmmovements:evidenceforrepresentationofkinematicparametersElectroencephalogr.Clin.Neurophysiol.93390–403

[23]GeorgopoulosAP,LangheimFJ,LeutholdACandMerkle

AN2005MagnetoencephalographicsignalspredictmovementtrajectoryinspaceExp.BrainRes.167132–5

[24]GeorgopoulosAPandMasseyJT1988Cognitive

spatial-motorprocesses:2.Informationtransmittedbythedirectionoftwo-dimensionalarmmovementsandbyneuronalpopulationsinprimatemotorcortexandarea5Exp.BrainRes.69315–26

[25]SalinasEandAbbottLF1994Vectorreconstructionfrom

firingratesJ.Comput.Neurosci.189–107

[26]TurnerRS,OwensJWJrandAndersonME1995Directional

variationofspatialandtemporalcharacteristicsoflimbmovementsmadebymonkeysinatwo-dimensionalworkspaceJ.Neurophysiol.74684–97

[27]KettnerRE,MarcarioJKandClark-PhelpsMC1996

Controlofrememberedreachingsequencesinmonkey:I.ActivityduringmovementinmotorandpremotorcortexExp.BrainRes.112335–46

[28]AmirikianBandGeorgopoulosAP2000Directionaltuning

profilesofmotorcorticalcellsNeurosci.Res.3673–9

[29]BaraducPandGuigonE2002Populationcomputationof

vectorialtransformationsNeural.Comput.14845–71

[30]TodorovE2002CosinetuningminimizesmotorerrorsNeural.

Comput.141233–60

[31]ShohamS,PaninskiLM,FellowsMR,HatsopoulosNG,

NormannRAandDonoghueJP2005Statisticalencodingmodelforaprimarymotorcorticalbrain–machineinterfaceIEEETrans.Biomed.Eng.521312–22

[32]NozakiD,NakazawaKandAkaiM2005Muscleactivity

determinedbycosinetuningwithanontrivialpreferreddirectionduringisometricforceexertionbylowerlimbJ.Neurophysiol.932614–24

[33]KalaskaJFandHydeML1985Area4andarea5:

differencesbetweentheloaddirection-dependentdischargevariabilityofcellsduringactiveposturalfixationExp.BrainRes.59197–202

Decodingtwo-dimensionalmovementtrajectoriesusingelectrocorticographicsignalsinhumans

[34]TairaM,BolineJ,SmyrnisN,GeorgopoulosAPandAsheJ

1996Ontherelationsbetweensinglecellactivityinthemotorcortexandthedirectionandmagnitudeof

three-dimensionalstaticisometricforceExp.BrainRes.109367–76

[35]SergioLE,Hamel-PˆaquetCandKalaskaJF2005Motor

cortexneuralcorrelatesofoutputkinematicsandkineticsduringisometric-forceandarm-reachingtasksJ.Neurophysiol.942353–78

[36]SchalkG,McFarlandDJ,HinterbergerT,BirbaumerNand

WolpawJR2004BCI2000:ageneral-purpose

brain–computerinterface(BCI)systemIEEETrans.Biomed.Eng.511034–43

[37]MillerKJ,MakeigS,HebbAO,RaoRP,DennijsMand

OjemannJG2007Corticalelectrodelocalizationfromx-raysandsimplemappingforelectrocorticographicresearch:the‘locationoncortex’(LOC)packageforMATLABJ.Neurosci.Methods162303–8

[38]FoxPT,PerlmutterJSandRaichleME1985A

stereotacticmethodofanatomicallocalizationforpositronemissiontomographyJ.Comput.Assist.Tomogr.9141–53

[39]TalairachJandTournouxP1988Co-PlanarSterotaxic

AtlasoftheHumanBrain(NewYork:ThiemeMedicalPublishers)

[40]LawrenceMarpleS1987DigitalSpectralAnalysis:With

Applications(EnglewoodCliffs,NJ:Prentice-Hall)

[41]WittenIHandFrankEibe2005DataMining:Practical

MachineLearningToolsandTechniques2ndedn(SanFrancisco,CA:MorganKaufmann)

[42]VaillancourtDE,ThulbornKRandCorcosDM2003Neural

basisfortheprocessesthatunderlievisuallyguidedandinternallyguidedforcecontrolinhumansJ.Neurophysiol.903330–40

[43]DavidsonPRandWolpertDM2005Widespreadaccessto

predictivemodelsinthemotorsystem:ashortreviewJ.Neural.Eng.2S313–9

[44]HeldmanDA,WangW,ChanSSandMoranDW2006

LocalfieldpotentialspectraltuninginmotorcortexduringreachingIEEETrans.Neural.Syst.Rehabil.Eng.14180–3

[45]DonchinO,GribovaA,SteinbergO,BergmanH,

CardosodeOliveiraSandVaadiaE2001LocalfieldpotentialsrelatedtobimanualmovementsintheprimaryandsupplementarymotorcorticesExp.BrainRes.14046–55

[46]O’LearyJGandHatsopoulosNG2006Earlyvisuomotor

representationsrevealedfromevokedlocalfieldpotentialsinmotorandpremotorcorticalareasJ.Neurophysiol.961492–506

[47]DonoghueJP,SanesJN,HatsopoulosNGandGaalG1998

NeuraldischargeandlocalfieldpotentialoscillationsinprimatemotorcortexduringvoluntarymovementsJ.Neurophysiol.79159–73

[48]HatsopoulosNG,OjakangasCL,PaninskiLand

DonoghueJP1998InformationaboutmovementdirectionobtainedfromsynchronousactivityofmotorcorticalneuronsProc.NatlAcad.Sci.USA9515706–11

[49]MaynardEM,HatsopoulosNG,OjakangasCL,AcunaBD,

SanesJN,NormannRAandDonoghueJP1999NeuronalinteractionsimprovecorticalpopulationcodingofmovementdirectionJ.Neurosci.198083–93

[50]DonchinO,GribovaA,SteinbergO,MitzAR,BergmanH

andVaadiaE2002Single-unitactivityrelatedtobimanual

armmovementsintheprimaryandsupplementarymotorcorticesJ.Neurophysiol.883498–517

[51]

SteinbergO,DonchinO,GribovaA,CardosadeOliveiraS,BergmanHandVaadiaE2002NeuronalpopulationsinprimarymotorcortexencodebimanualarmmovementsEur.J.Neurosci.151371–80

[52]RokniU,SteinbergO,VaadiaEandSompolinskyH2003CorticalrepresentationofbimanualmovementsJ.Neurosci.2311577–86

[53]PaninskiL,FellowsMR,HatsopoulosNGandDonoghueJP2004SpatiotemporaltuningofmotorcorticalneuronsforhandpositionandvelocityJ.Neurophysiol.91515–32[54]HatsopoulosN,JoshiJandO’LearyJG2004DecodingcontinuousanddiscretemotorbehaviorsusingmotorandpremotorcorticalensemblesJ.Neurophysiol.921165–74[55]

JonesMS,MacDonaldKD,ChoiB,DudekFEand

BarthDS2000Intracellularcorrelatesoffast(200Hz)electricaloscillationsinratsomatosensorycortexJ.Neurophysiol.841505–18

[56]

PesaranB,PezarisJS,SahaniM,MitraPPandAndersenRA2002Temporalstructureinneuronalactivityduring

workingmemoryinmacaqueparietalcortexNat.Neurosci.5805–11

[57]JacobsJoshua,KahanaMJ,EkstromADandFriedI2007Brainoscillationscontroltimingofsingle-neuronactivityinhumansJ.Neurosci.273839–44

[58]LopesdaSilvaFH1991Neuralmechanismsunderlyingbrainwaves:fromneuralmembranestonetworksElectroenceph.Clin.Neurophysiol.7981–93

[59]NiedermeyerE2004TheNormalEEGoftheWakingAdult5thedn(Baltimore,MD:WilliamsandWilkins)pp167–92[60]SchwartzABandMoranDW1999Motorcorticalactivityduringdrawingmovements:populationrepresentationduringlemniscatetracingJ.Neurophysiol.822705–18[61]MoranDWandSchwartzAB1999Motorcorticalactivityduringdrawingmovements:populationrepresentationduringspiraltracingJ.Neurophysiol.822693–704[62]MoranDWandSchwartzAB1999MotorcorticalrepresentationofspeedanddirectionduringreachingJ.Neurophysiol.822676–92

[63]MoranDWandSchwartzAB2000Onemotorcortex,twodifferentviewsNat.Neurosci.3963–5

[64]

WessbergJ,StambaughCR,KralikJD,BeckPD,

LaubachM,ChapinJK,KimJ,BiggsSJ,SrinivasanMAandNicolelisMA2000Real-timepredictionofhandtrajectorybyensemblesofcorticalneuronsinprimatesNature408361–5

[65]SchwartzAB,TaylorDMandTillerySI2001ExtractionalgorithmsforcorticalcontrolofarmprostheticsCurr.Opin.Neurobiol.11701–7

[66]

ReinaGA,MoranDWandSchwartzAB2001OntherelationshipbetweenjointangularvelocityandmotorcorticaldischargeduringreachingJ.Neurophysiol.852576–89

[67]

MerchantH,Battaglia-MayerAandGeorgopoulosAP2004Neuralresponsesduringinterceptionofrealandapparentcircularlymovingstimuliinmotorcortexandarea7aCereb.Cortex14314–31

[68]SchwartzAB,MoranDWandReinaGA2004DifferentialrepresentationofperceptionandactioninthefrontalcortexScience303380–3

[69]

AverbeckBB,ChafeeMV,CroweDAand

GeorgopoulosAP2005ParietalrepresentationofhandvelocityinacopytaskJ.Neurophysiol.93508–18

275