TECHNOLOGICAL REVOLUTIONS: ETHICSAND

POLICY IN THE DARK

1

(2006)NickBostrom
www.nickbostrom.com

[PublishedinNanoscale:IssuesandPerspectivesfortheNanoCentury,eds.NigelM.deS.
CameronandM.EllenMitchell(JohnWiley,2007):pp.129152.]

Abstract
Technologicalrevolutionsareamongthemostimportantthingsthathappentohumanity.
Ethicalassessmentintheincipientstagesofapotentialtechnologicalrevolutionfacesseveral
difficulties,includingtheunpredictabilityoftheirlongtermimpacts,theproblematicroleof
humanagencyinbringingthemabout,andthefactthattechnologicalrevolutionsrewrite
notonlythematerialconditionsofourexistencebutalsoreshapecultureandevenperhaps
humannature.Thisessayexploressomeofthesedifficultiesandthechallengestheypose
forarationalassessmentoftheethicalandpolicyissuesassociatedwithanticipated
technologicalrevolutions.

1.Introduction
Wemightdefineatechnologicalrevolutionasadramaticchangebroughtaboutrelatively
quicklybytheintroductionofsomenewtechnology.Asthisdefinitionisrathervague,it
maybeusefultocomplementitwithafewcandidateparadigmcases.

Someeleventhousandyearsago,intheneighborhoodofMesopotamia,someofour
ancestorstookupagriculture,beginningtheendofthehuntergathererera.
Improvedfoodproductionledtopopulationgrowth,causingaveragenutritional
statusandqualityoflifetodeclinebelowthehuntergathererlevel.Eventually,
greaterpopulationdensitiesledtovastlyacceleratedculturalandtechnological
development.Standingarmiesbecameapossibility,allowingtheancientSumerians
toembarkonunprecedentedterritorialexpansion.

1
IamgratefultoEricDrexler,GuyKahane,MatthewLiao,andRebeccaRoacheforhelpful
suggestions.
1

In1448,JohanGutenberginventedthemovabletypeprintingprocessinEurope,
enablingcopiesoftheBibletobemassproduced.Gutenbergsinventionbecamea
majorfactorfuelingtheRenaissance,theReformation,thescientificrevolution,and
helpedgiverisetomassliteracy.Afewhundredyearslater,MeinKampfwasmass
producedusinganimprovedversionofthesametechnology.

Brillianttheoreticalworkinatomicphysicsandquantummechanicsinthefirstthree
decadesofthe20
th
centurylaidthefoundationfortheManhattanprojectduring
WorldWarII,whichracedHitlertotheatomicbomb.Somebelievethatthe
subsequentbuildupofenormousnucleararsenalsbyboththeU.S.andtheUSSR
createdabalanceofterrorthatpreventedathirdworldwarfrombeingfoughtwith
conventionalweapons,therebysavingmanytensofmillionsoflives.Othersbelieve
thatitwasonlybyluckthatanuclearArmageddonwasavoided,whichwouldhave
claimedthelivesofmanyhundredsofmillions,perhapsbillions.Thesebeliefsmay
bothbetrue.

In1957,SovietscientistslaunchedSputnik1.Inthefollowingyear,theU.S.created
theDefenseAdvancedResearchProjectsAgencytoensurethattheU.S.wouldstay
aheadofitsrivalsinmilitarytechnology.DARPAbegandevelopinga
communicationsystemthatcouldsurvivenuclearbombardmentbytheUSSR.The
result,ARPANET,laterbecametheInternet,whichmadeavailabletheWorldWide
Web,email,andotherservices.Thelongtermconsequencesremaintobeseen.

Itwouldappearthattechnologicalrevolutionsareamongthemostconsequential
thingsthathappentohumanity,perhapsexceededintheirimpactonlybymoregradual,
nonrevolutionarytechnologicaldevelopments.Technologicalchangeisinlargepart
responsiblefortheevolutionofsuchbasicparametersofthehumanconditionasthesizeof
theworldpopulation,lifeexpectancy,educationlevels,materialstandardsofliving,the
natureofwork,communication,healthcare,war,andtheeffectsofhumanactivitiesonthe
naturalenvironment.Otheraspectsofsocietyandourindividuallivesarealsoinfluencedby
technologyinmanydirectandindirectways,includinggovernance,entertainment,human
relationships,andourviewsonmorality,cosmology,andhumannature.Onedoesnothave
toembraceanystrongformoftechnologicaldeterminismorbeahistoricalmaterialistto
acknowledgethattechnologicalcapabilitythroughitscomplexinteractionswith
individuals,institutions,cultures,andtheenvironmentisakeydeterminantoftheground
ruleswithinwhichthegameofhumancivilizationisplayedoutatanygivenpointintime.
Inthecourseofanormallifetimenowadays,wecanallexpecttobeinvolvedinone
ormoretechnologicalrevolutions:ifnotasinventor,funder,investor,regulator,oropinion
2
leader,thenatleastasvotingcitizen,worker,andconsumer.Giventhattechnological
revolutionshavesuchprofoundconsequences,onemightthinkthattheyshouldbethefocus
ofintenseethicaldeliberationandfeaturecentrallyinpublicpolicyanalysis.Ifsomuchisat
stake,itwouldseemtobehooveustodedicateacorrespondingamountofefforttoensuring
thatwemaketherightdecisions.Howishumanitymeasuringuptothischallenge?

2.ELSIresearch,andpublicconcernsaboutscienceandtechnology
OnecanperceiveaslowtrendsinceWorldWarIIofintensifyingendeavorstoconnect
scienceandtechnology(hereafter,S&T)policytoabroaderdiscussionaboutdesiredsocial
outcomes.Theriseofenvironmentalisminthe1960sfosteredthistrend,reflectingpublic
demandthatmoreS&Tresourcesbedevotedtothebettermentofwaterandairquality.The
congressionalOfficeofTechnologyAssessmentwascreatedtoimproveunderstandingof
thesocietalimplicationsoftechnologicalchoices.Diseaselobbieshaveformed,seeking,
amongotherthings,increasedfundingformedicalresearchintoavarietyofconditions.
Concernsaboutglobalwarminghaspushedgreatlyincreasedresourcesintoclimatescience
andledtocallsformorefundingforresearchintoalternativeenergysourcesaswellasmore
directinterventionstoreducegreenhousegasemissions.
Some3%ofthebudgetfortheHumanGenomeProjectwassetasideforstudyingthe
ethical,legal,andsocietalissues(ELSI)connectedtogeneticinformationnotmuchin
relativetermsbutstillenoughforArtCaplantodescribethemoveasthefullemployment
actforbioethics.Thereisnowaburgeoningofresearchinanumberoftechnologyrelated
fieldsofappliedethicssuchascomputerethics,neuroethics,andespeciallynanoethics.
Researchintotheethical,legal,andsocietalissuesrelatedtonanotechnology(NELSI)might
overtimeoutstripthatofthegeneticELSIprogram.
Thereistodayalsoalargeanddiversesetofgrassrootsorganizations,thinktanks,
anduniversitycentersthatworkontechnologyrelatedissues.Inrecentyears,ethicalissues
relatedtohumanenhancementtechnologyhaveparticularlycometothefore.Thereis
growingapprehensionthatanticipatedtechnologicaldevelopmentsincluding
nanotechnologybutalsoartificialintelligence,neurotechnology,andinformation
technologyarelikelyinthetwentyfirstcenturytohavetransformingimpactsonhuman
society,andperhapsonhumannatureitself.SomespeakofanNBICconvergence
(referringtotheintegrationoftheneurobioinformationandcognitivesciences),and
explicitlylinkthistotheprospectofhumanenhancement.
2

Whilethespectrumofopinionrepresentedinthesediscussionsisquitebroad,there
seemtobesomepointsofconsensus,atleastwithintheWesternmainstream:

2
Seee.g.(RocoandBainbridge2003;BainbridgeandRoco2006).
3
Technologicaldevelopmentwillhavemajorimpactsonhumansociety.
Thesedevelopmentswillcreatebothproblemsandopportunities.
Turningbackisneitherfeasiblenordesirable.
Thereisaneedforcarefulpublicexaminationofboththeupsidesanddownsidesof
newtechnologies,andforexplorationofpossiblewaysoflimitingpotentialharms
(includingtechnological,regulatory,intergovernmental,educational,and
communitybasedresponses).

InadditiontodisagreementsaboutthecontentofS&Tpolicy,therearealso
disagreementsabouttheprocesswherebysuchpolicyshouldbedetermined,withchallenges
beingraisedtotheofficialmodeloftheappropriaterelationshipbetweenscienceand
society,amodelwhichharksbacktotheEnlightenment.AccordingtotheEnlightenment
model,theonlyscientificcitizensarethescientiststhemselves.Forsciencetoengageinthe
productionofproperlyscientificknowledgeitmustliveinafreestateandinadomain
apartfromtherestofsociety.Historically,sciencesgriponTruthisseenashavinggrown
progressivelystrongerassocietysgriponsciencehasgrownprogressivelyweakerandever
morecloselycircumscribed.
3
TheEnlightenmentmodelpicturesscienceasthegoosethat
laysthegoldenegg,butonlywhenitisprotectedfromexternalinterference.
Thismodelhascomeunderincreasedscrutinysincethe1960s.InEurope,broad
effortsareunderwaytochangethesocialcontractbetweenscienceandsocietyinorderto
createalargerroleforpublicparticipationanddeliberationinsettingtheprioritiesand
limitationsofscienceandtechnology.Thenotionthatscienceisunproblematically
associatedwithprogressisnolongerwidelyaccepted,amorecriticalapproachhavingbeen
stimulatedbydevelopmentssuchasthenucleararmsraceduringthecoldwar,the
Chernobyldisaster,andtheincreasedsalienceofenvironmentalconcernsinlateryears.In
Britain,themismanagementoftheBSEoutbreak(madcowdisease)erodedpublic
confidenceinGovernmentsciencepolicy.TheexperiencesfromtheBSEcrisislaterhelped
fomentpublicoppositiontotheintroductionofgeneticallymodifiedcrops.
Initiativestobuildmoreopportunitiesforthepublictobecomeengagedinscience
andtechnologyissuescanbeseenasanefforttorebuildpublicconfidenceandtosecure
scienceslicensetooperate.Butbeyondsuchpublicrelationsgoals,therearealsomany
whoarguethattheS&Tenterpriseneedsmuchmoreguidancefromsocietyinorderto
ensurethatscientificandtechnologicalresearchisreallydirectedtoachievesocially
beneficialoutcomes.Theaimisnotnecessarilytorestrictresearch,ortocontestany
particularscientifictheory,buttoyokethescienceandtechnologybehemothtoendschosen
bythepeopleafterduedeliberationanddebate.IfS&Tissuchanimportantshaperofthe
modernworld,itshouldbebroughtunderdemocraticcontrol,thethinkinggoes,andits

3
(ElamandBertilson2002),p.133.
4
workingsshouldbecomemoretransparenttothepeoplewhohavetolivewiththe
consequences.
4

ThisviewisreflectedinarecentpaperbyMichaelCrowandDanielSarewitz:

WhenresourcesareallocatedforR&D[researchanddevelopment]programs,the
implicationsforcomplexsocietaltransformationarenotconsidered.The
fundamentalassumptionunderlyingtheallocationprocessisthatallsocietal
outcomeswillbepositive,andthattechnologicalcausewillleaddirectlytoadesired
societaleffect.TheliteraturepromotingtheNationalNanotechnologyInitiative
expressesthisview.
5

Theycontinue:

Thefactthatsocietaloutcomesarenotaseriouspartoftheframeworkseemsto
derivefromtwobeliefs:(1)thatthescienceandtechnologyenterprisehastobe
grantedautonomytochoseitsowndirectionofadvanceandinnovation;and(2)that
becausewecannotpredictthefutureofscienceortechnologicalinnovation,we
cannotprepareforitinadvance.Theseareoftarticulatedarguments,notstrawmen.
Yetthefirstiscontradictedbyreality,andthesecondisirrelevant.Thedirectionof
scienceandtechnologyisinfactdictatedbyanenormousnumberofconstraints
(onlyoneofwhichisthenatureofnatureitself).Andpreparationforthefuture
obviouslydoesnotrequireaccurateprediction;rather,itrequiresafoundationof
knowledgeuponwhichtobaseaction,acapacitytolearnfromexperience,close
attentiontowhatisgoingoninthepresent,andhealthyandresilientinstitutions
thatcaneffectivelyrespondoradapttochangeinatimelymanner.
6

Letuslookinalittlemoredetailatthesetwoissues,theautonomyoftheS&Tenterpriseand
unpredictability,startingwiththelatter.

4
Partofthisintellectualtrendistheconglomerationofscienceandtechnologyintotechnoscience,
theideabeing(roughly)thatscienceandtechnologyareinextricablylinkedandthatbotharesocially
coded,historicallysituated,andsustainedbyactornetworksconsistingofbothhumanandartifacts;
seee.g.(Latour1987).InthisessayIforthemostpartdonotsharplyseparatescienceandtechnology,
butitseemstomethatamorenuancedtreatmentwouldhavetodistinguishdifferentcomponentsof
thescienceandtechnologyenterprise(tousethetermfavoredbyCrowandSarewitz).
5
(CrowandSarewitz2001),p.97.
6
(CrowandSarewitz2001),p.98.
5
3.Unpredictability
Technologicalrevolutionshavefarreachingconsequencesthataredifficulttopredict.This
posesachallengefortechnologypolicy.Thechallenge,however,isnotuniquetotechnology
policy.Allmajorpolicychangeshavefarreachingconsequencesthataredifficulttopredict.
Theredoesnotexistanexactsciencethatcantelluspreciselywhatwillhappeninthelong
runwhenagovernmentdecidestoabolishslavery,gotowar,orgivewomentherightto
vote.
Formoremodestpolicychanges,suchasareductionofasalestaxorthe
introductionofstricterregulationonleadpaint,expectationsoftheneartermconsequences
aremoretightlyconstrainedbyeconomicandscientificmodelsandbyparallelexperiencein
othercountries.Butsocialsystemsarecomplex,andevensmallinterventionscanhavelarge
unanticipatedlongtermconsequences.Perhapsreducedleadlevelswillleadtoincreased
intelligenceinsomechildren,andsomeofthesemightthengrowuptobecomemore
successfulscientiststhantheywouldotherwisehavebeen.Someofthesescientistsmight
inventthefutureequivalentoftheatomicbomborantibiotics.Perhapsareducedsalestax
willincreaseprofitsinonesectoroftheeconomy,someofwhichmightbeusedascampaign
contributionsthatgetapoliticianelectedwhowillpasslegislationthatmay,inturn,have
widerangingandunpredictableramifications.
Eventhemosttrivialpersonaldecisionscanhavemonumentalconsequencesthat
shapethefateofnations.MaybeoneafternoonathousandyearsagoinsomeSwissvillage,a
youngwomandecidedtogoforastrolltothelake.Thereshemetalad,andlaterthey
marriedandhadchildren.Thusshebecamethegreatgreatgreatgreatgreatgreatgreat
greatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreat
greatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreatgreat
greatgrandmotherofAdolfHitler.Ifshehadgonetotheforestinsteadofthelake,the
HolocaustandperhapsWorldWarIIwouldnothavehappened.
7

Ontheotherhand,theunpredictabilityofthefutureshouldnotbeexaggerated.
CrowandSarewitzappeartoconcedethatwecannotpredictthefutureofscienceor
technologicalinnovation,butindoingsotheyconcedetoomuch.AsEricDrexlernotes:

Thefutureoftechnologyisinsomewayseasytopredict.Computerswillbecome
faster,materialswillbecomestronger,andmedicinewillcuremorediseases.
Nanotechnology,whichworksonthenanometerscaleofmoleculesandatoms,will
bealargepartofthisfuture,enablinggreatimprovementsinallthesetechnologies.
8

7
ThisexampleisborrowedfromJamesLenman(Lenman2000).Seealso(Bostrom2006).
8
(Drexler2003).Sometechnologyimpactsareequallypredictable,e.g.somenewmedicineswillbe
used,willsavelives,someofthosepeoplewhoseliveshavebeensavedwilldrawstatepensions,vote,
etc.
6

Predictabilityisamatterofdegree,andthedegreevariesradicallydependingonwhat
preciselyitisthatwearetryingtopredict.
Evensuchaseeminglyplatitudinousclaimasphysicsisbetteratpredictionthan
socialscienceisoncloserinspectionquiteproblematic.Physicscanpredictsomethingsand
notothers.Wecanusephysicstopredictwithconsiderableaccuracywheretheplanet
Jupiterwillbeintheyear2020.Butphysicsdoesnotenableustopredictwhichparticular
atomwillbeclosesttothecenterofgravityinthesolarsystemin2020.Socialsciencecanalso
predictsomethings:thepredictionthattherewillexistsomeinequalitiesinsocialstatusand
incomeintheU.S.in2020seemsaboutasreliableasthepredictionofwheretheplanet
Jupiterwillbeatthattime.
Ifwewanttomakesenseoftheclaimthatphysicsisbetteratpredictingthansocial
scienceis,wehavetoworkhardertoexplicatewhatitmightmean.Onepossiblewayof
explicatingtheclaimisthatwhenonesaysthatphysicsisbetteratpredictingthansocial
scienceonemightmeanthatexpertsinphysicshaveagreateradvantageovernonexpertsin
predictinginterestingthingsinthedomainofphysicsthanexpertsinsocialsciencehave
overnonexpertsinpredictinginterestingthingsinthedomainofsocialscience.Thisisstill
veryimprecisesinceitreliesonanundefinedconceptofinterestingthings.Yetthe
explicationdoesatleastdrawattentiontooneaspectoftheideaofpredictabilitythatis
relevantinthecontextofpublicpolicy,namelytheextenttowhichresearchandexpertise
canimproveourabilitytopredict.TheusefulnessofELSIfundedactivitiesmightdepend
notontheabsoluteobtainabledegreeofpredictabilityoftechnologicalinnovationandsocial
outcomesbutonhowmuchimprovementinpredictiveabilitytheseactivitieswillproduce.
Letushencesetasidethefollowingunhelpfulquestion:

Isthefutureofscienceortechnologicalinnovationpredictable?

Abetterquestionwouldbe,

Howpredictablearevariousaspectsofthefutureofscienceortechnologicalinnovation?

Butoften,wewillgetmoremileageoutofasking,

Howmuchmorepredictablecan(acertainaspectof)thefutureofscienceortechnological
innovationsbecomeifwedevoteacertainamountofresourcestostudyit?

Orbetterstill:

7
Whichparticularinquirieswoulddomosttoimproveourabilitytopredictthoseaspectsof
thefutureofS&Tthatwemostneedtoknowaboutinadvance?

Pursuitofthisquestioncouldleadustoexploremanyinterestingavenuesof
researchwhichmightresultinimprovedmeansofobtainingforesightaboutS&T
developmentsandtheirpolicyconsequences.
9

CrowandSarewitz,however,wishingtosidestepthequestionaboutpredictability,
claimthatitisirrelevant:

preparationforthefutureobviouslydoesnotrequireaccurateprediction;rather,it
requiresafoundationofknowledgeuponwhichtobaseaction,acapacitytolearn
fromexperience,closeattentiontowhatisgoingoninthepresent,andhealthyand
resilientinstitutionsthatcaneffectivelyrespondoradapttochangeinatimely
manner.

Thisansweristooquick.Eachoftheelementstheymentionasrequiredforthepreparation
forthefuturereliesinsomewayonaccurateprediction.Acapacitytolearnfromexperience
isnotusefulforpreparingforthefutureunlesswecancorrectlyassume(predict)thatthe
lessonswederivefromthepastwillbeapplicabletofuturesituations.Closeattentionto
whatisgoingoninthepresentislikewisefutileunlesswecanassumethatwhatisgoingon
inthepresentwillrevealstabletrendsorotherwiseshedlightonwhatislikelytohappen
next.Italsorequirespredictiontofigureoutwhatkindofinstitutionswillprovehealthy,
resilient,andeffectiveinrespondingoradaptingtofuturechanges.Predictingthefuture
qualityandbehaviorofinstitutionsthatwecreatetodayisnotanexactscience.
Itispossible,however,toreconstructCrowandSarewitzsargumentinawaythat
makesmoresense.Effectivepreparationforthefuturedoesrequireaccuratepredictionofat
leastcertainaspectsofthefuture.Butsomeaspectsarehardertopredictthanothers.Ifwe
despairofpredictingthefutureindetail,wemaysensiblyresorttocoursesofactionthatwill
doreasonablywellindependentlyofthedetailsofhowthingsturnout.Onesuchcourseof
actionistobuildinstitutionalcapacitiesthatareabletorespondeffectivelytofutureneeds
astheyarise.Determiningwhichinstitutionalcapacitieswillproveeffectiveinthefuture
doesrequireprediction,butthisisoftenamorefeasiblepredictiontaskthanpredictingthe
detailsofthesituationsthattheywillhavetorespondto.Themorethefutureisveiledin
ignorance,themoreitmakessensetofocusonbuildinggeneralpurposecapabilities.
Recastinthisway,theargumentismoredefensibleasfarasitgoes.Butitdoesnot
goveryfar.ItslimitationsbecomeclearwhenweconsideritwithinthecontextofS&T
policy.

9
Seeforexample(Tetlock2005).
8
ThetasksofS&Tpolicyincludesettingprioritiesfortheallocationoffundingto
researchprojects.Itishardtopredictwhichlinesofresearchwillbearfruitandwhichwill
not.Thereareseveralpossiblewaysofrespondingtothispredicament.Oneistoconcentrate
fundingonthoseresearchavenueswhichwecanbefairlycertainwillbearatleastsome
fruit.Anotheristodiversifytheresearchportfolioandfundalittlebitofeverything.Athird
approachistobetonafewresearchavenuesthatseemespeciallypromisingandacceptthe
riskoftotalfailure.Dependingonthefundersattitudetorisk,andotherfactors,some
mixtureoftheseapproachesmightbeoptimal.
10

Butthequestionofpredictabilitydoesnotgoaway.Ofcourse,itisnotpossibleto
fundalittlebitof(literally)everything,andspreadingoutfundingasevenlyaspossible
amongallseekersseemsunlikelytobethesmartestwayofgoingaboutthings.Sohow
muchfundingshouldbeconcentratedonafewpromisingfields,suchasnanotechnology?
Howtightlyfocusedshouldthatfundingbeonparticularapproaches,methods,and
researchcenters?Thereisnosimpleanswer.Theoptimalstrategywilldependonjusthow
confidentthefundersareintheirabilitytopickwinners,i.e.predictfutureadvances.
Thesituationbecomesevenmorecomplicatedifweconsiderthatsomeresearch
projectsmightnotsimplyfailtocometofruitionbutmightbearpoisonedfruitproduce
resultsthatwewouldbebetteroffwithout.Certainpossibleweaponstechnologiesfallinto
thiscategory,butsomecriticsoftechnosciencewouldarguethatitincludesagreatdealelse
beside.Presumably,themajoritysfeelingthathumanityoughttopursuescientificand
technologicalresearchrestsontheassumptionthatthevalueoftheconsequencesofsuch
advancesislikelytobe,onbalance,positive.Butifthisassumptionistrue,andifitisalso
grantedthatsometechnologicaladvanceswillprovedetrimental,thenagainthequestion
becomeswhetherwecanbeconfidentenoughinourabilitytopredictinadvancewhich
particulartreeswillproducepoisonedfruitinordertobejustifiedincuttingthemdown
now,orwhetherweshouldinsteadthemletallgrow,inthenameofourepistemicmodesty.
Universalcultivationseemstorequirethattherebejusttherightamountofpredictability:
enoughsothatwecanexpectthatonbalancetheorchardwillbebeneficialtohumanityand
thatourcultivationofitwillinfactpromoteitsgrowth,butnotsomuchthatwewouldbe
betteroffbychoppingdownselectedtreesbecausetheirgrowthmayinthelongruncause
harm.
Thecomplexityofourpredictionproblemincreasesevenfurtherwhenweconsider
thatthepayoffofanindividualresearchprojectisnotindependentofwhathappenswith
otherresearchprojects.Differentadvancesmayworksynergistically(asinthecaseofNBIC
technologies),oronemightpreemptanotherandmakeitobsolete.Whensuchdependencies
exist,thedevelopmentofanoptimalresearchportfoliobecomesmoredifficult.If

10
Theneedtomakethesekindsoftradeoff,ofcourse,isnotconfinedtofundingofthenaturalsciences
andtechnology,butappliestofundingofthesocialsciencesandELSIprogramstoo.
9
predictabilityislow,wemightdecidetoignoresuchdependencies;ifitishigher,onthe
otherhand,wewouldberemissnottotakethemintoaccountindecidingourresearch
priorities.Thequestionofdependenciesbetweenpotentialfutureadvancesmightalsohave
tobereflectedinwhatinstitutionalstructuresweshouldcreatefortheprocessofS&T
agendasettingandimplementation;forexample,whethertoestablishaseparatecommittee
foraparticularsubfield.Again,thequestionofthedegreeofpredictabilityofvariousaspects
ofthefuturefarfrombeingonethatcanbetriviallyansweredorsidesteppedisinfact
highlydifficultandcentraltomanyS&Tpolicyissues.
Fromthisbriefdiscussionofpredictabilitywecanalreadydrawseveral
conclusions.First,whilesomescientificadvancesandtechnologicalinnovationsarehardto
predictwithaccuracyfarinadvance,theproblemisnotuniquetothescienceand
technologycontext.Bigpolicydecisions,smallpolicydecisions,andtrivialpersonal
decisionsallhaveimportantconsequencesthatcannotbepredictedindetail.Second,there
aremanyaspectsofscientificandtechnologicaldevelopmentsthatcanbepredicted.Third,
allmeaningfulpreparationsforthefuturerely,explicitlyorimplicitly,onprediction.Fourth,
theissueoftherelativepredictabilityofdifferentaspectsofthefuture,andhowmuchthe
predictabilitycanbeimprovedbyvariouskindsofinvestment,isimportantinthinkingabout
howR&Dprogramsshouldbestructured.
Afurtherlessonisthatimprovementsinourabilitytopredictvariouspotential
S&Tadvancesandtheirconsequencescouldmakeaveryvaluablecontributiontoour
capacitytomakewiseS&Tpolicydecisions.

4.StrategicconsiderationsinS&Tpolicy
LetusnowturntoconsiderthesourceofanadditionallevelofcomplexityinS&T
governance:strategyandpolitics.Theseimposeconstraintsonwhatcanbedone,and
thereforeonwhatitwouldmakesensetoattempttodo.AsRalphWaldoEmersononce
wrote:

Webtoweave,andcorntogrind;
Thingsareinthesaddle,
Andridemankind.

Inparticular,oneneedstoquestionwhethermankindisreallyridingscienceand
technology,orwhetheritistheotherwayaround.
OneobvioussenseinwhichmankindisnotinthesaddleisthattheS&Tpolicy
decisionsonthisplanetarenotmadebysomeoneunifiedbodyofrationalandbeneficent
representativesofhumanitywhoaretryingtogetustosomeparticulardestination.Instead,
therearecountlessagents,pursuingdifferentandoftenopposingobjectives,influencing
10
variousaspectsofourspeciesS&Tactivitiesnationalandregionalgovernments,
corporations,privatephilanthropicfoundations,specialinterestlobbies,journaleditors,
researchcouncils,mediaorganizations,universitypresidents,prizecommittees,consumers,
voters,scientists,publicintellectuals,andsoforth.Morespecifically,weknowtwothings:
(1)thereisnounifieddecisionmakingentitythathasthepowertodirectorhaltallresearch
worldwideinanyarea,and(2)thatmanyofthedecisionmakingentitiesthatinfluenceS&T
policiesatvariouslocationsarethemselvessubjecttoinfluencefromavarietyofagentswith
diverginggoalsandagendas.Bothofthesefactshaveprofoundconsequencesforour
thinkingaboutS&Tpolicy.
Thefirstfact,theabsenceofglobalcontroloftheworldsS&T,makesitdifficultor
impossibletostopresearchandinnovationinaparticulardirectionevenifitwouldbea
goodthingtodoso.Forexample,evenifsomedetailedstudyorpublicconsultation
concludedthatananotechrevolutionwouldbedetrimentaltohumanity,thereisnoclear
pathtopreventingsucharevolutionfromhappeninganyway.Thegovernmentofone
countrymightrescindpublicfundingforresearchincertainareasthoughtlikelytoenable
advancesinnanotechnology;butresearchwouldcontinue(albeitperhapsataslowerpace)
usingfundingfromprivatesources.Thegovernmentmightthenbanallsuchresearch,
whetherpubliclyorprivatelyfunded,butothernationswouldalmostcertainlycontinueto
pushforward,andifthetechnologyisfeasible,itwilleventuallyseethedayanyway.Global
bansontechnologicaldevelopmentsareverydifficulttonegotiateandevenhardertopolice.
Thedifficultiesareamplifiedincaseswheresignificantincentivesexistforsomegroupsfor
movingforward,wheredevelopmentcanbeconductedwithmodestresources,where
concealmentispossible,andwherethereisnosalientdemarcationbetweenthe
hypotheticallyproscribedactivityandlegitimateresearch.Nanotechnologysatisfiesmanyof
theseconditions,sotheprospectofglobalrelinquishmentappearstobeclosetonil,atleast
inlieuofdramaticadvancesinbothsurveillancetechnologyandglobalgovernance.
11

TheinfeasibilityofhaltingcertainkindsofresearchisapointoftenrepeatedinS&T
policydiscussions:Ifourcountrydoesnotgoforwardwiththis,someoneelsewillandwe
willfallbehind,orIfnanotechnologyisoutlawed,onlyoutlawswillhave
nanotechnology.
12
Theappealtonationalcompetitivenessseemstobeoneofthe
rhetoricallymosteffectiveargumentsbothforincreasedspendingonresearch,andagainst
regulationthatwouldslowdevelopment.

11
Thelatterprovisoisnotinsignificantifwearethinkingaboutlongertimescales.Onemightalso
imaginethatsupportfortoughinternationalactioncouldincreasedramaticallyfollowingabig
disastersuchasanactofnuclearterrorism.Andofcourse,anotherwayinwhichnanotechnology
researchcouldcometoahaltisasaresultofacivilizationdestroyingglobalcatastrophe.Thisproviso
isalsonotinsignificant;see(Bostrom2002).
12
E.g.(Vandermolen2006).
11
Itisworthcomparingthisargumentfromeconomiccompetitivenesswithanother
appealingargumentformoreresearchfunding:thatresearchisaglobalpublicgoodand
shouldbesupportedoutofloveofhumankind.Thetwoargumentsstandinsometension.
Theglobalpublicgoodsargumentsuggeststhatitmightbeinanationsselfinteresttofree
rideonothernationsS&Tinvestment(particularlyfoundationalresearch,thebenefitsof
whichareespeciallydifficultfortheproducertomonopolize).Ifthisisthecasethen
nationalcompetitivenessmightactuallysufferfromthediversionofresourcesawayfrom
othersectorsofsocietytoS&Tresearch.Yetbothargumentscouldbetrue.Theremightbe
highreturnsforanationtoitsinvestmentsinR&D,andadditionalreturnsthatcannotbe
captureddomesticallyandinsteadbecomeapositiveexternalitybenefitingothernations.In
thiscase,boththeloveofhumankindandtheappealtonationaladvantagewouldworkin
tandemasreasonsforincreasingR&Dinvestment.
13

Asbothoftheseargumentsillustrate,thereareimportantconsequencesforS&T
policyfromthefactthatS&Tpolicyisnotperfectlygloballycoordinated.Butthecomplexity
ofthestrategicsituationincreasesvastlywhenwetakeintoaccountthatevenwithina
particularcountry,S&Tdecisionsarenotmadebyasingleunifiedperfectlyrationaland
perfectlybeneficentagency.Policyrecommendationsdirectedtoanimaginaryidealglobal
ornationaldecisionmakermayformusefulfocalpointsforinterimdiscussion,but
ultimatelytheyneedtobetransformedintorecommendationsaddressedtosome
identifiablerealagent.Atthatstage,recommendationsmusttakeintoaccountthe
limitationsofthatagentspowers,understanding,attention,andinterests.This
transformationofwhatshouldbedoneinanabstractsenseintosensiblerecommendations
toanagentthatcanactuallydothingsisfarfromstraightforward.
Forexample,oneargumentthathasbeengivenformovingforwardwith
nanotechnologyresearchasrapidaspossiblyisasfollows:
14

(1) Therisksofadvancednanotecharegreat.
(2) Reducingtheseriskswillrequireaperiodofseriouspreparation.

13
MoststudiesoftheeconomicreturnstoR&Dhavenotfocusedontheinternationaldimension.
Domestically,itappearsthatthesocialreturnsofR&Dalthoughtheyaredifficulttomeasureare
veryhighandthatoptimalR&Dinvestmentsubstantiallyexceedstheactuallylevel.Seee.g.(Jones
andWilliams1998;SalterandMartin2001).
14
See(Drexler1992a),p.242.Drexler(privatecommunication)confirmsthatthisreconstruction
correspondstothepointhewasmaking.Obviously,anumberofimplicitpremiseswouldhavetobe
addedifonewishedtopresenttheargumentintheformofadeductivelyvalidchainofreasoning.By
advancednanotechnologyIhererefertoapossiblefutureformofradicalnanotechnology,
sometimescalledmolecularnanotechnology,ormachinephasenanotechnology;seealso(Drexler
1992b).
12
(3) Seriouspreparationwillonlybeginoncetheprospectofadvancednanotechis
takenseriouslybybroadsectorsofsociety.
(4) Broadsectorsofsocietywillonlytaketheprospectofadvancednanotech
seriouslyoncethereisalargeresearcheffortunderway.
(5) Theearlieraseriousresearcheffortisinitiated,thelongeritwilltaketodeliver
advancednanotech(becauseitstartsfromalowerlevelofpreexistingenabling
technologies).
(6) Therefore,theearlieraseriousresearcheffortisinitiated,thelongertheperiod
duringwhichseriouspreparationwilltakeplace,andthegreaterthereduction
oftherisksthatwilleventuallyhavetobefaced.
(7) Therefore,aseriousresearcheffortshouldbeinitiatedassoonaspossible.

Ipresentthisargumentnotinordertoevaluateit,buttoillustratethepointaboutstrategic
complexity.Whatnaivelylookslikeareasonforgoingslowlyorstopping(therisksof
advancednanotechbeinggreat)endsup,onthislineofthinking,asareasonfortheopposite
conclusion.Whenoneattemptstointegratesuchsecondguessingoftheresponsesofvarious
actorsinonesR&Spolicyrecommendations,theresultmightdifferradicallyfromwhatone
wouldgetfromamoresimplemindedapproachthatignoresthestrategicdimensionsofthe
situation.
Itisinterestingtoconsidertowhomitisthatthiskindofargumentaddressesitself.
Thebroadsectorsofsociety,whichwillsupposedlybeginseriouspreparationonlyaftera
largeresearcheffortisalreadyunderway,arepresumablynottheintendedrecipientsofthe
message.Iftheywerecapableofandwillingtounderstand,agreewith,andactonan
argumentlikethis,thentheywouldnotneedtowaitforalargeresearchefforttoget
underwayinordertotaketheneedforpreparationseriously.Theargumentappearstobe
esoteric.Therearesomepeoplewhoareintheknowabouttheprospectsofadvanced
nanotechnology,andthesepeoplewouldhaveareason(sotheargumentgoes)todirecttheir
effortstowardsacceleratingtheimplementationofaseriousnanotechresearcheffortevenif
theythoughtthattherisksofadvancednanotechoutweighedthebenefits.
Onewaythecognoscenticoulddothiswouldbebypublicizinganotherargument
fortheaccelerationofnanotechresearch,suchastheargumentthatifwedontmoveforward
quickly,thensomebodyelse,perhapsahostilestate,willgettherefirstandthatwouldbethe
worstpossibleoutcomeofall.Notethateventhissecondargumentaddressesitselfnotto
everybodybuttoaselectgroupinthiscaseourcompatriots,oratleastthecitizensof
goodstates.Itwouldnotbedesirablethatthecitizensofbadstatesurgetheir
13
compatriotsandgovernmentofficialstolaunchacrashprogramforthedevelopmentof
advancednanotechsothattheygettherefirst.
15

Thereareactuallypeople(althoughperhapsnotmany)whothinkatthislevelof
sophisticationandattempttotakestrategicconsiderationssuchasthoseaboveintoaccount
indecidingwhattheyoughttodo.Someofthesepeoplearewellmeaningandhonestand
wouldnotconsenttoputtingforwardanargumentoranopinionthattheydidnotsincerely
holdtobetrue.Esotericargumentsdonotrequiredeception,orevenactiveconcealment,
becausetoasignificantextentaudiencesselfselectwhichargumentstheyhearandabsorb.
Thenanocognoscentimightbetheonlyoneswhoarereceptivetotheargumentabouta
largeresearchprogrambeingnecessarytogetbroadsectorsofsocietytotaketherisks
seriouslyandstartpreparing.Thecitizensofgoodnationsmightbemorelikelytofollow
theircompatriotsadviceontheneedtomoveforwardwiththeresearchtoavoidfalling
behindinafuturearmsracethanthecitizensofthebadnationswhowefearmight
otherwisetakethelead.(Buthowsurecanwebethatthisisalwaysthecase?)
Predictability,orthelackthereof,againemergesasanimportantissue.Clearly,
anticipatingtheresponsesofmanydifferentagents,howtheseresponseswillinteract,and
moregenerallyhowtheecologyofideasandopinionswillbeaffectedbythepromulgation
ofoneargumentoranother,isadauntingtaskinmanycasesevenmoredifficultthan
forecastingfuturedevelopmentsinS&T.

5.Limitingthescopeofourdeliberations?
Itistempingtoignoreallofthesedifficultiesandfocusonthesimplertaskoffiguringout
whatwehavemostreasontodo,subtlestrategicconsiderationsaside.Attheindividual
level,apersonmightsimplytrytodecide:isnanotechlikelytodomoregoodthanharm?If
yes,thenbeinfavorofitsdevelopment;ifno,opposeit.Oralternatively:istheresomepath
involvingnanotechdevelopmentcombinedwithcertainregulationsandorpublicpolicies
thatwouldbringgreatbenefitsandonlymoderaterisks?Ifyes,thenpromotethatpath.
Wemaynoteimmediatelythatthechoicehereisnotadichotomousone,eitherto
ignoreallstrategicconsiderationsorelsetotakethemallonboardwithoutlimitation.Even
inrelativelyuncomplicateddeliberations,somestrategicconsiderationsmightbeadmissible.
Forinstance,wemighttakeintoaccountthefactthatothernationswillproceedwith
nanotechdevelopmentevenifournationdoesnt,whilesettingasideallconsiderations
havingtodowiththecognitivelimitationsofthehoipolloiorthewaythatspecialinterests
arelikelytoinfluencetheimplementationofanyofficiallyadoptedpolicydirective.

15
Onemayofcourseinsistthatthegoodstatesshoulddeveloponlydefensivenanotechcapabilities.But
offensiveanddefensiveapplicationswouldrequirelargelythesameunderlyingtechnological
advances.
14
Atacollectivelevel,too,wemightdecidetoexcludecertainkindsofreasonsfrom
variouscontextsofpublicdeliberation.Thisistheidea,forexample,behindJohnRawls
conceptofpublicreason.
16
Indealingwithconstitutionalessentialsandmattersofbasic
justice,citizensabidingbytheideaofpublicreasonare,accordingtoRawls,entitledto
justifythepositiontheywantadoptedonlyonthebasisofreasonsthatcouldreasonablybe
acceptedbyothercitizenswhodonotnecessarilysharethesamemetaphysical,religiousor
culturalviews.
17
Thisconstrainthasbeencriticizedongroundsthatitwouldrequire
insincerity(notputtingforwardtherealreasonsforonesviews)andleadtoashallowness
inpublicdiscourseasparticipantsarerequiredtoconfinethemselvestothelowestcommon
denominatorofsharedbackgroundassumptions.Rawlsbelievesthatweshould
neverthelessabidebysuchaconstraint

givenourdutyofcivilitytoothercitizens.Afterall,theysharewithusthesame
senseofitsimperfection,thoughondifferentgrounds,astheyholddifferent
comprehensivedoctrinesandbelievedifferentgroundsareleftoutofaccount.Butit
istheonlyway,andbyacceptingthatpoliticsinademocraticsocietycanneverbe
guidedbywhatweseeasthewholetruth,thatwecanrealizetheidealexpressedby
theprincipleoflegitimacy:tolivepoliticallywithothersinthelightofreasonsall
mightreasonablybeexpectedtoendorse.
18

Asamuchmoremundaneexampleofdiscourserestriction,considertheconvention
againsttheuseofadhominemargumentsinscienceandmanyotherarenasofdisciplined
discussion.Thenominaljustificationforthisruleisthatthevalidityofascientificclaimis
independentofthepersonalattributesofthepersonorthegroupwhoputsitforward.
Construedasanarrowpointaboutlogic,thiscommentaboutadhominemargumentsis
obviouslycorrect.Butitoverlookstheepistemicsignificanceofheuristicsthatrelyon
informationabouthowsomethingwassaidandbywhominordertoevaluatethecredibility
ofastatement.Inreality,noscientistadoptsorrejectsscientificassertionssolelyonthebasis
ofanindependentexaminationoftheprimaryevidence.Cumulativescientificprogressis
possibleonlybecausescientiststakeontruststatementsmadebyotherscientists
statementsencounteredintextbooks,journalarticles,andinformalconversationsaroundthe
coffeemachine.Indecidingwhethertotrustsuchstatements,anassessmenthastobemade
ofthereliabilityofthesource.Cluesaboutsourcereliabilitycomeinmanyformsincluding

16
Forareviewofphilosophicalviewsontheideaofpublicity,includingthoseofRawls,see(Gosseries
2005).
17
ThisisasimplificationofRawlsview,butthedetailsarenotessentialforpresentpurposes;seealso
(Gosseries2005).
18
(Rawls1999),pp.242243.
15
informationaboutfactorssuchasfundingsources,peeresteem,academicaffiliation,career
incentives,andpersonalattributessuchashonesty,expertise,cognitiveability,andpossible
ideologicalbiases.Takingthatkindofinformationintoaccountwhenevaluatingthe
plausibilityofascientifichypothesisneedinvolvenoerroroflogic.
Whyisit,then,thatrestrictionsontheuseoftheadhominemcommandsuchwide
support?Whyshouldargumentsthathighlightpotentiallyrelevantinformationbesingled
outforsuspicion?Iwouldsuggestthatthisisbecauseexperiencehasdemonstratedthe
potentialforabuse.Forreasonsthatmayhavetodowithhumanpsychology,discoursesthat
toleratetheunrestricteduseofadhominemargumentsmanifestanenhancedtendencyto
degenerateintopersonalfeudsinwhichthespiritofcollaborative,reasonedinquiryis
quicklyextinguished.AdhominemargumentsbringoutourinnerNeanderthal.
Theinstrumentalproscriptionofadhominemandthemoredeeplynormative
Rawlsianideaofpublicreasonbothillustratethegeneralconceptofconstructingdiscourses
partlybyrulingoutofcourtsometypesofconsideration.Thereexistmanysuchdiscourse
boundaryconstraints,bothinscienceandinotherarenas,varyingfromcontexttocontext.It
isperhapsnotimplausibletosupposethatsomestrategicconsiderations,suchastheones
introducedabove,mayalsobeappropriatelyexcludedinsomecontextsofS&Tpolicy
deliberation.
Howshouldwedeterminewheretheappropriateboundariesforaparticular
discourseshouldbeplaced?Thecomplexitiesofthisquestionprotrudefarbeyondthescope
ofthisessay,butitmightbeworthlistingafewpotentiallyrelevantfactors.
19
Thesemight
servetohintathowwhichfutureinvestigationscouldexplorethisareafurther.

(1) Selfdeceptionandbias.Weknowthathumancognitionissusceptibletoself
deceptionandbiasesofvariouskinds.Certaintypesofconsiderationmightoffer
morefootholdthanotherstoirrationalityandprejudice.
(2) Manipulability.Similarly,inpublicdeliberation,certaintypesofconsideration
mightbeeasiertomanipulateforpartisanpurposes,suggestingthattheyshould
begivenlessweightorbarredaltogether.
(3) Unpredictability.Asarguedalready,thecomplexityofacertainlevelofstrategic
considerationissuchthatwemightquicklyloseourselvesinafogof
unknowabilityifweventurethere.
(4) Directvs.mediatedconsequences.Thisharksbackbothtounpredictabilityand
foregoingdiscussionofstrategicconsiderations.Butonemightalsomaintain,as
amatterofbasicethicaltheory,thatwearenotresponsiblefor(andshouldhence
nottakeintoaccount?)thoseconsequencesofouractionsthataremediatedby

19
ThesewouldbecomplementarytothemicrolevelmaximsidentifiedbyPaulGriceinhisworkon
conversationalimplicatures(Grice1975).
16
manyexternalcausalfactorsorotherbypeopleschoices,eveniftheyshouldbe
partlypredictable.
(5) Divisionoflabor.Inapublicdeliberation,itisnotalwaysnecessarythatall
partiesattempttoincorporateanddigestalltypesofconsideration.Asensible
majorityviewmightemergeevenifsomeorallpartiesarebiasedandblinkered
inwhatconsiderationstheytakeintoaccount.
(6) Accessibilitytostakeholders.Sometypesofconsiderationmighteffectively
excludesomestakeholdersfromparticipatingmeaningfullyinapublic
deliberation.Doesthissometimesmaketheappealtosuchconsiderations
inadmissible?
(7) Moralsideconstraintshonestyandtruthfulnessincluded,butperhapsthere
arealsoothermoralconstraintsanddesideratathatapply.
(8) Thepotentialforcumulativeprogress.Itmightbethatdiscussionofcertaintypes
ofconsiderationproduceresultsthatareeasiertopreserveandcarryoverto
futuredialogsandnewsituations,enablingcumulativeprogress.
(9) Thepotentialforconnectiontootherdomains.Workthatclarifiessometypesof
considerationmightmorereadilybeintegratedwithresultsfromother
disciplinesorcognitivedomains.Thismightbeanargumentforpayingmore
attentiontosuchconsiderations.

RecallthetwobeliefsoroftarticulatedargumentstowhichCrowandSarewitz
attributeresponsibilityforthelackofconsiderationgiventotheimplicationsforcomplex
societaltransformationintheallocationofresourcestoR&Dprograms.Onewasthe
argumentaboutunpredictability.Theotheristheargumentthatthescienceandtechnology
enterprisehastobegrantedautonomytochooseitsowndirectionofadvanceand
innovation.Thislatterargumentis,inCrowandSarewitzsview,

contradictedbyrealityThedirectionofscienceandtechnologyisinfactdictated
byanenormousnumberofconstraints(onlyoneofwhichisthenatureofnature
itself).

ItisunclearthathowthisissupposedtobeanobjectiontotheviewthattheS&Tenterprise
shouldbegrantedautonomy.ThosewhoholdthisviewmightwellagreewithCrowand
Sarewitzthatthespiritoffreeinquiryiscurrentlyfetteredbyanenormousnumberof
constraintswhileatthesametimeinsistingthatitwouldbepreferableifatleastsomeof
theseshacklescouldbeabolished,oratleastrelaxed,andthatimposingadditional
17
constraintswouldmakethingsworse.Oneneedstodistinguishthenormativequestion
abouthowthingsoughttobefromthepositivequestionofhowthingsare.
20

Democraciesoftenfinditadvisabletoinsulatecertainfunctionsfromdirectpublic
influence.TheU.S.constitution,forexample,deliberatelycreatesajudiciarythatis
substantiallyinsulatedfromdirectpoliticalinfluence.Centralbanksinmanycountriesare
similarlyautonomous.Inalldemocracies,thedaytodayoperationofmostgovernment
departmentsissignificantlyshieldedfromthedirectimpactofpublicopinion.Inthe
universities,seniorresearchstaffenjoyagreatdealofintellectualfreedomandfrequently
havetheopportunitytopursueresearchprojectsoftheirownchoosing,eventhoughtheir
salariesmightcomefromthetaxpayer.Intheprivatesector,corporationsandindividuals
arefreetopursuealmostwhateverdirectionofresearchtheydesire,providedtheyhavethe
fundingtodoso.Andindependentintellectualsandwritersareaccountabletonobodybut
themselves.
WhatadvocatesofincreasedpublicinvolvementanddirectdemocracyinS&Tare
arguingisnotthatmattersofscientificcontroversyoughttobesettledbypopular
referendum,orthatthepublicbebroughtintomicromanagetheconductofscientific
researchprojects.Noraretheymindedtorestrictfreedomorspeechoranyoftheother
intellectualfreedomscurrentlyenjoyedbyindividuals.Whatisbeingproposedisgenerally
somemoremoderateposition,forexamplethatthesettingoftheoverallparametersand
prioritiesofpubliclyfundedS&Tresearchoughttobemademoretransparentandmore
directlysubjecttodemocraticinputthaniscurrentlythecase.Effortsmightbeadvocatedto
improvepublicunderstandingofscienceandtocreatemoreopportunitiesforgenuine,two
waydialoguebetweenscientistsandthepublic.Often,thefocusisontheimplementation
sideoftheS&Tenterprise,urginggreaterdirectdemocraticcontroloverwhichnew
technologiesarepermitted,underwhatformsofregulation,andwithwhatancillarypolices
tomodulatetheirsocietalimpacts.Alsosometimesofconcernarespecificmethodsof
scientificresearch,suchasanimalexperimentationortheusehumanembryos.
ThequestionofgovernanceofS&Tissuescanprobablynotbeseparatedfrom
questionsofgovernanceingeneral.Myfocushere,however,isnotonthestructuresof
governancewithwhoshouldmakethedecisionsbutratheronthetermsofthediscourse:
whatkindofconsiderationsshouldbetakenintoaccount,andinwhatways.
21
Ofcourse,

20
Adistinctionthatisoftenoverlookedindiscussionsaboutthefuture.Thereisalamentabletendency
toletpredictionandevaluationblendintoaconfusedblobofwishful(orfearful)thinking.
21
Anothervariableherewouldbewhatwemaytermtheformatofdeliberation,e.g.whetheritshould
occurbehindcloseddoorsorinthefullglareofpublicity,andwhatthetoneorspiritofadeliberation
shouldbe,e.g.howmuchandwhatkindofemotionshouldbedisplayed.Itisplausiblethatsuch
formatfactorshaveasubstantialeffectonthenatureofthedeliberation.Thereissomedirectempirical
evidenceforthis.Forexample,onerecentstudyofthedeliberationsoftheFederalReservesFederal
OpenMarketCommitteefoundthattranscriptpublicationsuppressedtheexpressionofdissenting
18
whomakesdecisionsdoes,inreality,influencewhatconsiderationsgettakenintoaccount.
Butwecanstilldistinguishthenormativequestionofwhatkindsofreasonsoughttobe
guidingtheS&Tpolicy.
Wehaveconsideredsomepossiblegroundsforvariouslimitationsofwhatreasons
shouldbeconsideredinthedeterminationofS&Tpolicy.Letusendthesectionwiththe
wordsofMarieCurie:

Wemustnotforgetthatwhenradiumwasdiscoverednooneknewthatitwould
proveusefulinhospitals.Theworkwasoneofpurescience.Andthisisaproofthat
scientificworkmustnotbeconsideredfromthepointofviewofthedirectusefulness
ofit.Itmustbedoneforitself,forthebeautyofscience,andthenthereisalwaysthe
chancethatascientificdiscoverymaybecomeliketheradiumabenefitfor
humanity.
22

6.Expandingthescopeofourdeliberations?
ThequotebyMarieCurieexpressesanextremeversionoftheviewthatweoughttonarrow
theconsiderationstakenintoaccount.AccordingtoCurie,scientificworkmustbedonefor
itself,forthebeautyofscienceandwithnoviewtoitsdirectusefulness.
OnemaycontrasttheinnocenceofMarieCurieswordswithawellknownremark
madesometwoandahalfdecadeslaterbyanotherdistinguishedphysicist,Robert
Oppenheimer,whohadspearheadedthedevelopmentofthenuclearbomb:

Insomesortofcrudesensewhichnovulgarity,nohumor,nooverstatementcan
quiteextinguish,thephysicistshaveknownsin;andthisisaknowledgewhichthey
cannotlose.
23

TheexplosionofthefirstnuclearweaponintheTrinitytest,andthelaterusein
HiroshimaandNagasaki,aresometimesseenasemblemsofthefailureofscientiststo
concernthemselveswiththesocietalimplicationsoftheirwork.Infact,manyofthe
scientistsinvolvedintheManhattanproject(andothers,suchasLinusPauling,who
declinedtoparticipate)werequitedeeplyconcernedaboutsocietalimplications.Among
thoseagreeingtolendtheirskillstotheproject,amajormotivationwastheconcernthat
NaziGermanymightotherwisegettothefissionbombfirst.Thisisanexampleofastrategic

opinionsandstifleddebateovershortterminterestrates(Meade,Stasavage,andLondonSchoolof
EconomicsandPoliticalScience.CentreforEconomicPerformance.2004).
22
MarieCurie(18671934),LectureatVassarCollege,May14,1921.
23
RobertOppenheimer(19041967),PhysicsintheContemporaryWorld,lectureatM.I.T.November
25,1947.
19
considerationmentionedearlier,nationalcompetitiveness.OntheMarieCurieversionofthe
Enlightenmentmodel,neitherthisconsiderationnorscruplesoverhowthescientific
findingsmightlaterbeusedshouldbetakenintoaccount,atleastnotbythepeopledoing
thescientificwork.
Thereareseveralgeneralcounterargumentsagainsttheviewthatthescopeofour
deliberationsshouldbeinsomewaysrestricted.
First,ignoringconsiderationsthatareevidentiallyrelevanttopotentialoutcomes
thatwecareaboutmeansignoringrelevantinformation.Ignoringrelevantinformation
mightnotberational,andmightimpedeoureffectivenessinachievingourgoals.Themore
relevanttheinformation,andthemoreimportantthegoalstowhichitisrelevant,thegreater
thecostofsuchintentionalignorance.
Second,confiningdeliberationswithinasetoffixedconstraintscanyieldpowerto
thosewhodeterminewhattheseconstraintsshouldbe.Thispowercanbemisused.As
rhetoriciansandsophisticatedtechnocratsarewellaware,theframingofanissuewhich
implicitlydetermineswhatkindsofconsiderationwillbeseenasbeingappropriateand
havingabearingofteneffectivelydeterminestheconclusionadeliberationwillreach.In
politicaldiscourse,framingsareoftenfiercelycontested.Toacceptasetofdiscourse
constraintsmightmeanbuyingintosomeoneelsesagendawhichtiltsthedeliberationin
favorofsomepredeterminedposition.
Third,andrelatedtothefirsttwoarguments,onefairlylikelyeffectofadoptinga
narrowframeworkforourdeliberationsoftheS&Tenterpriseistocreateabiasinfavorofa
certainkindofconservatismconservatismnotinthesenseofpoliticalideology,butin
thesenseofapresumptioninfavorofbusinessasusual.Scoperestrictionsriskrulingout
radicalcritiques,onesthatchallengethefundamentalassumptionsbehindthecommonway
ofthinkinganddoingthings.InthecontextofS&T,theeffectofthiswouldnotbeto
perpetuatethestatusquo,becausechangeinscienceandtechnologyisbroughtforth
ineluctablybytheintellectualadvancesgeneratedbytheenterpriseitself.Rather,theeffect
couldbetodiminishthepossibilityofadeliberatechangeofcourse.Withoutrecoursetoradical
critique,thelocomotivewillrollalongitstrack,andthetrackmightturnleftorright;we
mightevenbeabletoflickaswitchhereandtheretoselectwhichbranchofabifurcationwe
take;butweexcludefromourmentalspaceahostofdiscontinuouspossibilities,suchas
gettingoffthetrainandcontinuingourjourneyviaanothermodeoftransport.
24

Radicalcritiquesmightchallengethemetaphysicalunderpinningsofourworld
view.Theymightchallengeourbasicvaluesormoralnorms.Theymightundermineour
confidenceintheentireS&Tproject.Alternatively,theymightsuggestthatourattemptsto
humanizetheS&Tenterprisewillhavetheoppositeeffectfromtheoneintended.They
mightarguethatoneparticularanticipatedtechnologicalbreakthroughwillhave

24
Inthecontextoffundamentalscience,suchacoursechangecouldbeaKuhnianparadigmshift.
20
consequencesovershadowingalltherest,andthatbyfailingtoactaccordinglyweare
grosslymisdirectingourattentionandourresources.Theymightcontendthatincreased
publicengagementandincreasedeffortstoanticipatethesocietalimplicationswillhave
obnoxiousconsequences.Theymightargueforrelinquishmentofbroadareasof
technologicalresearch,oralternativelythateverythingshouldbeputintoacceleratingsome
applications.Theymightsuggestnewwaysoffundingbasicresearchthatwouldsidestep
expertpanelsandbureaucraticprocedures.Theymightidentifycompletelynewwaysof
evaluatingandmeasuringprogress.Thepossibilitiesaremyriadandimpossibletospecifyin
advance.
25

Manyradicalcritiquesareutterlywrong,andsomeofthemwouldbeextremely
dangerousiftheybecamepopular.Nevertheless,ifwelookbackhistorically,andobserve
howmanywidelyheldconventionalwisdomsofthepastarerevealedasblinkeredor
deeplyflawedbyourcurrentlights,wemustsurelyadmitthatbyinductionitislikely
thatmanyofourowncentralbeliefs,too,aredeeplyflawed.Weneedtobuildintoour
processesofindividualandcollectivedeliberationsomeselfcorrectingmechanismthat
enablesustoquestionandrectifyevenourmostdeepseatedassumptions.
Onewayinwhichthescopeofourdeliberationscanbebroadenedisbytakinginto
accountthekindofmetalevelreflectionsthatIhavetriedtoillustrateinthisessay.More
specifically,onecouldarguethatmoreworkshouldbedoneonthenormativedimensionsof
theS&Tenterprise.
Onenormativedimensionisethics,andthisistosomeextentalreadypartofthe
officialprograms,forexampleastheEinELSIandNELSI,andasappliedethicsmore
generally.Onemighthaveoccasionalmisgivingsaboutthequality,depth,orimpactofthis
research,butatleastthereissomerecognitionofthesignificanceandrelevanceofthe
questionsitissupposedtoaddress.
Anothernormativedimensionthathasbeengivenratherlessattentioninthese
programsisthatofapplied(normative)epistemology.Thisencompassesanumberof
importantproblems.Onesuchproblemistodevelopbetterhigherorderepistemic
principlesfortheconductofscientificresearch.AssaidbyE.T.Jaynes:

Itappearstousthatactualscientificpracticeisguidedbyinstinctsthathavenotyet
beenfullyrecognized,muchlessanalyzedandjustified.Wemusttakeintoaccount
notonlythelogicofscience,butalsothesociologyofscience(perhapsalsoits
soteriology).Butthisissocomplicatedthatwearenotevensurewhetherthe

25
Althoughnotnecessarilyintendedasradicalcritiques,forafewrecentexamplesseee.g.(Joy2000;
Bostrom2003b,2003a,2005;Hanson2003).
21
extremelyskepticalconservatismwithwhichnewideasareinvariablyreceived,isin
thelongrunabeneficialstabilizinginfluence,oraharmfulobstacletoprogress.
26

Yettheepistemologicalproblemsgobeyondthechallengeofhowtomaximizescientificand
technologicaladvancement.Aswehaveseen,theyalsoarealsocentraltoourthinkingabout
theethicalandpolicyissuespromptedbytheS&Tenterprise.Appliedepistemologyalsolies
attheheartoftheproblemofhowtoevaluateradicalcritiquesofthisenterprise.
Therearemanyinterestingapproachestothesemattersinadditiontophilosophical
reflectionandtheoreticalanalysis.Hereisasample:

Studytheheuristicsandbiasesaffectinghumancognition,andfigureourwaysof
applyingthefindingstoimproveourjudgment.
27

Studythecorrelatesoftrueopinion,amongexpertsandthepublic,andusethis
informationascluestowhoisrightincasesofdisagreementandtohowwemight
improveourownepistemicsituation.
28

Informationtechnologies.Developourinformationinfrastructureinwaysthatwill
facilitatethecollection,integration,andevaluationofinformation.
Cognitiveenhancement.Improveindividualreasoningability(e.g.concentration,
memory,andmentalenergy)byeducational,pharmaceutical,andothermeans.
Studyhowvestedinterests,themassmedia,andothersocialrealitiesshapeandbias
(orfacilitate)theprocessescollectivedeliberation.
Publicdeliberation.Developprocedures,formats,orrhetoricalstandardstoimprove
thequalityofpublicdebatebyconfrontingplebiscitoryreason.
29

Subsidizeandimplementinstitutionalinnovationssuchasinformationmarkets,
whichhavebeenshowntooutperformexpertpanelsinmanypredictiontasks.
30

Theunifyingthemeistoexplorehowwecouldmakeourselvessmarterandwiser,both
individuallyandasanepistemiccommunity.Theresearchcouldbeslantedtowards
applicationsinS&Tassessmentbutitislikelytohaveimportantspilloverbenefitsinother
areas.Suchaprogramcouldbecombinedwithmorenarrowlyfocusedeffortstogatherand
analyzeinformationinareasofparticularconcern,suchasnanotechnology.
Finally,Iwanttocallattentiontoonemorenormativedimension,exceptthatitis
notreallyadimensionbutratherthespacespannedbyalltheothervectors.Iamreferringto

26
(JaynesandBretthorst2003),p.525.
27
Seee.g.(KahnemanandTversky2000;Gilovich,Griffin,andKahneman2002).Foranattemptto
applythiskindofinformationtoatechnologyrelatedissues,see(BostromandOrd2006).
28
Seee.g.(Tetlock2005).
29
(Chambers2004).
30
(Hanson1995;WolfersandZitzewitz2004;LeighandWolfers2006).
22
thechallengeofintegratingallcrucialconsiderationsintosomecoherentunitythatwilllet
usdeterminewhatwehavemostreasontodoallthingsconsidered.Thismighthaveto
accommodatepredictionsabouttechnology,socialimpacts,strategicconsiderations,value
judgments,ethicalconstraints,andassortedmetalevelthoughtsabouthowallthesethings
shouldfittogether.
Suchsyntheticworkisnotinfashionatthepresenttime.Itisdiscouragedbythe
disciplinarystructureofacademiaandbyprevailingnormsofacademicpublishing,andit
doesnotappeartobestronglynurturedbytheshorttermgrantsfundingavailablefor
interdisciplinaryprojectseither.Asaconsequence,syntheticworkisundertakenmainlyby
timepressuredjournalists,popularscienceauthors,retiredscientistsorseniorscholarswho
arenolongerwillingorabletodoseriouswork,crackpots,andmiscellaneouseccentrics.
Thequalityoftheensuingcontributionsisoftenpoor,andevenwhensomething
worthwhilecomesoutthereisnoguaranteethatitwillusedandbuiltuponbyothers.
LetmetrytoindicateslightlymoreclearlywhatkindofsynthetictaskIhavein
mind.Iamnotreferringtotheerectionofgrandphilosophicalsystemsrestingon
foundationsofindubitablefirstprinciples;nordoIrefertothepaintingofvisionsorthe
realizationofcomprehensiveideologicaloutlooks;norstillamIpushingforsomenew
kindofsystematizationofallknowledge,auniversaltaxonomy,orthecreationofan
encompassingdatabase,oralibraryofcommissionedstudies.Rather,whatIhaveinmindis
thetaskofattemptingtothinkthroughsomeofthebigchallengesforhumanityinaway
thatdoesnotleaveoutanycrucialconsiderationbywhichImeanaconsiderationsuchthat
ifitweretakenintoaccountitwouldoverturntheconclusionswewouldotherwisereach
abouthowweshoulddirectourefforts.Forexample,someofthestrategicconsiderations
relatedtothenanotechnologyinitiativethatImentionedearliermightbecrucialinthat
theymightplausibly,iftheyaresoundandoncetakenintoaccount,rationallydelivera
practicalconclusionpointingintheoppositedirectionfromtheonewemightotherwise
believeweoughttostrivetowards.Butnotonlystrategicconsiderations,butalsoother
empirical,epistemological,axiological,andmethodologicalconsiderationscouldbecrucial
inthissense.
Implicitly,weareconfrontingachallengeofintegratingallcrucialconsiderations
everytimeweareattemptingtomakeareasoneddecisionaboutsomematterwhichwe
thinkitisimportanttogetright.Whenourgoalisverylimited,wemightatleast
sometimessucceedinmeetingthechallenge(albeitnotusuallybyrelyingonreasoned
deliberationalone).Whenthegoalismoreopenended,suchasifweareattemptingto
decidewhatwehavemostreasontodowithourownlifeallthingsconsidered,orifweare
seekingtoformanopiniononatopicsuchaswhatpublicpolicyoughttobewithregardto
someanticipatedtechnologicalrevolution,thenthecomplexityofthesyntheticchallenge
growsenormously.Itisnotclearthatweevermanagetomeetitinanyrobustsense.Instead,
whatanswerweendupespousingmightdependmostlyoncontingentfactorssuchasthe
23
politicalinclinationsofourparents,theidiosyncraticviewsofourthesisadvisor,thecurrent
culturalclimateintheplacewehappentolive,orthemerefactthatwefailedtothinkof
somecruciallyrelevantconsiderationthatwouldhavecausedustocometoaverydifferent
conclusion.
Whatthisseemstoamounttoisthatwecanhaveverylittlerationalconfidencethat
ourefforts,insofarastheyareaimingultimatelyatimportantlongtermgoalsforhumanity,
arenotentirelywrongheaded.Ournoblestandmostcarefullyconsideredattemptstoeffect
changeintheworldmightwellbepushingthingsfurtherawayfromwheretheyoughttobe.
Perhapsaroundthecornerlurkssomecrucialconsiderationthatwehaveignored,suchthat
ifwethoughtofitandwereabletoaccordititsdueweightinourreasoning,itwould
convinceusthatourguidingbeliefsandourstrugglestodatehadbeenorthogonalorworse
tothedirectionwhichwouldthencometoappeartousastherightone.
Otherthanbecominggenerallymoreagnostic,whatcanwedoaboutthis?Itisnot
clearhoweffectivelythesynthetictaskcouldbebrokenintosubtaskswhichcanbe
delegatedtoateamofresearchersandthenstitchedbacktogether.Butitisalsodubiousthat
anysinglegeniuscanaccomplishthetaskalone.Perhapsroomcouldbecreatedforanopen
endedenquirybymanymindstofocusontheproblemsofintegration.Considerable
progressmightbemadeovertime,whichwouldbeinterestinginitsownright,whetheror
notitwouldsucceedincreatingapragmaticsynthesisofwhichonecouldbejustifiably
confidentthatitdidnotfailtorecognizeanddojusticetoanycrucialconsideration.A
decisivebreakthroughwouldmeanthatwewouldhavetheopportunitytomakeourchoices
inthefulllightofavailablefactsandreasons,andwithjustifiedconfidencethatwearenot
pushinginentirelythewrongdirection.
Whetherthingswouldbebetterthatway,wedonotknowforsure.Nordoweknow
forsurewhatpresentlyavailableactionswouldbestencouragesuchanoutcome,shouldwe
decidetostriveforit,forthisisofcourseoneofthebigopenendedgoalstowhichthe
argumentapplies:evenifwethinkhardandhonestlyaboutthisissue,weareapttoneglect
atleastonecrucialconsideration.Unlessanduntilweachieveadramaticenlightenmentin
ourcapacityforpragmaticsynthesis,thenonthisandonotherrevolutionaryprospects,we
willcontinuetostakeoutourethicsandpolicypositionsinthedark.

References
Bainbridge,WilliamSims,andMihailC.Roco,eds.(2006),ManagingNanoBioInfoCogno
Innovations.Dordrecht:Springer.
Bostrom,Nick(2002),ExistentialRisks:AnalyzingHumanExtinctionScenariosandRelated
Hazards,JournalofEvolutionandTechnology9.
(2003a),AreYouLivinginaComputerSimulation?PhilosophicalQuarterly53
(211):243255.
24
(2003b),AstronomicalWaste:TheOpportunityCostofDelayedTechnological
Development,Utilitas15(3):308314.
(2005),TheFableoftheDragonTyrant,JournalofMedicalEthics31(5):273277.
(2006),InfiniteEthics,Workingmanuscript.Availablefrom
http://www.nickbostrom.com/ethics/infinite.pdf.
Bostrom,Nick,andTobyOrd(2006),TheReversalTest:EliminatingStatusQuoBiasin
Bioethics,Ethics116(4):656680.
Chambers,S.(2004),Behindcloseddoors:Publicity,secrecy,andthequalityof
deliberation,JournalofPoliticalPhilosophy12(4):389410.
Crow,MichaelM.,andDanielSarewitz(2001),NanotechnologyandSocietal
Transformation,inAlbertH.Teich,StephenD.Nelson,CeliaMcEnaneyand
StephenJ.Lita(eds.),AAASScienceandTechnologyPolicyYearbook,Washington,DC:
AmericanAssociationfortheAdvancementofScience,89101.
Drexler,Eric(2003),NanotechnologyEssays:RevolutionizingtheFutureofTechnology
(Revised2006),AAASEurekAlert!InContextApril.
Drexler,K.E.(1992a),EnginesofCreation.Oxford:OxfordUniversityPress.
(1992b),Nanosystems:MolecularMachinery,Manufacturing,andComputation.New
York:JohnWiley&Sons,Inc.
Elam,Mark,andMargaretaBertilson(2002),Consuming,EngagingandConfronting
Science:TheEmergingDimensionsofScientificCitizenship,inPeterHealey(ed.),
STAGE(HPSECT200150003)FinalReport,121158.Availablefrom
http://www.stageresearch.net/STAGE/documents/STAGE_Final_Report_final.pdf.
Gilovich,Thomas,DaleW.Griffin,andDanielKahneman(2002),Heuristicsandbiases:the
psychologyofintuitivejudgement.Cambridge,U.K.:CambridgeUniversityPress.
Gosseries,Axel(2005),Publicity,inEdwardN.Zalta(ed.),TheStanfordEncyclopediaof
Philosophy.
Grice,Paul(1975),LogicandConversation,inPeterColeandJerryL.Morgan(eds.),Syntax
andSemantics,Vol.3,SpeechActs,NewYork:AcademicPress,4158.
Hanson,R.(1995),CouldGamblingSaveScience?EncouraginganHonestConsensus,
SocialEpistemology9:1:333.
Hanson,RobinShallWeVoteonValues,ButBetonBeliefs?2003.Availablefrom
http://hanson.gmu.edu/futarchy.pdf.
Jaynes,E.T.,andG.LarryBretthorst(2003),Probabilitytheory:thelogicofscience.Cambridge,
UK:CambridgeUniversityPress.
Jones,C.I.,andJ.C.Williams(1998),MeasuringthesocialreturntoR&D,QuarterlyJournal
ofEconomics113(4):11191135.
Joy,B.(2000),Whythefuturedoesntneedus,Wired8.04.
Kahneman,Daniel,andAmosTversky(2000),Choices,values,andframes.Cambridge,UK:
CambridgeUniversityPress.
25
26
Latour,Bruno(1987),Scienceinaction:howtofollowscientistsandengineersthroughsociety.
Cambridge,Mass.:HarvardUniversityPress.
Leigh,A.,andJ.Wolfers(2006),Competingapproachestoforecastingelections:Economic
models,opinionpollingandpredictionmarkets,EconomicRecord82(258):325340.
Lenman,James(2000),ConsequentialismandCluelessness,PhilosophyandPublicAffairs29
(4):342370
Meade,EllenE.,DavidStasavage,andLondonSchoolofEconomicsandPoliticalScience.
CentreforEconomicPerformance.Publicityofdebateandtheincentivetodissent
evidencefromtheUSFederalReserve.CentreforEconomicPerformance,LondonSchoolof
EconomicsandPoliticalScience2004.Availablefrom
http://cep.lse.ac.uk/pubs/download/dp0608.pdf.
Rawls,John(1999),Thelawofpeoples.Cambridge,Mass.:HarvardUniversityPress.
Roco,MihailC.,andWilliamSimsBainbridge(2003),Convergingtechnologiesforimproving
humanperformance:nanotechnology,biotechnology,informationtechnologyandcognitive
science.Dordrecht;Boston,Mass.:KluwerAcademicPublishers.
Salter,A.J.,andB.R.Martin(2001),Theeconomicbenefitsofpubliclyfundedbasic
research:acriticalreview,ResearchPolicy30(3):509532.
Tetlock,Philip(2005),Expertpoliticaljudgment:howgoodisit?howcanweknow?Princeton,
[N.J.]:PrincetonUniversityPress.
Vandermolen,ThomasD.(2006),MolecularNanotechnologyandNationalSecurity,Air&
SpacePowerJournal20(3):96106.
Wolfers,J.,andE.Zitzewitz(2004),Predictionmarkets,JournalofEconomicPerspectives18
(2):107126.