7/16/2016

Neural networks and deep learning

Whatthisbookisabout
Neuralnetworksareoneofthemostbeautifulprogrammingparadigms
everinvented.Intheconventionalapproachtoprogramming,wetellthe
computerwhattodo,breakingbigproblemsupintomanysmall,precisely
definedtasksthatthecomputercaneasilyperform.Bycontrast,inaneural
networkwedon'ttellthecomputerhowtosolveourproblem.Instead,it
learnsfromobservationaldata,figuringoutitsownsolutiontothe
problemathand.
Automaticallylearningfromdatasoundspromising.However,until2006
wedidn'tknowhowtotrainneuralnetworkstosurpassmoretraditional
approaches,exceptforafewspecializedproblems.Whatchangedin2006
wasthediscoveryoftechniquesforlearninginsocalleddeepneural
networks.Thesetechniquesarenowknownasdeeplearning.They've
beendevelopedfurther,andtodaydeepneuralnetworksanddeeplearning
achieveoutstandingperformanceonmanyimportantproblemsin
computervision,speechrecognition,andnaturallanguageprocessing.

NeuralNetworksandDeepLearning
Whatthisbookisabout
Ontheexercisesandproblems
Usingneuralnetstorecognize
handwrittendigits
Howthebackpropagationalgorithm
works
Improvingthewayneuralnetworks
learn
Avisualproofthatneuralnetscan
computeanyfunction
Whyaredeepneuralnetworkshardto
train?
Deeplearning
Appendix:Isthereasimplealgorithm
forintelligence?
Acknowledgements
FrequentlyAskedQuestions
Ifyoubenefitfromthebook,please
makeasmalldonation.Isuggest$3,but
youcanchoosetheamount.

They'rebeingdeployedonalargescalebycompaniessuchasGoogle,

Microsoft,andFacebook.
Thepurposeofthisbookistohelpyoumasterthecoreconceptsofneural

Sponsors

networks,includingmoderntechniquesfordeeplearning.Afterworking
throughthebookyouwillhavewrittencodethatusesneuralnetworksand
deeplearningtosolvecomplexpatternrecognitionproblems.Andyou
willhaveafoundationtouseneuralnetworksanddeeplearningtoattack
problemsofyourowndevising.

Aprincipleorientedapproach
Oneconvictionunderlyingthebookisthatit'sbettertoobtainasolid
understandingofthecoreprinciplesofneuralnetworksanddeeplearning,

Thankstoallthesupporterswhomade
thebookpossible,withespecialthanks
toPavelDudrenov.Thanksalsotoall
thecontributorstotheBugfinderHallof
Fame.

ratherthanahazyunderstandingofalonglaundrylistofideas.Ifyou've
understoodthecoreideaswell,youcanrapidlyunderstandothernew
material.Inprogramminglanguageterms,thinkofitasmasteringthecore
syntax,librariesanddatastructuresofanewlanguage.Youmaystillonly
"know"atinyfractionofthetotallanguagemanylanguageshave
enormousstandardlibrariesbutnewlibrariesanddatastructurescanbe
understoodquicklyandeasily.

http://neuralnetworksanddeeplearning.com/about.html

Resources
BookFAQ
Coderepository
MichaelNielsen'sproject
announcementmailinglist
DeepLearning,draftbookin
preparation,byYoshuaBengio,Ian
Goodfellow,andAaronCourville

1/3

7/16/2016

Neural networks and deep learning

Thismeansthebookisemphaticallynotatutorialinhowtousesome
particularneuralnetworklibrary.Ifyoumostlywanttolearnyourway
aroundalibrary,don'treadthisbook!Findthelibraryyouwishtolearn,
andworkthroughthetutorialsanddocumentation.Butbewarned.While
thishasanimmediateproblemsolvingpayoff,ifyouwanttounderstand
what'sreallygoingoninneuralnetworks,ifyouwantinsightsthatwill

ByMichaelNielsen/Jan2016

stillberelevantyearsfromnow,thenit'snotenoughjusttolearnsomehot
library.Youneedtounderstandthedurable,lastinginsightsunderlying
howneuralnetworkswork.Technologiescomeandtechnologiesgo,but
insightisforever.

Ahandsonapproach
We'lllearnthecoreprinciplesbehindneuralnetworksanddeeplearning
byattackingaconcreteproblem:theproblemofteachingacomputerto
recognizehandwrittendigits.Thisproblemisextremelydifficulttosolve
usingtheconventionalapproachtoprogramming.Andyet,aswe'llsee,it
canbesolvedprettywellusingasimpleneuralnetwork,withjustafew
tensoflinesofcode,andnospeciallibraries.What'smore,we'llimprove
theprogramthroughmanyiterations,graduallyincorporatingmoreand
moreofthecoreideasaboutneuralnetworksanddeeplearning.
Thishandsonapproachmeansthatyou'llneedsomeprogramming
experiencetoreadthebook.Butyoudon'tneedtobeaprofessional
programmer.I'vewrittenthecodeinPython(version2.7),which,evenif
youdon'tprograminPython,shouldbeeasytounderstandwithjusta
littleeffort.Throughthecourseofthebookwewilldevelopalittleneural
networklibrary,whichyoucanusetoexperimentandtobuild
understanding.Allthecodeisavailablefordownloadhere.Onceyou've
finishedthebook,orasyoureadit,youcaneasilypickuponeofthemore
featurecompleteneuralnetworklibrariesintendedforuseinproduction.
Onarelatednote,themathematicalrequirementstoreadthebookare
modest.Thereissomemathematicsinmostchapters,butit'susuallyjust
elementaryalgebraandplotsoffunctions,whichIexpectmostreaders
willbeokaywith.Ioccasionallyusemoreadvancedmathematics,but
havestructuredthematerialsoyoucanfollowevenifsomemathematical
detailseludeyou.Theonechapterwhichusesheaviermathematics
extensivelyisChapter2,whichrequiresalittlemultivariablecalculusand
linearalgebra.Ifthosearen'tfamiliar,IbeginChapter2withadiscussion
ofhowtonavigatethemathematics.Ifyou'refindingitreallyheavy
http://neuralnetworksanddeeplearning.com/about.html

2/3

7/16/2016

Neural networks and deep learning

going,youcansimplyskiptothesummaryofthechapter'smainresults.
Inanycase,there'snoneedtoworryaboutthisattheoutset.
It'srareforabooktoaimtobebothprincipleorientedandhandson.ButI
believeyou'lllearnbestifwebuildoutthefundamentalideasofneural
networks.We'lldeveloplivingcode,notjustabstracttheory,codewhich
youcanexploreandextend.Thiswayyou'llunderstandthefundamentals,
bothintheoryandpractice,andbewellsettoaddfurthertoyour
knowledge.
Inacademicwork,pleasecitethisbookas:MichaelA.Nielsen,"NeuralNetworksandDeepLearning",
DeterminationPress,2015

Lastupdate:FriJan2214:09:502016

ThisworkislicensedunderaCreativeCommonsAttributionNonCommercial3.0UnportedLicense.Thismeans
you'refreetocopy,share,andbuildonthisbook,butnottosellit.Ifyou'reinterestedincommercialuse,please
contactme.

http://neuralnetworksanddeeplearning.com/about.html

3/3