Escolar Documentos
Profissional Documentos
Cultura Documentos
Streaming Analytics
Sheetal Dolas
Principal Architect, Hortonworks
19
Page 1Feb 2015
Hortonworks Inc. 2011 2014. All Rights Reserved
Who am I ?
Principal Architect @ Hortonworks
Most of the career has been in field, solving real life
business problems
Last 5+ years in Big Data including Hadoop, Storm etc.
Co-developed Cisco OpenSOC ( http://opensoc.github.io )
sheetal@hortonworks.com
@sheetal_dolas
Historic Analytic
Tools
Hive /
R / Python
Sources Flume / Kafka Storm HDFS
Custom BI Tools
Search
Syslog Topology
Agent A Topic A
A Elastic Search
Web
Machine Data Services
Topology / Solr
Agent B Topic B
B REST API
External
Low Latency
Streams
NoSql Web Apps
Topology
Other Agent N Topic N
N HBase
Alerting
Systems
New Data
Pre-compute Data
Batch View
Views Access
Data
Stream Query
Speed Layer
Common
ly ContextuSoftware
ReusableSolution Problem
Occurrin al Design
g
Data Stream
Architectural Functional
Management Security
Patterns Patterns
Patterns Patterns
Real-time Stream Joins External Message
Streaming Lookup Encryption
Top N
Near-real-time (Trending) Responsive Authorized
Streaming Shuffling Access
Rolling
Lambda Windows Out-of- Secure Cluster
Architecture Sequence Authentication
Events
Kappa
Architecture
Data Stream
Architectural Functional
Management Security
Patterns Patterns
Patterns Patterns
Real-time Stream Joins External Message
Streaming Lookup encryption
Top N
Near-real-time (Trending) Responsive Authorized
Streaming Shuffling Access
Rolling
Lambda Windows Out-of- Secure Cluster
Architecture Sequence Authentication
Events
Kappa
Architecture
Always Fetch
Cache
Everything
Partition and
Cache on the
go
Partitioner Bolt FL NC OH
Instance n Fraud Analyzer
Bolt
Instance n
Static Shuffle
Responsive
Shuffle
Result Operational
Latency
Accuracy Ease
Drop
Wait
Fan Out
Ordered
Typical
Event Filter events
Source Spout Processing
Bolt Bolt
Out-of-
Sequence
events
Based on
complexities in
processing, this
Special
can be extended
Handling Bolt as different
topology
Encryp
Encrypt Event(s)
t AES
Event event(s) Envelope
key w/ Decryp
w/ AES Decryp
RSA t
t AES
Event(s) event( Event
key w/
Envelope s) w/
RSA
Event(s) Envelope AES
Encrypted AES
Key (w/ RSA)