Akka persistence

(message sourcing in 30 minutes)

Konrad 'ktoso' Malawski

Scalar 2014 @ Warsaw, PL

Konrad `@ktosopl` Malawski

typesafe.com

geecon.org

Java.pl / KrakowScala.pl

sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl

meetup.com/Lambda-Lounge-Krakow

Konrad `@ktosopl` Malawski

typesafe.com

geecon.org

Java.pl / KrakowScala.pl

sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl

meetup.com/Lambda-Lounge-Krakow

Konrad `@ktosopl` Malawski

typesafe.com

geecon.org

Java.pl / KrakowScala.pl

sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl

meetup.com/Lambda-Lounge-Krakow

Konrad `@ktosopl` Malawski

typesafe.com

geecon.org

Java.pl / KrakowScala.pl

sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl

meetup.com/Lambda-Lounge-Krakow

Konrad `@ktosopl` Malawski

typesafe.com

geecon.org

Java.pl / KrakowScala.pl

sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl

meetup.com/Lambda-Lounge-Krakow

hAkker @

Konrad `@ktosopl` Malawski

typesafe.com

geecon.org

Java.pl / KrakowScala.pl

sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl

meetup.com/Lambda-Lounge-Krakow

akka-persistence
mainly by

Martin Krasser
!https://github.com/krasserm
!

(as contractor for Typesafe)

!

inspired by:

https://github.com/eligosource/eventsourced

dependencies

libraryDependencies ++= Seq(!

"com.typesafe.akka" %% akka-actor"
% "2.3.0",!
"com.typesafe.akka" %% "akka-persistence-experimental" % "2.3.0"!
)

Show of hands!

Show of hands!

Show of hands!

Show of hands!

sourcing styles
Command Sourcing
msg: DoThing
msg persisted before receive
imperative, do the thing
business logic change,
can be reflected in reaction
Processor

Event Sourcing

sourcing styles
Command Sourcing

Event Sourcing

msg: DoThing

msg: ThingDone

msg persisted before receive

commands converted to events,

must be manually persisted

imperative, do the thing

past tense, happened

business logic change,

can be reflected in reaction

business logic change,

wont change previous events

Processor

EventsourcedProcessor

Plain Actors

Actor
count: 0
!
!

Actor
An Actor that keeps count of messages it processed

count: 0
!
!

Actor
An Actor that keeps count of messages it processed

count: 0
!
!

Actor
An Actor that keeps count of messages it processed
Lets send 2 messages to it

count: 0
!
!

Actor
An Actor that keeps count of messages it processed

count: 0
!
!

Lets send 2 messages to it

(its commands)

Actor
!
!
class Counter extends Actor {!
var count = 0!

def receive = {!
case _ => count += 1!
}!
}

Actor

count: 0
!
!

Actor

count: 0
!
!

Actor

count: 1
!
!

Actor

count: 1
!
!

crash!

Actor

crash!

Actor
restart

Actor
restart

count: 0
!
!

Actor
restarted

count: 0
!
!

Actor
restarted

count: 1
!
!

Actor
restarted

count: 1
!
!

Actor
restarted

count: 1
!
!

wrong!

expected count == 2!

Processor

Processor
Journal

(DB)

!
!
!

var count = 0
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 0
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 0
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 0
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 1
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 1
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 1
!
def processorId = a
!

crash!

Processor
Journal

(DB)

!
!
!

Processor
Journal

(DB)

!
!
!

restart

Processor
Journal

(DB)

!
!
!

restart
var count = 0
!
def processorId = a
!

Processor
Journal

(DB)

!

restart
var count = 0
!
def processorId = a
!

!
!

replay!

Processor
Journal

(DB)

!
!
!

var count = 0
!
def processorId = a
!

Processor
Journal

(DB)

!

var count = 0
!
def processorId = a
!

!
!

replay!

Processor
Journal

(DB)

!
!
!

var count = 1
!
def processorId = a
!

Processor
Journal

(DB)

!

var count = 1
!
def processorId = a
!

!
!

replay!

Processor
Journal

(DB)

!
!
!

var count = 1
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 2
!
def processorId = a
!

Processor
Journal

(DB)

!
!
!

var count = 2
!
def processorId = a
!

yay!

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!
}!
}

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!
}!
}

counter ! Persistent(payload)

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!
}!
}

counter ! Persistent(payload)

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!
}!
}

counter ! Persistent(payload)

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!
}!
} persisted!
is already

counter ! Persistent(payload)

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!
sequenceNr
}!
(generated by akka)

} persisted!
is already

counter ! Persistent(payload)

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case notPersisted =>!
// will not replay this msg!!
count += 1!
}!
}

counter ! payload

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case notPersisted =>!
// will not replay this msg!!
count += 1!
}!
}

wont persist
counter ! payload

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case notPersisted =>!
// will not replay this msg!!
count += 1!
}!
}

wont persist
counter ! payload

wont replay

Processor
import akka.persistence._!
!
class CounterProcessor extends Processor {!
var count = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// payload already persisted!
count += 1!

!
case notPersistentMsg =>!
// msg not persisted - like in normal Actor!
count += 1!
}!
}

Processor
Upsides

Persistent Command Sourcing out of the box

Pretty simple, persist handled for you

Once you get the msg, its persisted

Pluggable Journals (HBase, Cassandra, Mongo, )

Can replay to given seqNr (post-mortem etc!)

Processor
Downsides

Exposes Persistent() to Actors who talk to you

No room for validation before persisting

Theres one Model, we act on the incoming msg

Lower throughput than plain Actor (limited by DB)

EventsourcedProcessor

EventsourcedProcessor

(longer name == more flexibility)
;-)

super quick domain modeling!

super quick domain modeling!

Commands - input from user, send emails, not persisted
sealed trait Command!
case class ManyCommand(nums: List[Int]) extends Command

super quick domain modeling!

Commands - input from user, send emails, not persisted
sealed trait Command!
case class ManyCommand(nums: List[Int]) extends Command

Events - business events emitted by the processor, persisted

sealed trait Event!
case class AddOneEvent(num: Int) extends Event!

super quick domain modeling!

Commands - input from user, send emails, not persisted
sealed trait Command!
case class ManyCommand(nums: List[Int]) extends Command

Events - business events emitted by the processor, persisted

sealed trait Event!
case class AddOneEvent(num: Int) extends Event!

State - internal actor state, kept in var

case class State(count: Int) {!
def updated(more: Int) = State(count + more)!
}

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

Command

!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

Generate
Events

!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

E1

!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

E1

Event
!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

E1
!
!

Journal

EventsourcedProcessor
var count = 0

C1

!
def processorId = a
!

ACK
persisted
E1
!
!

Journal

EventsourcedProcessor
var count = 1

C1

!
def processorId = a
!

E1

E1
!
!

Journal

EventsourcedProcessor
var count = 1

C1

!
def processorId = a
!

E1

Apply
event
E1
!
!

Journal

EventsourcedProcessor
class MultiCounter extends EventsourcedProcessor {!
!
var state = State(count = 0)!
!
def updateState(e: Event): State = {!
case event: AddOneEvent => state.updated(event.num)!
}!
!
// API:!
!
def receiveCommand = ??? // TODO!
!
def receiveRecover = ??? // TODO!
!
}!

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
persist(AddOneEvent(command)) { state = updateState(_) }!
}

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
persist(AddOneEvent(command)) { state = updateState(_) }!
}

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
persist(AddOneEvent(command)) { state = updateState(_) }!
}

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
async
persist
persist(AddOneEvent(command)) { state = updateState(_) }!
} that event

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
async
persist
persist(AddOneEvent(command))
{ state
async called
after = updateState(_) }!
} that event
successful persist

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
async
persist
persist(AddOneEvent(command))
{ state
async called
after = updateState(_) }!
} that event
successful persist

It is guaranteed that no new commands will be received by a processor

between a call to `persist` and the execution of its `handler`.

EventsourcedProcessor
def receiveCommand = {!
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }!
!
case command: Command =>!
persist(AddOneEvent(command)) { state = updateState(_) }!
}

EventsourcedProcessor
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }

EventsourcedProcessor
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }

style fix

EventsourcedProcessor
case ManyCommand(many) =>!
for (event <- many map AddOneEvent)!
persist(event) { state = updateState(_) }

style fix

case ManyCommand(many) =>!

persist(many map AddOneEvent) { state = updateState(_) }

Eventsourced, recovery

/** MUST NOT SIDE-EFFECT! */!

def receiveRecover = {!
case replayedEvent: Event => !
updateState(replayedEvent)!
}

Eventsourced, snapshots

Eventsourced, snapshots
/** MUST NOT SIDE-EFFECT! */!
def receiveRecover = {!
case SnapshotOffer(meta, snapshot: State) => !
this.state = state!
!
case replayedEvent: Event => !
updateState(replayedEvent)!
}

Eventsourced, snapshots
/** MUST NOT SIDE-EFFECT! */!
def receiveRecover = {!
case SnapshotOffer(meta, snapshot: State) => !
this.state = state!
!
case replayedEvent: Event => !
updateState(replayedEvent)!
}

Eventsourced, snapshots
/** MUST NOT SIDE-EFFECT! */!
def receiveRecover = {!
case SnapshotOffer(meta, snapshot: State) => !
this.state = state!
!
case replayedEvent: Event => !
snapshot!?
updateState(replayedEvent)!
}
how?

Eventsourced, snapshots
/** MUST NOT SIDE-EFFECT! */!
def receiveRecover = {!
case SnapshotOffer(meta, snapshot: State) => !
this.state = state!
!
case replayedEvent: Event => !
snapshot!?
updateState(replayedEvent)!
}
how?

def receiveCommand = {!
case command: Command =>!
saveSnapshot(state) // async!!
}

Snapshots

Snapshots

(in SnapshotStore)

Snapshots
sum of states
E1

E2

E3

E4

E5

E6

! E7

Journal

E8

Snapshots
E1

E2

E3

E4

state until [E8]

E5

E6

! E7

Journal

E8

S8

Snapshots
E1

E2

E3

E4

state until [E8]

E5

E6

! E7

E8

Journal

S8
snapshot!

!
!

Snapshot Store

Snapshots
E1

E2

E3

E4

state until [E8]

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

S8

Snapshots
E1

E2

E3

E4

state until [E8]

E5

E6

! E7

E8

Journal

S8

S8
crash!

!
!

Snapshot Store

Snapshots
E1

E2

E3

E4

E5

E6

! E7

E8

Journal

S8

crash!

!
!

Snapshot Store

Snapshots
restart!

E1

E2

E3

E4

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

bring me up-to-date!

Snapshots
restart!
replay!

E1

E2

E3

E4

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

bring me up-to-date!

Snapshots
restart!
replay!

E1

E2

E3

E4

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

bring me up-to-date!

Snapshots
restart!
replay!

E1

E2

E3

E4

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

bring me up-to-date!

Snapshots
restart!
replay!

E1

E2

E3

E4

bring me up-to-date!

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

S8

Snapshots
restart!
replay!

E1

E2

E3

E4

state until [E8]

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

S8

Snapshots
E1

E2

E3

E4

state until [E8]

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

S8

Snapshots
We could delete these!
E1

E2

E3

E4

state until [E8]

E5

E6

! E7

E8

Journal

S8

!
!

Snapshot Store

S8

Snapshots, save
trait MySummer extends Processor {!
var nums: List[Int]!
var total: Int!
!
def receive = {!
case "snap"
=> saveSnapshot(total)!
case SaveSnapshotSuccess(metadata)
=> // ...!
case SaveSnapshotFailure(metadata, reason) => // ...!
}!
}!

Snapshots, save
trait MySummer extends Processor {!
var nums: List[Int]!
Async!
var total: Int!
!
def receive = {!
case "snap"
=> saveSnapshot(total)!
case SaveSnapshotSuccess(metadata)
=> // ...!
case SaveSnapshotFailure(metadata, reason) => // ...!
}!
}!

Snapshots, success
trait MySummer extends Processor {!
var nums: List[Int]!
var total: Int!
!
def receive = {!
case "snap"
=> saveSnapshot(total)!
case SaveSnapshotSuccess(metadata)
=> // ...!
case SaveSnapshotFailure(metadata, reason) => // ...!
}!
}!

Snapshots, success
trait MySummer extends Processor {!
var nums: List[Int]!
var total: Int!
!
def receive = {!
case "snap"
=> saveSnapshot(total)!
case SaveSnapshotSuccess(metadata)
=> // ...!
case SaveSnapshotFailure(metadata, reason) => // ...!
}!
}!
final case class SnapshotMetadata(!
processorId: String, sequenceNr: Long, !
timestamp: Long)

Snapshots, success
trait MySummer extends Processor {!
var nums: List[Int]!
var total: Int!
!
def receive = {!
case "snap"
=> saveSnapshot(total)!
case SaveSnapshotSuccess(metadata)
=> // ...!
case SaveSnapshotFailure(metadata, reason) => // ...!
}!
}!

Snapshot Recovery
class Counter extends Processor {!
var total = 0!
!
def receive = {!
case SnapshotOffer(metadata, snap: Int) => !
total = snap!
!
case Persistent(payload, sequenceNr)
}!
}

=> // ...!

Snapshots
Upsides

Simple!
Faster recovery (!)
Allows to delete older events

known state at point in time

Snapshots
Downsides

More logic to write

Maybe not needed if events replay fast enough

Possibly yet another database to pick

snapshots are different than events, may be big!

Views

Views
Journal

(DB)

!
!
!

Processor
!

def processorId = a

Views
Journal

(DB)

!

Processor
!

def processorId = a

!
!

View
!

def processorId = a

!
!
!

Views
Journal

(DB)

Processor
!

def processorId = a

!
!

pooling

View
!

def processorId = a

!
!
!

Views
Journal

(DB)

Processor
!

def processorId = a

!
!
!

pooling
!

pooling

View
!

def processorId = a

!
!
!

View
!

def processorId = a

!
!
!

Views
Journal

(DB)

Processor
!

def processorId = a

!
!
!

different ActorPath,
same processorId
!

pooling
!

pooling

View
!

def processorId = a

!
!
!

View
!

def processorId = a

!
!
!

View
class DoublingCounterProcessor extends View {!
var state = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// state += 2 * payload !

!
}!
}

Akka Persistence Plugins

Akka Persistence Plugins

Plugins are Community maintained!
(not sure how production ready)

http://akka.io/community/#journal_plugins

Akka Persistence Plugins

Plugins are Community maintained!
(not sure how production ready)

http://akka.io/community/#journal_plugins

Journals - Cassandra, HBase, Mongo

Akka Persistence Plugins

Plugins are Community maintained!
(not sure how production ready)

http://akka.io/community/#journal_plugins

Journals - Cassandra, HBase, Mongo

SnapshotStores - Cassandra, HDFS, HBase, Mongo

SnapshotStore

http://akka.io/community/#journal_plugins

Plugins!

Links
Official docs: http://doc.akka.io/docs/akka/2.3.0/scala/persistence.html

Patriks Slides & Webinar: http://www.slideshare.net/patriknw/akkapersistence-webinar

Papers:

Sagas http://www.cs.cornell.edu/andru/cs711/2002fa/reading/
sagas.pdf

Life beyond Distributed Transactions: http://www-db.cs.wisc.edu/

cidr/cidr2007/papers/cidr07p15.pdf

Pics:

http://misaspuppy.deviantart.com/art/Messenger-s-Cutie-Mark-AFlying-Envelope-291040459

Mailing List
groups.google.com/forum/#!forum/akka-user

Links
Eric Evans - the DDD book

http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215

!
!
!
!
!
!

Vaughn Vernons Book and Talk

http://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon/dp/0321834577

video: https://skillsmatter.com/skillscasts/4698-reactive-ddd-with-scala-and-akka

!
!
!

Dziki!

Thanks!



!
!

ping me:
ktoso @ typesafe.com
twitter: ktosopl

github:
ktoso

blog:
project13.pl

team blog: letitcrash.com

Scalar 2014 @ Warsaw, PL