Scaling with Stambecco

David Pollak @dpp

th
Scala BASE, April 12 2010
http://stambecco.org
@goat_rodeo
 About DPP

● Wrote Beginning Scala

● Started Lift in 2007
● Started Mesa in 1991
● Coding for money, 1977
 Problems

● Cache Invalidation
● Concurrency
● Solved well by RDBMS
– Until it's not
– Then huge pain
 Scaling & Distribution == Hard

● It's a different way of coding

– References don't work
– Non-logic failure is common
– Lack of synchronicity
● Hard to simulate locally
● Like sailing the high seas
 Stambecco

● Naming: the other hard problem

● Workers (Actor-ish)
– Asynchronous/Immutable Messages
– Ask/Answer (Requests)
– Inter-worker Transactions
– Per-worker SQL store
● Not about financial transactions
 What's the Goat good for?

● Smart Caching
– Discrete entities
– Per entity cache strategies
● e.g. Social Networks
 Workers

● Typed/Type-safe
● Messages deposited in mailbox
● Worker scheduled
● Single message processed at a time
● Lifecycle: create/unfreeze/freeze/destroy
● case class Follow(who: UserId) extends UserMsg
 Async/Immutable Messages

● Message send is asynchronous

● Messages are compiler guaranteed
– Immutable
– JSON Serializable
● Timeouts on Ask/Answer & Xaction
● Sent to Worker proxy
● Messaging turtles all the way down
 Ask/Answer

● Request designates response type

● Answer Function executed
– On answer
– On timeout
● case class GetTimeline(first: Int,
count: Int) extends UserMsg with
MsgWithResponse[ListOfMessages]
 Ask/Answer Code

● for {
user <­ WorkerMgr.find(UserId(user))
frst = r.param("first").asInt openOr 0
cnt = r.param("count").asInt openOr 20
timeline <­ (OutsideOfWorker on user ask
       GetTimeline(frst,cnt) future).value
} yield timeline.asJson
 Inter-worker Transactions

● Only within Workers

● Actions: Pre/Post-Commit/Rollback
● No external messages within xaction
● Integrated with SQL
● Nested, recursive
● transaction on otherWorker complete 
{result = 99} send MyMessage(123)
 Per Worker SQL

● Private SQL Store

● Backed by H2
● Integrated with transactions
● Can also access external SQL stores
● Can also access NoSQL
 Run modes

● Test: in memory
● Single Node: H2 & Filesystem
● Multiple Node
– Messages: RabbitMQ
– Transactions: ZooKeeper
– SQL: H2 & Cassandra
● Same Worker & Client code
 Worker & Client code the same

● Run mode change, but

– No changes to worker code
– No changes to client code
● Changes to WorkerMaster
– Local vs. Remote
– H2 URL
– Message send/receive mechanics
 Why does it scale?

● Right paradigm: Async & Immutable

● Logic & Distribution: Separate
● No single point of failure
● Fail-over & re-balancing support
● Load Shedding & Message Prioritization
 Housekeeping

● Runs on Scala 2.8

● OSGi coming
● Not tied to Lift
● Apache 2.0 License
● Commercial modules: balancing, load
shedding, smart msg delay, mgt GUI, etc.
● Nope, no big-name users... yet
 Where to get the Goat

● http://stambecco.org
● @goat_rodeo

● http://groups.google.com/group/goat-rodeo/
Questions?

Thanks!!