Você está na página 1de 75

Redis: Persistence Power

Nick Quaranto / @qrush / nick@quaran.to

Tuesday, August 10, 2010


What is Redis?

“advanced key-value store”

REmote DIctionary Server

data structures server

Tuesday, August 10, 2010


YOUR APP

Tuesday, August 10, 2010


YOUR APP

REDIS

Tuesday, August 10, 2010


the basics

persist data as you think of it

in memory, sync to disk in background

ridiculously fast

master-slave replication

keys = strings, value = data structures

Tuesday, August 10, 2010


http://try.redis-db.com

Tuesday, August 10, 2010


use it: redis-cli

% ./redis-cli SET user:1:name qrush


OK

% ./redis-cli GET user:1:name


"qrush"

Tuesday, August 10, 2010


use it: redis-rb
% gem install redis
% irb -rubygems -rredis

>> $redis = Redis.new


=> #<Redis client v2.0.3 connected...

>> $redis.set "user:1:name", "qrush"


=> "OK"

>> $redis.get "user:1:name"


=> "qrush"
Tuesday, August 10, 2010
FEATURE
SWITCHES
STRINGS

based on
http://github.com/blog/677
http://github.com/bvandenbos/redis_feature_control
Tuesday, August 10, 2010
Rediswitch.features << :super_secret
Rediswitch.features << :payment_gateway
Rediswitch.features << :twitter

if Rediswitch.enabled?(:twitter)
# post to twitter
else
# failwhale ahoy!
end

Tuesday, August 10, 2010


begin
# take some money
rescue PaymentGateway::TotallyDown => ohno
Rediswitch.disable(:payment_gateway)
# notify the troops
end

Tuesday, August 10, 2010


class Rediswitch
def self.enabled?(feature)
$redis.exists(feature)
end

def self.enable(feature)
$redis.incr(feature)
end

def self.disable(feature)
$redis.del(feature)
end
end

Tuesday, August 10, 2010


feature switch lessons

the real win: no-deploy configuration

fast enough to be transparent

next step: separate users into buckets with sets

http://github.com/jamesgolick/rollout

Tuesday, August 10, 2010


RATE
LIMITER
STRINGS

soon to be in place at
http://hoptoadapp.com
Tuesday, August 10, 2010
class Choker
def restrict?
track
count_for > 60
end
end

Tuesday, August 10, 2010


class Choker
def count_for
$memcache.get(key, true).to_i
end
end

Tuesday, August 10, 2010


class Choker
def track
if !$memcache.get(key, true)
$memcache.add(key, "0", 1.minute.from_now, true)
end
$memcache.incr(key)
end
end

Tuesday, August 10, 2010


class Choker
def track
if !$redis.exists(key)
$redis.setex(key, 60, 0)
end
$redis.incr(key)
end
end

Tuesday, August 10, 2010


class Choker
def count_for
$redis.get(key).to_i
end
end

Tuesday, August 10, 2010


rate limiter lessons

expire semantics are changing in redis 2.2

benchmark the crap out of it

could use a sorted set instead of strings

Tuesday, August 10, 2010


API USAGE
LOGGING
STRINGS
SORTED SETS

based off
http://www.production-hacks.com/2010/07/10/redis-api-access-logger/

Tuesday, August 10, 2010


# one way to do it
class ActionHit < ActiveRecord::Base
# t.string :controller
# t.string :action
# t.integer :counter
end

class UserHit < ActiveRecord::Base


# t.string :controller_action
# t.integer :user_id
# t.integer :counter
end

Tuesday, August 10, 2010


# for all controllers
{
"statuses#update" => 1410,
"users#create" => 931,
"home#index" => 2936
}

# users hitting an action


{
"101" => 42,
"102" => 13,
"103" => 34
}

Tuesday, August 10, 2010


class StatusesController < ApplicationController
def update
$redis.incr "statuses#update"
$redis.incr "statuses#update:#{user.id}"
end
end

Tuesday, August 10, 2010


class StatusesController < ApplicationController
def update
key = "statuses#update"
$redis.zincrby "actions", 1, key
$redis.zincrby "users:#{key}", 1, user.id
end
end

Tuesday, August 10, 2010


# hits for a specific user
>> $redis.zscore "users:statuses#update", 1001
=> 42

# list all the controller actions, sorted


>> $redis.zrevrange "actions", 0, -1,
:with_scores => true

=> ["home#index", "2936",


"statuses#update", "1410",
"users#create", "931"]

Tuesday, August 10, 2010


api usage logging
lessons

sorted set = high score list

bad at historical usage, trends

good for a simple heartbeat or pulse

Tuesday, August 10, 2010


JOB QUEUE
LISTS

based on
http://github.com/defunkt/resque
Tuesday, August 10, 2010
class Staple
@queue = :default

def self.perform(post_id, tempfile)


# complex image resizing, cropping
end
end

Tuesday, August 10, 2010


class Post < ActiveRecord::Base
after_save :process_with_stapler

def process_with_stapler
Resque.enqueue(Staple, self.id, @tempfile)
end
end

Tuesday, August 10, 2010


module Resque
extend self

def push(queue, item)


redis.rpush "q:#{queue}", encode(item)
end

def pop(queue)
decode redis.lpop("q:#{queue}")
end
end

Tuesday, August 10, 2010


class Resque::Worker
def work
loop do
if job = Resque.pop(queue)
job.perform
else
sleep 5
end
end
end
end

Tuesday, August 10, 2010


module Resque
extend self

def bpop(queue)
decode redis.blpop("q:#{queue}")
end
end

Tuesday, August 10, 2010


class Resque::Worker
def work
loop do
job = Resque.bpop(queue)
job.perform
end
end
end

Tuesday, August 10, 2010


job queue lessons

guaranteed atomic actions, no row locking

blocking commands simplify daemons

many more queue commands in redis itself!

Tuesday, August 10, 2010


GLOBAL
ERRORS
SETS
MULTI/EXEC

a new feature at
http://hoptoadapp.com
Tuesday, August 10, 2010
# text :globals, :default => '', :null => false

class Project < ActiveRecord::Base


def has_global?(name)
@globals ||= globals.gsub(/,/,' ').split
@globals.include?(name)
end
end

Tuesday, August 10, 2010


# MORE TABLES!!!!

class Global < ActiveRecord::Base


belongs_to :project
end

class Project < ActiveRecord::Base


has_many :globals
end

Tuesday, August 10, 2010


# Project#global_errors

["MySQL::Error",
"MemCache::Error",
"Net::HTTPFatalError"]

Tuesday, August 10, 2010


class Project < ActiveRecord::Base
def global_key
"project-#{id}-globals"
end

def has_global?(name)
$redis.sismember(global_key, name)
end
end

Tuesday, August 10, 2010


class Project < ActiveRecord::Base
after_save :save_globals

def save_globals
$redis.del global_key
@globals.each do |g|
$redis.sadd global_key, g
end
end
end

Tuesday, August 10, 2010


SISMEMBER SISMEMBER

[Mysql::Error, MemCache::Error, Net::HTTPFatalError]

Tuesday, August 10, 2010


DEL

[]

Tuesday, August 10, 2010


DEL

SADD

[Mysql::Error]

Tuesday, August 10, 2010


DEL

SADD

SADD

[Mysql::Error, OpenURI::HTTPError]

Tuesday, August 10, 2010


DEL

SISMEMBER

SADD

SADD

[Mysql::Error, OpenURI::HTTPError] []

Tuesday, August 10, 2010


MULTI

DEL

SADD
SISMEMBER
SADD
EXEC

[Mysql::Error, OpenURI::HTTPError]

Tuesday, August 10, 2010


class Project < ActiveRecord::Base
after_save :save_globals

def save_globals
$redis.multi do
$redis.del global_key
@globals.each do |g|
$redis.sadd global_key, g
end
end
end
end

Tuesday, August 10, 2010


global error lessons

avoid joins for simple data

consider race conditions

use append-only file (AOF)

Tuesday, August 10, 2010


MULTIPLAYER
NOTEPAD
PUB/SUB

based on
http://github.com/laktek/realie
Tuesday, August 10, 2010
# usage: ruby pub.rb room username

data = {"user" => ARGV[1]}


loop do
msg = STDIN.gets
$redis.publish ARGV[0],
data.merge('msg' => msg.strip).to_json
end

Tuesday, August 10, 2010


# sub.rb

$redis = Redis.new(:timeout => 0)


$redis.subscribe('rubyonrails', 'rubymidwest') do |on|
on.message do |room, msg|
data = JSON.parse(msg)
puts "##{room} - [#{data['user']}]: #{data['msg']}"
end
end

Tuesday, August 10, 2010


% ruby pub.rb rubymidwest qrush
i give up, i hate markdown

% ruby sub.rb
#rubymidwest - [qrush]: i give up, i hate markdown
#rubyonrails - [railsn00b]: undefined method posts_path? wtf?
#rubymidwest - [turbage]: seriously.

Tuesday, August 10, 2010


multiplayer notepad
lessons

combine with other data structures

can subscribe to channels via patterns

concurrency in ruby is hard

use eventmachine! (or node.js)

Tuesday, August 10, 2010


more to learn

know your data! (via @antirez)

command reference on the wiki

active IRC, mailing list

Tuesday, August 10, 2010


AKASENTAI.com
redis in the cloud
Tuesday, August 10, 2010
Thanks!
http://redis.io @qrush
http://rediscookbook.com
http://scr.bi/redispower
Tuesday, August 10, 2010
BONUS ROUND!
I prepared way too many examples. Jackpot!

Tuesday, August 10, 2010


URL
SHORTENER
STRINGS

based on
http://github.com/mattmatt/relink
Tuesday, August 10, 2010
require 'sinatra'
require 'redis_url'

post '/' do
RedisUrl.new(params[:url]).save
end

Tuesday, August 10, 2010


class RedisUrl
attr_accessor :url, :id

def initialize(url)
@url = url
@id = seed # unique string algorithm
end

def save
$redis.set("relink.url|#{@id}", @url)
$redis.set("relink.url.rev|#{@url}", @id)
end
end

Tuesday, August 10, 2010


get %r{/(.+)} do |url|
u = RedisUrl.find(url)
if u
u.clicked
redirect u.url
else
status 404
end
end

Tuesday, August 10, 2010


class RedisUrl
def self.find(id)
u = $redis.get("relink.url|#{id}")
if u
redis_url = RedisUrl.new(u)
redis_url.id = id
redis_url
end
end

def clicked
$redis.incr("relink.url.clicks|#{@id}")
end
end

Tuesday, August 10, 2010


url shortener lessons

common pattern: namespacing

incr/decr assumes value is an integer

wrap behavior into ActiveRecord-like objects

next step: store URLs in a list

Tuesday, August 10, 2010


LIVE
DEBUGGING
LISTS

based on
http://github.com/quirkey/redisk
Tuesday, August 10, 2010
def after_save
begin
# make request to external service
rescue Exception => ex
logger.error "this shouldn't ever happen!"
logger.error ex
logger.error ex.backtrace
end
end

Tuesday, August 10, 2010


# config/initializers/logger.rb

require 'redisk'
path = "#{Rails.env}.log"
config.logger = Redisk::Logger.new(path)

Tuesday, August 10, 2010


class Redisk::IO
def write(string)
redis.rpush "#{name}:_list", string
end

def self.readlines(name)
redis.lrange("#{name}:_list", 0, -1)
end
end

Tuesday, August 10, 2010


live debugging lessons

enables real-time data about your system

dump serialized/marshalled data fast

run the redis instance on a different box

dive deeper: hummingbird

Tuesday, August 10, 2010


COUNTING
DOWNLOADS
STRINGS
SORTED SETS
HASHES

based on
http://github.com/rubygems/gemcutter
Tuesday, August 10, 2010
# bad idea, dude

class Download < ActiveRecord::Base


belongs_to :rubygem
end

class Rubygem < ActiveRecord::Base


has_many :downloads
end

Tuesday, August 10, 2010


class Download
def self.incr(rubygem)
$redis.incr("all")
$redis.incr(rubygem)
$redis.zincrby("today", 1, rubygem)
end
end

Tuesday, August 10, 2010


class Download
def self.rollover(version)
$redis.rename "today", "yesterday"

dls = Hash[*$redis.zrange("yesterday",
0, -1,
:with_scores => true)]

dls.each do |key, score|


$redis.hincrby key, Date.today, score
Rubygem.find_by_name(key).increment!(:downloads, score)
end
end
end

Tuesday, August 10, 2010


get "/api/v1/downloads/rails.json" do
$redis.hgetall("rails").to_json
end

# returns...
{
"2010-07-09" => 1908,
"2010-07-10" => 1032,
"2010-07-11" => 1091,
}

Tuesday, August 10, 2010


counting downloads
lessons

hybrid approach does work!

redis is really not for search

test your migration away from SQL

Tuesday, August 10, 2010