Managing Big Data With Percona Server, XtraBackup, and Tungsten

Managing Big Data with
Percona Server,
XtraBackup, and Tungsten
Robert Hodges CEO, Continuent,

Vadim Tkachenko, Founder & CTO, Percona
Agenda
/ Introductions
/ What Is The Big Data Problem?
/ Percona Server
/ XtraBackup
/ Tungsten
/ Commercial Services and Support
© Copyright 2011 Percona and Continuent

About Percona
/ Percona is the oldest and largest independent MySQL

support and consulting company
/ Our products:
• Percona Server
• Percona XtraBackup
/ We provide consulting, support, training and
development for MySQL and LAMP

About Continuent
/ Continuent is the leading provider of data replication

and clustering for open source relational databases
/ Our Products:
• Tungsten Replicator - High-performance, MySQL replication
• Tungsten Enterprise - Commercial replication and data
management solution for MySQL and PostgreSQL
/ Our Services:
• Consulting on Tungsten plus replication and clustering in general
• Subscriptions for commercial products

What Is the Big Data
Problem?

The Big Data Problem
/ Large quantities of data

• Application performance
• Slow server starts
/ High transaction loads
• Resource contention
• SQL processing overhead
/ Distributed data
• Datasets replicated across servers and sites
• Master failover and slave promotion
• Slave lag
/ Continuous, 24x7 Operation
• Hot backup and restore
• System upgrades

Percona Server

Percona Server
/ Drop-in replacement for MySQL

/ High performance
/ Diagnostics
/ Operations / data management
/ NOSQL

High Performance
/ Tuned for multi-core multi-disks systems

/ SSD / Flash optimizations

Diagnostic
/ Advanced statistic on query execution

/ # User@Host: mailboxer[mailboxer] @ [192.168.10.165]
/ # Thread_id: 11167745 Schema: board
/ # QC_Hit: No Full_scan: No Full_join: No Tmp_table: Yes Disk_tmp_table: No
/ # Filesort: Yes Disk_filesort: No Merge_passes: 0
/ # Query_time: 0.000659 Lock_time: 0.000070 Rows_sent: 0 Rows_examined: 30 Rows_affected: 0 Rows_read: 30
/ # InnoDB_IO_r_ops: 1 InnoDB_IO_r_bytes: 16384 InnoDB_IO_r_wait: 0.028487
/ # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
/ # InnoDB_pages_distinct: 5 select count(distinct author_id) from art87.article87 force index (forum_id) where
forum_id = 240215 and thread_id = '710575'
/ Additional INFORMATION_SCHEMA tables

CLIENT_STATISTICS
INDEX_STATISTICS
INNODB_BUFFER_POOL_PAGES
INNODB_BUFFER_POOL_PAGES_BLOB
INNODB_BUFFER_POOL_PAGES_INDEX
INNODB_BUFFER_POOL_CONTENT
INNODB_INDEX_STATS
INNODB_IO_PATTERN
INNODB_RSEG
INNODB_SYS_INDEXES
INNODB_SYS_TABLES
INNODB_TABLE_STATS
PROCESSLIST
TABLE_STATISTICS
USER_STATISTICS

Operations / management
/ Export / Import InnoDB tables

• Move shard from one server to another
• Try copy 100GB user1.table1 from Server1 to Server2
/ Fast warmup
• Periodic dump and load at start content of buffer pool
• Very important for big (memory, data) slaves
/ Integration with FlashCache
• Hints for queries to ignore caching
• FlashCache is Flash/SSD cache layer over regular HDD/RAID

NOSQL
/ HandlerSocket plugin
• Access to data without SQL queries / overhead
• Wide range of operations on PK/ index
• Batch, range ( < , >, <= X <= ) access
/ Outstanding performance
• Reported 750,000 request/sec

Percona Server plans
/ Cloud support
• Scalability + performance
/ Flash/SSD
• Continue research and improve performance
/ Diagnostic
• You never can have enough statistics 

Introducing Percona
XtraBackup

Percona XtraBackup
/ OpenSource hot-backup for MySQL/InnoDB

/ Free

Designed with big data in mind
/ Non-blocking for InnoDB tables

/ Streaming directly to backup server / host
/ Incremental ( differential ) backups
/ Partial backups
/ Parallel copy
/ Minimal actions to setup slave

Non-Blocking
/ Copy InnoDB files in background

/ Server continues to process workload

Streaming
/ Tar-compatible file
• innobackupex-1.5.1 --stream=tar ./ > backup.tar
/ Compress
• innobackupex-1.5.1 --stream=tar ./ | gzip - > backup.tar.gz
/ Copy to remote host

• innobackupex-1.5.1 --stream=tar ./ | ssh vadim@desthost "cat - > /data/vol1/mysqluc/backup.tar“

Incremental/Differential
/ Differential
• Copy ONLY changes since last full backup
/ Incremental
• Copy ONLY changes since last incremental / differential

Partial Backup
/ Copy only selected databases / tables

• Databases:
user1
user2
…
user1000
• You want to copy only user567 database from server1 to server2

• Create new database or import into existing

Setup slave
/ innobackupex-1.5.1 --stream=tar /tmp/ --slave-info | ssh

user@DESTSERVER "tar xfi - -C /DESTDIR“
/ innobackupex-1.5.1 --apply-log --use-memory=2G
/DESTDIR

Percona XtraBackup plans
/ Cloud/S3
• Copy / restore data from cloud storage
/ Compression on table level
/ Improve incremental and partial backup operations

Introducing Tungsten
© Continuent 2010
2011 © Copyright 2011 Percona and Continuent
What Is Tungsten?
/ Tungsten Replicator is a fast, open-source

replication engine for MySQL
/ Tungsten Enterprise is a complete data
management solution based on replication
1. Ensure reliable, fast access to data
2. Simplify common management tasks
3. Increase performance and resource utilization
…Without changing applications

Tungsten Replicator Architecture
(Replicator configuration)
Master
replicator.
Tungsten properties
MySQL Tail binlog or Replicator
login as client (replicator)
Transport via
Transaction TCP/IP
Binlogs History Log connection
(Transactions + Metadata)
Apply using JDBC

Transaction Tungsten
History Log Replicator
(replicator)
replicator.
MySQL
Slave
properties

Tungsten Replicator: So Many Features
/ Global transaction IDs

/ Transaction filters
/ Replicate from MySQL to PostgreSQL & Oracle
/ Time-delay replication
/ Backup and restore integration
/ Cross-version replication: 5.5 -> 5.1 -> 5.0 -> 4.1
/ Row and statement replication
/ Automatic consistency checks
/ Auto-provisioning
/ Multi-master replication
/ Parallel replication
Tungsten Replicator Pipelines (Slave)
Tungsten Replicator Process
Pipeline
Stage Stage
Extractor Filters Applier Extractor Filters Applier
THL
Master THL Slave
(via network) DBMS
Transaction History Log

Tungsten Enterprise Architecture
Apache/Mod_PHP Apache/Mod_PHP
libmysqlclient.a libmysqlclient.a
Connector Connector
DBMS DBMS DBMS
Replicator Replicator Replicator

Manager Manager Manager
Slave Master Slave

Tungsten Enterprise Management
Business Cctrl Client
Rules Manager
Group
Communications
Manager Local DBMS

(Coordinator) Resources
Broadcast commands
and monitoring data Cctrl Client
Local DBMS
Resources
Manager
Cctrl Client
Local DBMS Resources

Tungsten Replicator vs. Enterprise
Feature Replicator Enterprise

Simple admin with trepctl X X
(MySQL) Row/Statement 5.0/5.1/5.5 X X
Multi-master replication X X
Parallel apply (2.0) X X
Replication to other DBMS types X X
Distributed management X
Automated failover and recovery X
Zero-downtime upgrade X
SQL routing and load-balancing X
Floating IP support X

Moving Data:
Tungsten Replicator
© Continuent 2010
2011 © Copyright 2011 Percona and Continuent
Big Data and Multi-Master Replication
/ Big data is spread across multiple servers and sites

/ Many-to-one: consolidate updates from multiple
servers into one slave
• Roll-up reporting from remote servers
/ Bi-directional: Updates from two masters propagate
to each other
• High-volume merchant apps and distributed directories
/ Cross-site/Cross-Host: Processing distributed over
multiple hosts and sites
• Sharded, multi-tenant datasets with centralized data

Multi-Master Data Consolidation
Master Pipeline Slave Pipeline
Tungsten Tungsten
MySQL Replicator Replicator
MySQL
Tungsten Tungsten
MySQL Replicator Replicator
Tungsten
MySQL Replicator
master1
MySQL
master2
MySQL Direct Pipeline,

Multiple Services
Bi-Directional Replication
Replicator Replicator
sjc1 nyc-remote sjc-remote nyc1
njc-local nyc-local
sjc2 sjc-local nyc-local nyc2
Site SJC Site NYC
Cool feature: Tungsten slaves remember

positions on remote and local masters

Big Data and Failover
/ Slave promotion is a severe problem for high-volume

applications with multiple slaves
/ Tungsten global transaction IDs eliminate binlog
hacks to reconfigure slaves
/ Tungsten epoch numbers prevent re-connection
when logs diverge after failover
/ Say goodbye to binlog hacks for failover!

Global ID and Epoch Number Concepts
Master Slave
Transfer log from master

Replicator Log Replicator Log

Seqn Epoch Transaction Backup reloadable Seqn Epoch Transaction
o on any server o
1 1 UPDATE CUST.. 1 1 UPDATE CUST..
2 1 INSERT INTO 2 1 INSERT INTO
CALL_TAB… CALL_TAB…
3 3 DELETE FROM
Backup
3 3 DELETE FROM
CALL_TAB… seqno=3 CALL_TAB…
4 3 UPDATE CUST… 4 3 UPDATE CUST…
5 3 UPDATE CALL_TAB…
Epoch numbers prevent
inconsistent slaves from
connecting

DR Site Enabling with Global IDs
Seqno 35
Epoch 1
Seqno 25 Slave
Floating IP
Master Epoch 1 (Relay)
Manager Manager
Seqno 21
Seqno 28 Epoch 1
Slave Epoch 1 Slave
Manager Backup Backup Manager
Backup Backup
Backups Backups
Main Cluster DR Cluster
Reconfiguration After Failover
Seqno 35
Epoch 1
Old (epoch number Seqno 25 Slave
Master prevents Epoch 1 (Relay)
reconnect as
Replicator slave) Replicator
Floating IP
Manager Manager
Seqno 21
New Epoch 1 Slave
Master
Seqno 29
Epoch 29
Manager Backup Backup Manager
Backup Backup
Backups Backups
Main Cluster DR Cluster
Big Data and Resource Utilization
/ Single-threaded MySQL replication suffers from

impedance mismatch between CPU and I/O for high-
volume data replication
/ Tungsten optimizes resource utilization for large-
scale data transfer
/ Pipeline parallelism - Spread work across stages
/ Service parallelism -- Spread work across different
replicators and replication services within replicators
/ Parallel apply -- Spread apply operations across
multiple threads

Parallel Slave Apply -- Internals View
Tungsten Replicator Process
Pipeline
Stage Stage
Stage
Extractor Applier
Parallel
Extractor Applier Extractor Applier Extractor Applier
Queue
Extractor Applier
Transaction
History Log
shard.list
Master THL THL file Slave
(via network) DBMS

Demo: Parallel Apply with TPC-B Tests
TPC-B TPC-B TPC-B

Test Test Test
Slave
Master
db1 db2 db3 db1 db2 db3

Future Plans
Tungsten Replicator will go fully

GPL V2 in 2011!
Initial Replicator-Only Builds

available in February
Keep Watching
www.continuent.com

Commercial Services and
Support

Continuent Product Subscriptions
/ Per-server subscriptions for Tungsten Enterprise;

pricing based on number of DBMS servers
/ Software-only -- Software license; use consulting for
support; bug-fixes are free
/ Software + Support -- Software license + 24x7
support (SLAs negotiable)

Continuent Consulting
/ Practical consulting by industry experts on any

aspect of Tungsten or replication/clustering for open
source databases
/ Hourly support contracts
• Advice, troubleshooting, bug fixes
/ Projects
• Cluster design, roll-outs, upgrades, special replication problems
/ Sponsored development
• Implementation of features large and small
/ Offered for both open source and commercial
products

Percona Software and Services
/ Consulting
• Our specialties include MySQL architecture and design,
diagnosing and solving performance problems, recovering
corrupted data, setting up high availability, clustering, and much
more.
/ Support & Maintenance
• Expert technical support for MySQL
• Insurance against bugs in MySQL
• Customized MySQL server binaries
/ Training
• Public and custom onsite
/ Software
• Percona Server with XtraDB 5.1 and 5.5 coming
• Percona XtraBackup

Conclusion
/ Big data has a unique set of challenges for MySQL

deployments
• Large volumes
• High transaction rates
• Distribution across servers and sites
• Constant availability
/ Percona Server, XtraBackup, and Tungsten address
big data problems
/ Combine Percona and Continuent for your next
project!

Contact Information
Continuent, Inc. Percona, Inc.

560 S. Winchester Blvd., Suite 500 +1-888-316-9775 or
San Jose, CA 95128 +1-208-473-2904 (USA)
Tel +1-866-998-3642 +44-208-133-0309 (UK)
Fax +1-408-668-1009 http://www.percona.com/contact/
e-mail: sales@continuent.com sales/
Continuent Web Site:

http://www.continuent.com
Percona Web Site:

http://www.percona.com

Managing Big Data With Percona Server, XtraBackup, and Tungsten

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Managing Big Data With Percona Server, XtraBackup, and Tungsten

Enviado por

Direitos autorais:

Formatos disponíveis

Managing Big Data with

Robert Hodges CEO, Continuent,

© Copyright 2011 Percona and Continuent

/ Percona is the oldest and largest independent MySQL

© Copyright 2011 Percona and Continuent

/ Continuent is the leading provider of data replication

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

/ Large quantities of data

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

/ Drop-in replacement for MySQL

© Copyright 2011 Percona and Continuent

/ Tuned for multi-core multi-disks systems

© Copyright 2011 Percona and Continuent

/ Advanced statistic on query execution

/ Additional INFORMATION_SCHEMA tables

© Copyright 2011 Percona and Continuent

/ Export / Import InnoDB tables

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

/ OpenSource hot-backup for MySQL/InnoDB

© Copyright 2011 Percona and Continuent

/ Non-blocking for InnoDB tables

© Copyright 2011 Percona and Continuent

/ Copy InnoDB files in background

© Copyright 2011 Percona and Continuent

/ Copy to remote host

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

/ Copy only selected databases / tables

• You want to copy only user567 database from server1 to server2

© Copyright 2011 Percona and Continuent

/ innobackupex-1.5.1 --stream=tar /tmp/ --slave-info | ssh

© Copyright 2011 Percona and Continuent

© Copyright 2011 Percona and Continuent

/ Tungsten Replicator is a fast, open-source

© Copyright 2011 Percona and Continuent

Apply using JDBC

© Copyright 2011 Percona and Continuent

/ Global transaction IDs

Extractor Filters Applier Extractor Filters Applier

© Copyright 2011 Percona and Continuent

DBMS DBMS DBMS

Replicator Replicator Replicator

Slave Master Slave

Manager Local DBMS

Local DBMS Resources

Feature Replicator Enterprise

© Copyright 2011 Percona and Continuent

/ Big data is spread across multiple servers and sites

© Copyright 2011 Percona and Continuent

MySQL Direct Pipeline,

Site SJC Site NYC

Cool feature: Tungsten slaves remember

© Copyright 2011 Percona and Continuent

/ Slave promotion is a severe problem for high-volume

© Copyright 2011 Percona and Continuent

Transfer log from master

Replicator Log Replicator Log

© Copyright 2011 Percona and Continuent