etdTT.27 Gallagher-etdTT.27 PDF

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved. 1

ECS ARCHITECTURE
DEEP DIVE
#EMCECS
ROADMAP INFORMATION DISCLAIMER
EMC makes no representation and undertakes no obligations with
regard to product planning information, anticipated product
characteristics, performance specifications, or anticipated release
dates (collectively, Roadmap Information).
Roadmap Information is provided by EMC as an accommodation to the
recipient solely for purposes of discussion and without intending to be
bound thereby.
Roadmap information is EMC Restricted Confidential and is provided
under the terms, conditions and restrictions defined in the EMC Non-
Disclosure Agreement in place with your organization.
#EMCECS
unstructured
@scale global
accessible many devices

web
#EMCECS
MODERN APPS ARE CHANGING
INDUSTRIES
#EMCECS
MODERN APPS ARE BREAKING
TRADITIONAL STORAGE
NOT DESIGNED FOR MODERN APPLICATIONS AND HYPERSCALE
Architecture is too complex

Locking, replication, High Availability, geo-
distribution is complex
Not Web or firewall friendly

Distributed (WAN) access is complex
Storage silos impede development

Different hardware for every data type and
access protocol
#EMCECS
NEW DEMANDS FOR HYPERSCALE
SOFTWARE-DEFINED STORAGE AND COMMODITY ARE FOUNDATIONAL
The Potential
Reduce storage costs with

7:1 Savings
commodity off-the-shelf (COTS)
components and standard, open
technologies
The Reality
Individual components provide

lower performance, lower
reliability, etc.
You need sufficient scale, the right
software, and operational expertise Storage Appliance Commodity Hardware
to provide a reliable, scalable

infrastructure #EMCECS
Copyright 2014
2015 EMC Corporation. All rights reserved. Source: IDC DSS Tracker Q4 2014 , Dec. 2014 8
WHAT DO CUSTOMERS NEED?
Customer need simple, API-

accessible, software-based
storage
that can be deployed on
commodity off-the-shelf
components
in a hyperscale architecture
#EMCECS
CUSTOMER PAIN POINTS
Help Me Give Me I Want a

Reduce Freedom of Path to
Costs Choice the Cloud
#EMCECS
ECS DESIGN PHILOSOPHY
Software
Defined Storage
#EMCECS
RACK & STACK MODEL
SCALE OUT ARCHITECTURE
#EMCECS
ECS SOFTWARE STACK
#EMCECS
GEO-DISTRIBUTED CLOUD STORAGE
ACTIVE-ACTIVE ARCHITECTURE
#EMCECS
SYSTEM OVERVIEW
User & System data persisted

/bucket
through the storage engine
/images Scalable Key-Value pair
/data Multi-protocol access
/metadata
/docs
/data
/metadata
#EMCECS
ECS CONCEPTS
Object User
Namespace
Admin
Namespace 1 Namespace 2
Replication Group
Sys Admin
VDC 1 VDC 2
Storage Pool
Storage Pool
#EMCECS
BUILD YOUR OWN ECS (DIY)
HW & OS agnostic
Self-contained
Docker containers
Timeline
Dev/Test (now)
Production (EOY 2015)
#EMCECS
DEEP DIVE
#EMCECS
Copyright
Copyright 2015
2015 EMC
EMC Corporation.
Corporation. All
All rights
rights reserved.
reserved. 18
CONSISTENCY SEMANTICS
#EMCECS
CONSISTENCY
Strong consistency
Resilient to temporary site outage
#EMCECS
LAYERS IN SYSTEM
#EMCECS
DATA PATH
#EMCECS
TEMPORARY SITE FAILOVER
#EMCECS
GLOBAL STATE
MANAGEMENT
#EMCECS
GLOBAL STATE MANAGEMENT (VNEST)
Service discovery
Distributed lock management
Paxos based coordination
#EMCECS
ECS DESIGN
Performance optimized for small and large data sets

File system on Key-Value
#EMCECS
PARTITIONING SCHEME
A logical table is shared into
Partition Table partitions, based on hashing for
Partition Owner random distribution of load.
ID Node
A partition is owned by one node at
P1 N1 a given time. A node can own many
P2 N1 partitions.
P3 N2
Node 1 Node 2
P4 N2
P1 P3
P2 P4
#EMCECS
PARTITION COMPONENTS
A partition presents a table

of key-value.
Node 1
A node that owns a
partition, does the
P1
execution of operations
that are associated with
the keys in that partition.
BTree Journal
The key-value information
is persisted on chunks
using BTree and Journal. Chunks
#EMCECS
PERFORMANCE
#EMCECS
#EMCECS
CHUNK WRITE (PARALLEL SYNC
WRITE)
Acks
Node
Buffered Writer
#EMCECS
ECS AT EMC WORLD
ECS is Everywhere at EMC World!
On The Show Floor
ECS at CTD/ETD Booth 813 and kiosks #8 and #9
Breakout Sessions
EMC ECS: ECS and Unstructured Content Strategies for Web
Scale IT
EMC ECS: ECS and VMware vCloud Air - Architecting and
Deploying an Exabyte+ Cloud
EMC ECS: ECS Architecture Deep Dive
EMC ECS: ECS Storage for 3rd Platform and What's New in
2015
HOLs (Hands-on Labs)

EMC ECS: Empowering Storage Admins for Cloud Storage
#EMCECS
ECS SOFTWARE STACK
Object HDFS ECS Portal Services

Storage Services
Licensing, Authentication, Multi-tenancy and Provisioning

Services
ECS Provisioning Services
Cluster manager, health, configuration management upgrade,

installer, alerting
Fabric
OS + networking + tools
Infrastructure
#EMCECS
TEMPORARY SITE FAILOVER
Bucket ownership transferred to site-2
Successful operations
(secondary)
Failed Operations
Create Object x Create bucket Read / Write objects succeed
List Bucket x Create User
Edit Bucket x Create Namespace
List Buckets in Successful operations
namespace on site-1 Read and Update
Read and Update Object Failed Operations
Objects created on
created on site-1 & x Create Bucket
site-1 & site-2
site-2 x Create User
List Bucket
x Create Namespace
Create Objects
Site-1
Edit Bucket
Bucket A Site-2
Bucket A
Read and Update
Objects created on Failed Operations
site-1 & site-2 x Create Bucket
List Bucket x Create User
Create Objects x Create Namespace
Edit Bucket
Bucket A
Site-3 #EMCECS
TEMPORARY SITE FAILURE (EXISTING
BEHAVIOR) Bucket A is owned by Site-1 (primary)
Outage leads to temp site failure of Site-1
ECS detects the temp site outage
Create Object Failed Operations Successful Failed Operations
List Bucket List Bucket created on operations x Create bucket,
Read and Update Object site-2 Read and Update user, namespace
created on site-1 Read and Update Object Object created on x Edit bucket
created on site-2 site-2 x Create Object
Create bucket x List Bucket
Edit bucket x Read and Update
Object created on
site-1
Site-1 Site-2
Bucket A Bucket A
#EMCECS
TEMPORARY SITE FAILOVER 2 SITE SCENARIO
(secondary)
Read / Write objects succeed
Successful operations Failed Operations Successful operations Failed Operations

Create Object x Create bucket Read and Update x Create Bucket
List Bucket x Create Namespace Objects created on x Create Namespace
Edit Bucket x Create User site-1 & site-2 x Create User
List Buckets in List Bucket
namespace on site-1 Create Objects
Read and Update Object Edit Bucket
created on site-1 &
site-2
Site-2
Site-1
Bucket A Bucket A
#EMCECS
RECONCILIATION (RESYNC)

Objects created at site-1 and site-2 will be
retained
Objects edited at site-2 will be the copy
retained
Creating buckets, user, namespace succeed
Create Object Successful operations
List Bucket Read and Update Object
Read and Update Object created on site-1 &
created on site-1 & site-2
site-2 Create / edit Bucket
Create / edit Bucket List Bucket
Site-2
Site-1
Bucket A Bucket A
#EMCECS

etdTT.27 Gallagher-etdTT.27 PDF

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

etdTT.27 Gallagher-etdTT.27 PDF

Enviado por

Direitos autorais:

Formatos disponíveis

#EMCECS

Copyright 2015 EMC Corporation. All rights reserved. 1

accessible many devices

Architecture is too complex

Not Web or firewall friendly

Storage silos impede development

Reduce storage costs with

Individual components provide

to provide a reliable, scalable

Customer need simple, API-

Help Me Give Me I Want a

User & System data persisted

Performance optimized for small and large data sets

A partition presents a table

HOLs (Hands-on Labs)

Object HDFS ECS Portal Services

Licensing, Authentication, Multi-tenancy and Provisioning

Cluster manager, health, configuration management upgrade,

Successful operations Failed Operations Successful operations Failed Operations

Você também pode gostar