Testing SAP:

Modern Testing
Methodologies
Ethan Jewett

But first!

Whats the problem?

Enterprise project trade-offs

Software development
trade-offs

Client requests
Build Faster
And produce more Value
Cheaper
with less Risk

Remember

Are there hidden

assumptions?
Yes.
Methodology
People and Culture
Technology

Any other assumptions?

The topic
Over the next hour and a half, well talk
about how testing methodologies,
techniques, and technologies can help us
Change these underlying assumptions.
Enable modern project methodologies.
Control the basic project trade-off.

The focus

The goal
More value
Faster
With less risk
(and more fun)

For example

V-Model methodology

http://en.wikipedia.org/wiki/File:V-model.JP

More parallel, less repetition

http://en.wikipedia.org/wiki/File:V-model.JP

Faster

http://en.wikipedia.org/wiki/File:V-model.JP

More iterative

http://en.wikipedia.org/wiki/File:V-model.JP

How do we get there?

Modern methodology
Modernquality assurance and testing
Leading practice in custom
development, open source, and agile
projects
Test driven development (TDD), behavior driven
development (BDD), readable tests, executable
requirements, continuous integration

Overview of the SAP test automation

technologies
Road we're going to take through testing
SAP

Leading Practice

Context
Questions and problems
Approaches
Tools

Leading Practice

Context
Questions and problems
Approaches
Tools

Context
The context of modern software
development is above all open,
agile and decentralized.
Open source projects are usually the
furthest along this path, and as such
these projects use (or are forced to
invent) tools that fit their processes.

Openness
Requirements, specifications, issues
reports, test cases, source code, and
project tools and metrics are readily
open and available to all participants
in the project.
(Not necessarily the same as open
source.)
Open Collaboration within Corporations Using Software Forges http://www.riehle.org/publications/2009/open-collaboration-within-corporations-using

Agility
The values of the Agile Manifesto:
Individuals and interactions over
processes and tools
Working software over comprehensive
documentation
Customer collaboration over contract
negotiation
Responding to change over following a
plan
http://agilemanifesto.org/

Decentralization
Open-source projects are decentralized
by nature, but most organizations
today have some element of
decentralization
Challenges:
Quality control
Communication
Maintaining commitment

Leading Practice

Context
Questions and problems
Approaches
Tools

Questions and problems

How do we
enforce
disciplined
Do we use unit,
testing?
functional,
integration, or
acceptance
testing? Or all of
them?
How do we ensure
that the build
complies with the
tests?

How do we track
100s or 1000s of
bugs (including
How do we duplicates)?
make tests
How does
relevant?
testing
integrate with
issue
How do we
tracking?
determine the
Manual or
cause of test
automatic
failure?
testing?

Leading Practice

Context
Questions and problems
Approaches
Tools

Approaches
Test Coverage
Test Automation
Test Driven Development
Behavior Driven Development
Spec as Test (the test/spec
convergence)
Exploratory Testing
Continuous Integration

Test coverage
The percentage of code or
development that is tested.
In code, this might be measured by
determining if every branch of code
can result in a failing test condition
(Branch coverage)

Wikipedia http://en.wikipedia.org/wiki/Code_coverage

Test automation
Accepted as gospel modern dev
communities
Regardless of how good they are at
testing, accept that they should
automate as much as possible
Can lead to ignoring non-automatable
testing

In the SAP world we havent even

accepted that full test coverage is
desirable, much less automation

Test automation pushback http://railspikes.com/2008/7/11/testing-is-overrated

http://michaelfeathers.typepad.com/michael_feathers_blog/200
8/06/the-flawed-theo.html
Automated testing story on SDN (never completed) https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4103

A note on the following

slides
Im using bowling as an example because
Im copying from the internet. I dont
know much about bowling, so the
examples are probably wrong. My
assumption is that a game is usually 10
frames of two balls per frame. If the last
frame is a spare or a strike, you get at
most two additional frames.
Or something like that.

Test Driven Development

(TDD)

Technical specification

TDD Example
require test/unit
require test/unit/assertions
require bowling
class TC_Bowling <
Test::Unit::TestCase
def setup
bowling = Bowling.new
end
def gutter_game
20.times { bowling.hit(0) }
assert bowling.score == 0
end
end

Behavior Driven Development

(BDD)
Focus on the external behavior of the
code
Abstract away implementation
details
Additionally, BDD libraries tend to
codify the best practices of unit
testing and TDD

BDD example
require spec
require bowling
describe Bowling do
it "should score 0 for gutter
game" do
bowling = Bowling.new
20.times { bowling.hit(0) }
bowling.score.should == 0
end
end

Spec as Test or writing features

Recently, the Ruby community has
begun developing testing frameworks
that are even closer to natural
language.
Cucumber

These frameworks wrap TDD and

BDD frameworks and allow for
business-writable, businessreadable, executable test scripts.

Feature example
Feature (visible to the user)
Scenario: Gutter game
Given I have bowled 20 balls
and I have knocked over 0 pins per ball
When I check the score
Then I should have 0 points

Implementation (not visible to user)

Feature example cont.

Note that we can now implement
many more tests with no more
code/implementation:
Scenario:
Perfect game
Given I have bowled 12 balls
and I have knocked over 10 pins per ball
When I check the score
Then I should have 300 points
Scenario: Bad game
Given I have bowled 20 balls
and I have knocked over 2 pins per ball
When I check the score
Then I should have 40 points
Scenario: Lots of spares

Side-by-side
TDD
require test/unit
require test/unit/assertions
require bowling

BDD
require spec/expectations
require bowling
describe Bowling do

class TC_Bowling <

Test::Unit::TestCase
def setup
bowling = Bowling.new
end
def gutter_game
20.times { bowling.hit(0) }
assert bowling.score == 0
end
end

it "should score 0 for gutter

game" do
bowling = Bowling.new
20.times { bowling.hit(0) }
bowling.score.should == 0
end
end

Spec as Test
Scenario: Gutter game
Given I have bowled 20 balls
and I have knocked over 0 pins
per ball
When I view the score
Then I should have 0 points

TDD, BDD, and Test-as-Spec

References
Test Driven Design/Development
http://en.wikipedia.org/wiki/Test-driven
_development
http://www.agiledata.org/essays/tdd.html

Behavior Driven Development

http://behaviour-driven.org/

Spec as test
Cucumber - http://cukes.info/
Rspec - http://rspec.info/
http://www.pragprog.com/titles/achbd/th
e-rspec-book

Exploratory Testing
The practice of trying to break things
Career security for klutzes

Exploratory testing appears informal,

but can be structured and is a very
important aspect of software testing.
Probably the most neglected form of
testing in open source projects

http://www.kohl.ca/blog/archives/000185.html
http://www.kohl.ca/articles/ExploratoryTesting_Musicof

Continuous Integration
The practice of automating not only
your tests but your full commit-buildtest cycle
1. A commit new or changed code
2. Triggers a full system build
3. And an execution of the entire test
suite
Cruisecontrol.rb http://rubyforge.org/projects/cruisecontrolrb
http://cruisecontrolrb.thoughtworks.com/
Hudson https://hudson.dev.java.net/
http://www.softwarebloat.com/2008/11/19/continuous-integratio
n-blueprints-how-to-build-an-army-of-killer-robots-with-hudso

Leading Practice

Context
Questions and problems
Approaches
Tools

Tools
Source version control (svn, git)
Testing libraries (rspec, cucumber,
junit, ABAPunit, jspec)
Continuous Integration tools
(cruisecontrol.rb, Hudson)
Issue tracking (Sourceforge, trac,
Lighthouse, Google Code)
Agile methodologies (Scrum, XP,
etc.)

Modern methodology
Modernquality assurance and testing
Leading practice in custom development,
open source, and agile projects
Test driven development (TDD), behavior driven
development (BDD), readable tests, executable
requirements, continuous integration

Overview of the SAP test automation

technologies
Road we're going to take through testing
SAP

The SAP World

Manual
Unit testing
ABAP Unit
Java

Functional testing
eCATT

ABAP Unit
Modeled on Java Unit
Excellent (if a bit unwieldy) for
traditional unit test automation or
classes
Works well for TDD

ABAP Objects (object-oriented ABAP)

TDD Demo ZBOWLING &
ZBOWLING_TEST
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/a
2/8a1b602e858645b8aac1559b638ea4/frameset.htm

Non-SAP
We can use just about anything via
RFCs
For Web Dynpros, BSPs, or Portal
applications, there are a lot of
options using web-drivers that can
automatically drive browsers
Address in depth in future sessions

Modern methodology
Modernquality assurance and testing
Leading practice in custom development,
open source, and agile projects
Test driven development (TDD), behavior driven
development (BDD), readable tests, executable
requirements, continuous integration

Overview of the SAP test automation

technologies
Road we're going to take through
testing SAP

The Road to the Future

Developing techniques to support
agile, open, decentralized
development in an SAP landscape
Using SAP tools and 3rd party tools
Shorter, more focused and hands-on
sessions

Start with this

http://en.wikipedia.org/wiki/File:V-model.JP

Get more parallel, less

repetitive

Spec = Test

http://en.wikipedia.org/wiki/File:V-model.JP

Get faster

Test
Automati
on

http://en.wikipedia.org/wiki/File:V-model.JP

Get iterative
Continuo
us

Integrati
on
http://en.wikipedia.org/wiki/File:V-model.JP

And end up here

(or somewhere similar)

http://en.wikipedia.org/wiki/File:Scrum_pro

Thanks

General references for

inclusion
Open Collaboration within Corporations Using Software Forges - http://www.riehle.org/publications/2009/open-collaboration-within-corporations-using-software-forges/
Continuous integration tools
http://rubyforge.org/projects/cruisecontrolrb - http://cruisecontrolrb.thoughtworks.com/
https://hudson.dev.java.net/
http://www.softwarebloat.com/2008/11/19/continuous-integration-blueprints-how-to-build-an-army-of-killer-robots-with-hudson-and-cucumber

Exploratory testing
http://www.kohl.ca/blog/archives/000185.html
http://www.kohl.ca/articles/ExploratoryTesting_MusicofInvestigation.pdf

The ongoing revolution in software testing

http://www.kaner.com/pdfs/TheOngoingRevolution.pdf

ABAP Unit
Spotlight on ABAP Unit Part 1 - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/1088

Load testing
http://dannorth.net/2007/02/monkey-business-value

Joel Spolsky 12 steps to better code - http://www.joelonsoftware.com/articles/fog0000000043.html

Automated testing story on SDN - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4103
JUnit - http://www.junit.org/
Open Source
http://www.paulgraham.com/opensource.html (What business can learn from open source)

Watir
http://wtr.rubyforge.org/

Spec as test
Cucumber - http://cukes.info/
Rspec - http://rspec.info/
http://www.pragprog.com/titles/achbd/the-rspec-book

SAP testing
http://www.beteoblog.com/beteo-alm-miniguides/software-quality/
http://raa.ruby-lang.org/project/saprfc/
Integration Tests in ABAP Development und Tool Support - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/7131
XSLT - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/12173

Test automation as panacea (not)

http://railspikes.com/2008/7/11/testing-is-overrated
http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html

BDD - http://behaviour-driven.org/
Blue Ruby - https://sap.na.pgiconnect.com/p16473929/ - http://www.slideshare.net/schmerdy/blue-ruby-sdn-webinar-1260181 https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/408a9a3b-03f9-2b10-b29c-f0a3374b19d8