SQA Test Process for Femtotality

www.ccpu.com

Confidential and Proprietary

Verification Process
Development
Unit Testing Cover the functional aspects of a component. Ensure that all the error/failure scenarios are handled properly. Integration with other components. FACT test cases verification

QA
Sanity Endto-End System Testing Functional Regression Performance Load Stress Recovery Usability Deployment Installation User documentation Upgrade Rollback SOAK MPS

IOT
End-to-end Testing with Customer specific Equipment Hand held Devices RAN Core

Customer Acceptance

Verification and Acceptance Test Live Trials

SIT (System integration Testing) Check-in Verification Daily nightly build Regression Comprehensive release testing including NE-SM Integration Sanity testing on the nightly build

LAV (Live Air Verification) End-to-end, RF-related functionality and performance Home Trials

www.ccpu.com

Confidential and Proprietary

QA Process
FSRS
Feature Systems Requirement Specification

QA Test Plan
QA testplan template Review testplan Meeting minutes Master Control
Approve testplan Testplan repository Testplan version control

Requirements Tracibility Matrix

Clearquest Database Requirements Tracking Testcase Results

Feature Acceptance Test (FACT)

Test Start through Test First Pass (TS-TF)

Test First Pass through Test Done (TF-TD)

First Customer Shipment through General Availability (FCS-GA)

www.ccpu.com

Confidential and Proprietary

QA Test Plan
1. The format of QA test plans are derived from a test plan template to ensure consistency in all test plans and allows for imports into the test cases database. The template includes required sections such as: a. feature description b. product coverage c. test methodology d. restriction/limitations e. test bed set-up f. appendix for required meeting minutes of test plan review

2.

www.ccpu.com

Confidential and Proprietary

QA Entrance Criteria example (1 of 3)

Marketing/Program Management 1. Project Plan/PRD is complete and has been updated Hardware 1. 0D version (or other approved version) of hw available to QA 2. Hardware version control in place Software 1. All Functional Specifications complete and have been updated to accurately reflect the implementation, including diagnostics 2. All Design Specification complete and have been updated to be sync with the implementation 3. All Code complete and frozen Code changes allowed only for bug fixing but not features 4. Software version control in place 5. Unit test plan (s) and SIT test plan (s) are reviewed and approved by QA 6. 100% unit tests executed and passed 7. Unit level test plan(s) reviewed and approved by QA

www.ccpu.com

Confidential and Proprietary

QA Entrance Criteria (2 of 3)
FACT

1. 2.

FACT test plan(s) reviewed and approved by Dev/QA First Pass Test cases ~70% (if less than 70% the feature need to go back to development for UT and IIT)

3.
4. 5. 6.

100% FACT tests executed and 95% passed or deferred (as approved by QA) since start or since a revert back to start
No more than 1 crash/week averaged since FACT start or since last revert. 72 hours of soak time on the QA load No P1 bugs open without QA agreement

7.
8.

No P2 bugs open without QA agreement

Less than 50 P3 and P4 bugs open

www.ccpu.com

Confidential and Proprietary

QA Entrance Criteria (3 of 3)
QA 1. 2. 3. 4. 1. 2. Test Strategy document completed and approved by all relevant groups (SW Development, QA, System Eng where applicable) Test Plan(s) completed and approved and entered into database TC execution plan in place Cross- referencing between test cases to requirements in place Documentation updates will be available for QA testing, Deployment, Upgrade/Rollback and CLI Guide should be available at the Test start. Online help needs to be included in all major drops to QA

Tech Publication

www.ccpu.com

Confidential and Proprietary

Feature Acceptance Test (FACT)

1. The purpose of the FACT is to ensure the new feature and release meets a minimum level of quality before it enters formal QA testing.

a. The FACT test plan is jointly created by QA and Development who make up the FACT team. It is essential that test cases are chosen carefully to cover the feature and its interaction with the existing code and other depended features.
b. QA supplies equipment and is responsible for execution of FACT as the acceptance criteria 2. A branching and drop strategy for multiple drops to QA is agreed between development and QA. a. Each drop must have a FACT plan to get formally released to QA testing.

3.

When the code base passes FACT and meets the other QA entrance criteria, QA testing begins.
a. Development is responsible for passing FACT and meeting the handoff dates to QA.
Confidential and Proprietary

www.ccpu.com

QA Test Cycles
There are 3 cycles of testing and each cycle has its own entry and exit criteria:
Test Cycle TS to TF (TestReady to TestFirstPass) Goal 100% of test-cases are executed and retested to achieve a 95% pass rate. The first pass rate should be more than 80% per feature. Bug fix validation and regression are executed to achieve a 99% pass rate All soaks with call model should pass and all agreed KPI/KCIs should be met. FCS to GA Product acceptance in customer environment

TF to TD

(TestFirstPass to TestDone)

(First Customer Shipment or FOA to General Availability)

www.ccpu.com

Confidential and Proprietary

QA Test Cycles: TS to TF
Test Start (TS) to Test First Pass (TF) Goal 1. Before TF, all the test cases from the test plan(s) will be executed. 2. Failed test cases will be re-run to achieve 95% pass rate.
Exit Criteria: 1. All test cases will be executed at least once 2. First Pass rate at 70% 3. 95% pass rate 4. No P1s 5. < 10 P2s, all documented and accepted by QA 6. < 50 P3/P4s 7. < 2 crashes/week between TS to TF (from last revert, if necessary) 8. 72 hours of uninterrupted soak on the soak/capacity test bed

www.ccpu.com

Confidential and Proprietary

10

QA Test Cycles: TF to TD
Test First Pass (TF) to Test Done (TD) Goal: 1. Validate bugs and retest failed test cases. 2. A Regression set of test cases will be re-executed 3. 98% of test cases should pass Exit Criteria 1. All the test cases executed/addressed 2. Pass 98% of all the test cases 3. Compliance matrix will be provided. 4. Test cases are updated in test Database with updated test procedure 5. No crash in last two weeks of all testing (as recorded in the test report) 6. 0 P1, 0 P2s, no more than 30 P3/P4s 7. 72 hours of uninterrupted soak on the soak/capacity test bed 8. All the resolved bugs must be in closed state 9. Cross-functional group agreement of release notes covering all open bugs with assessment of customer impact
Confidential and Proprietary 11

www.ccpu.com

QA Test Cycles: FOA/CUR to GA

First Customer Shipment for FOA to General Availability (GA) Goals 1. Product acceptance in customer environment Exit Criteria 1. The product is released to the customer. 2. All required customer bugs fixed 3. Regression tests successful 4. 2 week acceptable soak of final release

www.ccpu.com

Confidential and Proprietary

12

Rational Clearquest: Bug and Testcase Tracking

1. Rational Clearquest is our bug tracking system a. b. 2. Integrated with Clearcase software version control Code check-ins are cross-linked to bugs

Rational Clearquest is our QA testcase database tracking system a. b. The database tracks test results: Pass/Fail Testcase failures are cross-linked to Clearquest bugs

www.ccpu.com

Confidential and Proprietary

13

Bug Priority & Severity Definitions

Priority 1 Effect on Scheduling of Fixes - Product in QA: Resolve Immediately Must be fixed for next build of code to SQA - Product in field: patch or maintenance release candidate for immediate delivery. 2 - Product in QA: High Attention Try to fix for next build to QA. - Product in field: Fix for next release to field. Criteria - Blocks testing progress - Change poses major risk to large amounts of code. - Unacceptably hinders customer use of product. 2 - Blocks a limited area of testing - Change poses some risk, but not for major destabilization - Important function is not working for customer 3 Severit y 1 Criteria - Box or board crash or hang - Major function not working, no workaround - Large Resource leak (memory, buffers) - Memory corruption - Major function not working, workaround - Minor function not working, no workaround - Significant performance issue - Minor function not working, workaround - Operation inconvenient or inconsistent - Cosmetic imperfection Fix before lesser Severities of same Priority and after higher Severities of same Priority Fix last of this Priority Side-effect on Fix Scheduling Fix before lesser Severities of same Priority

- Trivial problem, but makes us look silly.

3 - Product in QA: Normal Queue Try to get a fix in before final QA Cycle check in. - Product in field: Schedule for fix in an upcoming release to field. 4 - Product in QA or the field: Low Priority - When all higher priority bugs are fixed. - Blocks testing of only that small, specific item of functionality. - Doesnt hinder use of the product for required purpose. - Doesnt meet any of above criteria.

www.ccpu.com

Confidential and Proprietary

14

Unit Test Process

Entry Criteria: 1. Final reviewed version of FRS, FSRS document. This provides feature level requirements. 2. Final reviewed version of FAS document. This provides the agreed-upon feature requirements, architecture and external interfaces for the feature. 3. Reviewed version of FS document. This provides component level specification for the feature. 4. Reviewed version of OAM&P document. User Interface document related to feature reviewed. 5. Reviewed version of DS document that defines the design for the component. 6. Compiled and running software on target hardware for the component / feature. 7. Code review for the software (Under Test) conducted and review comments incorporated. Exit Criteria: 1. 100% success on passing the entire unit tests for the component. Unit test cases and results should be archived as quality record. 2. Purify memory leak and coverage results should be recorded in Project notebook. 3. Project notebook should be updated with unit test results. 4. Problems found during unit tests are fixed and submitted in Clearcase. 5. Number of problems found during unit testing should be recorded in Project Notebook. 6. Component level regression testing should be conducted and anomalies should be corrected. 7. Component in ready to check-in in feature branch.

www.ccpu.com

Confidential and Proprietary

15

System Integration Test (SIT) Process

Check-in Verification 1. Basic sanity check of the software components functionality as well as end-to-end verifications of the overall system functionality. 2. New software component, features or a bug fix is verified in an end-to-end system environment before check-in to code base in SIT branch. 3. The developer fills a check-in request form after the software component, feature or a bug fix has been unit tested, integrated with other software components and tested by the developer. 4. The check-in request form contains pertinent details regarding the check-in and its testing. 5. Once the check-in form is reviewed and approved by the software manager, the new software is put to test in an end-to-end SIT test bed. 6. After the SIT verification and testing the view is merged with rest of the code in SIT branch. The merged code is then checked-into SIT. Daily nightly build Regression 1. Daily Smoke Test: maintains the quality of software and ensure that previous days check-ins did not break the system in any way. 2. Nightly build regression test plan ensures that all aspects of the system have been verified.

Release Testing and Regression 1. Regression Testing: to ensure that new release does not break features from previous release. 2. New Feature Testing 3. Capacity and Performance Testing

www.ccpu.com

Confidential and Proprietary

16

Streams/Drops
Code Complete Design Test Complete

CIT = Check In Test (includes smoke test & P1 of FACT) FACT = Feature Acceptance Test (section of QA Feature TestPlan)

Coding Design TestPlan Unit Test Functional Test Integration Test Stress Test CIT merge final merge LOCK FACT

FACT Complete

Release Branch

QA Feature Test

QA Feature TestPlan High Level View Feature B Drop2 Feature C Feature B Drop1 Feature A
Bundle #1 CheckIn synchronized w hen ALL features in bundle are ready, so FACT start is synchronized Bundle #2

NOTES: 1.Design owns milestones up to FACT Complete. 2.Joint cooperation & learning between Design & QA during FACT. 3.EMS-NE functionality must be available for FACT to start, but CheckIn of NE or EMS can precede indpedendently. 4.CRs written for FACT failures/deferrals. 95% pass required for FACT complete. 5.End-to-end equipment required for FACT. (details to be worked) 6. A feature could have multiple Drops (e.g. IRSHO). 7. Features are bundled together. All features within a bundle are synchronized so checkin & FACT starts with same image (this way Integration is with preexisting features + features within bundle) 8. QA regression test etc. picks up the load once FACT complete for all features within bundle (tbd, this could be changed).

Release Branch

FACT A FACT B (1) QA Feature Test A

QA Feature Test B (1)

FACT C FACT B (2) QA Feature Test C

QA Feature Test B (2)

QA QA QA QA

Regression System Test Load & Soak Test LAV & Performance Test

www.ccpu.com

Confidential and Proprietary

17

Thank You

www.ccpu.com

Confidential and Proprietary

18