Understanding and (Re)Designing Software Development Processes

Walt Scacchi
Institute for Software Research University of California, Irvine Irvine, CA 92697-3425 USA http://www.ics.uci.edu/~wscacchi/Presentations/JPL-ProcessOct01.ppt

Recurring problems
What is the best way to organize software development project? How to speed up development, reduce costs and improve software quality? How to achieve the quickest development effort, lowest effective development cost, and best available product quality?

Possible solutions
Attain and improve maturity of software development capabilities (image) Get best people to practice standardsbased development process supported by IDEs (Re)Design your software production architecture (SPA) to optimize use of development resources, processes, and people.

Goals
Present an approach for how to optimize software production Identify key concepts, techniques, and tools that enable better optimization Describe optimization transformations from business process redesign studies Describe opportunity areas for exploitation and use

Definitions and Differences

Software production: enterprise processes and resources that produce software Production strategies: business strategies guiding overall approach to building software Production architecture: configuration of enterprise capabilities to enact strategies Optimizing production: minimizing enterprise configuration to maximize strategic options

 What first: to-be goal vs. as-is mess?

(Re)designing software production

If you dont know where you are, any road will do (proverb) Observation: people at work cannot describe the processes they do with high fidelity (tacit knowledge) Redesign necessitates understanding as-is, to-be, and here-to-there

Creating high-performance work groups

Empowerment, participation, incentivization (resource sovereignty), and recognition
W. Scacchi, Redesigning Contracted Service Procurement for Internet-based Electronic Commerce: A Case Study, J. Information Technology and Management, 2(3), 313-334, 2001.

Software Production Strategies

Reduce costs Reduce cycle time Improve cash flow Customer satisfaction Increase sales Improve customer service Increase productivity Open new markets Open new channels Be innovation leader Increase market share Enable just-in-time service delivery

 A composite model that interrelates

software system architecture software process architecture development organization architecture network infrastructure and development tools/environment configuration documentation architecture customer-support knowledge base architecture P. Mi and W. Scacchi, A Meta-Model for Formulating Knowledge-Based Models of
Software Development, Decision Support Systems, 17(4):313-330, 1996.

Software Production Architecture

Optimizing Software Production

Strategies provide global constraints or opportunities for optimizing software production Constraints and opportunities realized in software production enterprise Constraints and opportunities are distributed across the software production architecture

Optimizing Software Production

Optimization must address composite architecture of software production Local optimization of any component architecture does not guarantee global optimality of software production Diagnostic analyses and transformation heuristics applied to composite architectural models lead to optimization opportunities

Optimizing Software Production

Transformation heuristics classified taxonomically Taxonomy classifies domain-independent and domain-specific hueristics DI transformations applied in any software production setting DS transformations applied to specific component architectures

Optimizing Software Production

DI transformation classes (sample):
Job scope Worker empowerment Organization design Workflow streamlining Information technology (IT)

Research grant justification and approval process at Office of Naval Research (c. 1995)

W. Scacchi and J. Noll, Process-Driven Intranets: Life Cycle Support for Process Reengineering IEEE Internet Computing, 1(5):42-49, 1997.

Optimizing Software Production

IT transformation sub-classes (sample):
Extend IT-based support to manual process steps Extend IT-based communication facilities to encourage information sharing activities Extend IT-based automation to incorporate new kinds of application packages Extend IT-based integration to interconnect and interrelate existing "islands of automation"

As-is vs. to-be process

Redesign/Optimization Results
Reduction in procurement process cycle times of 20X, annual operational savings of $10-15M.
Via transformation and realignment of information systems, business processes, corporate strategy, and work practices in a 1-2 year time frame. Participatory design, development and refinement of computational models of new work processes, resource configurations and work practices, together as an organizational system.

W. Scacchi, Redesigning Contracted Service Procurement for Internet-Based Electronic Commerce: A Case Study, Information Technology and Management, 2(3):313-334, 2001.

Tools and Techniques

Software process redesign case web Knowledge web for software production Process-driven intranets Organizational transformation

Case study: Software development teamwork

Comparative analysis of software specification teamwork (in complex setting)
Five teams, 5-7 members, two-week (parttime) process that incorporates planning, formal notation, automated tool use, reusable assets, documentation tasks, and team shared responsibility.

What to Understand
Work Structures and Shifts: Resource arrangements, historical circumstances, division of labor and expertise, etc. Work Processes: Routine, habitual or emergent patterns of how work flows among people through/onto work structures Work Practices: Behavioral discourse and social dynamics enacted through work processes

What to Understand
Structures are domain independent constructs
Prescriptive/descriptive abstractions

Processes are classes of workflow

Descriptive and derived Prescriptive and composed

Practices are instances

Descriptive, historic and situated

Comparative analysis of software specification teamwork

Six work structure types observed: Negotiated, Integrated, Replicated, Delegated, Prediscriminated and Separated Three structural shift types observed:
anticipated ->, unanticipated -->>, role shift within work structure +.

Team ID Team Size Reusable Exemplar

Work structures and shifts (data)

T1 6 no T2 7 yes T3 7 yes T4 7 yes N->R->I N I I P (D,I,I) D R D I -->> N -->> R -->> I I+ N -->> D D N R D N -->> -->> I -->> -->> I N->R->I N I I+ P (D,I,I) D R I I+ N->R->I N I I P (D,I,I) D R I I+ N->R->I N I -> S S P (D,S->I,I) D D D I+ -> S+

T5 5 yes

PROCESS A. Pre-planning task B. Planning task c. d. e. f. g. h. C. Develop preliminary (informal) specification D. Develop formal (processable) spec. E. Document write-up F. Documentation integration G. Document review H. Prepare for Delivery

N->R->I N I -> S S P (D,D,I) D D D I -> S+

I+ P+ D+ R I+

I+ P+ D+ R N -->> I+

S+ -->> S+ N -->> N -->> P(D,D,I) P(D,S->I,D) D+ D+ N -->> R N -->> I+ D D+

Findings
Highest (lowest) quality product (measured by automated tools): T1 (T5) Highest (lowest) productivity (self reported time expended): T5 (T1) Note the coincidental relationship Effectiveness of planning, automated tool use, asset reuse not clearly associated with high(low) quality or high(low) productivity

Findings
Teams falling primarily into Negotiative and Integrative structures had higher quality Teams falling primarily into Delegative, Pre-discriminative or Separative structures had higher productivity Computer supported work environments must account for teamwork structures as a usage parameter.

Tools and Techniques

Software process redesign case web Knowledge web for software production:
Software production ontology Taxonomy for as-is diagnosis, redesign heuristics Best practices and lessons learned crosslinked

Process-driven intranets W. Scacchi and A. Valente, Developing a Knowledge Web for Business Process Redesign, Presented Organizational transformation at the 1999 Knowledge Acquisition Workshop, Banff, Canada, October 1999.

Current field study

Understanding open source software practices and processes in different domains
Academic research vs. Commercial development
Deep Space Astronomy, Software Architecture Internet infrastructure, Networked computer games

To produce and compare case studies using narrative, hypertext, and computational renderings.

Tools and Techniques

Software process redesign case web Knowledge web for software production: Process-driven intranets:
Model, prototype, and enact software production architectures

Organizational transformation

Process-driven intranets
Enable rapid configuration of virtual enterprises (VEs) across multiple organizational domains Virtual enterprises for wide-area software development have been demonstrated Software production in VEs supported and enacted via process navigation (process surfing) J. Noll and W. Scacchi, Supporting Software Development in Virtual Enterprises, Journal
of Digital Information, 1(4), February 1999.

Process-driven intranets
Accommodate organizational autonomy and computer-supported cooperative work Accommodate heterogeneous repositories of development artifacts (source code, etc.) Accommodate use of local tools and development environments

Process-driven intranets

Also see, J. Noll and W. Scacchi, Specifying Process-Oriented Hypertext for Organizational Computing, J. Network and Computer Applications, 24(1):39-61, 2001

Process-driven intranets

Tools and Techniques

Software process redesign case web Knowledge web for software production Process-driven intranets Organizational transformation
Whats involved? How long does it take? How much bang for the buck?

Organizational transformation
Collaborative participation to understand as-is, to-be, and here-to-there SPAs within constraints and contexts. Timeframe: 6-18 months Baseline investment (ROI):
External: Invest $1 to realize $10 savings/gain. Internal: Staff time, infrastructure usage, etc.

Conclusions
Software production can be optimized Optimizing software production is a strategic option/choice that can be realized Software production architectures and supporting technologies enable optimization Process improvement and production optimization are complementary efforts

References
S. Bendifallah and W. Scacchi, Work Structures and Shifts: A Study in Software Specification Teamwork, Proc. 11th. Intern. Conf. Software Engineering, Pittsburgh, PA, IEEE Press, 260-270, 1989. R. Conradi and A. Fuggetta. Software process improvement: what can be improved? Politecnico di Milano, Dipartimento di Elettronica e Informazione, Submitted for publication, 2001. W. Scacchi and P. Mi, Process Life Cycle Engineering: A Knowledge-Based Approach and Environment, Intelligent Systems in Accounting, Finance and Management, 6:83-107, 1997. W. Scacchi, Understanding Software Process Redesign using Modeling, Analysis, and Simulation, Software Process--Improvement and Practice, 5(2/3), 183-195, 2000. J.S. Choi and W. Scacchi, Modeling and Simulating Software Acquisition Process Architectures, Journal of Systems and Software, to appear, 2001

Available at http://www.ics.uci.edu/~wscacchi/publications.html