Software Engineering Process Paradigms – Process Models – Incremental and Evolutionary models, Agile methodology – Process and Project Metrics – Software estimation – Empirical estimation models – Cost/Effort estimation – Planning – Risk analysis – Software project scheduling, Control and Monitoring.
1.1 Software Processes Model or Software Engineering Paradigm 1.1.1 Why use a Life Cycle Model ?
The primary advantage of adhering to a life cycle model is that it encourages development of software in a systematic and disciplined manner. When a program is developed by a single programmer, he has the freedom to decide the exact steps through which he will develop the program. However when an software product is developed by a team, it is necessary to have a precise understanding among the team members as to- when to do what. Otherwise it may lead to chaos and failure.
IEEE defines a process model as ‘a framework containing the processes, activities, and tasks involved in the development, operation, and maintenance of a software product, spanning the life of the system from the definition of its requirements to the termination of its use’.
Process model for software engineering is chosen based on :
i) Nature of project and application.
ii) The methods and tools to be used.
iii) Controls and deliverables that are required.
1.1.2 Linear Sequential Model or Classic Life Cycle or Waterfall Model
The linear sequential model encompasses following activities
1. System / information engineering and modeling
i) As a software is a part of system (or business) work begin by establishing all requirements to software,
ii) System engineering and analysis encompasses requirements gathering at system level with a small amount of top level design and analysis.
iii) Information engineering encompasses requirements gathering at the strategic business level and at the business area level.
2. Software requirement analysis
i) Requirement gathering process is intensified and focused specifically on software
ii) To understand the nature of programs to be built, the software engineer (analyst) must understand information domain for software ; as well as required function, behavior, performance and interface.
iii) Requirements for both the system and software are documented and reviewed with customer.
i) It is a multistep process that focuses on four distinct attributes of a program:-data structure, software architecture, interface representation, and procedural (algorithmic) detail
ii) The design process translates requirements into a representation of software that can be assessed (estimated) for quality before coding begins,
iii) It is documented and part of software configuration.
4. Code generation
i) Design translated into a machine readable form
ii) If design performed in a detailed manner, code generation can be accomplished mechanistically.
i) If focuses on logical internals of software.
ii) i.e. conducting tests to uncover errors and ensures that defined input will produce actual results that agree with required results.
i) Software will undergo change, after it is delivered to customer (a possible exception is embedded software);
ii) Change will occur because errors have been encountered (e.g. change in new o.s., peripheral devices) or because customer requires functional or performance enhancements.
iii) Software support/maintenance reapplies each of preceding phases to an existing program rather than a new one.
Formal review of the end of each phase allows maximum management control.
Users get a system that meets, the needs as understood by the developers; this may not be what was really needed.
This approach creates considerable system documentation.
Documentation is expensive and time consuming to create. It is also difficult to keep current.
Formal documentation ensures that system requirements can be traced back to stated business needs.
Often, user needs go unstated or are misunderstood.
It produces many intermediate products that can be reviewed to see whether they meet the users’ needs and conform to standards.
Users cannot easily review intermediate products and evaluate whether a particular product (e.g., data flow diagram) meets their business requirements.
Why linear model does sometimes fails ?
i) It leads to “blocking states” in which some project team members must wait for other members of team to complete dependent task. In fact, time spent on waiting can exceed the time spent on productive work.
ii) Real projects rarely follow sequential flow that the model process. Since, linear model can accommodate iteration indirectly; changes can cause confusion as project team proceeds.
iii) It is often difficult for customer to state all requirements explicitly .The linear sequential model requires this and has difficulty to accommodate natural uncertainty that exists at the beginning of many projects.
iv) The customer must have patience. A working version of programs will not be available until late in project time span. A major blunder (mistake), if undetected until the working program is reviewed, can be disastrous.
Conclusion : If requirements are well understood, then this model is most suitable.
1.1.3 Prototyping Model
A prototype usually exhibits limited functional capabilities, low reliability, and inefficient performance compared to the actual software. A prototype is usually built using several shortcuts. The shortcuts might involve using inefficient, inaccurate, or dummy functions.
Prototype : ideally, prototype serves as a mechanism for identifying software requirements.
Working of model
i) It begins with requirement gathering,
ii) Developer and customer both together define the overall objectives for software, identify whatever requirements are known, and outline areas further definition is mandatory,
iii) A quick design then occurs, focuses on a representation of those aspects of software that will be visible to customer/user (e.g. input approaches and output formats),
iv) The quick design leads to the construction of a prototype,
v) The prototype is evaluated by customer / user and used to refine requirements for software to be developed. Iteration occurs as the prototype is turned to satisfy the needs of the customer, while at same time enabling developer to better understand what need to be done.
How cost gets reduced in prototyping ?
i) For prototyping for the purposes of requirement analysis to be feasible, its cost must be kept low. Consequently, only those features are included in prototype that will have a valuable return from the user experience. Exception handling, recovery, and conformance to some standards and formats are typically not included in prototypes.
ii) In prototyping, as the prototype is to be discarded, there is no point in implementing those part of requirements that are well understood. Hence, the focus of the development is to include those features that are not properly understood with the focus on quick development rather than quality.
iii) Because the prototype is to be thrown away, only minimal documentation needs to be produced during prototyping. For e.g., design document, test plan, and a test case specification are not needed during the development of prototype.
iv) Reduce testing because testing consumes a major part of development expenditure during regular software development.
When to use ?
i) Often, customer defines a set of general objectives for software, but does not identify detailed input, processing, or output requirements,
ii) In other case, the developer may be unsure of efficiency of an algorithm, the adaptability of an o.s., or the form that human / machine interaction should take,
i) From customer side : It is unaware that in rush to get working version of software no one has been considered overall software quality or long-term maintainability.
ii) From developer side : an inappropriate operating system or programming language may be used simply because it is available and known; an inefficient algorithm may be implemented simply to demonstrate capability. After a time, the developer may become familiar with these choices and forget all the reasons why they were inappropriate.
iii) Resist pressure to extend a rough prototype into a production product. Quality almost always suffers as a result.
Users can try the system and provide constructive feedback during development.
Each iteration builds on the previous iteration and further refines the solution. The makes it difficult to reject the initial solution as inappropriate and start over. Thus, the final solution will be only incrementally better than the first.
An operational prototype can be produced in weeks.
Formal end-of-phase reviews do not occur. Thus, it is very difficult to contain the scope of the prototype, and the project never seems to end.
Users become more positive about implementing the system as they see a solution emerging that will meet their needs.
System documentation is often absent or incomplete, since the primary focus is on development of the prototype.
Prototyping enables early detection of errors and omissions.
System backup and recovery, performance and security issues can be overlooked in the haste to develop a prototype.
1.1.4 Evolutionary Software Process Model
i) All complex system, evolve over a period of time.
ii) Business and product requirements often change as development proceeds, making a straight path to an end product unrealistic.
iii) Tight market deadlines make completion of comprehensive software product impossible, but a limited version must be introduced to meet competitive or business pressure.
iv) A set of core product or system requirement is well understood, but details of product or system extensions have not yet to be defined.
v) In these and similar situation, software engineer need a process model that evolves over a time.
vi) Evolutionary models are iterative. They are characterized in a manner that enables software engineer to develop increasingly more compete version of software.
e.g. incremental model, spiral model, winwin spiral model, concurrent development model.
The Evolutionary SLCM consists of planned development of multiple releases of a product (the product evolves). Five properties of software systems motivating Evolutionary SLCMs have been identified (Lehman and Belady, 1985). These properties are summarized in Table 1.1. The Evolutionary life cycle model entails continual overlapping of development activities (Fig. 1.3), which produce a succession of software releases.
Table 1.1 : Properties of Software Evolution
Properties of software system
Continuing change, Degradation
Software systems continually change and degrade, becoming less and less useful.
Due to continual change, software complexity increases
Programs, programming processes and measures of project and system attributes are statistically self-regulating with determinable trends and invariance’s.
Invariant work rate
Rate of activity in large software is statistically invariant (eg., a property such as the average number of changes per cycle is approximately the same)
Incremental growth limit
During the life of a large software system, the volume of modifications in successive releases is statistically invariant.
22.214.171.124 Incremental Model
i) Combines elements of linear sequential model (applied repetitively) with iterative philosophy of prototyping
ii) Each linear sequence produces a deliverable “increment” of software
Increment 2 : more sophisticated editing and document production capability
Increment 3 : spelling and grammar checking
Increment 4 : advanced page layout capability.
iv) It should be noted that process flow for any increments can incorporate the prototyping paradigm.
v) First increment is often a core product, i.e. basic requirements are addressed, but many supplementary features (some known, other unknown) remain undeliverable. Core product is used by customer (or undergoes detailed review).
vi) As a result of use and/or evaluation, a plan is developed for the next increment. The plan addresses the modification of core product to better meet the need of customer and delivery of additional features and functionality.
vii) This process is repeated following the delivery of each increment, until the complete product is produced.
viii) Early increments are stripped down (to open, release) versions of final product, but they do provide capabilities that serves the user and also provide a platform for evaluation by user.
When to use incremental model ?
i) Useful when staffing is unavailable for a complete implementation by business deadline that has been established for project
ii) Early increments can be implemented with fewer people
iii) If core product is well received, then additional staff (if required) can be added to implement next increment
iv) Increments can be planned to manage technical risk.
Example : A major system might require availability of new hardware i.e. under development and whose delivery date is uncertain. It might be possible to plan early increments in a way that avoids the use of that hardware, thereby enabling partial functionality to be delivered to end users without inordinate delay.
Key point : the incremental model delivers software in small but usable pieces called “increments”. In general, each increment builds on those that have already been delivered.
Advice : When you encounter difficult deadline that can not be changed, the incremental model is a good paradigm to consider.
Advantages of the incremental model
1. Avoids the problems resulting in risk-driven approach in the software.
2. Understanding of the problem increases through successive refinements.
3. Performs cost-benefit analysis before enhancing software with capabilities.
4. Incrementally grows in effective solution after each multiple iteration.
5. Does not involve a high complexity rate.
6. Early feedback is generated, because implementation occurs rapidly for a small sub-set of the software.
Disadvantages of the incremental model
1. Requires planning at the management and technical level
2. Becomes invalid when there is time constraint in the project schedule or when the users cannot accept the phased deliverables.
126.96.36.199 Spiral Model
i) Model that couples the iterative nature of prototyping with the controlled and systematic aspects of linear sequential model.
ii) Software is developed in a series of incremental releases
iii) During early iterations, the incremental release might be a paper model or prototype. During later iterations, increasingly more complete versions of engineered systems are produced.
iv) A spiral model is divided into a number of framework activities also called as task regions.
Six task regions
a) Customer communication
Task required to establish effective communication between developer and customer.
Task required to define resources, timelines and other project related information
c) Risk analysis
Task required to assess (to estimate) both technical and management risk.
Task required to build one or more representations of applications.
e) Construction and release
Task required to construct, test, install, and provide user support (e.g. documentation and training)
f) Customer evaluation
Tasks required to obtain customer feedback based on evaluation of software representation created during the engineering stage and implemented during installation stage.
i) Each of regions is populated by a set of work task called a “task set”. In all cases umbrella activities are applied.
ii) As evolutionary process begins, software engineering team moves around spiral in a clockwise direction, beginning at the center.
iii) Unlike classical process models that end when software is delivered, spiral model can be adapted to apply lifecycle of computer software.
iv) Each cube placed along project entry point axis can be used to represent starting point for different types of projects.
1. Direct consideration of management/technical risk.
2. Maintains systematic stepwise and approach mentioned by classic lifecycle + iterative framework.
3. Realistic approach to development of large scale system software.
4. Uses prototyping as a risk reduction mechanism.
5. Unlike prototyping as a risk reduction mechanism.
6. Unlike classical process models that ends when software is delivered.
1. It may be difficult to convince customer’s (particularly in contract situations) that evolutionary approach is controllable.
2. It demands considerable risk assessment expertise and relies on this expertise for success.
3. If a major risk is not uncovered and managed, problems will undoubtedly occur.
4. Finally, the model has not been used as widely as linear sequential or prototyping paradigm.
1.1.5 RAD (Rapid Application Development) Model
When to use RAD model ?
1. It is an incremental software development process model
2. It emphasizes on extremely short development cycle
3. It is a high speed adaptation of linear sequential model in which rapid development achieved using component based construction.
4. If requirements are well understood and project scope is constrained , RAD enables project team to create a “fully functional system” within very short time like 30 to 60 days.
5. It used primarily for information system application
The RAD approach encompasses following phases
The information among business function is modeled in a way that answers following questions
i) What information drives the business process ?
ii) What information is generated ?
iii) What information is generated ?
iv) Who generate it ?
v) Where the information does goes ?
vi) Who process it ?
i) Information flow is refined into a set of data object.
ii) The characteristics (attributes) of each objects are identified.
iii) Relationship between these objects defined.
i) The data objects are transformed to implement a business function to achieve the information flow necessary.
ii) Processing descriptions are created for adding, modifying, deleting, or retrieving a data object.
i) RAD assumes the use of fourth generation technique.
ii) RAD process works to reuse existing program components (when possible) or create reusable components (when necessary).
iii) In all cases automated tools are used to facilitate construction of software.
Testing and Turnover
i) Since, RAD emphasizes reuse, many of program components have already been tested.
ii) This reduces overall testing time.
iii) However, new components must be tested, and all interfaces must be fully exercised.
1. For large but scalable projects, RAD requires sufficient human resources to create the right number of RAD teams.
2. RAD requires developers and customers who are committed to rapid fire activities necessary to get a system complete in much abbreviated time frame. If commitment is lacking from either side, RAD project will fail.
3. Not all types of applications are appropriate for RAD. If a system can not be properly modularized, building the components necessary for RAD will problematic. If high performance is an issue and performance is to be achieved through tuning the interfaces to system component, the RAD approach may not work.
4. RAD is not appropriate when technical risks are high. This occurs when a new application makes heavy use of new technology or when new software requires high degree of interoperability with existing computer program.
For appropriate projects, this approach puts an application into production sooner than any other approach.
This intense SDLC can burn out systems developers and other project participants.
Documentation is produced as a by product of completing project tasks.
This approach requires systems analysts and users to be skilled in RAD system development tools and RAD techniques.
RAD forces teamwork and lots of interaction between users and stakeholders.
RAD requires a larger percentage of stakeholders’ and users’ time than other approaches.
1.1.6 Object Oriented Model or Component Based Development Model (CBD)
i) Object oriented technology provides technical framework for this model, which emphasizes the creation of classes that encapsulates both data and algorithm used to manipulate data.
ii) CBD model incorporates many of the characteristics of spiral model. However, component based development model composes application from prepackaged software components called classes.
iii) The engineering activity begins with identification of candidate classes. [This is accomplished by examining the data to be manipulated by application and algorithm that will be applied to accomplish the manipulation.]
iv) Corresponding data and algorithm are packaged into a class. Classes created in past SE project are stored in a Class library or Repository.
v) Once candidate classes are identified, class library is searched to determine if they already exists. If they do, they are extracted from library and reused. If no, it is engineered using object oriented methods.
vi) The first iteration of application to be built is then composed using classes extracted from library.
vii) Process flow then returns to spiral model and ultimately re-enter the component assembly iteration during subsequent passes through engineering activity.
i) Model leads to software reuse. Reusability provides software engineer with a number of measurable benefits.
ii) Based on study of reusability, QSM Associates, Inc reported component assembly leads to
70% (↓) reduction in development cycle time
84 %(↑) reduction in project class
Production index of 26.2 %(↑) compared to industry norm of 16.9%
Unified software development process is representation of a number of component based development models, using UML.
Demerit of OO model
Risk involved with reusability may produce defects.
1.1.7 Methodology and Tools
We need to distinguish between a methodology and a process. A process covers all the activities beginning from product inception though delivery and retirement. Additionally, it addresses the broader issue of reuse, documentation, testing, parallel work carried out by team members and coordination with the customer, and so fourth. In contrast, a methodology covers only a single activity or at best a few individual steps in the development. For e.g., both testing methodology and design methodology address only a single activity in their respective areas in a lifecycle. Thus a process scales up a methodology to the entire life cycle.
Methodology defined : The way something gets done. The strategy, steps, directions, or actions.
Methodologies can be :
combination of both
Classifications of Methodologies
Structured Analysis and Design
Prototyping is a technique - (some say that it is a methodology)
The Traditional Methodology (1950s - now)
Applicable for small teams on small projects
Functional perspective of problem domain
Informal, unstructured, unrepeatable, unmeasurable, ad-hoc way
Tools used to support it are okay
Traditional methodology tools
TECHNIQUES and TOOLS REPRESENTING
Play script, Program
Flowcharts, HIPO Charts
Structured Analysis and Design Methodology (mid-1970s - now)
Data Flow methodology (synonym).
Compliments Structured Programming.
Very popular - perhaps the leading one.
Can be repeatable, measurable, and automated.
CASE brought significant assistance : 1) Yourdon, and 2) Gane and Sarson.
Functional perspective of problem domain.
Describes the real world as data flowing through the information system, being transformed from inputs to outputs.
Structured Analysis and Design Methodology Tools
TECHNIQUES and TOOLS REPRESENTING
Data Flow Diagram
Interviews, User Reviews, JAD sessions
Decision Tree/Table, Structured English,
Information Modeling Methodology (early-1980s - now)
Data modeling and information engineering (synonyms)
Describes the real world by its data, the data’s attributes, and the data relationships
Describes the real world by its objects, the attributes, services, and relationships
Data and functions are encapsulated together
Object-Oriented Methodology Tools
TECHNIQUES and TOOLS REPRESENTING
Object Model Attributes
Interviews, User Reviews, JAD Sessions, Brainstorming
Decision Tree/Tables, Structured English
Object technology principles
Encapsulation (Information Hiding)
Common Methods of Organization
What is a methodology ?
Integrates tools and techniques
Usually an underlying philosophy
Justified by experience
A methodology may or may not prescribe a Life Cycle Model.
Methodology is not a life cycle method
A LCM specifies and orders the development activities
A methodology offers the tools
Although most methodologies specify a LCM, some don’t.
Why use a methodology ?
Distilled experience/best practice
Ensures user involvement
Helps inexperienced analysts
Provides planning and control
1.1.8 Selection Criteria for Software Process Models
Project type and associated risks : one of the key features of selecting a process model is to understand the project in terms of size, complexity, funds available, and so on. In addition, the risks which are associated with the project should also be considered. Note that only a few process models emphasize risk assessment.
Requirements of the project : the most essential feature of any process model is to understand the requirement of the project. In case the requirements are not clearly defined by the user or poorly understood by the developer, the developed software leads to ineffective systems. Thus, the requirements of the software should be clearly understood before selecting any process models.
Users : Software is developed for the user. Hence, the users should be consulted while selecting the process model. The comprehensibility of the project increases if users are involved in selecting the process model. It is possible that a user is aware of the requirements or has a rough idea of the requirements. It is also possible that a user wants the project to be developed in a sequential manner or an incremental manner (where a part is delivered to the user for use).
Table 1.3 : Selections on the basis of requirements of the project
Requirements of the project
Requirements are defined early in SDLC
Requirements are easily defined and understandable
Requirements are changed frequently
Requirements indicate complex system
Table 1.4 : Selections on the basis of the user
Requires limited user involvement
User participation in all phases
No experience of participating in similar projects
1.2 Agile Methodology
Agile development methodologies have been designed to address the problem of delivering high quality software on time under constantly and rapidly changing requirements and business environment. Agile development processes are characterized by extensive coding practice, intensive communication between key stakeholders, small and flexible teams and fast iterative cycles.
1.2.2 Where to use Agile Development ?
Agile development methods are used in the following circumstances :
There is no ‘Requirement Freezing’. Change in requirements are acted upon.
Incremental and iterative approach is used for modeling.
Clients are actively take part in modeling.
Prioritization of tasks is being done by key stakeholders and the team works on the highest priority requirement.
Every one in the team is an active participant and everyone’s input is welcome.
The content of the model is being recognized as being more significantly important than the representation or format.
1.2.3 Where not to use Agile Development ?
Agile methods are not used under the following circumstances :
Customers have limited involvement in the development efforts.
Your have to prepare documents at each stage which would be signed off by key stakeholders.
You hand over documents at each stage the next team then takes care of your developing the system further.
Case tool is used to specify the design of the software but not for generating the software.
1.2.4 Advantages of Agile Development
Shortened development cycles
Higher stability of deliverables
Improved utilization of resources
Better quality of deliverables due to the involvement of the customer
1.2.5 Limitations of Agile Development
Agile development approach works well for small to medium sized teams
Agile development methods do not scale. Due to the number of iterations involved it would be difficult to understand the current project status.
This approach requires highly motivated and skilled individuals which would not always be existent.
All software development methods, including the Agile ones, have some sort of underlying project life cycle.
Some of the Agile methods don’t make a big deal of it, and others do. Some have such abstract life cycles that it is actually hard to know what activities to schedule.
The key elements are a Project Initiation and Project Plan followed by one or more Releases made up on one or more timeboxes (although usually 3).
There are also more or less optional phases to Elaborate Requirements and for the Architecture - often done in parallel with the Project Plan.
Perhaps not so strangely, this life cycle looks pretty conventional.
Fig. 1.8 a) Project initiation
Set up and justify the project. Determine initial high level requirements.
b) Project plan
Plan out the project, everything after Project Initiation, including whether you’re going to have the optional phases.
If you’re using the optional phase to Elaborate the Requirements you’re likely to have a draft plan after the Project Initiation and then a more complete plan after the Requirements are complete.
DSDM makes this explicit and mandatory with its “Outline Plan” (created in Project Initiation) and “Development Plan” (created after the Requirements have been fleshed out).
XP has something similar in the “Big Plan” done to justify a project, which evolves into a Release plan.
c) Elaborate requirements (optional)
Very high level requirements are gathered during Project Initiation. These can optionally be expanded during an phase to Elaborate Requirements. Even is the requirements are “elaborated” they are still high level.
There is no separate Elaboration phase in XP and Scrum, although the Customer is expected to elaborate as much as necessary at the start of an XP Iteration (Timebox).
Crystal Orange and DSDM have a separate Elaboration phase. Facilitated workshop(s) to identify “High level requirements” which are then collated into a document (Requirements Document / Business Area Definition (BAD)) that contains use Cases and non-functional requirements.
d) Architecture (optional)
The Agile approach is generally design for today and refactor tomorrow (at least in XP). An Architecture Phase is optional in XP and Crystal Orange.
However, In DSDM An architecture phase is compulsory and results in a “System Architecture Document”. The SAD is created in parallel to the BAD, i.e. requirements elaboration and architecture occur at the same time.
A Release is a piece of development where the customer gets some new software. Releases can be from 2 weeks to 6 months, but are usually 3 months long.
Release have one or more timeboxes. Unlike other methods, in DSDM one Release is the norm, i.e. there is only one release to the customer in the entire project.
f) Time box
A Timebox is 1 – 6 weeks long, but usually 3 – 4 weeks. The most important thing about a timebox is that the delivery date is fixed.
188.8.131.52 Project Initiation
There are some important questions to be addressed at the start of any project. Project Initiation is the phase where these questions are addressed.
Scope Project : Should we invest more ?
Probably the key question during project initiation is “Should we invest more?”, i.e. is the project worth going ahead with ? Normally, if this question is addressed at all, it is covered as part of a business case.
Not all Agile methods address this question. DSDM includes a business case in the Feasibility Report. XP addresses cost benefit for the overall project with a “Big Plan” in the early stages. Scrum and Crystal Orange don’t mention it.
Choose the right method
Having decided to continue with the project the next question is what method to use.
XP says to use XP for vague or changing requirements and leaves it at that.
The DSDM manual includes a list of suitability characteristics and the PM must use these to decide if DSDM is appropriate for the project (and document this decision in the “Feasibility Report”).
Scrum claims to be for any project. A big claim!!
Crystal Orange says to pick a method that is barely sufficient to match the needs of the project. Cockburn offers a categorisation of methods and projects to help with this decision.