SQA in Theory

A comprehensive definition of Software Quality Assurance SQA is presented here, in the form of a short history of the Software Quality Movement, an example of a theoretical CMMI SQA\SQC implementation and an example of what a typical Software Application QA engineer does day to day.


SQA, SQC and CMMI Definitions

Having positioned Software Quality Assurance SQA and Software Quality Control SQC (see SQA Definition) within their historical context,
this paper outlines an example implementation of SQA and SQC, within a CMMI context that matches the formal definitions of these terms.

The formal definitions that are used within this paper are:-

Software Quality Assurance
The function of software quality that assures that the standards, processes, and procedures are appropriate for the project and are correctly implemented.

Software Quality Control
The function of software quality that checks that the project follows its standards, processes, and procedures, and that the project produces the required internal and external (deliverable) products.
A further definition of SQA and SQC, by way of example can be found here.

CMMI
A process improvement approach to software development.

CMMi identifies a core set of Software Engineering process areas as:-

  • Requirements Development
  • Requirements Management
  • Technical Solution
  • Product Integration
  • Verification
  • Validation

    CMMI also covers other process areas, such as Process Management, Project Management and Support but only the core Software Engineering development processes are used here by way of example.

    It is also interesting to note that SQA and SQC are processes defined within CMMI, they are under the Support process area.
    In CMMI SQA\SQC is defined as Process and Product Quality Assurance.

    CMMI is an approach to process improvement, in which SQA\SQC play a major but not exclusive role.
    Everyone in a software development organization takes part in both the CMMI processes and any improvement initiatives for those processes.

    Each of the main Engineering process areas is now described together with the role that SQA\SQC plays within those areas.





    • SQA and SQC roles in CMMI Requirements Development

    The CMMI Requirements Development process area describes three types of requirements:-
    customer requirements, product requirements, and product-component requirements.

    SQA role To observe (audit) that documented standards, processes, and procedures are followed.
    SQA would also establish software metrics in order to measure the effectiveness of this process.
    A common metric for measuring the Requirements process would be the number of errors (found during system testing) that could be traced to inaccurate or ambiguous requirements (note: SQC would perform the actual system testing but SQA would collect the metrics for monitoring and continuous improvement).

    SQC role SQC takes an active role with Verification (this is a process itself that is described later).
    Verification of the requirements would involve inspection (reading) and looking for clarity and completeness.
    SQC would also verify that any documentated requirement standards are followed.

    Note there is a subtle difference between SQA and SQC with regard to standards, SQC’s role is in verifying the output of this process (that is the Requirement document itself) while SQA’s role is to make sure the process is followed correctly.

    SQA is more of an audit role here, and may sample actual Requirements whereas SQC is involved in the Verification of all Requirements.

    The type of requirement need not be just the functional aspect (or customer\user facing requirements) they could also include product and\or component requirements.

    The product requirements e.g. Supportability, Adaptability and Reliability etc. are characteristics discussed here (as part of the FURPS+ model).
    The respective roles of SQC and SQA is the same for all types of requirement (customer and product) with SQC focusing on the ‘internal deliverable’ and SQA focusing on the process of how the internal deliverable is produced, as per the formal definition.

    SQA and SQC roles in CMMI Requirements Management



    The purpose of (CMMI) Requirements Management is to manage the requirements of the project's products and product components and to identify inconsistencies between those requirements and the project's plans and work products.

    This process involves version control of the Requirements and the relationship between the Requirements and other work products. One tool used in Requirements Management is a Traceability Matrix.

    The Traceability Matrix maps where in the Software a given requirement is implemented, it is a kind of cross reference table.
    The traceability matrix also maps which test case verify a given requirement.

    There are other processes within Requirements Management and CMMI should be referenced for further information.

    SQA role To observe (audit) that documented standards, processes, and procedures are followed.
    SQA would also establish metrics in order to measure the effectiveness of this process.
    A common metric for measuring the Requirements Management would be the how many times the wrong Version was referenced.
    Another measure (for the Traceability Matrix) would be lack of test coverage,
    that is defects detected in the shipped product that were not tested due to the fact that they were not referenced in the Traceability matrix that referenced the requirements.

    SQC role As with the actual Requirements Development, SQC would be involved in inspecting the actual deliverable’s (e.g. Traceability Matrix) from this process.

    SQC may also get involved at this stage as they will be the people doing the actual Testing (Verification and Validation) so for their test coverage a complete Traceability Matrix is essential.

    SQA and SQC roles in CMMI Technical solution

    The purpose of (CMMI) Technical Solution is to design, develop, and implement solutions to requirements.

    Solutions, designs, and implementations encompass products, product components, and product-related life-cycle processes either singly or in combinations as appropriate.

    This is the main Design and Coding processes.
    CMMI puts the design and build together. Other important processes, e.g. Configuration Management are listed in other process areas within CMMI.

    SQA role To observe (audit) that documented standards, processes, and procedures are followed.
    SQA would also establish metrics in order to measure the effectiveness of this process. Clearly testing the end product against the requirements (which is itself a SQC activity) will reveal any defects introduced during this (the Technical solution) process.
    The number of defects is a common measure for the Design\Build phase. This metric is usually further quantified by some form of scope, for example defects per 100 lines of code, or per function.

    It is important that the defect may not always be a functional (or Customer facing defect) it could be that a required adaptability scenario is absent from the Design and\or coded solution.
    The FURPS+ model references typical Software Metrics that are used for specifying the total (both functional and non-functional) software requirements.

    SQC role The major SQC role during this process will be testing (see Validation and Verification).

    The finished product does not have to be present before testing can begin. Unit and Component can both take place before the product is complete. Design and Code reviews are also something that SQC could get involved with.

    The purpose of the review has to be clearly stated, i.e. to verify standards are followed or to look for potential Supportability (part of the Product Requirements) issues. The Supportability metric is the time it takes for a defect in a system to be fixed. This metric is influenced by the complexity of the code, which impacts the developer’s ability to find the defect.

    SQA and SQC roles in CMMI Product Integration

    The purpose of Product Integration is to assemble the product from the product components, ensure that the product, as integrated, functions properly, and deliver the product. Note this is the final Integration and ‘move to production’ or product delivery. For large Software packages (consider SAP, Oracle Financials etc.) the assembly process is huge and the potential for errors is high. This process does not involve any coding but pure integration and\or assembly. SQA role: To observe (audit) that documented standards, processes, and procedures are followed. SQA would also establish metrics in order to measure the effectiveness of this process. One measurement for this would be the defects found that resulted from the interface specifications (part of the Product requirements), potential process improvements could be to find other, perhaps less ambiguous ways, of specifying interfaces. For example a development team may move to XML or Web Services for all interfaces, SQA could then measure the defects and report back to management and development as to the effectiveness of this change. SQC role: Again testing would be a large role played by SQC. Systems Integration Testing (SIT) would be carried out by SQC. Also install ability testing would be done during this process.

    SQA and SQC roles in CMMI Verification

    The purpose of Verification is to ensure that selected work products meet their specified requirements Theses activities are only carried out by SQC, the role of SQA would be to make sure that SQC had documented procedures, plans etc. by audit. SQA would also measure the effectiveness of the Verification processes by tracking defects that were missed by SQC during Verification. Note the term Verification, as opposed to Validation (see below). In essence Verification answers the question ‘Are we building the product correctly’ while Validation answers the question ‘Are we building the correct product’. Validation demonstrates that the product satisfies its intended purpose when place in the correct environment while Verification refers to building to specification. The FURPS+ model identifies both Customer and Product requirements; Verification applies to both these types of requirements and can be applied to the intermediary work products. Design or Code reviews are examples of Verification. These terms Verification and Validation are often mixed, CMMI makes this comment about the distinction:- Although “verification” and “validation” at first seem quite similar in CMMI models, on closer inspection you can see that each addresses different issues. Verification confirms that work products properly reflect the requirements specified for them. In other words, verification ensures that “you built it right.” While SQC carries out all the Verification activities the Verification process itself is still subject to SQA and process improvement.

    SQA and SQC roles in CMMI Validation

    Validation confirms that the product, as provided, will fulfill its intended use. In other words, validation ensures that “you built the right thing.” As with Verification, Validation is mainly the domain of SQC. The term Acceptance Test could also apply to Validation, in most cases the Acceptance test is carried out by a different group of people from the SQC team that performed Verification, as the product was being built. In the case where an application is going to be used internally, then the end user or business representative would perform the Acceptance testing. Wherever this is done it is in essence a SQC activity. As with Verification, SQA makes sure that these processes conform to standards and documented procedures. The Validation process itself is subject to continuous improvement and measurement.

    Conclusion

    Although only a high level snapshot has been given of how some SDLC processes are subjected to SQA and SQC a clear pattern can be seen. In all cases the SQA and SQC do not get involved in building any of the products. SQC are only involved in Verification and Validation. The role of SQA is even more removed from development; they are mainly providing the role of an Auditor. In addition SQA will collect measurements of the effectiveness (and cost) of the processes in order to implement continuous process improvement. This separation of SQC from development and SQA from SQC ensures objectivity and impartiality. In an ideal environment theses (Development, SQC and SQA) would be three separate organization units reporting to different managers. Some of the benefits of SQA\SQC can be achieved in a less formal environment. This hybrid approach is typically used by small development groups. An example of this hybrid approach is documented here at SQA in Practice.

    No guarantee (or claim) is made regarding the accuracy of this information. Any questions or comments should be sent to:-