Testing Process

Understanding the testing process requires starting from the software life cycle. Different development models call for different testing approaches, and various testing models help organize testing activities in a structured way. This article covers software development models, testing models, and the main types of testing performed across the software lifecycle.

Software Life Cycle

What process does software testing go through? This starts with the software life cycle.

The software life cycle (also called software life span or system development life cycle) is the life cycle of software from creation to retirement. The entire life cycle includes problem definition and planning, requirements analysis, system design, software programming, software testing, software maintenance, and other phases.

Within this cycle, both development and testing rely on some model as a basis to effectively improve development and testing efficiency.

Software Development Models

In the practice of software development, many development models have been summarized to describe and represent complex development processes — such as the Waterfall model, the Rapid Prototype model, the Spiral model, and others.

Software testing has a close relationship with software development models. As a tester, you should fully understand the software development model, find your position quickly, and begin contributing your value as soon as possible.

Waterfall Model

The Waterfall model is a type of linear model, occupying an important position among all models and serving as the foundation for all others.

The Waterfall model is like the process of building a house at a construction site: it uses milestones and strictly defines the inputs and outputs of each development phase. If the required output is not met, the next phase does not begin.

From a testing perspective, after development is complete, sufficient time must be reserved for testers — otherwise testing may be insufficient, causing many problems to surface only in the later stages of a project.

Advantages

• Clearly divides each phase of the software life cycle.
• Emphasizes early software planning; requirements analysis is important.
• Clear workflow makes it easy to divide tasks and collaborate.
• Suitable for product development with stable requirements.
• Each phase has a checkpoint.

Disadvantages

• Linear development process carries significant risk.
• Relies on early requirements investigation; unsuitable for changing requirements, and the single flow is irreversible.
• Risks may only surface in later phases and cannot be corrected, leading to project failure.
• Cannot guarantee that user product requirements won’t change; the development process cannot be altered. For example, if you build a foundation for a 7-story building according to blueprints and the user suddenly wants 5 more floors, the foundation must be rebuilt and the existing structure demolished — obviously an impossible operation.

Improvements

The linear thinking of the Waterfall model is retained, but phases are further refined. Iterative thinking is incorporated between certain critical phases.

Problem Definition and Planning Determine the purpose of software development and its feasibility; formulate a development plan.

Requirements Analysis After confirming feasibility, perform detailed analysis of the functions the software needs, clarify client requirements, and output prototype diagrams.

Software Design High-level design: architectural implementation, describing module functions, module interfaces, and data transfer. Detailed design: in-depth analysis of each described module, requiring code-level detail — describing the specific implementation of the program, as well as database design.

Software Coding Based on the detailed module function description, programmers write computer-executable code.

Rapid Prototype Model

Before developing the real system, construct a prototype. Based on that prototype, gradually complete the development of the entire system.

The first step is to build a rapid prototype to enable user-system interaction. Users evaluate the prototype and further refine the requirements of the software to be developed. By gradually adjusting the prototype to satisfy users, developers can determine what users truly need.

The second step, based on the first step, is to develop a software product that satisfies users.

Advantages

Overcomes the weaknesses of the Waterfall model. Better satisfies user requirements and reduces project development risk caused by unclear software requirements. Suitable for software development where requirements cannot be precisely defined in advance. Suitable for developing small, highly flexible systems.

Disadvantages

Not suitable for large-scale system development. Requires an existing demonstrable product prototype as a reference, which may limit developer creativity to some extent.

Spiral Model

Divides the development process into spiral cycles. Each spiral cycle corresponds to the Waterfall model, rotating along the spiral line. The four quadrants of the coordinate axis represent four activities.

Proposed in 1988 by Barry Boehm as an upgraded version of a software development model, it incorporates the advantages of the Waterfall model.

The core of the Spiral model is that you don’t need to define everything clearly at the start. Start light: define the most important features, implement them, then listen to client feedback, and move on to the next stage. This cycle repeats until a final product satisfying users is obtained.

• Plan: Determine software goals, select implementation plans, clarify project development constraints.
• Risk Analysis: Analyze and evaluate selected plans; consider how to identify and eliminate risks.
• Engineering: Implement software development and verification.
• Customer Evaluation: Evaluate development work, propose correction suggestions, formulate the next plan. The Spiral model is largely a risk-driven approach, because risk assessment must be performed before each phase and before each recurring cycle.

Advantages of the Spiral Model Introduces risk analysis not present in other models, allowing software to stop when risks are encountered, reducing losses. Requires building prototypes at each iteration phase and performing software testing to reduce project development risk. The entire process has high flexibility; any phase of the development process can freely respond to changes.

Disadvantages Requires testers to have extensive risk assessment experience and professional knowledge to identify risks in time and reduce software defect losses. Too many review iterations create development cost pressure.

Agile Development Model

External link

Software Testing Models

Software testing is part of software engineering, and an indispensable part of the overall software engineering process.

In companies where software engineering, project management, and quality management are standardized, software testing will proceed more smoothly and deliver greater value.

We need to focus on the relationship between software engineering, project management, quality management, configuration management, and software testing, and understand how to perform software testing under different software development models.

As software testing has evolved, testers have summarized some testing models through extensive practice, abstracting testing activities — such as the V-model and W-model.

V-Model

The V-model is the most representative testing model. It was first proposed by Paul Rook in the late 1980s and published by the British National Computing Centre, aiming to improve the efficiency and effectiveness of software development.

Before the V-model was introduced, people generally treated testing as a phase that came after requirements analysis, high-level design, detailed design, and coding were all complete. Even though at that time some testing work had already been shown to occupy half the project timeline, most people still considered testing merely a wrap-up activity. The V-model was introduced to change this common perception.

The V-model itself is a variant of the Waterfall model in software development. It reflects the relationship between testing activities and analysis and design.

The V-model marks the different phases that exist in the testing process. From left to right, it describes the corresponding relationships between development phases.

The V-model and the Waterfall development model share some common characteristics. The V-model’s process from left to right describes the basic development process and testing behaviors.

The value of the V-model lies in clearly marking the different levels that exist in the testing process, and clearly describing the correspondence between these testing phases and the phases during the development process.

Limitation: it treats testing as the last activity after coding, meaning errors produced during early phases like requirements analysis cannot be discovered until later acceptance testing.

Advantages of the V-Model

• The V-model includes both low-level and high-level testing:
  • Low-level testing, such as unit testing.
  • High-level testing, such as system testing.
• The V-model clearly marks each phase of software development.
• The V-model’s top-down, stepwise refinement approach divides the entire development process into distinct phases with clear work at each phase, making it easy to control the development process. When all phases are complete, the software development process ends accordingly.

Disadvantages of the V-Model

• The disadvantage is also obvious — its inherent sequentiality means that testing only begins after reaching the testing phase. But some errors are only discovered at that point, or may not be discovered at all; irreparable damage may have already been done.
• Additionally, in actual development, it is difficult to fully clarify user requirements during the requirements phase. When requirements change, phases must be repeated (re-analyzing requirements, design, coding, testing, etc.), causing enormous rework and making the model relatively inflexible.

Improvement: Just like the improvements to the Waterfall model, perform small iterative tasks within relevant steps.

W-Model

IEEE Std 1012-1998, Software Verification and Validation, established the principle that testing activities must also occur during the software requirements and design phases.

The W-model was proposed by the company Evolutif. In software development, one V for development and one V for testing combine to form a W. Therefore the W-model is also called the dual-V model.

Testing accompanies the entire software development cycle, and the test subjects are not only programs — requirements and design phases also require testing.

The development-testing coordination diagram shows the development V in green and the testing V in orange.

Advantages of the W-Model

• Testing accompanies the entire software development life cycle; test subjects are not only programs — requirements and high-level design also require testing.
• Earlier test involvement can better discover defects in requirements and design, at lower repair cost.
• Phase-by-phase work makes it easy to control the project process.

Disadvantages of the W-Model

• Still relies on the linear sequential relationship between software development and testing; still cannot solve the problem of requirements change adjustments. That is, it cannot support iteration, spontaneous activity, and requirements change adjustments.
• For some projects where documentation is not produced during execution, the W-model is basically inapplicable (small companies may produce prototype diagrams directly, without writing requirements specifications).
• The technical complexity of the W-model is high, placing high demands on requirements design and testing personnel; the threshold for practice is relatively high.

In general, only medium to large enterprises use the W-model.

H-Model

In practice, people have found that although requirements, design, coding, and other activities in software development are divided into phases, they are not executed completely in series — more often they are cross-cutting and iterative.

Therefore, some experts proposed the H-model, which completely separates activities into fully independent processes, while also clearly representing testing preparation and test execution.

H-Model Testing Process

• Test preparation: preparation for all test execution activities; determine whether the testing readiness point has been reached.
• Testing readiness point: the test entry criteria — the condition for whether testing can begin.
• Test execution: the specific programs executed during testing.

Other Processes

• Specific processes in development, such as design processes and small iterative tasks.

Advantages of the H-Model

• The H-model reveals that software testing requires much more work beyond test execution.
• Software testing is completely independent, runs throughout the entire life cycle, and proceeds in parallel with other processes.
• Software testing activities can enter the preparation phase as early as possible, be executed early, and have high flexibility.
• Software testing can be multi-level, phased, and sequenced based on the different test subjects, and can also be iterated.

Disadvantages of the H-Model

• High management requirements: due to the model’s flexibility, correspondingly clear planning and management systems are required. Otherwise the testing process will be very difficult to manage and control.
• High skill requirements: the H-model requires a good ability to define the scope of each iteration — neither too large nor too small.
• Difficulty analyzing testing readiness points: during the testing phase, it is unclear how prepared testing preparation needs to be — that is, the readiness point is hard to grasp. For example, where is the readiness point? What is the standard for readiness? This creates great difficulty for test execution.
• Very high personnel quality requirements for the entire project: under standardized systems, work efficiency is high, but insufficient personal skills can cause a single point to create problems and disrupt the entire project’s progress.

While the H-model is good, it is rarely used because the demands it places on everyone — from management to individual project team members — are very high.

Summary of V, W, and H Models

The V-model is suitable for small and medium enterprises.

The W-model is suitable for medium and large enterprises, because of its high requirements for project team members.

The H-model places very high demands on project team members and is rarely adopted by companies.

Test Classification

Classification by Software Testing Role

• Functional testing: verifying whether the main functionality of the current software is usable.
• Automated testing: using programs to test programs (testing APIs), freeing up manual effort.
• Performance testing: locating system bottlenecks, ensuring good system performance and stability.
• Black-box testing (data-driven testing): does not focus on underlying code implementation; tests application functionality and verifies that results are correct. Only cares about software inputs and outputs.
• White-box testing (structural testing): the test subject is the underlying code; it doesn’t focus on whether the external interface looks OK, only that underlying functionality is implemented and logic is correct. In other words, the box is opened to study the internal source code and program structure.
• Gray-box testing: between black and white box testing — interface testing.

Classification by Whether the Test Subject Is Executed

• Static testing: does not actually execute the software under test; statically inspects program code, interfaces, or documents for possible errors. For example, testing an interface document — it’s all text, so how do you execute it? These testers may include a moderator, internal reviewers, the author, observers, user representatives, a recorder, and technical staff all meeting together.
• Dynamic testing: actually runs the software under test, inputs corresponding test data, and checks whether the output matches expectations.

Classification by Software Testing Function

By Test Subject

• Black-box testing
• White-box testing
• Gray-box testing

The “box” in these three methods refers to the subject under test.

By Testing Method

• Manual testing: the core is the person; based on individual subjective judgment. Advantage is the ability to flexibly change testing operations and environment.
• Automated testing: testers use scripts to drive testing tools to automatically complete tests, with some human cooperation still required. Also uses third-party tools to test against the subject under test. Advantage is the ability to efficiently achieve operations that humans cannot (such as testing a website’s concurrency load).

By Testing Phase

• Development phase testing:
  • Unit testing (component, module, file, function, class testing)
  • Integration testing (assembly testing, interface testing, data structure testing)
• Testing of an existing formed product:
  • System testing (combining software with computer hardware, support software, data, personnel, etc. to test the product)
• Post-production testing — after the product is working:
  • Acceptance testing (formal acceptance testing, Alpha testing, Beta testing)

By Test Content

• Functional testing
• UI testing
• Security testing (banking and financial enterprises): verifying that software only provides functionality to authorized users.
• Compatibility testing: verifying that software is usable in different environments.
• Usability testing
• Performance testing: verifying software’s resource consumption and output capability.
• Stress testing
• Recovery testing
• Smoke testing: the term “smoke testing” originates from the hardware industry — after changing or repairing a hardware component, directly power up the device. If there is no smoke, the component passed the test. In software, smoke testing describes the process of verifying code changes before embedding them into the product’s source tree. After reviewing code, smoke testing is the most cost-effective method of determining and fixing software defects. Smoke testing is designed to confirm that changes in code will run as expected and not break the stability of the entire build.

Testing Phases

Generally, the tests that software goes through from development to delivery include:

• Black-box testing
• White-box testing
• Unit testing
• Integration testing
• System testing
• Acceptance testing

But the roles of testers are constantly changing:

• Developers perform at different development stages: black-box testing, white-box testing, unit testing.
• Testers perform during the testing cycle: black-box testing, integration testing, system testing.
• Users perform: acceptance testing.

Based on different scopes, testing can be broadly divided into unit testing, integration testing, system testing, and acceptance testing. This reflects a testing process that moves from small to large, from inside to outside, step by step — the principle of divide and conquer.

Unit Testing

Unit testing (UT) refers to inspecting and verifying the smallest testable unit in software. The meaning of “unit” in unit testing generally depends on the actual situation. Unit testing has the smallest granularity and is generally performed by development teams using the white-box method, primarily testing whether units conform to the design.

Black-Box Testing

Black-box testing does not consider the internal structure and logical structure of the program. It is mainly used to test whether the system’s functions meet the requirements specification. Generally there will be an input value compared against expected values. Black-box testing is also called functional testing — it tests whether each function can work properly. During testing, the program is treated as an unopened black box. Without considering the internal structure and characteristics of the program at all, testing is performed at the program interface. It checks whether the program functions correctly according to the requirements specification, and whether the program can appropriately receive input data and produce correct output information. Black-box testing focuses on the program’s external structure, not internal logical structure, and is mainly directed at software interface and software functionality. It provides reports on the software’s actual usage effects.

For example, a TV remote control is a standard black-box test — you don’t need to know whether it uses LCD or other technology. Multiple sets of data and multiple tests are required to obtain an accurate report.

White-Box Testing

White-box testing is mainly applied in the unit testing phase. It is mainly code-level testing, targeting the internal logic structure of programs. Testing methods include:

• Statement coverage
• Decision coverage
• Condition coverage
• Path coverage
• Condition combination coverage

White-box testing (also called structural testing or logic-driven testing) tests the program according to its internal structure. It tests whether the internal actions of the product proceed normally according to the design specification, and checks whether every path in the program can work correctly as expected. This method treats the test subject as an open box. Testers design or select test cases based on the program’s internal logical structure, test all logical paths of the program, and determine whether the actual state is consistent with the expected state by checking the program’s state at different points.

Compared to black-box testing, white-box testing is more rigorous. It tests the source code and architecture of the software and requires source code, flowcharts, and other design documents.

Black-box and white-box testing complement each other: after white-box testing passes, the software still needs to run to check performance quality, UI aesthetics, usability, and so on.

Gray-Box Testing

Gray-box testing sits between white-box and black-box testing. It maintains the focus of black-box testing while having insight into the internal structure of white-box testing, without examining the internal program and operations in detail. Gray-box testing combines elements of both white-box and black-box testing.

Integration Testing

Integration testing (IT, system integration test), also called assembly testing, sits between unit testing and system testing and serves a bridging role. It is generally performed by development teams using a combination of white-box and black-box methods, verifying both the design and the requirements. It is primarily used to test interfaces between modules, as well as some main business functions.

Integration testing is the logical extension of unit testing. In its simplest form: two already-tested units are combined into a component, and their interface is tested. In practice, many units are combined into components, which are further aggregated into larger parts of the program. The approach is to test combinations of segments, eventually expanding to processes — testing modules together with modules from other groups. Finally, all modules making up a process are tested together. Additionally, if a program consists of multiple processes, they should be tested in pairs rather than all integration test processes simultaneously.

System Testing

System testing (ST) has the largest granularity. It is generally performed by an independent testing team using the black-box method, primarily testing whether the system conforms to the requirements specification. After confirmation through the above phases, the entire system is simulated in the customer’s environment for testing.

System testing integrates confirmed software, computer hardware, peripherals, networks, and other elements for various assembly and validation tests. Its purpose is to compare against system requirements, discover contradictions or inconsistencies between the developed system and user requirements, and propose more complete solutions. Its task is to check out program errors as thoroughly as possible and improve software system reliability. It can be divided into three steps:

• Module testing: testing whether each module’s program has errors.
• Assembly testing: testing whether interfaces between modules are correct.
• Validation testing: testing whether the entire software system meets user functional and performance requirements.

Acceptance Testing

Acceptance testing (AT) is similar to system testing; the main difference is the testers — acceptance testing is performed by users. It is generally divided into:

• Alpha testing (closed beta / internal testing): Alpha testing simulates acceptance testing in an actual operating environment. The software is only a preliminary product that may have many bugs and will not go live for user access. For example, a closed beta test with character wipes.
• Beta testing (open beta / public testing): Beta testing means the system has passed internal testing and most errors have been fixed. It is about to be officially released and is released in multiple real environments. For example, an open beta without character wipes. Compared to the Alpha version there are significant improvements, but some bugs may still exist and large-scale testing is needed.
• Gamma version: Indicates the release candidate version of the software, nearly identical to the final release. The Gamma version is also called the RC (Release Candidate) version.
• UAT testing (User Acceptance Testing): The UAT phase of testing is no longer done by the software developer’s own testing team, but by customers using the software according to their actual business scenarios to test specific functions.

Random Testing

Random testing is also called exploratory testing.

It mainly involves retesting some important functions of the software under test, and also includes testing the parts not covered by current test cases. In addition, it focuses on testing updates and new features of the software, especially testing special situations, special usage environments, concurrency, and so on. It also includes retesting major bugs discovered in previous testing.

Random testing can be combined with regression testing.

Regression Testing

Regression testing refers to re-testing after modifying old code to confirm that the modification did not introduce new errors or cause other code to produce errors. Automated regression testing significantly reduces costs in system testing, maintenance, and upgrade phases.

Regression testing, as a component of the software life cycle, accounts for a large proportion of the workload throughout the software testing process. Multiple rounds of regression testing are performed at various phases of software development. In progressive and rapid iterative development, continuous release of new versions makes regression testing increasingly frequent. In extreme programming, multiple regression tests are required each day. Therefore, selecting the correct regression testing strategy to improve regression testing efficiency and effectiveness is very meaningful.

Regression Testing Process The following process applies to unit testing, integration testing, and system testing:

• During the testing strategy formulation phase, formulate the regression testing strategy.
• Determine which versions require regression testing.
• Release the regression testing version; execute regression testing according to the regression testing strategy.
• If regression testing passes, close the defect tracking ticket.
• If regression testing does not pass, the defect tracking ticket is returned to developers. Developers re-fix the issue and submit again for testers to perform regression testing.

Canary Release (Gray Release / Testing)

Canary release (also called gray release or gray testing) is a type of release that enables smooth transition between old and new versions. A/B testing can be performed on it — letting some users continue using product feature A while other users start using product feature B. If users have no objections to B, the range is gradually expanded until all users are migrated to B.

Canary release gradually expands the user group for a product release — also called gray rollout. Canary release ensures overall system stability; problems can be discovered and adjusted at the beginning of the canary phase to ensure limited impact.

Canary period: The period from the beginning to the end of canary release is called the canary period.

Significance of Canary Release

Canary release enables early collection of user feedback, improvement of product features and quality, user participation in product testing, enhanced user interaction, and reduction of the user impact scope of product upgrades.

Steps of Canary Release

• Define goals
• Choose strategies: including user scale, release frequency, feature coverage, rollback strategy, operations strategy, old/new system deployment strategy, etc.
• Screen users: including user characteristics, user count, frequently used features, user scope, etc.
• Deploy the system: deploy the new system, deploy user behavior analysis systems (web analytics), set traffic splitting rules, analyze operations data, fine-tune traffic splitting rules
• Release summary: user behavior analysis reports, user questionnaires, social media opinion collection, forming a product feature improvement list
• Product improvement
• Next round of canary release or full release

Exhaustive Testing

Although we have proposed so many test cases, there will certainly still be testing methods not covered.

Exhaustive testing refers to a testing method that contains all possible combinations of software input values and preconditions. A system that completes exhaustive testing should have no unknown software defects remaining, because more testing must be done to find them.

However, in the vast majority of software projects, testing is constrained by time and economic costs, making exhaustive testing impossible. Instead, it is based on a risk-driven model, focusing test case design on seeking a balance between defects and development costs.

Comparison of Unit, Integration, and System Testing

Different Testing Methods

• Unit testing belongs to the white-box testing category.
• Integration testing belongs to the gray-box testing category.
• System testing belongs to the black-box testing category.

Different Scope of Examination

• Unit testing primarily tests data structures, logical control, and exception handling within a unit.
• Integration testing primarily tests interfaces between modules, interface data transfer relationships, and the overall functionality of combined modules.
• System testing primarily tests the degree to which the entire system conforms to requirements.

Different Evaluation Benchmarks

• The evaluation benchmark for unit testing is primarily logical coverage rate.
• The evaluation standard for integration testing is primarily interface coverage rate.
• The evaluation benchmark for system testing is primarily the coverage rate of test cases against the requirements specification.

Software Testing Classification and Phases

Software testing has many classifications, with varying terminology, and the classifications have a certain degree of interoperability and overlap.

https://img2018.cnblogs.com/blog/1168165/201911/1168165-20191105004226647-1736627202.png