Test Planning & Estimation

Recently I answered a query on Test Estimation and Planning on a very popular forum.

On TEST ESTIMATION here is what I suggested

1. Understand the domain thoroughly.
2. Break the domain you understood into as small a pieces as possible say Login, Add Customer,… & so on.
3. Compile a list of domain aspects where clarity is lacking in terms of poor documentation, clarifications pending, & so on.
4. Compile a list of assumptions you made in terms of understanding the domain.
5. For each small piece of functionality – classify it as Simple, Medium & Complex in terms of domain complexity.
6. Try & list the non-functional requirements (NFR) separately (testing across x browsers/platforms like, usability requirements, testing across languages & so on).
7. For each granular piece (functional) try & approximate the no. of test cases/scenarios required.
8. Now arrive at effort (based on past experience, skills) for writing & executing test cases/scenarios being say x & y test cases per day. These will be different for the complexity factors. You may need to factor the impact of test data here in terms of env. setup, design & execution time.
9. Consider Review & Rework effort as % of effort for writing the test cases.
10. Likewise arrive at effort for executing the test cases. Ensure that you include a % of effort for regression testing, exploratory testing, defect reviews done by test lead, reporting & client updates.
11. Consider how the build will be delivered to the test team. If they are delivered in an iterative fashion you will need to compute effort for each piece incrementally along with regression, integration testing (with previous delivered pieces) and system test cycles at the end. If they are delivered as a single piece then effort for system testing & regression testing will be computed. How many cycles of System Test will be done is again an approximation dependent on factors like past experience, dev./test team maturity, customer expectations of deliverable & so on.
12. Consider Domain understanding by team as a % of Design + Execution.
13. Consider effort for testing the non-functional requirements (using logical & common-sensical approach).
14. So eventually you will have effort in buckets of Domain Understanding, Design (FR & NFR – Preparation, review, rerwork), Env. Setup, Execution across x cycles (FR & NFR – test case exec., defect reviews, exploratory, regression) and Reporting.
15. For the ‘blurred’ domain aspects (point 3) you listed separately you may either not compute effort or factor it within a contingency bucket as a % of total effort.

There are other approaches related to Function Points, COCOMO but they are as good as any one else. My advise is speak with people who are system matter experts and better still if they have tested it since their inputs would be critical.

Please understand that estimation is inexact, an approximation, fallible and prone to debates that go nowhere. They cause headaches sometimes but they need to be done using an approach which is justifiable and understandable.

Please note that despite the above you may find one day that all you have is a miniscule portion of the estimated effort left for you to test & deliver the goods and you will have to do the best within that time to test as effectively and efficiently as allowed by the delivered system and how well it is understood, designed & developed, people at your disposal to test & the skills / minds they possess, business dynamics and the management support for testing.

Here are my suggestions for PLANNING/SCHEDULING.

1. At the outset please note that there is nothing sacrosanct about planning in the software development world. While you must plan and undoubtedly planning is key to execution and success, one has to be continually focussed on the ground situation vis-a-vis the plan. And in tune with it, be ready to re-plan and re-adjust your priorities should the need arise Eg. delayed build delivery, poor design/deliverable full of bugs, key personnel resigning, requirements changed by client & so on.
2. As has been pointed out risk-based testing is done by most but its effectiveness can be achieved dependent on how well you know the domain and how efficiently you collaborated with the customer, subject matter experts to unravel the ups & downs of the application.
3. If the product being tested is large, it is a good idea to appoint module leads to focus learning and testing better. That way you also foster the development of the next line of leaders. You may have module leads for even special/non-functional areas also so that any study required can be done say for testing accessibility or say usage of tool requiring an understanding before implementation. These module leads can even backup the Primary Lead & help in reviews, reporting & client coordination.
4. Please ensure that the environment & data aspects are understood and set in motion during the planning state and the dialog carried out with development team (build mode, env prerequisites etc.) and customer (production/representative data, env aspects etc.)
5. Please pay attention to domain understanding by the team. The big picture understanding & in-depth understanding to areas handled by individual test personnel is critical to success especially for domain intensive applications. to It is a good idea to brainstorm, ensure reverse presentation of key areas, participation of test personnel in download sessions by Business Analysts / SME’s, encourage queries and thinking aloud on functionality & so on. The test team must be vibrant on knowledge and vibrate with passion to dig and make a difference.
6. It is a good idea to keep swapping test personnel across functional areas after one has spent a reasonable time testing it because ‘fresh eyes generally find fresh problems’.
7. In planning aspects it is important to seek the team’s inputs from time to time on application behaviour & complexity aspects so that right decisions may be made. Transparency is also as important so that everyone knows about looming milestones ahead giving them some leeway to make their individual plans to achieve team goals.
8. It is a common practice to leave a period with no tasks within the test plan for contingencies although it gets eaten up whatever you do such are the vagaries and unpredictability of software development.
9. Continual and day-one onwards involvement is strongly suggested of test personnel with the happenings in the project with the development personnel & the customer representatives to resolve queries and clarifications and build up knowledge of the domain/application.

Here are my suggestions for Test planning/scheduling.
1) At the outset please note that there is nothing sacrosanct about planning in the software development world. While you must plan and undoubtedly planning is key to execution and success, one has to be continually focussed on the ground situation vis-a-vis the plan. And in tune with it, be ready to re-plan and re-adjust your priorities should the need arise Eg. delayed build delivery, poor design/deliverable full of bugs, key personnel resigning, requirements changed by client & so on.
2) As has been pointed out risk-based testing is done by most but its effectiveness can be achieved dependent on how well you know the domain and how efficiently you collaborated with the customer, subject matter experts to unravel the ups & downs of the application.
3) If the product being tested is large, it is a good idea to appoint module leads to focus learning and testing better. That way you also foster the development of the next line of leaders. You may have module leads for even special/non-functional areas also so that any study required can be done say for testing accessibility or say usage of tool requiring an understanding before implementation. These module leads can even backup the Primary Lead & help in reviews, reporting & client coordination.
4) Please ensure that the environment & data aspects are understood and set in motion during the planning state and the dialog carried out with development team (build mode, env prerequisites etc.) and customer (production/representative data, env aspects etc.)
5) Please pay attention to domain understanding by the team. The big picture understanding & in-depth understanding to areas handled by individual test personnel is critical to success especially for domain intensive applications. to It is a good idea to brainstorm, ensure reverse presentation of key areas, participation of test personnel in download sessions by Business Analysts / SME’s, encourage queries and thinking aloud on functionality & so on. The test team must be vibrant on knowledge and vibrate with passion to dig and make a difference.
6) It is a good idea to keep swapping test personnel across functional areas after one has spent a reasonable time testing it because ‘fresh eyes generally find fresh problems’.
7) In planning aspects it is important to seek the team’s inputs from time to time on application behaviour & complexity aspects so that right decisions may be made. Transparency is also as important so that everyone knows about looming milestones ahead giving them some leeway to make their individual plans to achieve team goals.
8) It is a common practice to leave a period for contingencies although it gets eaten up whatever you do such are the vagaries and unpredictability of software development.
9) Continual and day-one onwards involvement is strongly suggested of test personnel with the happenings in the project with the development personnel & the customer representatives to resolve queries and clarifications and build up knowledge of the domain/application.