Test Confidence

Someone on Linkedin asked a question on how-to derive a “Test Confidence” ratio? Here is how I responded…

I think a “Test Confidence” is best derived by experiencing the application first-hand over a period of time or soaking the expert opinion of someone or some people who have. Whether it is bugs or issues or plain-speak about application behaviour / functionality / performance / usability / other ‘ities’ what you/they feel based on tests that have been performed will be very helpful indeed. Maybe a questionnaire can be compiled with some rating system for the tested application. If this is answered by people who know the business that the application supports and have diligently traversed its length & breadth like a passionate tourist would in a city he has visited armed with atlases and guides’ in tow then you have information which is reliable and can both support & justify your cause.

This information is rarely captured and presented since management trust numbers being the easier to present n peek and graphs based on it are such pretty things.

Numbers/Metrics will have to be analysed and interpreted carefully.

If it is a decreasing count of defects across various phases of testing or across different cycles of the same phase (say System Testing) you will have to determine that the environment is the same (different browsers, versions, OS, versions, etc.), testers are the same or having the same skill, build has been more or less consistent in terms of features developed / delivered for testing and fixes have been brilliant and not caused any regression and there has been as much time spent actually on testing as planned by you. If this is the situation you are in – your job is done.

But as you know, it seldom is.

Metrics with weights for each factor mentioned above to make the eventual number less or more maybe the way to go if there is a lot of pressure on you to produce a ‘Test Confidence’ number. The risk is that the numbers more likely gets convoluted beyond a point and may stop telling the story you set out to. If this happens, then defending the numbers rather than providing an no-holds barred assessment of the application tested would I think jeopardize the existence of both the testing team and the organisation it belongs to!