There is a thing about acceptance tests which sometimes became a talking point with me involved that tended to gravitate away from logic. There is an apprehension (which is normal and understandable) associated with these tests and the bugs logged with thoughts that some would likely source nasty emails asking questions about ‘quality’. The tests and the emails do necessarily carry weight since they are courtesy those who know their applications and the way around it like their own house. The discussions with top management people around these acceptance tests and how to prevent the ‘dirty’ emails even kept me somewhat amused at times.
In one place I worked at – a delivery head thundered in one of the meeting rooms that customers better inform about the test cases exactly as they will execute by documenting and sharing it upfront. His view was that being an application which had high number of permutations in terms of business scenarios and underlying complexity it was impossible to test them all and the acceptance tests would stretch on and on with a stockpile of bugs if the scope was not restricted and the customer not tied by a contractual clause of not going beyond the documented test cases…
I had tried to reason this on 3 fronts:
- any client worth his salt may not agree to acceptance testing being restricted and limited to ‘documented’ testing.
- the business users would test for a product satisfying as many scenarios (as normally executed in their daily lives) as possible within the available time. Business users would also given the way they think and work tend to test the application in exploratory form and not want their exploration/testing to be tied down by scripted test cases and the regimentation it requires.
- testing anyway is never 100% but we from our side must test as many business critical scenarios by interacting with the business folks from an early stage before releasing it for acceptance.
I stressed on solid preparedness rather than trying to push anyone in a corner.. What happened after surprised me but only just.. During project negotiations the clause for acceptance testing being bound by documented-and-shared test cases was inserted in the contract and signed off. But the clause never ever was adhered to by the customer when the acceptance tests commenced. I do not recall anyone from our side (including the delivery head) bringing up this clause when bugs not related to the test cases were raised. The acceptance tests were delayed somewhat but completed successfully owing to the well-developed and well-tested release for acceptance testing.
We succeeded mainly due to good domain understanding, focused development, efficient testing and on-going collaboration about the business scenarios shared by the client and not because of ‘a clause in the contract’ which no one remembered or cared about because it was still-born really.
Testing with tied hands is not possible (as it was tried here during acceptance testing) just as thinking cannot be stopped if you close your eyes.
Well not unless you are a Yogi meditating in the Himalayas seeking self-realisation!