There was an article on Test Architecture Framework in a recent issue of Star Tester which made interesting reading.
I had some apprehensions about the framework and I wrote to the authors to clarify about them. I am reproducing the Q&A below with answers from Nathalie Rooseboom de Vries van Delft (Senior Consultant and Expert Group Leader of the cluster Testing Technologies and Processes at Capgemini Netherlands).
SM: What steps do you take to ensure that the framework is kept lean and relevant? In other words how do you drive its innovation? I guess collaboration between Test Architects and Test team cannot be limited to just the planning stage rather it will have to be for the project duration if the TAF(c) has to evolve? But is that practical?
Nathalie: Sure it is – in fact we have a four-stage process model for development of a test architecture. But the fifth and possibly most important stage of the process model is the guiding and monitoring of the test architecture’s usage in real life, checking results and benefits, setting requirements for a next release of the test architecture to be developed. In fact a test architect’s role could be compared to an architect’s role in real estate development. Setting the standards, getting a design best suited to all stakeholders and supervising realization.
SM: How do you ensure flexibility or tailorability within your framework?
Nathalie: The test architecture’s design has to be produced with a full rationale – all contents are shown to be needed, together with the reasons why. In this way the test architecture doesn’t contain superfluous or redundant products or activities – it’s kept as lean and mean as possible. And since all rationale is specified, any change in environmental factors (like change of development architecture, like change in business risks, like …) may be translated to parts that may need change.
SM: How do you deal with a client who wants less of the framework/process since he thinks the test effort is getting bloated and he is paying extra? In other words how do you justify the use of the framework in terms of RoI to a customer who wants bang for the buck vis-a-vis process circumvention/dilution?
Nathalie: You are quite right that the client will not be interested in a framework. The client doesn’t get the framework – he gets the results of the test architect using the test architecture framework. Usually in the form of a set of web-communicated testing handbooks, dedicated standards, templates, guidelines, test process descriptions and procedures, etc.
SM: While a Test Architect would study a new project specifics and create the process applicable for that project how does the organisation ensure compliance? Are there Test Audits? Who does the Audits? Would it be the Project SQA or are there dedicated team members from the test fraternity within CapGemini that does it? (I guess the latter doing it would be worthwhile to help understanding and feeding the system with relevant improvement suggestions).
Nathalie: The test architect is specifying the way testing should be done. In this way he should remain accountable for results produced by the test organization. For me the test architect will be a representative for stakeholders involved – since the test architect will be responsible for translating stakeholder requirements into acceptance criteria and into test requirements / test standards complying these acceptance criteria. A test architect should be the one that will give a signoff for test products / test results – giving a statement that required tests actually have been executed and that these tests have been executed the way that were specified in the test architecture. Relevant products in the test architecture are: Master Acceptance Approach and Master Test Approach. For a specific project the implementation of these would be the Master Acceptance Plan and the Master Test Plan.
SM: Are there guidelines available for the Architect to suggest the project specific process picked from the framework?
Nathalie: There will be – we are developing these.
SM: Are there specific instances when the framework has to be sidestepped or subverted in the interest of customer management / project delivery? Do the normal deviation process apply in this case too?
Nathalie: No not really – the test architect’s approach (as described in the Test Architecture Framework) is designed to solve the issues you mention here. The process of the test architect is aimed at construction of an optimized test process (and guidelines, standards, templates, infrastructure) under specified circumstances. Of course IT and Development architecture are changing – meaning that in a number of cases we need to invent test techniques, new test approaches for specific programmes. But the way we determine a test architecture remains the same …
SM: Since Estimation is such a hotbed and a influential impacting factor cutting across quality, effort and cost considerations what special steps does the framework include to minimise the impact of an estimate which is adverse to the interests of the organisation or the customer.
Nathalie: Transparency is a main goal for a test architect – a test architect can’t anyone specialized in testing alone. A test architect should have a firm knowledge, experience in all areas that may be relevant for risk assessment. A test architect should (monitor and) provide estimation parameters, estimation procedures – telling how much resources (budget, time, manpower, infrastructure) will be needed if a certain approach in testing will be required. And getting agreement of stakeholders with respect of risks concerned.
SM: As Metrics is such an important part to track, manage and improve the process what steps does the framework take (in terms of tools or techniques) to ensure that this is seamless, unobtrusive and painless while still relevant? How is buy-in ensured from the stakeholders?
Nathalie: Metrics will remain a painful issue, since anyone will have the feeling that we don’t have the time or energy to provide metrics of real-life results> It remains however a required activity for a test architect – else it will be impossible to provide evidence on test results and resources required for it.
SM: Does the test architecture concern itself with aspects like re-strategising due to delayed delivery from development and testing time reduction, risks arising from it, SLA between dev teams and test teams in terms of delivery, fixes delivery and so on?
Nathalie: You mention why a test architect should be an experienced professional – we all know that in complex programmes delays will occur, that required quality standards won’t be met all the time. The test approach needs to have provisions and procedures for incident handling, planning etc.
SM: What does the test architecture suggest in terms of development conducted testing? Are the defect leakages arising from dev to test stages analysed equivalent to leakages into production?
Nathalie: This depends on the principles applied in the test architecture – I strongly recommend to set rigorous standards on quality standards on every milestone / internal or external release. In a number of cases we set formal criteria for acceptance (from development to functional testing, from functional testing to acceptance testing, etc.) and an escalation procedure if delivery was done not meeting criteria. After a first hick-up this proved to be a major quality enhancer for delivery to operational services.
SM: Are aspects like Structures for Test Organisation within the ambit of TAF(c)?
Nathalie: Not yet …
SM: Who conducts the causal analysis of defects if it is suggested as a best practice by the TAF(c)?
Nathalie: This is a required part of incident management – one of the best practices required by a test architect.
hmm… As I said – interesting!