Steps in an Acceptance Process
Requirement Specification and Acceptance Criteria
Potential Types of Acceptance Activity/Evidence
Issues associated with the Timing of Development of Acceptance Criteria
Acceptance Criteria for System Effectiveness Parameters
Acceptance Criteria for System Performance Parameters
Acceptance Criteria for System Design Parameters
Acceptance Criteria for Standards
Acceptance of Goods bought off the shelf
Acceptance of subcontracted items
Acceptance of Study Outputs
Acceptance of Bought in
Labor
Items for potential inclusion in Acceptance Management Plan
Acceptance Policy Type Statements
Issues to be considered in Detailed Acceptance Planning
Repeat item acceptance
Issues associated with Acceptance after a period of In-service
Use
The Role of Computer Modeling in Acceptance
The role of Quality Assurance in Acceptance
Making it Work
Introduction
Acceptance is a formal process of demonstrating and agreeing that obligations (usually contractual) have been met. Where the contract is for the production of a system these obligations will generally relate to demonstrating that the requirements for the system have been met.
Acceptance is carried out in the context of a customer accepting from a supplier. Customer and supplier could be part of the same
organization, though in this case the acceptance process will usually be less formal. There are often payment milestones associated with specific acceptance activities.
Acceptance criteria must always be in the forefront of any contractual agreement. It is only through having agreed acceptance criteria that the contract is clearly defined. Having paid attention to acceptance criteria during contract specification it must be planned for as an inherent and important part of the system engineering activity.
The steps involved in this process are typically those described in
Fig Acc.1. For a simple product there may only be only one acceptance activity. For a complex system project there are usually many different acceptance activities considering the different aspects of the system and demonstrating that different parts of the requirement have been met.
The acceptance criteria themselves are associated with the
requirement specification, as discussed in Fig Acc.2.
The generic types of criteria are identified in Fig Acc.3,
and some issues associated with their development discussed in Fig
Acc.4.
Some specific examples of, and issues associated with,
acceptance criteria for different aspects of a system, and for different types
of contract, are given in Figs Acc.5 to Acc.12, which are called up in Fig
Acc.2.
For large system projects it is usual to have an Acceptance
Management Plan, or similar document, which defines the process and
responsibilities for Acceptance Management. The sort of items that would
typically be included within such a plan are given in Fig
Acc.13. Underpinning the plan, and possibly also the contract itself,
will be key Acceptance Policy statements – see Fig Acc.14.
As a result of the policy and management arrangements, more
detailed planning will take place. Issues associated with acceptance planning are listed in
Fig Acc.15.
Where the requirement is for a number of identical or similar
items then there is a concept of design proving, production process proving, and
repeat production item proving. There will therefore be far more onerous
‘acceptance’ associated with the first item or items, where the design and
production processes are being proven, than for repeat items, where acceptance
is focused on ensuring the production processes have not degraded or introduced
faults. The distinction between ‘first of batch’, and repeat item
acceptance however is not always as simple as it might appear – see Fig
Acc.16.
It is not uncommon for some elements of acceptance to
continue after the customer has already started to use the system. Issues
associated with this are discussed in Fig Acc.17.
Acceptance testing on the product itself is expensive and
could risk damage to the product. Use of modeling enables a design to be
‘tested’ more cheaply and more extensively than testing on the produce, albeit
with issues associated with model validity – Fig Acc.18.
Even with modeling however it is rarely possible to include
as part of acceptance proving every aspect of a system or product. The
customer will also need to have confidence in the underlying processes used to
produce the product, as controlled through Quality Assurance – Fig
Acc.19.
Fig Acc.20 considers a number of issues
associated with making Acceptance processes work in practice.
Steps in an Acceptance Process
Preparation of Contract Requirement Specification and Acceptance Criteria
The acceptance process begins at the time the requirements themselves are specified. As part of agreeing to the requirement specification there must be a clear understanding of how the satisfaction of the requirement is to be demonstrated, ie. of the Acceptance Criteria associated with each requirement. This is discussed in
Fig Acc.2.
These acceptance criteria could be specified by the Customer but would more normally be specified by potential Suppliers as part of their costed proposal for meeting the Customer’s requirement. Should the proposal be accepted by the Customer then these acceptance criteria become part of the Contractual agreement between Customer and Supplier.
Even where both Customer and Supplier are part of the same organization
it is still appropriate to identify acceptance criteria. The difference may be a more relaxed acceptance by the Customer of shortfalls against the requirement.
The detail to which acceptance criteria need to be identified at the time of contractually agreeing the requirements is discussed in
Fig Acc.4.
Production of Acceptance Management Plan
For a complex system project the acceptance process will involve many different acceptance activities. An Acceptance Management Plan will be required to ensure the myriad of acceptance activities are conducted in an integrated fashion. Items for potential inclusion in an Acceptance Management Plan are listed in
Fig Acc.13.
An initial issue of such an Acceptance Management Plan will generally form part of the contract agreement process – see
Management Plans. During the contract this will then be developed in more detail as the Acceptance Criteria are themselves developed in more detail.
Acceptance Planning
Acceptance planning is carried out in association with individual acceptance activities. It is necessary that everyone who will be involved knows their involvement and that all the appropriate facilities etc. are made available.
The need for given acceptance activities will be identified within the Acceptance Management Plan. Each activity then needs itself to be planned for, and this will be done using individual Acceptance Plans for each Acceptance Activity.
Acceptance plans are usually prepared in outline well in advance of the acceptance activity itself. As the time gets closer the plan is prepared in greater detail (if required).
The preparation of the acceptance plan will usually be the responsibility of the supplier, within whose
organization an individual will be responsible for the plan, and for ensuring appropriate actions are undertaken to make the activity happen in accordance with the plan. The individual might change as the process progresses. Ultimately an individual will be overall coordinator for the Acceptance Activity when it occurs.
If the Acceptance Activity also requires customer facilities (other than simply manpower to attend) then the customer will also appoint an individual responsible for ensuring the customer responsibilities are fulfilled. In these circumstances the customer may be a joint approval signatory to the Acceptance Plan document.
Issues to be considered in acceptance planning are identified
in Fig Acc.15, including potential involvement of customers.
Formal Agreement of Acceptance Tests
During the acceptance planning process formal agreement will be required on the detail of the tests to be carried out and the pass/fail criteria for each test.
The general criteria will have already been established, but may not of themselves be in enough detail to allow an unambiguous pass/fail decision to be made. It is often not until close to the time of the tests themselves that sufficient detail will be available to identify these detailed pass/fail criteria.
The customer will be required to agree the detail pass/fail criteria prior to the conduct of the tests. There should be sufficient understanding at the time the initial Acceptance Criteria are agreed (as part of the contractual agreement) to allow agreement on the detailed criteria – see
Fig Acc.2.
Conduct of Acceptance Tests
Formal acceptance tests will be carried out in accordance with the acceptance test plan, and judged against the agreed criteria.
Formal acceptance tests will often be proceeded by tests internal to the supplier. In general a supplier will only look to carry out tests with the customer once he is sure they will be successful. The failure of tests with the customer both leads to expensive retesting and to weakened customer confidence. It is however not always possible to carry out full testing prior to the customer acceptance tests, for example due to the need to use customer supplied facilities.
Signed agreement on outcome of Acceptance Tests
It is necessary that the customer sign to both his having witnessed the tests and as to his agreement on the outcome.
The acceptance tests may be judged:
- fully successful
- having failed and thus requiring repetition once the supplier has made changes
- accepted with reservations, which might lead to some agreement such as a future repetition of a part of the test or other tests after the supplier has taken future action.
The supplier should look to get the customer signature as soon as possible after the tests, and preferably as the tests are conducted. Provision should be made on the test documentation for recording the outcome of the tests and for signatures confirming both that the test has been witnessed and the outcome.
For certain types of complex system there may be different levels of acceptance such as:
- Accepted for use with respect to a subset of the full functionality, or under defined conditions.
- Accepted as ‘Safe to Use’, possibly with constraints.
In these cases the degree of acceptance must be explicitly stated such there is clear agreement on what still remains to be demonstrated and accepted.
Delivery of Acceptance Evidence and Product
As a result of the acceptance test some deliverable from supplier to customer will generally be required. This will include the documentation recording the acceptance evidence, and at some point will include the product(s) itself. If payment is associated with the acceptance test then it will be more precisely associated with the delivery of the acceptance evidence rather than the activity itself. Suppliers should thus ensure clear responsibilities for this within their own
organization and prompt tidying up of and distribution of the relevant documentation.
Requirement Specification and Acceptance Criteria
Every statement of requirement should have acceptance criteria associated with it. It is only generally possible to be clear about a requirement statement by being clear about how its satisfaction will be contractually demonstrated. It could be argued that statements of how a requirement will be demonstrated are more important than the requirement statement themselves.
For example a requirement may be for a user-friendly interface. In practice however only a limited set of features of the interface will be demonstrated, these being the ‘Acceptance Criteria’. Once these criteria have been met, the requirement for the user-friendly interface has been met; irrespective of whether or not there are features of the interface with which the customer is not happy. Whilst there do exist ‘fitness for purpose’ obligations in law, these only protect the customer from gross inadequacies in the product, not against shortfalls in expectation.
Where the techniques associated with acceptance of a given type of requirement statement are already well established then there is not likely to be a need to explicitly identify the acceptance criteria at the time the requirement specification is agreed. If however the techniques are not well established then it is vitally important that the acceptance criteria are established.
The importance of having clear acceptance criteria is often unappreciated, and can lead to significant argument, delays, and extra costs towards the end of a project.
The customer in particular must ensure that the acceptance criteria agreed do
genuinely cover the intended scope of the requirement.
The type of ‘Acceptance Criteria’ differs depending upon the type of product/service. Of primary interest in this book are acceptance criteria associated with a complex system. With respect to complex systems, examples of areas of requirement which typically have acceptance criteria set are:
- Measure of effectiveness, probably in agreed scenarios and/or mission
profiles – Fig Acc.5.
- Measures of system performance, again in agreed scenarios and/or mission
profiles – Fig Acc.6.
- System Design parameters – Fig Acc.7.
- Design and production standards – Fig Acc.8.
Other examples of the need to consider acceptance criteria, which may occur within a system project include:
- Acceptance of goods bought off the shelf (Fig Acc.9)
- Acceptance of subcontracted items (Fig Acc.10)
- Acceptance of study outputs (Fig Acc.11)
- Acceptance of bought in labor (Fig Acc.12).
Potential Types of Acceptance Activity/Evidence
- Design Reviews and presentation of design information
- . For example reviewing design documentation, calculations, and design/production drawings.
- Model Results.
-
- Note that before the outputs of models can be
- accepted as evidence there must be some confidence in the validity of the model.
- Inspections, for example of the product or parts of it. Could include the witnessing of certain production activities.
- Tests and Trials.
-
- As carried out on either prototypes,
- mock-ups, or production versions of the product or parts of the product.
- Audits (eg. of procedures, implementation of procedures, documentation and records). Note that audits could be carried out by the customer, or the supplier may offer up evidence from his own audit activities (with the customer possibly having audited the operation of the audit activities).
- Accumulation of evidence
- , such as the accumulation of ARM information over a long period of time.
Some observations with respect to different types of evidence:
|
|
Positive |
Negative |
|
Design Review |
– Early enough in the design process to allow modifications to be relatively cheaply incorporated. – Comprehensive, in that all aspects of the design can be reviewed. – A wide range of experts can be involved in the review process. |
– Only reflects what is intended to be done, not what has been done. – Difficult to judge the adequacy of expected behavioral – Very much dependent upon the quality of the |
|
Modeling |
– Relatively cheap. – Allows a wide range of circumstances to be investigated. – Often provides useful insights which go beyond the planned purpose. |
– Significant work often required to gain high confidence in the models used. – Costs may not be insignificant. – Not |
|
Prototype/Mock-up Tests |
– Can be used early in the design phase when changes to the design can be relatively cheaply incorporated. – Can provide reasonably fidel representations of the intended system. |
– Fidelity of representations can be misleading. – Costs need to be allocated. |
|
Factory Tests/Trials |
– Comprehensive test of the item can be undertaken. – Faults found can be fixed before the item has left the factory. |
– Only investigates part of the system in isolation. – May be difficult representing operational conditions. |
|
Integration Facility Tests/Trials |
– Provides means of demonstrating that items integrate with other items. |
– Not generally fully representative of operational environment. |
|
(Within) Operational (Environment) Tests/Trials |
– Demonstration of the system in its intended environment. |
– Very expensive. – Generally requires access to customer personnel/facilities, for which there might be conflicting needs. – Only practical to demonstrate a (possibly very) limited aspect of the full requirement (particularly in terms of full environmental conditions). – Relatively expensive for corrections/modifications to be made as a result of identified shortfalls. |
|
Audits |
– Audit type activities need to be carried out anyway, and the additional cost of using evidence for acceptance is low. – Can catch problems very early on, even before they become problems. |
– Are not a systematic check for problems, and thus many problems may go unidentified. |
Issues associated with the Timing of Development of Acceptance
Criteria
Ideally ‘Acceptance Criteria’ should be specified at the same time as the requirement. In practice however the detailed acceptance criteria are left till later during the project for reasons of:
- Specifying the ‘Acceptance Criteria’ in detail requires extensive effort. Management is often reluctant to spend such effort early during the
program, preferring to wait until a clearer view has emerged of the most cost-effective manner of proving the requirement. - The precise ‘Acceptance Criteria’ which will be appropriate may not be apparent at the beginning of the
program, and may be dependent upon the design produced in response to the
requirement. - The expertise needed to specify and agree the ‘Acceptance Criteria’ might not be available early during the
program.
There is a trade off associated with the best time to detail the acceptance criteria. If it is conducted early there is likely to be much nugatory effort because the criteria may need modifying in the light of
the design itself, contract changes, or the availability of facilities to support the acceptance process. If it is conducted too late then a detailed understanding of the product will have emerged leading both sides to have conflicting prejudices about what acceptance criteria should be applied. The practical compromise is an evolving agreement on the acceptance criteria, with an outline of what they should be early on in the project, but the detailed planning of the acceptance tests being left till late on.
Acceptance Criteria for System Effectiveness Parameters
System Effectiveness is a measure of how well a system carries out its purpose – see
EFFECTIVENESS.
If measures can be defined which both adequately specify this purpose in quantitative terms and which can be measured prior to the system being put into operation then there should be little difficulty in identifying acceptance criteria (which will involve direct measuring of the features of the system of interest).
However for complex system projects this is often not possible, and it can be very difficult to identify quantitative acceptance criteria prior to the system being used by the customer. The acceptance criteria for effectiveness are thus frequently:
- Demonstrations using the system after use, with possible retention of part of the payment until satisfactory demonstration. See
Fig Acc.17. - Providing evidence that the system appears theoretically to have the necessary features to satisfy the effectiveness requirements.
In the case of (b), or if it is not possible to identify reasonable measurable acceptance criteria, then other evidence must be provided giving at least some degree of confidence that satisfactory effectiveness would be expected. In this case acceptance criteria might be associated with theoretical
modeling. It will be necessary to address the process by which confidence is gained in the results of the model – see
MODELING.
Note that it is particularly important for the supplying organization
to ensure that effectiveness, and fitness for purpose more generally, is judged against the perception of purpose at the time the requirement was placed. For a system with a long development and manufacture period the customers perception of what he wants from the system may change. The design/manufacturer should not however in general be held responsible for this.
Acceptance Criteria for System Performance
Parameters
System Performance is the primary areas for which a system requirement can often be practically accepted.
These are the ’emergent properties’ of the design which are expressed in terms
relating to how the system is expected to perform in its operational environment.
The primary opportunity for proving the requirement will be:
- Planned trials within the operational environment, following ‘run-plans’ specially formulated to demonstrate the given performance characteristics of interest.
- Use of integration facilities or mock-ups into which either pre-production or production versions are equipments are put and demonstrated to work together.
- Modeling to demonstrate that proven design performance leads to the required
(emergent) system performance. This will often thus also require a program of model validation.
Where some given system performance function is satisfied within a particular component part of the system, then the acceptance criteria may consist of:
- Demonstrating that the component part of the system has the required performance, and
- Demonstrating that the performance is not degraded as a result of its fitting within the wider system (and thus the potential for environmental interferences).
Acceptance Criteria for System Design Parameters
Design parameters are physical characteristics of the design
which tend to be easily measured.
It should be noted however that by specifying design
parameters as part of the requirement the customer is potentially compromising
the achievement of system performance or effectiveness.
Acceptance Criteria for Standards
Standards may have acceptance criteria as an inherent part of the standard itself.
Given the nature of many types of standards it is not possible to carry out comprehensive tests/inspections to demonstrate that a standard has been complied with. It is often necessary to ensure a method is put in place which ensures that standards are applied where relevant, and conduct audits to demonstrate that the method is being effectively applied. The acceptance criteria against standards may thus be related to demonstration that processes are in place and being adopted rather than actual tests themselves.
Acceptance of Goods bought off the shelf
Goods bought off the shelf do not have an acceptance process as such. The supplier will provide a detailed description of the product and it is for the customer to decide whether it will suit his purpose or not. The supplier will give a demonstration if appropriate. It may be possible to arrange for the product to undergo customer tests though there is not generally guidelines for this and each case must be considered on its own merits.
Should the product fail to conform to its specification then the customer will be protected by law, at least for a minimum period of time (eg. 1 year). It is important that the customer uses this period of time to ensure that the product does conform to its specification.
Where a large number of relatively low value goods are bought
organizations tend to have their own procedures for checking/testing these as part of goods
inwards. This often involves sampling.
Acceptance of subcontracted items
Covered under PROCUREMENT and SUB-CONTRACTING.
Acceptance of Study Outputs
Consider an activity requiring a study, with the output being a study document with conclusions and recommendations. Ultimately the acceptance of the study output must have a large subjective element. The following are aspects of a study which can provide some degree of objectivity to the acceptance process.
-
Format. Objective acceptance against the document format is clearly possible, and measurable standards should be specified in the study requirement. These standards may be company standards which themselves will have been indirectly approved.
However, although format is clearly important with respect to easing the reading of the document, it is not what is being paid for.
-
Scope of coverage. The requirement for the study will identify what issues the study must address. A comparison can and should be made to ensure all the issues were either addressed, or there is a reason for why not. (Possibly as agreed during the study itself.) This also applies to agreements reached during the study for coverage of certain issues. It is important that such agreements are either minuted in formal meetings or are otherwise written and agreed.
-
Effort expended. A study requirement will usually include an estimate of the effort to be expended on the project. This identification of effort is an implicit agreement of the level of detail to which the study will be undertaken. It places a degree of commitment on both sides with respect to acceptance. The customer cannot expect a study to a degree of detail which is obviously not possible with the effort identified. Similarly the supplier is committed to going to a level of detail which should be possible with the effort expended.
This should not be taken to imply that the agreement is simply to spend a given amount of effort and then stop. The statements of requirement must still be met even when these lead to greater than estimated effort being expended.
In estimating effort expended it is thus necessary to include provision for supporting activities such as document preparation, document reviews, quality reviews, etc.
-
The customer can reasonable expect the issues he has raised and the supplier has accepted to be addressed. This issues could be either raised during meetings or the result of identifying other contacts or relevant documentation.
-
Progress meetings provide a means of identifying any conflicting viewpoints as the study progresses rather than leaving them until acceptance of the final documentation is required.
-
Prior to final issue of the document a draft is usually produced for comment. Assuming that the draft is relatively complete the customer cannot object to issues in the final report(s) which he did not raise at the draft stage.
What is very difficult to accept against is the quality of the study findings themselves. Customers attempt to ensure they get appropriate quality by:
- Going to organizations with a reputation for quality.
- Ensuring they have some say with respect to who works on the study.
Most organizations understand that giving a customer a bad service invariably does far more damage than the benefit of any short term gain in profit associated with the given project.
On the other hand if the customer refuses to accept what is a good service given the contractual agreements then the supplier can also make complaints against the customer. This will frequently not be to the customers long term interest as an individual within his
organization.
Thus there is some interest on both sides in reaching a compromise on the agreement of the quality of the output.
In some special cases there may be a case for holding the supplier of the study responsible for the study findings. For example a study into the safety of a product which clears the product but for which there are subsequently found to be significant safety problems. If it can be shown that these problems should have been uncovered by the study then the customer may have some comeback in law – for example under the
law with respect to provision of services.
In general, although the outputs are subjective, amicable agreement on acceptance can be reached. In particular if attention is paid to:
- Ensuring a clear statement of study requirement.
- Ensuring format standards are clearly specified.
- Ensuring progress on the study is reported and formal minutes kept.
- Ensuring agreement is reached on the scope of the deliverables and the degree of detail to be recorded as early as is reasonably possible.
- No significant attempt is made, on either side, to exploit the study beyond the implicitly agreed scope (the supplier by trying to get away with putting significantly less effort into the study than required or by putting on inferior staff than suggested; the customer by trying to insist that the study is not acceptable in order to keep getting extra work undertaken beyond that implicitly agreed at the time of contract placement.)
Acceptance of Bought in Labor
A requirement for provision of labor can do little more than identify the task(s) required to be carried out, ask that suitably qualified/experienced persons are provided, and reserve the right to ask that persons be replaced if there work is not adequate.
Unless otherwise stated, acceptance is implicitly an ongoing activity on the part of the customer, who by not formally rejecting the individual is implicitly accepting them and their work.
The supplier would generally look to gain stage payments since this ensures there is an incentive for prompt payment by the customer, who presumably still has need of the service. Once the service is complete there is little incentive for prompt payment. In particular the supplier must avoid a situation where the customer pays at the end of a long period of work on the assumption of satisfactory service.
Items for potential inclusion in Acceptance Management Plan
Responsibilities and Dependencies
Clear identification of the respective responsibilities of the supplier and the customer and what dependencies exist between them. This needs to include specific agreement on the format in which evidence will be passed and the precise procedures leading to formal sign-off.
More detail of organizational structures may be appropriate. For example the customer may be required to involve a number of other parties such as safety authorities, or may wish to involve other authorities such as specialist
organizations or the user. The role of such authorities in the acceptance activities must be clearly defined. (Noting that both customer and supplier must present a single line of authority to one another.)
Acceptance Activities
High level definition of the acceptance activities that will be undertaken, and how these relate to give a total acceptance, together with an associated showing key dates and deliverables.
Standards and Methods
Identification of any standards or methods which are associated with the acceptance process itself.
Documentation/Method of Presentation of Evidence
Identification of what documentation is required to be produced, identifying who is to produce it and when, and who is to
authorize it.
Sub-contract acceptance
How will (the supplier’s) subcontracted items be accepted?
What conditions will be passed onto subcontractors with respect to their methods of acceptance from their subcontractors?
What visibility will the customer be given of subcontractor acceptance?
Repeat items
How will ‘repeat items’ be accepted?
In-service Acceptance
It is becoming common for contracts to include an element of retained payment for after a period of in-service use. Is this appropriate and if so how should it be scoped?
Acceptance Policy Type Statements
Acceptance policy should identify a system wide approach to acceptance of sub-systems into the system and acceptance proving of the system to the customer. A clear distinction however must be made between acceptance to the customer and acceptance of sub-systems into the system. There may be some overlap where the customer has specified requirements with respect to given sub-systems.
The acceptance policy is concerned with acceptance of the system by the customer and thus identifies to the sub-system authorities the degree to which they are expected to contribute to this activity.
Sub-System contracting should include an element of proving the sub-system in the system environment. The sub-system authority should not be considered to have fully meet his contractual requirements until the system has itself been accepted (and maybe beyond).
There may be some overlap between system acceptance to the customer and sub-system acceptance by the system authority. For example the same trials might be used for both.
The following are examples of the types of statements that may be relevant to an acceptance policy.
- Acceptance evidence will be gathered at the earliest opportunity adequate evidence becomes available, or
- Acceptance evidence will be limited to that which can be provided based on field trials.
- The need for customer supplied facilities will be kept to a minimum, or
- The need for customer supplied facilities will be identified at least two years in advance of the requirement for their use, and will be costed to the project.
- The totality of the acceptance process will be managed by the Acceptance Manager, who will be responsible for ensuring that the total acceptance
program is carried out in an integrated and cost-effective manner, or
- Individual system/project areas will each be responsible for the management of their acceptance activities.
- Evidence from sub-contract acceptance activities will be used to support contract acceptance itself wherever possible.
- Wherever possible acceptance evidence is to be derived from activities already required to be undertaken as part of the design/manufacture/test activities.
Issues to be considered in Detailed Acceptance Planning
Detailed acceptance planning might need to consider any or all of the following:
- What facilities will be used and are they available. Includes considerations of for example electrical supplies.
- Who needs to be involved, and what are their specific roles and responsibilities (from both customer and supplier
organizations and anyone else they need to involve). - What facilities are required to support the people involved (eg. office space, hotel arrangements).
- PHS&T arrangements for items required.
- What consumables and spares should be available (note that lack of availability of a relatively inexpensive part could postpone a test/trial with very large cost implications).
- What procedures are required. Are they available, or is there a feasible program
for their production. Who is responsible for ensuring they will be approved and available for when required.
Some level of contingency planning may also be necessary, for example if weather conditions delay a given trial.
Someone in particular needs to be given explicit responsibility for individual or groups of acceptance activities, and ensuring that all the above are planned for and provided in plenty of time for the activities themselves. The level of planning will become more detailed as the test/trial approaches, and for example might include the production and maintenance of a Master Movements Chart (for personnel movements and arrangements for travel, etc.)
Detailed acceptance plans may need to be approved by the customer, particularly if their are obligations on the customer beyond those of simply witnessing the tests. For example acceptance proving may require access to customer facilities. (The requirement for customer furnished facilities should have been agreed, at least in outline, at the time of contract placement. This will then also include contractual agreements about what happens in the event of the customer failing to make the facilities available or for shortfalls/failure of the facilities.)
Repeat item acceptance
The first item or items will be used to prove the design and the production process. This will require extensive testing/acceptance-proving. However once these are proven, subsequent proving of repeat
items need only focus on ensuring that the production activities have been correctly carried out.
This simple consideration can be more complex in practice due to:
- Some design testing could be delayed until second or later builds due to, for example, production or test
program considerations.
- Some changes in requirement could have occurred which were not practical to implement in earlier builds. Thus different items could have different configurations.
- Some changes could have been made as a result of testing associated with the first build, but the changes may have not been incorporated into the first build.
- Some design or production process testing may only be possible through an accumulation of evidence requiring evidence to be gathered from more than one item. This is particularly the case with, for example, reliability measurements.
Issues associated with Acceptance after a period of In-service
Use
It is becoming common for contracts to include an element of retained payment for after a period of in-service use.
Items which often require a period of in-service use before final confidence can be gained include:
- Building up statistical evidence of functional performance.
- Obtaining sufficient ARM information to confirm ARM specification.
- Operation in certain environmental conditions.
- Adequacy of any support provisions included in the contract.
However cognizance needs to be taken of the following:
- Misuse or damage by the customer/user may affect operation, and thus perceived shortfalls may not be the fault of the designers/manufacturers.
- Once the system is in use normal ageing affects may degrade performance.
- Once the system is in use the customer/user’s perception of what he wants will change and may be very different from what was originally contracted for. His assessment of what is acceptable may thus be against different criteria than originally used as the basis for the contractual requirements.
- There may be a conflict between measuring performance for acceptance proving and operational usage.
Where in-service assessments are being used as part of acceptance then the following is required:
- To maintain a clear definition of the requirement and acceptance criteria against which the system must be assessed.
- Ensure clear procedures are in place for the recording of evidence to be used for acceptance purposes.
- Maintain a record of system usage whilst there are still contractual responsibilities on behalf of the designer/manufacturer.
The Role of Computer Modeling in Acceptance
Computer modeling techniques have developed to the extent that they can have a very significant role in the acceptance process.
There are two main roles computer modeling has associated with acceptance:
a) As providing quantification evidence in its own right
The models may be used directly, for example as an implicit part of the definition of the design, and thus acceptance of design information may be through acceptance of
modeling information (eg. looking at layouts within 3D CAD or virtual reality environments).As intended representation of the real system performance, with validation of the model having been demonstrated (for example using trial evidence).
This later approach is becoming common for most complex systems, with real tests being done both as a demonstration of the system but also as providing data to validate the computer models. Having obtained significant confidence in the computer model it is then used to provide a much wider scope of acceptance evidence.
b) Supporting other acceptance activities
Modeling can be used both to help plan trial activities, and to help
analyze the result afterwards. In planning trial activities for example
modeling can help ensure that the run plans are feasible given the known system characteristics.
Potential benefits in the use of computer modeling for acceptance
vis-à-vis tests/trials include:
- Computer ‘tests’ are very significantly cheaper than real tests.
- Computer ‘tests’ can investigate situations which real tests can not, for example extreme conditions, unsafe situations, or military engagements.
- Many more computer ‘tests’ can be carried out than real tests.
- Computer test situations can usually be more closely matched to the requirement than can real situations (which may introduce many more
uncertainties): for example with regards to operation in abnormal
environments.
In order to be able to use modeling for the above however there must be sufficient confidence in the model. Ie. the model must have been validated – see
VALIDATION.
The role of Quality Assurance in Acceptance
Written into most major contracts is an auditing by the customer of the supplier’s quality control process. This can be seen as adjunct to the acceptance process which is intended to give the customer confidence in the product.
Quality control is the process which ensures a quality product and addresses the methods by which the product is designed, built, etc. Various standards and ‘management plans’ will describe how the supplier conducts his business. The objective of customer quality assurance is to satisfy the customer that these standards and plans are adequate and that they are being carried out.
Part of this quality assurance may involve third party assessments – eg. ISO 9000.
See QUALITY.
Quality assurance is generally outside of the direct acceptance process. As part of general terms and conditions associated with the contract the customer will have audit rights and if these show up shortfalls in the quality process the customer will generally have rights to insist they are put right.
However for certain types of requirement, such as the demonstration of compliance with standards, it might be appropriate to make use of information gathered as part of the quality control process to demonstrate that standards are being applied appropriately.
Making it Work
The following are issues which must be borne in mind:
Compromises during Acceptance
Frequently the demonstrated product may fail to meet a given criteria. The customer then clearly has the right to reject the product in accordance with the contractual conditions. It is often however not in the customers interest to be completely hard headed about this.
In general the customer will himself be keen to obtain acceptance since it will be a critical milestone on the way to his having
procession of the product.
It is thus often in the interests of both parties to reach a compromise. Types of compromise might include:
- Accept the shortfall on a given ‘build’ with agreement for modification on subsequent builds.
- A conditional acceptance, where acceptance is granted but there is agreed action still to be undertaken, eg. rectification of defect, or future modification. A conditional acceptance might typically be identified as allowing the product to be used as though it had been accepted, albeit with contractual actions still outstanding.
- A shortfall may be accepted by the customer. And if the shortfall is not important to him he may well be advised to do this because it does not then force the supplier to use limited resources in an area which is of limited or no importance to the customer. There may well be other areas on which the customer must take a firm stand and by not forcing the supplier to use resources elsewhere, more, in principle, should be available.
The customer may look to build up a register of shortfalls and use acceptance of these to trade-off against other aspects of the contract.
Payment Milestones and their link to Acceptance
Payment of product/services will often be linked to acceptance milestones.
Where payment is linked to acceptance milestones there are additional pressures/interests on both parties to ensure the payments occur within given
expected profiles. The planning of acceptance milestones must therefore be integrated with cost planning. The cost planning must take into account the
RISK associated with the acceptance milestones not being achieved, or being
delayed.
Lack-of-clarity/Disagreement as to whether requirement has or has not been met
It is of course of overriding importance that the acceptance criteria be clear such that there is no doubt as to whether they have been met or not. This however may not always be the case, for example:
- the environmental conditions on the day may not be those which match the stated environmental conditions. Since it may not be practical to expect ‘ideal’ conditions it may be necessary to ‘interpret’ the results obtained to determine whether they were consistent with what would have happened had the ideal conditions occurred.
- Many acceptance criteria are statistical. These can be difficult to interpret since statistical tests can lead to rejection in a test when the system does in fact meet its requirement, and acceptance in a test where the system fails to meet the requirement – see
Accept-Reject Decisions under TESTING.
For these and other reasons there may be disagreements as to whether or not the evidence presented indicates that the requirement has been met.
Whilst it is in general in both parties interests to come to an amicable agreement, for reasons given in the point above, on occasion the stakes may be too high on both sides, in which case appeal may need to be made to a higher authority.
There always is a higher authority. Clearly if both customer and supplier are from the same
organization then at some point there is someone responsible for both, ultimately for example the Managing Director. Where, as is usual, the customer and supplier are from different
organizations then the law provides a higher authority. It is however rarely in either parties interests to let the law decide and better to come to some compromise.
Who pays when tests fail or are delayed?
Consider the case of an acceptance test requiring significant involvement of both supplier and customer personnel:
-
If the customer is unable to provide the facilities on the day and the acceptance tests are delayed, does he compensate the supplier for consequential costs? How might such consequential costs be estimated? Clearly the costs will depend upon how much warning the supplier is given of the need to delay. If the supplier is ready to demonstrate then delays in being able to do so could be very expensive, since it may force him to effectively hold up further progress until the demonstration takes place.
It is the costs associated with delays that often leads suppliers to move ahead with acceptance even when they are unsure of complete success.
-
If the supplier is unable to carry out the demonstration as and when agreed, does he compensate the customer for consequential costs? The costs associated with rescheduling could be very high.
-
Who pays in the event of the test not being able to take place for reasons outside of either the suppliers or the customers control? For example due to adverse weather conditions.
-
What if a test fails because of failure of customer supplied items? Does the customer pay all supplier’s costs associated with the failure.
The recognition of the potential occurrence of the above must be written into the contract between customer and supplier. Any obligations on the customer must be clearly identified and the consequences of failure to meet those obligations.
Whilst the customer will look to minimize obligations on himself, this may lead to increased costs for the supplier. It may be of lower overall cost to the customer (at least after a risk assessment has been taken into account) if he retains certain obligations.
Making maximum use of trials time
Actual use of the end product as part of acceptance is both
the most convincing aspect of acceptance and the most expensive. It is
important that this time is not wasted. Some ways of maximizing use of
this time include:
1 Use of Video Recorders
One common problem with acceptance activities is recording precisely what happened. Video technology
is a cheap and reliable method for recording precisely what happened (assuming it is used appropriately and the recorders are placed: where they will not interfere with the acceptance activities themselves,
and where they will record the essential information.
2 Use of a Black Box
In circumstances where a system is assessed in its operational environment it may be difficult to keep records of precisely what the system has done. If assessment is critically dependent upon this knowledge then it may be appropriate to provide appropriate automatic recording.
This could be done using conventional recording facilities, or using facilities such as a ‘Black Box.’ For a first of class system the use of a Black Box has the additional advantage that it might provide valuable information in the case of a major problem.
The above can be invaluable where their may be potential doubts about liability. This can occur during the early use of a system where the supplier retains liability but where the system is being used by the user who may take the system outside of its design limits.
3 Maximizing of Trials for different purposes.
With a large complex system project it can be that many trials get planned in isolation to demonstrate particular parts of the system. With careful planning it is usually possible to
rationalize trials such that although a given sequence of trials might be primarily for a given purpose, additional information can be gathered which can be used to support other acceptance activities.
4 Change of runplan after the event
It can be difficult carrying out precise runplans. Rather than reject results from tests which have
not kept to their original runplan it may be appropriate to assess whether enough information has been provided to meet the overall objectives of the test. This may be possible if:
-
- there is adequate information to reconstruct what actually happened.
- there is the means for deriving acceptance criteria associated with the changed run plan.
This approach, which is feasible where the runplan can be
simulated within a modeling environment, can save large amounts of money and time.
Progressive acceptance or confidence building
For a large complex system contract it may be many years between agreement on the requirement/contract and its
realization as a physical system. Leaving all acceptance until a physical version of the system exists is generally very risky from the viewpoints of both the customer and the supplier: for the customer it means he does not know until very late whether he is going to get the type of system he wants and expects; for the supplier he does not know the extent to which the system is going to be acceptable to the customer.
It is generally in both parties interest to be able to assess the acceptability of the evolving design as early as possible in the design process whilst both sides can still take meaningful action. Even if there is a requirement for contractual changes these are likely to be significantly less costly than changes not identified until a version of the system has been physically built.
As a result there may be a number of acceptance activities associated with a given requirement. For example a requirement associated with the providing adequate layouts may involve a partial acceptance during the design phase to approve general features of the layouts with final acceptance delayed until completion. The final acceptance will be associated with ensuring that the agreed design features have been correctly implemented.
Liabilities after acceptance
It must be noted that acceptance activities do not necessarily mean the customer has accepted all responsibility from the supplier. The law protects both customer and supplier from sharp practices, or unreasonable demands.
In addition to the passing of ACCEPTANCE tests there may be a period of warranty or guarantee associated with a product. The supplier may thus retain some liability for the product and should it fail in a specific manner then he may be expected to fix the product free of charge. An approach sometimes adopted is for the customer to retain part of the final payment for a period of time after delivery.
A particular area often covered by warranty/guarantee is reliability. It is difficult to test a system for reliability as part of a short period of ACCEPTANCE testing.
In setting warranty/guarantee terms however there may be a problem with proof of liability. Clearly if the product is used by the customer outside of its design limits and a fault occurs then it may be difficult to be sure of who is responsible. It may also be difficult to know if the fault is the cause of the product being taken outside of its limits or even proving whether or not it has been.
Customer Visibility
In addition to actual acceptance activities/evidence, there may be benefits to both parties for the customer to be given visibility of progress towards meeting the contractual requirements.
From the customers viewpoint:
- It provides early confidence that the requirements are going to be met, or early indication that there are going to be potential problems.
From the suppliers viewpoint:
- It provides early indications of potential problems that might arise associated with acceptance.
- It provides the opportunity for drawing upon the customer organization’s experience.
Visibility however must be a managed process. Note in particular:
- If the customer looks to have unrestricted access to the evolving design/manufacture process then he may significantly distract designers/manufactures.
- Unless there is strong management within the supplier organization, the supplier personnel may react to all customer comments and therefore loose coherency to the overall design process.
- The customer needs to be careful that in gaining visibility or commenting he is not implicitly then accepting some liability.
A program of visibility should therefore focus on areas that either the customer and/or the supplier perceive risks.