Auditor NL 5/12

IS-BAO NEWSLETTER | May 2012

As of 1 May 2012 there were 646 registrants:

• Stage 3 – 56 — 9% of total
• Stage 2 – 155 — 24%
• Aviation Operating Certificate holders – 184 — 28%
• Non-US – 147 — 23%

Noteworthy: One-third of registrants now have productive SMSs, i.e. they have progressed beyond the formative stage of the program.

Three new policies effective 15 March 2012 have been released:

Policy 2012-01, Auditor Accreditation Requirements and Procedures

This is an update to the 2010 version of this policy. The new helicopter auditor designation requirements are described.

Policy 2012-02, Auditor Currency Requirements

The policy requires auditors:

• not familiar with both flight operations and maintenance aspects of IS-BAO to take a brief examination prior to their first audit in the subject area for which they have little or no experience. After passing the examination act as an auditor under the observation of a fully qualified auditor during at least one audit. After completing the audit the lead auditor must attest to the ability of the new auditor to adequately conduct a complete audit.
• not conducting an audit or completing an IS-BAO workshop within the previous 12 months must present evidence of completing an approved workshop or other course or to provide implementation support to an operator and take a written requalification examination.

Policy 2012-3, SMS Progress Requirements for Operators

This was developed to clarify the SMS ongoing performance requirements for IS-BAO registration, and to provide for provisional operator registration under certain circumstances. Specifically, the policy covers issues of failure to progress to or maintain higher stage levels of registration.

Also, note IBAC-Bulletin-12-03.pdf and IBAC-Bulletin-12-04.pdf

IS-BAO standards and recommended practices are being constantly reviewed to better remain current with ICAO standards and industry best practices. Each January scores of changes are published in a completely revised edition of the standard and interim changes are promulgated throughout the year via policy statements like those above.

Most auditors are providing better comments for advanced stage audits, especially SMS comments, but some reports and protocols continue to provide insufficient information to provide meaningful feedback to operators and to convince the audit review group of an operator’s readiness for an advanced stage designation. This is important – in case you missed a previous article on this, see Making a Case for Stage 2 and 3 Audits.

Just because it is an advanced stage audit doesn’t mean that we don’t want comments throughout the protocols, either. What’s new, changed, different, better, worse…? Granted, the focus of these audits is SMS performance and maturity but sampling of all aspects of the operation is required; tell us something about what you observe during the sampling process.

“…findings are specific instances of non-conformity with the requirements of IS-BAO, operator manuals or operator directives. The findings may relate to one or more minor non-conformities or major non-conformity, but should be of such significance as to warrant urgent action (APM 6.1).”

Some auditors continue to cite individual non-conformities as findings when these items are really evidence of a larger issue that may be properly designated a finding.

For example, non-conformities such as, “4.2.3 – PIC duties not fully described”, “5.3.1 – tems d, h and j of emergency procedures training not mentioned” and “6.4 Weather Minimums – 6.4.2, takeoff weather minimums not specified” are all non-conformities but not necessarily findings when considered individually. Rather, they may constitute a pattern of missing items that could be considered a finding under standard 10.1, requiring an operator to provide an operations manual containing the previously mentioned items. On a larger scale, management’s commitment and responsibilities may be involved if a larger pattern of missed requirements is evident – see standard 3.2.1.

Do all non-conformities have to be turned into findings? Not really: APM 6.2.3 provides guidance: “If it is determined that one or more minor non-conformities are simply slips, lapses or omissions that can easily be corrected and there is no significant underlying problem, rather than making a Finding the non-conformity should be noted in the Analysis of Non-conformities section of the Audit Protocol along with the comment, ‘Action Required’.”

“The organization shall develop and maintain the means to verify the safety performance … shall be verified in reference to the safety performance indicators and safety performance targets of the SMS.”The operator should generate a set of measurement tools to verify performance. The secret to this one is to set measurable targets/goals. Unfortunately, many SMS goals contain few, if any, means of quantifying progress toward goals. For example, a safety goal to reduce overall risk to its lowest possible level is not measurable unless quantified. A more realistic safety goal may be to — reduce risk to its lowest possible level by:

• Conducting risk analyses for all operational issues
• Actively identifying and mitigating hazards
• Conducting quarterly internal evaluation program evaluations
• Ensuring each employee is evaluated on their safety program participation
• Etc…

This is based on and validated by the management axiom: if you can’t measure it, you can’t manage it.

The majority of audits received are now are for the advanced stages. These require evidence of performance, not just having a system in place designed to prepare an operator for conformance. A number of audits describe varying levels of performance but also describe relatively high levels of basic oversights and non-conformities that should have been accomplished at stage 1.If an operator has been operating under IS-BAO for two years yet all personnel have not attended required training courses, safety communication is lacking, SOP are not universally used and/or key standards and limitations are missing from the operations manual, it is hard to believe that safety management is actively targeted and safety risks are being effectively managed (APM 5.4.4). Taken alone these non-conformities are bad enough but together they indicate a pattern that indicates a lack of SMS conformance and effectiveness, and are therefore a significant finding. See APM 6.3.2 and 6.3.3 for additional information.

No one wants to tell an operator they don’t qualify for the next stage, but not doing so may permit a false sense of well-being to continue within the operator’s organization.

Advanced stage audits concentrate on SMS performance but not to the exclusion of the remainder of the IS-BAO standards. The percentage guidelines for emphasis on program effectiveness shown in APM 5.4 are not intended to exclude the necessity of ensuring conformance to all standards. APM 4.2.3 provides guidance in this issue. Checking the program basics, especially performance of essential items, is a required subset of the audit designed to ensure the program fundamentals leading to SMS conformance are indeed in place and working correctly.

At the end of each 2012 protocol chapter the term “Analysis of Non-conformities” has been substituted for the old “Summary of Observations” header. This was done to encourage auditors to do just what the new header says: analyze the evidence, determine severity and justify the finding with objective evidence, as appropriate. Comments and remarks should be placed in the right-hand column of the form. However, if you have no findings to analyze, feel free to use the new analysis section for summary comments for the chapter but preface your remarks with the term “Comments:”

• 2012 IS-BAO forms and protocols must be used for audits performed after 1 July 2012
• We are beginning the annual revision process for all IS-BAO documents and would like to have your input regarding any changes you would recommend for improving these documents. Please send you comments to sbates@ibac.org prior to 15 June 2012.
• When completing the 8.4 audit report form ensure the accuracy of the operator contact information, including company name, operator contact name and email address and telephone number. Inaccurate information often requires time-consuming research to correct.
• Your best source of information regarding whether the SMS is implemented, in use and understood is through interviews with a variety of flight department personnel – they are interesting and emphasize the importance of IS-BAO for the interviewee.
• Management company audits must consider and report on issues shown in IS-BAO Policy 2010-5, Management Company IS-BAO Registration

The following open-book questions are designed to keep you current on the standards and audit procedures. Don’t send us the answers; this is just for you:

1. What items should be covered in the closing meeting?

2. When must a pilot advise the operator of the aircraft’s departure and arrival times?

3. How does an operator know whether its fatigue management system is effective?

4. Is an operator required to explain why safety actions are taken and why safety procedures are changed?

5. If hazard reports are not being used to improve company procedures, training, etc., is this considered to be a critical deficiency for a stage 2 audit?

To confirm your expertise, see the bottom of the page for the references.

Be a discerning and insightful auditor,

John Sheehan
IBAC Audit Manager
auditmanager@ibac.org