Thailand has a high prevalence of childhood caries and low levels of dental service utilization among children. The percentage of dental caries in the permanent teeth of 12-year-old children has increased steadily from 45.8% in 1987 to 57.3% in 2002 [1]. In order to alleviate this problem, the school dental sealant program, which is recommended by the American Association of Community Dental Programs [2], was launched more than ten years ago. Community sealant programs, in particular school sealant programs, have been successfully employed in many countries over long periods [3]. In Thailand, such programs have been implemented in many areas in order to increase access to sealants. However, the outcomes of the use of sealants have generally been poor [4] [5] [6] [7] due perhaps to differences among the quality of the service given by different providers. As a result, training programs have been launched to solve the effectiveness problem. However, the results are still unsatisfactory [7]. Therefore, a new strategy is needed to solve this problem.

Audit and feedback is a strategy widely used to monitor improvements in the performance of physicians. The definition endorsed by the National Institute for Clinical Excellence is "A quality improvement process that seeks the improvement of patient care and outcomes through systematic review of care against explicit criteria and the implementation of change" [8]. Selected aspects of the structure, process, and outcomes of care are systematically evaluated against explicit criteria. Where indicated, changes are implemented at an individual, team, or service level and further monitoring is used to confirm improvements in healthcare delivery. Many studies have reported on the successful use of audit and feedback systems to improve performance in health and health related programs, for instance in the reduction of cesarean delivery rates [9] [10] [11], the prevention of venous thrombo-embolism [12] [13] [14] and the reduction of surgical site infection [15] [16]. Jamtvedt et al. [17] who conducted a review of a large number of studies involving audit and feedback systems noted that:

"... audit and feedback can be effective in improving professional practice. When it is effective, the effects are generally small to moderate. The relative effectiveness of audit and feedback is likely to be greater when baseline adherence to recommended practice is low and when feedback is delivered more intensively."

Clearly, audit and feedback systems must be carefully and intensively designed if they are to be effective.

Literature contains a small number of studies of the use of audit and feedback systems in dental health [18] [19]. Bahrami et al. measured the effectiveness of dental practices who received audit and feedback relating to the management of impacted and unerupted third molars compared to a control group consisting of dental practices who were not subject to audit and feedback [18]. The study found that there was no significant difference in the proportion of patients whose treatment complied with the guidelines between the intervention group and the control group. Palmer and Dailey studied the experience of general dental practitioners in a clinical audit of the prescribing of antibiotics [19]. This was a qualitative study which documented changes in antibiotic prescribing practices and improvements in patient care.

The purposes of this study were (1) to establish the baseline awareness of sealant effectiveness among dental nurses, and (2) to examine the effect of an audit and feedback system on the dental nurses' awareness of sealant effectiveness.

Study design
The study was designed as a randomized cluster trial. The clusters were groups of dental nurses who provided school-based sealant programs in each health care unit in Songkhla province, Southern Thailand. Simple random sampling was performed to select 12 out of 15 health care units and these were then divided into two main groups; the intervention group which consisted of six clusters with 23 dental nurses and the control group which included another six clusters with 22 dental nurses.

Instruments
The instruments used in this study were sealant and caries record forms, a self-administered questionnaire to establish baseline data regarding the subjects' awareness of dental sealant effectiveness, and a semi-structured interview guideline based on a set of open-questions for the focus group discussions (FGD).

The audit was done by examining of sealant retention and caries on the sealed surface. Sealant retention was classified as either, fully retained or loss of sealant retention, which included partially retained and missing sealant according to Simonsen's criteria [20]. The criterion used to detect caries was based on those of Möller and Poulsen [21]. The reliability of intra-examiner and inter-examiners were performed in 15 children. The kappa values for intra-examiner agreement were 0.75–0.8 for sealant retention and 0.80–0.85 for caries. The inter-examiner agreements were 0.75 for sealant retention and 0.82 for caries.

The content validity of the self-administered questionnaire and the FGD open question guidelines were performed by the two experts.

Procedure
This study was done during 2009 and 2010. The whole process of this study was summarized in (Figure 1). The design of the audit and feedback system began at a preliminary workshop, in which the objectives of the program were set out and discussions held on the feasibility of the system. After the workshop, the school sealant program was conducted in the usual way. The baseline data of their awareness of sealant effectiveness collected from the dental nurses. The audit system was set-up to examine the children's teeth six months after sealant application to establish the extent of sealant retention and the presence of caries as an indication of service quality.

The data from the first audit was analyzed and the feedback phase was conducted by two methods. The first method involved conducting workshops among the dental nurses in the intervention groups to discuss the results based on sealant retention and the development of caries. The qualitative data on sealant retention failures and caries were classified into six scenarios and the dental nurses discussed and identified the common causes of these scenarios. At the request of the dental nurses further workshops were conducted aimed at identifying solutions to these causes.

The second method by which feedback was conducted was through the presentation of quantitative data which included the sealant retention rate, and the rate of caries on the surface of sealed teeth. All data were presented at both a provincial level and a cluster level. Therefore, the providers in each cluster were able to compare their results to the overall result. The second feedback data was given confidentially to each cluster in the workshop in a sealed envelope.

After implementation of the audit and feedback, FGD were conducted for each cluster in the intervention group. The scope of these discussions included their awareness of the effectiveness of the sealant they applied and how they used the feedback data to improve the quality of school sealant program.

The places and times of the FGDs were chosen by the participants. All of the participants were asked to speak openly and honestly which is an essential feature of the audit and feedback system and open-minded and relaxed discussions were encouraged. Each FGD started with a brief introduction about the objectives of the program and its process, and ended with a summary of the main points arising from the discussion. The number of participants in each FGD was between three and seven depending on the number of providers in each cluster. The discussions occupied between 45 minutes and 3 hours and 30 minutes. The open question session followed the introduction and the researcher then followed-up the informants' answers by asking follow-up questions to clarify and go deeper into the subject under discussion. The semi-structured interview guide was used to ensure that the discussion covered all the key areas. All the discussions were audio-recorded and were later transcribed verbatim by the interviewer.

The sample size necessary to maintain statistical validity was calculated based on the guidelines suggested for cluster randomized trials by Cosby et al. [22]. The total number of children estimated to be required was 1500. The actual acquired sample size in this study was 1704 children.

Data analysis
The data analysis was conducted using the R program, version 2.12.0 [23] employing descriptive statistics and logistic regression. The descriptive statistics derived from the data included frequencies and percentages. Logistic regression was used to assess the influence of the dental nurses' awareness of sealant effectiveness on the recorded effectiveness of the use of sealant. The dependent variables in the regression analysis were sealant retention and caries on sealed surfaces, and the main independent variable was the dental nurses' awareness of sealant effectiveness controlled for service settings and the children's characteristics.

Thematic content analysis was performed on the qualitative data [24] [25]. The process involved analyzing transcripts, identifying themes and gathering examples of those themes. This process was conducted both at the time of data collection and also during data analysis. During data collection, participant validation was effected by summarizing the key points of the discussion for them and asking them to verify that the summary was accurate. Peer review verification was conducted independently during data analysis by two experienced researchers who were involved in the collection of data at other FGDs.

Baseline data of the dental nurses and their awareness of sealant effectiveness
There were 45 dental nurses in 12 clusters. There was no difference of these dental nurses' characteristics between control and intervention groups. Forty-three of them were female. Two-thirds of them were more than 30 years old (Table 1). Most of them had experience of sealant training courses and sealant evaluation, implemented the school sealant program both in hospitals and in school-based settings. Nearly, 40% of dental nurses from both groups had low awareness of sealant effectiveness.

Baseline data of sealed teeth and sealant effectiveness
The characteristics and sealant effectiveness of the 4121 sealed teeth from 1703 children are given in Table 2. There were significant differences of caries experience, tooth position and sealant retention of sealed teeth between control and intervention groups. High caries experience and high number of lower teeth were sealed in the control group more than the intervention group. Moreover, higher percentage of full sealant retention found among children in control group.

Factors influencing sealant effectiveness
Tables 3 and 4 give the results from the logistic regression analysis. There was significant indication of the influence of dental nurse baseline awareness of sealant effectiveness. Low awareness on sealant effectiveness had more sealant loss compared to high awareness group. The intervention group had 2.31 times loss of sealant retention compared to the control group. The factors significantly related to sealant retention were the setting in which sealant was applied, the gender of the child, caries experience and tooth position. Application of sealant in a hospital setting, male gender, application on lower teeth and low caries experience all tended to increase the rate of sealant loss significantly (Table 3) whereas there were no significant factors which influenced the rate of caries on sealed surfaces (Table 4).

The awareness of the sealant quality results from FGD
The participating dental nurses were asked, how they felt about the results from the audit and feedback and what they did when they received feedback indicating poor sealant quality. It became apparent that the participants had two distinct reactions to such feedback. The impression emerging from their direct statements was of a conflict between the quantity of children who were treated and the quality of the service they received. On the other hand, their indirect statements indicated their wish to identify problems and to find ways of solving the problems identified by the data in the feedback.

Direct statements indicating awareness about service quality
All the dental nurses told us that they were dissatisfied with their sealant effectiveness and that they worried more about cases of partial sealant retention which were more prone to caries than about cases where the sealant was totally lost. They also talked about the conflict between the number of cases treated and the quality of the school sealant service. Dental nurses in Cluster I2 (intervention 2), for instance, conducted their own evaluation of sealant effectiveness, and obtained a similar result to that which they received from the feedback given in this study and they found hard to believe.

"There was high sealant loss in our area. We got a similar result from our own sealant evaluation and this made us feel very dissatisfied. We had not only wasted our material but also wasted our time. We realized that if our use of sealants was ineffective, we would just be waiting for caries to occur so that we could provide fillings...". (Cluster I2)

"When we saw the results in the feedback, we discussed what we wanted to provide - quantity or quality? What should we do next? Should we get the names of children with partial sealant retention and caries and provide fillings for them? We were afraid that they might just get more caries". (Cluster I4)

"Quantity is important but quality is more important. If we apply sealant in 8 or 10 cases without quality, we achieve nothing at all. We have wasted our time and money but we still have children with caries". (Cluster I6)

Indirect statements indicating awareness of the problem and the desire to solve it.

The current policy
The dental nurses in all clusters complained that the policy, which aims to maximize the number of cases where sealant is applied, has resulted in poor service quality because the goal of the policy is not based on the actual situation in terms of the manpower available, the overall workload, the number of children needing to be treated and the condition of their teeth. Key performance indicators (KPI) have been set according to the policy and this affects the way in which the program is promoted. This problem is being dealt with by trying to allocate more manpower to the school sealant program.

"There are so many schools with so many children in our area. There are about 40 schools. Last year, the sealant target group was grade 1 which was about 1,000 children. The goal was for 75% of all children to have their teeth sealed. This year, the policy also includes 25% of grade 6 children". (Cluster I6)

"Some children were uncooperative when we tried to apply sealant to their teeth. We felt very stressed; how many cases were we expected to treat in an one hour?" (Cluster I5)

Dental Assistants
The most commonly identified cause of sealant failure was a lack of adequate qualified chair-side assistants. They do not act as chair-side assistants because of a lack of manpower, whilst there are general dental assistants. Two clusters (Clusters I2 and I5) had unqualified dental assistants who did not have a good attitude towards their role and provided ineffective assistance.

"He did not assist me effectively and could not control saliva contamination. He did not pay attention to the work and did not love his job. When I tried to advise him, he was not interested in my advice". (Cluster I2)

"We have two dental nurses and only one assistant available for the school sealant program. The dental assistant works in a general capacity. Only in case one of the children is uncooperative will the dental assistant help as a chair-side assistant". (Cluster I1)

Equipment
The last problem that the dental nurses identified was with the quality of the equipment they used. The mobile school sealant program was implemented in order to increase the number of children who have access to sealant. The equipment is moved and set-up at each school the service visits. However, the system for maintaining the equipment is inadequate leading to broken and inefficient equipment in all the clusters. The participating dental nurses requested a workshop on routine maintenance and basic repair techniques for both dental assistants and dental nurses.

"We just have to repair our own equipment. Since we started the outreach sealant program, we have applied sealant in schools. This semester, the equipment was moved from place to place more than 17 times". (Cluster I5)

"We need training on basic repair and maintenance techniques. We do not know much about the equipment. Once I had to call back to the hospital and ask how to solve a problem. That time, the solution was just turning on a switch". (Cluster I2)

This study found poor sealant retention and a high rate of caries on sealed surfaces among school children. There was the evidence that the dental nurses' baseline awareness of dental service quality influenced sealant effectiveness and nearly 40% of the dental nurses had low awareness on sealant effectiveness. The dental nurses' awareness of service quality was apparent from their reactions after receiving feedback.

The effectiveness of the use of sealant found in this study was similar to that found in other studies in similar contexts [6] [7] [8] [9] [10]. This is much lower than that found in studies in developed countries where around 87% effectiveness has been recorded within one year [2]. Even though, in Thailand, the school sealant program has been operating for more than ten years and the program has been found to be ineffective in many studies [6] [7] [8] [9] [10]. Some efforts have been made to improve the sealant effectiveness achieved by the school sealant program. Tianviwat et al. [10] conducted a continuing education program providing training in sealant application techniques and the results showed a small improvement in sealant effectiveness [5]. However, a recent review of literature suggests that most of the studies relate to the causes of sealant failure due to clinical factors while neglecting other important factors such as service providers, policy and workload [2].

As previously noted, audit and feedback is a performance improvement strategy which can be effective in cases where baseline adherence to recommended practice is low and where feedback is delivered more intensively [22]. In the present study, the intensive feedback was based on the dental nurses' own suggestions. They wanted to know about their effectiveness and the most common failures found in their context. They also wanted to be able to compare their effectiveness with the overall result achieved by all the clusters in the study as a benchmark for their performance. The most common types of failures noted by the researchers during their evaluation were presented to them as scenarios, such as the thickness of sealant applied and the most commonly fractured areas.

Two-thirds of the dental nurses in this study had experience of sealant evaluation which was one of the processes in the audit system. However, in the past, the results of these evaluations had not been communicated to them. Therefore, they did not realized that there was a quality problem.

Their perception of sealant effectiveness was different from the actual data given to them during the feedback sessions. The FGD findings revealed that the data given to them during feedback increased their awareness of the issue of service quality. The dental nurses discussed the issue of quantity versus quality within the school sealant program and were able to discover the causes of their own failures. The causes of failure discovered in this study were not generally found in other studies. They reflected the problems related to the specific context of this study and extended beyond purely clinical factors.

Implications for practice
In Thailand, the school health care programs which included dental and general health care have been implemented in many areas in order to increase accessibility of school children. The usual evaluation based on the coverage of schools or children. Therefore, there was a problem of the quality of programs.

Health care systems have invested many resources in developing training courses to improve quality of care which showed a small improvement. The intensive audit and feedback system designed by whom who received the feedback showed the change of their awareness and identification of their problems. For problem solving, a partnership between the local provider, the health provincial office and the university is needed to solve the complex of the poor service quality which is beyond the clinical factors.

There were a limited number of clusters allocated to the intervention and control groups, and the numbers of sealed teeth and the number of children whose teeth were sealed was different in each cluster. The strong point of the use of FGDs in this study is that they were able to deal with sensitive issues and to clarify them. However, their weakness is that the results which emerged relating to the causes of sealant failures are unlikely to be capable of being generalized beyond the specific context of the study, although generalizability was not the main purpose of this study. The next phase of this study should examine whether performance improvement occurs following the audit and feedback or not.

The intensive audit and feedback system was able to improve the dental nurses' awareness of dental service quality of school sealant program and initiate their problem solving.

This study was supported by the Office of the Higher Education Commission and the Thailand Research Fund (MRG5280196). The opinions expressed in this paper are those of the researchers, and not those of the Office of the Higher Education Commission or the Thailand Research Fund.

1. Dental Division. The 5th Thailand national oral health survey report. Bangkok: Health Department (Thailand), Ministry of Public Health 2002.    
2. Gooch BF, Griffin SO, Gray SK, et al. Preventing dental caries through school-based sealant programs: updated recommendations and reviews of evidence. J Am Dent Assoc 2009 Nov;140(11):1356–65.   [Pubmed]    
3. Beauchamp J, Caufield PW, Crall JJ, et al. Evidence-based clinical recommendations for the use of pit-and-fissure sealants: A report of the American Dental Association Council on Scientific Affairs. J Am Dent Assoc 2008 Mar;139(3):257–68.   [Pubmed]    
4. Boonyaves K. The efficiency of using dental sealant in 6–9 year old children at different depth of fissure. Taksin Med J 1994;13(1):7–15. (In Thai).    
5. Tianviwat S, Chukadee W, Sirisakulveroj B, Leewanon R, Larsen MJ. Retention of pit and fissure sealants under field conditions after nearly 2-3 years. J Dent Assoc Thai2001;51(2):115–120.    
6. Thipsoonthornchai J. The comparative study on retention rate and caries prevention between Glassionomer and Resin using as pit and fissure sealant in mobile dental service, Buriram province. J Dent Public Health 2003;15(2):62–77.    
7. Tianviwat S, Chongsuvivatwong V, Sirisakunweroj B. Loss of sealant retention and subsequent caries development. Community Dent Health 2008 Dec;25(4):216–20.   [Pubmed]    
8. National Institute for Clinical Excellence: Principles for Best Practice in Clinical Audit. Abingdon: Radcliffe, Medical press 2002. in Foy R, Eccles MP, Jamtvedt G, YoungJ, Grimshaw JM, Baker R. What de we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. Website:http://www.biomedcentral.com/1472–6963/5/50. Access on March 2010.    
9. Bickell NA, Zdeb MS, Applegate MS, Roohan PJ, Sui AL. Effect of external peer review on cesarean delivery rates: A statewide program. Obstet Gynecol 1996 May;87(5 Pt 1):664–7.   [CrossRef]   [Pubmed]    
10. Althabe F, Belizán JM, Villar J, et al. Mandatory second opinion to reduce rates of unnecessary caesarean sections in Latin America: A cluster randomized controlled trial. Lancet 2004 Jun 12;363(9425):1934–40.   [CrossRef]   [Pubmed]    
11. Liang WH, Yuan CC, Hung JH, et al. Effect of peer review and trial of labor on lowering cesarean section ration rates. J Chin Med Assoc 2004 Jun;67(6):281–6.   [Pubmed]    
12. Michota FA. Bridging the gap between evidence and practice in venous thromboembolism prophylaxis: The quality improvement process. J Gen Intern Med 2007 Dec;22(12):1762–70.   [CrossRef]   [Pubmed]    
13. Labarere J, Bosson JL, Brion JP, et al. Validation of a clinical guideline on prevention of venous thromboembolism in medical inpatients: A before-and -after study with systematic ultrasound examination. J Intern Med 2004 Oct;256(4):338–48.   [CrossRef]   [Pubmed]    
14. Cohn SL, Adekile A, Mahabir V. Improved use of thromboprophylaxis for deep vein thrombosis following an educational intervention. J Hosp Med 2006 Nov;1(6):331–8.   [CrossRef]   [Pubmed]    
15. Richards C, Emori TG, Peavy G, Gaynes R. Promoting quality through measurement of performance and response: Prevention success stories. Emerg Infect Dis 2001 Mar-Apr;7(2):299–301.   [CrossRef]   [Pubmed]    
16. Schneeberger PM, Smits MH, Zick RE, Wille JC. Surveillance as a starting point to reduce surgical-site infection rates in elective orthopaedic surgery. J Hosp Infect 2002 Jul;51(3):179–84.   [CrossRef]   [Pubmed]    
17. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Audit and feedback: Effects on professional practice and health care outcomes (review). The Cochrane Library 2007;4:1–23.    
18. Bahrami M, Deery C, Clarkson JE, et al. Effectiveness of strategies to disseminate and implement clinical guidelines for the management of impacted and unerupted third molars in primary dental care, a cluster randomized controlled trial. Br Dent J 2004 Dec 11;197(11):691–6.   [CrossRef]   [Pubmed]    
19. Palmer NA, Dailey YM. General dental practitioners' experiences of a collaborative clinical audit on antibiotic prescribing: A qualitative study. Br Dent J 2002 Jul 13;193(1):46–9.   [CrossRef]   [Pubmed]    
20. Simonsen RJ. Retention and effectiveness of dental sealant after 15 years. J Am Dent Assoc 1991 Oct;122(10):34–2.   [Pubmed]    
21. Moller IJ, Poulsen S. A standardized system for diagnosting, recording and analyzing dental caries data. Scand J Dent Res 1973;81(1):1–11.   [Pubmed]    
22. Cosby RH, Howard M, Kaczorowski, Willan AR, Sellors JW. Randomizing patients by family practice: Sample size estimation, intracluster correlation and data analysis. Fam Pract 2003 Feb;20(1):77–82.   [CrossRef]   [Pubmed]    
23. R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3–900051–07–0. Available at: URL http://www.R-project.org. Accessed January 2010.    
24. Burnard P, Gill P, Stewart K, Treasure E, Chadwick B. Analysing and presenting qualitative data. Br Dent J 2008 Apr 26;204(8):429–32.   [CrossRef]   [Pubmed]    
25. Rabiee F. Focus-group interview and data analysis. Proc Nutr Soc 2004 Nov;63(4):655–60.   [CrossRef]   [Pubmed]