A systematic review of health promotion in schools demonstrated the varying success of self-help programmes designed to change risky behaviour, such as smoking or substance misuse (Reference Lister-Sharp, Chapman and Stewart-BrownLister-Sharp et al, 1999). Internet and computer-based assessment and interventions offer an alternative method to be used alongside more traditional approaches. CD–ROMs have shown promise for changing behaviour and promoting better psychological health in the USA (Bosworth et al, Reference Bosworth, Gustafson and Hawkins1994, Reference Bosworth, Espelage and Dubay2000).
The internet is an accessible resource (Reference McAlindon, Formic and KabbaraMcAlindon et al, 2003) and media, such as television, can provide information in a non-stigmatising way (Reference Sanders, Montgomery and Brechman-ToussaintSanders et al, 2000). The internet also facilitates individual autonomy (Reference Theodosiou and GreenTheodosiou & Green, 2003), which is an important part of adolescent development. The internet can also provide self-help assessment and interventions to people who do not seek help for their mental health problems (Reference Marks, Mataix-Cols and KenwrightMarks et al, 2003; Reference Christensen, Griffiths and JormChristensen et al, 2004; Reference Kenwright, Marks and GegaKenwright et al, 2004).
There is hardly any knowledge of how self-assessment questionnaires completed online compare with paper questionnaires (Reference Ritter, Lorig and LaurentRitter et al, 2004). Frequent concerns are that online data may not be secure or reliable because the rater may not understand the instructions, may not take as much care when completing the forms online, or may be influenced by peers to give higher or lower ratings if the online assessment takes place in too public an area, such as an open library, classroom or drop-in centre.
With these issues in mind, the website ru-ok.com was developed in 2003 with the aid of focus groups of teenagers. The research project was funded by the Health Foundation. Focus groups were positive about the final result and the site went live in 2003. The site allows teenagers to assess their strengths and weaknesses and obtain self-help advice. The site includes, among other psychological self-help material, a widely used educational resource, ‘Retracking’ (Reference BatesBates, 1997) and a self-assessment questionnaire based on the Salford Needs Assessment Schedule for Adolescents (Reference Kroll, Woodham and RothwellKroll et al, 1999). There are a range of other features, including interactive stories, cartoons about depression, eating problems, teenage pregnancy and school problems, and some games.
The aims of this study were: (a) to examine whether teenagers find the website acceptable and useful; (b) to validate the RU–OK self-assessment questionnaire by comparing it with standard questionnaires.
Method
Participants
The sample was drawn from a Salford comprehensive school and a special hospital school for children with depression and anxiety.
Measures
The RU–OK self-assessment questionnaire examines problems and strengths of teenagers. The teenager, using a secure method, enters their chosen user name and password. There are 26 questions about problems, each question rated on a 5-point scale from 1 to 5, zero equating to ‘not rated’. The questions are taken from the Salford Needs Assessment Schedule for Adolescents (Reference Kroll, Woodham and RothwellKroll et al, 1999). A score of 3 or more is judged to indicate a significant problem and is based on validity data from the original research study. Data can be saved at different time points and reviewed on revisiting the site. The online questionnaire takes about 5–10 min to complete and advice is available for each problem area depending on the score. Low scores would usually lead to advice about self-help, such as using downloadable material from the site or links to other recommended sites. Higher scores would stress the importance of discussing problems with a mentor, parent or professional.
A qualitative questionnaire asking about acceptability, ease of use and usefulness of parts of the website was administered. Each of the questions is rated on a 5-point scale.
The adolescent self-completion version of the Strength and Difficulties Questionnaire (SDQ; Reference GoodmanGoodman, 1997) consists of 25 items scored on a 3-point scale. In addition to a total score, there are sub-scales for difficulties in the areas of hyperactivity, conduct, emotions and social interactions. There is also one prosocial scale. For epidemiological research studies, cut-off points on each scale can be set in order to classify teenagers into positive and negative groups depending on the research requirements.
The adolescent Mood and Feelings Questionnaire (MFQ; Reference Costello and AngoldCostello & Angold, 1988) is designed to screen for depression. It consists of 34 questions scored on a 3-point scale. A cut-off point of 27 is often used, giving a sensitivity and specificity of about 80% compared with a gold standard research interview.
Sub-scales from the RU–OK questionnaire were generated so that Pearson correlation could be assessed with the MFQ and SDQ. To do this, scores from certain areas were added together. The sub-scales were:
-
• violence: consisting of two areas, violence to property and persons
-
• emotional: consisting of seven areas, depression, anxiety, trauma, self-harm, eating problems, self-esteem and paranoid thinking
-
• relationships: consisting of two areas, peer problems and family problems
-
• drugs and alcohol: consisting of two areas, drugs and alcohol
-
• total score: consisting of all 26 areas.
The sub-scales and individual scores were checked to ensure normal statistical distribution.
Data collection and loss
The study received local ethical approval and was then explained to teenagres by two research psychiatrists, S.E. and A.K., in a personal, social and health education (PSHE) lesson. The parent or guardian was then informed and teenagers completed consent forms. The following week, teenagers supplied demographic details and completed the SDQ and MFQ. They were given a postcard with a research identification number and instruction about the website. The next PHSE lesson was held in the information technology room and teenagers completed (in confidence) the RU–OK questionnaire and evaluation questionnaire. The same method of data collection was used with the teenagers who attended the hospital school.
Of the teenagers approached, only four did not agree to participate in the study. About 40% of evaluation questionnaires were not completed owing to lack of time in the lesson; the final numbers of questionnaires ranged from 64 to 67. A total of 105 teenagers completed the SDQ and 101 the MFQ questionnaires. A total of 103 completed the RU–OK self-assessment questionnaire but about 30% of the data were lost to analysis because of failure to track pupils’ private identification numbers on the questionnaires. The complete set of questionnaires for analysis of correlation between RU–OK and the SDQ and MFQ was thus 69.
Results
Participants
A total of 105 teenagers were included in the final analysis, 46 boys and 59 girls; 91 teenagers were aged 13–14 years and came from year 9 in the comprehensive school; 14 teenagers came from the hospital school of which 13 were 14- to 15-year-olds (year 10 and 11), the other being from year 9.
Responses
Table 1 shows the responses to the three questionnaires. The sub-scale scores for the SDQ are not significantly different from the national SDQ norms (http://www.sdqinfo.com). The numbers of teenagers classified as having a significant problem on the MFQ and the RU–OK questionnaire are also shown.
Table 1. Responses of participants to the three questionnaires
| Questionnaire | Mean | (s.d.) | Below cut-off, n | Above cut-off, n |
|---|---|---|---|---|
| SDQ, n=105 | ||||
| Total score | 11.8 | (5.2) | ||
| Emotional sub-scale | 2.6 | (2.0) | ||
| Conduct | 2.7 | (1.7) | ||
| Hyperactivity | 4.3 | (2.3) | ||
| Peer problems | 2.1 | (1.6) | ||
| Prosocial score | 6.0 | (1.7) | ||
| MFQ, n=101 | ||||
| Total score | 91 | (10) | ||
| RU-OK, n=103 | ||||
| Looking after self | 91 | 10 | ||
| Cooking | 85 | 16 | ||
| Physical illness | 95 | 6 | ||
| School attendance | 93 | 8 | ||
| School work | 89 | 12 | ||
| Employment | 4 | 7 | ||
| Peer | 91 | 10 | ||
| Family | 88 | 13 | ||
| Cultural/racial | 94 | 3 | ||
| Sexual activity | 94 | 3 | ||
| Alcohol | 100 | 1 | ||
| Drugs | 95 | 4 | ||
| Depression | 87 | 14 | ||
| Self-harm | 96 | 5 | ||
| Anxiety | 91 | 11 | ||
| Self-esteem | 87 | 14 | ||
| Bullying | 96 | 5 | ||
| Anorexia/bulimia | 96 | 5 | ||
| Psychological trauma | 97 | 4 | ||
| Sexual health/knowledge | 97 | 4 | ||
| Paranoid thinking | 99 | 2 | ||
| Money | 91 | 10 | ||
| Neighbourhood | 91 | 10 | ||
| Leisure activities | 91 | 10 | ||
| Violence to others | 87 | 14 | ||
| Defiance, making trouble | 94 | 7 |
Correlation between the RU–OK questionnaire subscales, the MFQ and SDQ total and sub-scale scores are shown in Table 2. This shows modest correlations between scales. The MFQ, SDQ and RU–OK questionnaire emotional sub-scales and conduct sub-scales had better correlations as expected. We thus performed a standard statistical 2 × 2 table screening procedure of two areas (Reference WarnerWarner, 2004), despite the small sample size. A cut-off point of 4 on the SDQ for emotional and conduct subscales was used. Using the SDQ emotional sub-scale as the gold standard, the RU–OK ‘depression’ sub-scale gave a sensitivity of 17.4% and specificity of 97%, with a positive predictive value of 80% and a negative predictive value of 70%. Confidence intervals were wide. Comparison with the MFQ showed much lower sensitivity and specificity. When the RU–OK ‘violence to others’ problem area was compared with the conduct sub-scale of the SDQ, the sensitivity was 48%, specificity 94%, positive predictive value 77% and negative predictive value 80%.
Table 2. Correlations between MFQ, SDQ and RU–OK sub-scales
| SDQ sub-scales and total score n=105 | MFQ n=101 | RU—OK sub-scales and total score n=69 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hyperactivity | Conduct | Emotional | Prosocial | Peer problems | SDQ total score | MFQ total score | RU-OK total score | Emotional sub-scale | Violence sub-scale | Relationships sub-scale | Drugs/alcohol | |
| SDQ | ||||||||||||
| Hyperactivity | 1 | |||||||||||
| Conduct | 0.565** | 1 | ||||||||||
| Emotional | 0.283** | 0.170 | 1 | |||||||||
| Prosocial | -0.293** | -0.337** | 0.090 | 1 | ||||||||
| Peer problems | 0.102 | 0.074 | 0.415** | 0.034 | 1 | |||||||
| Total score | 0.770** | 0.672** | 0.694** | -0.197* | 0.543** | 1 | ||||||
| MFQ total score | 0.337** | 0.231* | 0.584** | 0.110 | 0.347** | 0.557** | 1 | |||||
| RU—OK | ||||||||||||
| Total score | 0.196 | 0.148 | 0.275* | -0.043 | 0.442** | 0.387** | 0.373** | 1 | ||||
| Emotional | 0.131 | 0.015 | 0.420** | 0.016 | 0.534** | 0.403** | 0.537** | 0.814** | 1 | |||
| Violence | 0.329** | 0.337** | 0.051 | -0.310** | 0.157 | 0.326** | 0.188 | 0.690** | 0.422** | 1 | ||
| Relationship | 0.107 | 0.057 | 0.173 | 0.058 | 0.451** | 0.281* | 0.278* | 0.822** | 0.575** | 0.469** | 1 | |
| Drug and alcohol | 0.313** | -0.280* | 0.017 | -0.134 | -0.067 | 0.215 | 0.180 | 0.260** | 0.012 | 0.222* | 0.218* | 1 |
The summary results of the questionnaire are shown in Table 3. The 5-point scales were reduced to two categories as shown. Excellent, good and OK were combined into ‘positive’, and poor and bad into ‘negative’. Table 3 shows that 70–80% of respondents were positive about the site. There were no differences between the main-stream school and the specialist hospital school.
Table 3. Summary results of the questionnaire evaluating the ru-ok website
| Positive view | Negative view | Total n | |||
|---|---|---|---|---|---|
| Evaluation of site | n | % | n | % | |
| Navigation of site | 59 | 88.1 | 8 | 11.9 | 67 |
| Self-assessment area | 52 | 78.8 | 14 | 21.2 | 66 |
| Your overall opinion of site | 58 | 86.6 | 9 | 13.4 | 67 |
| Graphics/pictures on site | 56 | 87.5 | 8 | 12.5 | 64 |
| Customising the site | 57 | 85.1 | 10 | 14.9 | 67 |
| Games | 50 | 76.9 | 15 | 23.1 | 65 |
| Interactive stories | 54 | 83.1 | 11 | 16.9 | 65 |
We analysed website traffic; in the 20 months since the site went live there were 18 382 visits: 2956 people had logged on to the site and 640 completed the RU-OK questionnaire. Of activity on the site, 23% was dedicated to downloading advice and information files and 12% to looking at the cartoons about depression, eating disorders, school problems and pregnancy. On average, there are about 900 visitors per month, of which 150 logged on to the secure section of the site containing the online questionnaire. There continues to be a steady increase in activity; the number of visitors to the site is now about 730 per week.
Discussion
This study shows that the ru-ok.com website was positively evaluated by a sample of representative teenagers aged 13–14 years. Website traffic indicates a steady rise in use, as more teenagers become aware of the existence of the site. The self-assessment questionnaire shows acceptable correlations with established paper-and-pencil screening questionnaires.
Adolescent mental health problems in school are increasingly recognised and questionnaires can be used to screen for these (Reference Stansfeld, Haines and HeadStansfeld et al, 2004). The advantage of the RU–OK questionnaire is that it is based on a needs assessment methodology; the paper version (Reference Kroll, Woodham and RothwellKroll et al, 1999) has good face validity and acceptability with teenagers. The RU–OK questionnaire does not require any summary analysis to interpret the results or to obtain downloadable advice and information about interventions. Some questionnaires, however, can be misused and misunderstood (Reference Clark and HarringtonClark & Harrington 1999; Reference WarnerWarner, 2004). Many also require scoring and interpretation.
Shortcomings of this study include the limited age range of teenagers studied and the small sample size. Thus, formal screening analysis and receiver operator curve analysis was not possible. Ideally, the RU–OK questionnaire should be compared with a gold standard diagnostic interview and other observer-rated methods using a larger sample of teenagers.
Schools face an increasing challenge of first identifying pupils with problems and then delivering acceptable interventions at an appropriate tiered level. Stepped care models are being proposed for certain conditions (National Institute for Clinical Excellence, 2004), and it would be useful to study further the role of screening questionnaires and self-assessment in schools who use a stepped care approach.
Declaration of interest
L.K. contributes to the management and development of the ru-ok.com website.
Acknowledgements
We thank the pupils and staff of the two schools who participated in this study, the Health Foundation who funded the website development, and the website development research team, Jenny Bates, Aaron Davidson, Leo Kroll, John Rowley, Hilary Wait and Heather Sacco.
Open access
eLetters
No eLetters have been published for this article.