The impact of involving young children in improving 1 the understanding of a new eye test

research


Introduction
The UK National Advisory Group promoting public involvement in health and social care research, INVOLVE, defines Patient and Public Involvement (PPI) as research which is …"carried out 'with' or 'by' members of the public rather than 'to', 'about' or 'for' them 1 .This does not include people in their role as research subjects or participants in a study, but rather as advisers or collaborators in the research 2 .PPI continues to be of growing importance for research across disciplines.Involvement of children in research is encouraged in the guiding principles of the United Nation's convention on the Right of the Child.Article 12 acknowledges that children and young people have the right to be involved in decisions that affect their life, express their views and have their views listened to 3 .
Involving children in research from design to dissemination is therefore no longer a preferred approach but a requirement 4 .
Traditionally, children's perspectives have been filtered through interpretations of parents and carers rather than children being involved themselves with their unique insights into their own reality 5 .Despite the change in policy and vision, examples of involvement of children in research are very limited, especially across the range of healthcare provision 4,6,7 .
For instance, on 28 th April 2019, the INVOLVE evidence library 8 included 321 works of involvement of people in designing research in the health sector, 19 of which relate to children's research and PPI.Of these 19, only seven describe original research that involves children, and none outline involvement of children under 6 years old.Dunn and colleagues 4 concluded that despite many "efforts to include children's voices, translation into research and pedagogical practice is still evolving".
Whilst securing involvement from adults may be more straightforward, involving children in research is challenging.There is the intrinsic issue of generational divide and power differences between researchers and children they endeavour to work with 4 .Specifically, younger children are arguably at more risk of difficulty with understanding questions that ask for their feedback.The difficulty of choosing age-appropriate involvement methods for young children has been seen as a barrier to involvement 4 and good practice methodology on how PPI with children should be achieved is lacking 2 .Some guidelines are available 5,9 but they only advise on running discussion groups with young people, generally above 12 years old.In sum, while toolkits and case studies from national bodies provide starting points from which research groups could promote PPI practice, a large evidence base containing research that implements this does not yet exist.Furthermore, no evidence is available on the impact of PPI with young children upon research to drive the development of standards of good practice 6 .Taken together, there is an urgent need for evidence on how young children can be involved in research and what impact can be achieved.

Aims
The current paper describes how children were involved in improving children's understanding of ASTEROID, a new 3D eye test, after initial prototype development.The purpose is to highlight the importance and impact of involving young children in the development of medical technology.Most crucially, this paper aims to contribute to the slowly growing pool of research that involves children and young people and provide evidence of its impact on research.Such an evidence base is the allimportant next step to ensure that Public and Patient Involvement becomes a fundamental part of all kinds of research.

General methods
The ASTEROID study Stereopsis, stereoscopic or simply stereo vision all refer to the perception of depth via binocular vision, or vision using both eyes.Normal stereoscopic vision is associated with correct development of visual functions and alignment of the eyes [10][11][12] .Measuring stereopsis is therefore common in children with suspected amblyopia or strabismus.However, current ways of measuring stereopsis can be unengaging and under-sensitive (e.g.Huynh, Ojaimi, Robaei, Rose, & Mitchell, 2005).To address this, a new test, known as "Accurate STEReotest On a mobile Device" or ASTEROID, was developed on a glasses-free 3D tablet (see Figure 1 and Study 1 Methods for a detailed description, see also 14 ).The 3D tablet eradicates the need for children to wear 3D glasses during testing, something that has proven to be an issue with younger children with the currently available stereotests 15 .To engage children and young people, ASTEROID took the form of an animated game.
ASTEROID was validated against a gold standard stereotest and normative data were collected.
Anecdotal evidence and feedback gathered during data collection highlighted some issues.Key to this is ensuring that each participant understood what "seeing the 3D image" would be like to ensure that any failures were genuinely due to problems with stereo vision rather than understanding test instructions.These initial outcomes triggered an iterative PPI approach.

Design of the studies
An iterative approach was undertaken to improve ASTEROID (see Figure 1).Each Study started with a data collection stage in which children completed ASTEROID in the context of our larger validation study and we observed an issue related to poor understanding of ASTEROID (pre-measure).In the second stage, this issue was then explored in depth during patient and public involvement (PPI) with a different group of children or adults.By collecting feedback from participants with no previous experience of ASTEROID, we were able to avoid learning effects.PPI outcome triggered changes to ASTEROID (Stage 3).Subsequently, the new version of ASTEROID was evaluated in a separate sample (post-measure, Stage 4).This enabled us to collect quantitative evidence of the impact of our PPI.
Verbal feedback and observations during Stage 4 highlighted other issues and subsequently triggered another cycle.In this paper we describe two such cycles that aimed to improve understanding of ASTEROID (Study 1 and 2).Our secondary aim was to monitor how engagement levels changed following improvement of ASTEROID (Study 3).Instruments.ASTEROID (version 0.932 and 0.933) is a stereotest that runs on a 3D tablet computer with an autostereoscopic display, meaning no 3D glasses were required for use.The stereotest is embedded in an enjoyable game designed to keep children engaged and responsive.A Bayesian staircase was implemented to adjust the difficulty level of each trial, depending on the user's ability.
The test takes the form of four dynamic random-dot stereograms (Figure 2).A disparate target  Results.39% of the children showed a poor understanding of the ASTEROID task by solving less than 80% of the practice trials correctly.The average proportion of correct practice trials was 0.76 and on average 10 additional prompts were needed per child to explain the task.We particularly observed difficulties in the transition between the practice trials and the stereo trials.were invited to join a PPI session for 5-10 minutes.No ethical approval was required for PPI 16 .
Children were thanked for their feedback with a 'Junior Scientist' certificate and a sticker.
Level and nature of involvement.We organised three informal drop-in PPI sessions (Table 1).We setup a stall with engagement activities around vision testing and neuroscience in an area of the museum with high traffic (Figure 3).The stall contained games including visual distortion goggles, bean-bags and examples of visual illusions.The competitive nature of some of the activities engaged children and their siblings, whilst certificates and other small prizes were available as a reward.Outcome measures.
1) Observations of the participant's progress (e.g.focus, hesitation, difficulties etc.) were collated on each trial.
2) Individual verbal feedback from participants: We asked questions like what they found difficult, what was unclear to them, whether they enjoyed ASTEROID, etc.People were free to give any feedback.This information was collected in Session 3 only.

Outcomes of PPI.
Observations of some children indicated a lack of understanding of the gameplay.
For example, for a 2-year old boy it was noted that he "only played a few trials and showed no understanding of what to do".A similar observation was made for another 2-year old girl: "Happy to play but didn't understand", and for a 1.5 year old girl: "Didn't understand, mum modelled a lot".In Results.To solve the first problem, we replaced the colour cue by a frame (Figure 4).The second problem was solved by adding a card shuffle animation.At the end of the trial, the stereograms would flip around, move to the centre of the screen, mix up, move out again to the four corners of the display and flip back face up (see video clip at https://youtu.be/w8q-4uejwdk ).The animations mimicked playing cards being shuffled and dealt out.We believed this would explain to young children that in the next trial the target could appear in any of the four locations.Procedures.Ethical procedures, data collection procedures and outcome measures were the same as in Stage 1.
Impact of PPI.4% of the children showed a poor understanding of the ASTEROID task (defined as less than 80% correct on the practice trials) after we implemented the shuffle animation and the frame cue in the practice trials, compared to 39% before the changes were made.The proportion of correct practice trials significantly increased from an average of 0.76 in the pre-measure to 0.97 in the post-measure (Welch two-sample t-test: t(231)=14.29,p<.001).However, we observed a similar average number of verbal instructions that needed to be given (mean in both pre and post-measures = 10; Welch two-sample t-test: t(77)=0.81,p =.42).Level and nature of involvement.The public was involved through eight PPI sessions (Table 2).All people involved were given the opportunity to try out ASTEROID (version 0.938-0.94) in drop-in sessions similar to those described in Study 1. Children under 12 were asked for individual feedback and engagement ratings.Sessions with adults were set up to obtain verbal reports on understanding the ASTEROID stereotest.The outcome measures were the same as in Study 1.

Study 2: Instructions
Outcomes of PPI.Adults described the target as 'a square', 'centre that sort of sticks out', 'popping out and behind'.The outline of the square seems to become more difficult to see at lower disparities when adults described it as 'a circle appears', 'something looks off', ' I can see 3D without square' and even 'going with the gut'.We used these variety of ways to describe the 3D target to the children but observed that children find it difficult to understand any verbal description of a 3D target.However, once they have seen the target in 3D for the first time, they understand what they need to look out for in the subsequent trials.Results.We decided to provide the children with a visual and tactile aid to explain them what our 3D target looks like.We therefore made a cardboard demo of an ASTEROID trial.The demo show a print screen of a trial as the background.In the bottom right location, a square with the same dot pattern as the background was glued on top of the background with a 2mm cardboard layer in between (Figure 5).Impact of PPI.6% of the participants showed a poor understanding of the ASTEROID task (defined as less than 80% correct on the practice trials) after we implemented the cardboard instructions compared to 4% without the cardboard instructions.There was no significant increase in average proportion of correct practice trials between pre-and post-measure (mean proportion correct in pre-and post-measure = 0.97; Welch two-sample t-test: t(92)=-0.02,p=.98).However, we found a significant decrease in the average number of additional verbal instructions given from 10 to 8

Methods
Five groups of children and adults were involved in evaluating engagement (Table 3).The groups are described in detail above.Group 4 was a subset of the children and adults involved in Stage 2 of Study 2 (session 4-5).Different outcome measures were collected from different groups (see Table 3).We monitored the number of encouragements given by the researchers during game play, such as 'Keep going', 'Well done', 'Just have a go', etc.In other groups, children rated their level of enjoyment on a smiley face rating scale with five levels (Figure 6).Their ratings were converted to scores between 1 (saddest face) and 5 (happiest face).

Results
ASTEROID was rated as a fun game by the children with an average rating of 4.8 in Study 1 and 4.7 in Study 2. The implementation of the frame and the shuffle animation did not change the rating (Welch two-sample t-test: t(68) = 0.91, p = 0.36).Although engagement was not the primary aim of the PPI sessions, occasionally positive engagement with ASTEROID was spontaneously noted by the research assistants, for instance 'Really enjoyed it, laughed at triangle/prize coming up', 'Happy to play', 'Liked simple [game] but preferred chicken story better'.We found a significant increase of average number of encouragements from 9.6 to 13.8 after we implemented the shuffle animation and the frame cue (Welch two-sample t-test: t(64)=8.25,p<.001).This indicated that children needed more encouragement and ASTEROID was less engaging after Study 1, possibly because of the additional time occupied by the shuffle animation.However, the average number of encouragements significantly reduced to 8.0 after we added the cardboard demo (Welch two sample t-test: t(103)=8.32,p<.001).

Discussion
Our aim was to improve children's understanding of the instructions of a new stereotest that measures 3D vision, ASTEROID.Through two PPI studies, we consulted 233 children and adults, an unprecedented number in studies with children.The insights based on their feedback were extremely valuable for the development team and we made three major changes in subsequent versions of ASTEROID: (1) we replaced the colour cue by a frame cue; (2) we added a shuffle animation to explain that the target could appear in any location on the next trial; (3) we made a visual and tactile cardboard demo to explain to the children what target they were looking for in ASTEROID.
The involvement of the public and the changes we made to ASTEROID following their feedback had major impact on the level of understanding of ASTEROID.We noticed a substantial increase in the average proportion of practice trials answered correctly after implementing the frame cue and the shuffle animation.In addition, the percentage of children that showed a poor understanding of ASTEROID (< 80% correct on practice trials) reduced.The impact on the number of additional verbal instructions given to the child during ASTEROID is less straightforward.We observed a slight increase after making the first set of changes, but this decreased again after we included the cardboard demo.This seems to indicate that the cardboard demo was able to replace some of the verbal instructions, while the frame cue and shuffle animation were not.Given the nature of the problem that these changes were trying to solve and the type of verbal instructions given, this is not that surprising.Most verbal instructions were variations on what target to look out for: 'Tap the one that is different', 'Which one is sticking out', 'Do any look like they are popping out'?The cardboard demo was included because children had difficulties understanding any verbal descriptions of the 3D target.So, including a visual and tactile aid removed the need for additional verbal instructions.On the other hand, the frame cue and the shuffle animation were solving rather different problems, not directly related to the verbal instructions given.
 The drop-in nature of our sessions required very little commitment of the children and the parents, which lowered the barrier for involvement.
 Besides ASTEROID we brought a range of activities and games related to vision and the brain to engage the public, parents and siblings.This certainly helped to attract and engage people.
 We set up our stall/table with activities in a central place in the venue with high volume of passage rather than opting for a quiet separate room as is usually preferred for research in public places.
 The experience of our team in working with children -both clinically and in schools -was extremely valuable.They were able to make children feel at ease, bring over their enthusiasm and be sensitive to a child's individual needs.
In conclusion, children's contributions have measurably impacted on the development of ASTEROID, a new stereotest for children.By increasing accessibility and through creative methods we gathered feedback from 165 children between 1 and 12 years old and 68 older children and adults.The changes implemented following their feedback significantly improved our stereotest.Our approach can be inspirational for future researchers and contribute to an evidence pool of good-practice in involvement of children and young people in research.

Figure 1 .
Figure 1.Schematic representation of how Patient and Public Involvement was embedded in the development of ASTEROID.
changes are tested in postmeasure appears randomly in one of these stereograms.The child is verbally instructed to look at each of the four "squares" individually to see which one sticks out.The viewing distance is monitored by the front camera.To ensure that the child understands the task, ASTEROID starts with non-stereo practice trials.In these trials, a colour is added on top of the stereo cue, to make the target clearly visible.This cue is also visible even to colourblind individuals, due to the change in luminance.If the child solves the practice trials correctly, the colour cue gradually fades away over the next few trials and will eventually disappear, leaving only the stereo cue.There are a minimum of 25 stereo trialsthe number of non-stereo practice trials increases with the number of mistakes made in the practice trials.Technical details of ASTEROID are described elsewhere14 .

Figure 2 :
Figure 2: Screenshot of a practice trial of ASTEROID (version 0.933).The tests show four dynamic random-dot stereograms.One of the four stereograms has a square with a different disparity and appears to float above the display.In the practice trials this square also has different coloured dots.The colour cue is removed in the disparity-only trials.

Stage 2 :
PPI sessions People involved.111 children between 2 and 12 years old (median age = 4, IQR = 3, 53 girls, 55 boys, 3 gender not recorded) were involved in the PPI sessions.They attended local science and history museums in Newcastle-upon-Tyne (United Kingdom) between October 2016 and January 2017 and Crucially, children were invited to try out ASTEROID (version 0.932 and 0.933) and comment on any aspect of the design.Opinions were gathered by research assistants: CB, TC, JH, and SR.CB and SR had experience in engaging children for vision testing in their previous roles as an optometrist and orthoptist respectively.TC has a postgraduate degree in Health Psychology and had worked with in a primary school as a Teacher Training Programme Coordinator.JH worked as a primary school teacher and completed an MSc in Psychology before joining the research team.

Figure 3 .
Figure 3.Our set-up for the drop-in PPI activities at Discovery museum (A) and Centre for Life (B) many children we observed hesitation on how to proceed at the end of the non-stereo practice trials, indicating a gap in understanding on what to do during the non-stereo and stereo stages of the game.In addition, comments made during sessions indicated that the colour cue in combination with the colourful dots in the non-stereo trials primed children to look for a local colour change in a few dots instead of an overall depth change in the stereo trials.Second, some children had difficulties understanding that on each trial the locations of the target was randomly determined.In subsequent trials, these children would tap all four stereograms in alternation.For instance, if they incorrectly tapped the top left stereogram in one trial, in the next trial they would not revisit this location but instead try for instance the top right location.If the target was not found there, they would move to yet another position in the third trial.This seemed to indicate that children did not understand that the placement was random, but imagined that the target location remained the same until it was found.Stage 3: Implementing changes Methods.Feedback from Stage 2 was summarised by research assistants and discussed at a crossdisciplinary meeting attended by vision scientists, computer scientists and game developers.Once consensus was reached, the proposed changes were implemented in a new version of ASTEROID.

Figure 4 .
Figure 4. Screenshots (version 0.937) of the changes made after the first Public and Patient Involvement round: a frame cue (top) and a shuffle animation (bottom, see also video at https://youtu.be/w8q-4uejwdk ).

Stage 1 :
Pre-measureThe post-measure data of Study 1 served as the pre-measure data of Study 2.Stage 2: PPI sessionsPeople involved.122 children and adults were involved in the PPI sessions of Study 2. Children were between 1 and 12 years old (median age = 4, IQR = 4).37 of them were girls and 24 of them were boys.Age and gender was not recorded for the adults.The children attended local science and history museums in Newcastle-upon-Tyne (United Kingdom) in February 2017 and were invited to join a PPI session for 5-10 minutes.Adults were either recruited via our research volunteer pool and registered to attend a session at Newcastle University or they attended a drop-in session at a public venue (Newcastle University Medical School foyer or local museum).Children were thanked for their feedback with a 'Junior Scientist' certificate and a sticker.

Stage 3 :
Implementing changesMethods.Just as in Study 1, feedback from Stage 2 was discussed at a cross-disciplinary meeting and consensus changes were implemented in a new version of ASTEROID.

Figure 5 .
Figure 5. Cardboard demo added after the second Public and Patient Involvement round.

Figure 6 .
Figure 6.Rating scale for engagement with ASTEROID.The scale was accompanied by the question: Did you enjoy playing the Asteroid game?

Table 1 .
Overview of PPI sessions for Study 1 2 Local museum with free entrance;2Local museum with entrance fee; N = number of people involved

Table 2 .
Overview of PPI sessions for Study 2 2Local museum with free entrance;2Local museum with entrance fee; N = number of people involved Outcome measures.

(
Welch two-sample t-test: t(103)=3.72,p<.001).An age difference between the pre-and postmeasure might possibly confound this effect, we therefore rerun our analyses including only subjects younger than 9 years old (n=22, median age = 6.3,IQR = 3.4, 8 boys and 14 girls).This indeed resulted in an insignificant difference in the number of additional verbal instructions given (Welch

Table 3
Groups of people involved in evaluating engagement with ASTEROID