Identifying variation in GP referral rates: an observational study of outpatient headache referrals.

Objective: This study provides a framework methodology for identifying GP surgeries with unexpected rates of referral to specialist services, using headache referrals to outpatient neurology as an example. Design: This is a retrospective observational study using routinely collected and open-source data. Setting and participants: Data was collected from a single consultant outpatient neurology clinic and 202 GP surgeries across seven CCGs in the Northwest of England. Primary Outcome: The number of headache referrals from each GP surgery during the study period of three and a quarter years was used as the primary outcome in a generalised linear model. The standardised residuals from this model were then used to identify GP surgeries that were likely to have referred unexpected patient numbers for headaches to an outpatient neurology clinic during the study period. Results: In the model using data from the CCG in which the outpatient neurology clinic is located we identified one GP surgery referring more headache patients than expected. The model showed that the clearest predictor of headache referrals was the number of referrals for other types of neurological disorders. In the model using data from all seven CCGs we identified four GP surgeries with unexpected numbers of referrals. This model showed that there were two predictors of headache referral, namely other neurology referrals and the distance from the clinic. Conclusion: We have developed a flexible methodology for identifying GP surgeries with unexpected numbers of referrals to specialist services. This methodology was demonstrated using headache referrals but could be adapted to any type of referral or geographical area.


Referral
General Practitioners (GPs) provide a number of key services, including referral to specialist treatment when needed. Referrals can be made for a number of reasons including for investigation, diagnosis, management or reassurance. [1] In the UK referral rates from GPs vary for a number of complex reasons such as resource availability, population health needs, patient pressure, and lack of consensus on which conditions benefit most from specialist input. [1,2] Understanding variability in referral rates from GPs to specialist services has been an area of interest in health research for many years and has been investigated using differing statistical approaches. Two Canadian studies demonstrate the utility of regression models in the investigation of referrals from primary care to specialist services. Ryan et al used multi-level negative binomial regression on primary care data to describe referrals to all medical specialities.
[3] They found that both patient and GP surgery characteristics influenced referral patterns. Chan et al also found that patient and GP level variables had an impact on referral to specialist care using a multi-level modelling approach, including geographic characteristics such as socioeconomic status. [4] Other work has focused on investigating interventions that may reduce unnecessary variations in referral and has come to contradictory conclusions. In 1993 Fertig et al concluded that 'inappropriate referrals' were not the cause of variation, and that guidelines would therefore be unlikely to reduce referral numbers. [5] However, since then numerous studies have been conducted, and in 2006 Akbari et al undertook a systematic review of interventions designed to 'change outpatient referral rates or improve outpatient referral appropriateness'. [6] This study found that effective interventions included targeted dissemination of guidelines and involvement of consultants in educational activities for GPs.
Within the National Health Service (NHS) there has been an effort in recent years to identify potential areas for improvement, both in terms of patient outcomes and reducing costs. Two recent initiatives include RightCare and Getting It Right First Time (GIRFT). The aim of GIRFT is to 'improve medical care within the NHS by reducing unwarranted variations'. [7] The GIRFT Neurology project divides England into 'neuroscience regions' for analysis and also relies on visits to NHS Trusts to 'deep dive' into local issues. [8,9] The NHS RightCare initiative seeks to help CCGs 'Diagnose the issues and identify the opportunities with data, evidence and intelligence; develop solutions, guidance and innovation; and deliver improvements for patients, populations and systems'.
[10] However, the RightCare methodology has not been without criticism, including the way in which similar CCGs are identified being flawed, and that the way in which CCGs are compared resulting in overestimation of differences. [11] The methodology we develop in this paper for identifying . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint GP surgeries of interest is not intended as a replacement for either RightCare or GIRFT methodology, but offers an alternative approach.

Headache Referral
In this study we focus on the specific issue of referrals from primary care to neurology outpatient care for headaches (including migraine).
Headache is a common and disabling condition, with migraine representing the second largest contribution to global disability of all neurological conditions. [12] Headache, including migraine, accounts for a large proportion of consultant neurologist appointments in the UK. [13][14][15][16][17] It has been reported that GPs refer up to 2 to 3% of the headache patients they see in primary care, [18] and that GPs experience pressure from patients to refer to specialist care. [19] This is despite the fact that headache conditions are often best managed at primary care level. [20] This puts pressure on already stretched outpatient neurology services. [21] GPs are the gatekeepers for referral to specialist care, including neurological outpatient care. Brilla et al (2008) found that interventions made at the neurology service level for reducing 'inappropriate referrals', such as email triage, are not effective. [22] This study concluded instead that interventions should be made at the point of referral by enhancing guidance for referral decisions for GPs. In the same year Davies et al also emphasised the benefit of interventions at primary care level, recommending improved education for GPs to help reduce the burden of headache. [23] Most recently Huang et al found that an online headache referral guideline for GPs was successful in reducing the number of referrals to neurology services. [24] This study also raised the important point that their positive results may be due to both the accessibility of the guideline and the awareness of GPs to its existence.
If interventions such as structured guidelines and education are best applied at the point of referral, then it follows that identifying the GP surgeries where these interventions may be most effective would be of interest. In addition, identifying GP surgeries where there are lower referral rates may offer further insights into the spectrum of variation in referral patterns, as under-referral can also potentially signal the need for intervention to improve care.

Specific Objectives
This study provides a framework for identifying GP surgeries with unexpected rates of referral. Although the specific case study used is that of headache referrals, the aim of this study is to provide a broad methodology which is flexible and can be applied to any type of referral both within and outside of the neurology specialty.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint

Study Design
This study used routinely collected data from outpatient appointments alongside open access data in a retrospective observational study. We recorded the number of patients referred for headache to, and offered an appointment in, a single consultant-delivered neurology clinic over a period of three years and four months (18 th September 2015 to 9 th January 2019).
The study received relevant approvals, including by the NHS Research Ethics Committee

Data Sources
Data regarding the number of referrals from GPs within the catchment CCGs were taken from records at a neurology outpatient clinic at the Royal Preston Hospital (RPH), which is part of the Lancashire Teaching Hospitals NHS Foundation Trust (LTHTR). As the clinic is dedicated to adult care no paediatric referrals were included. The data covers all GPs within 7 Clinical Commissioning Groups (CCGs): Greater Preston; Chorley and South Ribble; East Lancashire; Fylde and Wyre; Blackpool; Blackburn with Darwen; and Lancashire North. There were a small number of referrals for headache from outside this catchment area, but their small number made them unsuitable for inclusion in the analysis (6 referrals from Cumbria CCG, and 1 referral each from West Lancashire and Wigan Borough CCGs).
Data regarding GP surgery characteristics was downloaded from NHS Digital open access repositories https://digital.nhs.uk/services/organisation-data-service/data-downloads/gp-andgp-practice-related-data This study did not use the latest data available as, although the data is updated regularly, we felt it more appropriate to use data from the start of our study period (October 2015). This allowed us to capture information for GP surgeries that have subsequently been closed or amalgamated with other locations.

GP surgery variables
The outcome of interest was the number of headache referrals from each GP surgery over the study period. Explanatory variables were chosen for both their relevance and their availability. These variables included GP surgery list size (adults over 14 years of age); proportion of the surgery list that are male; mean age of the surgery list; number of other . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint neurology referrals made; distance of the surgery from the clinic at Royal Preston Hospital (RPH); weighted Index of Multiple Deprivation (IMD); and the standard deviation of the weighted IMD.

Weighted IMD calculation
The Index of Multiple Deprivation (IMD) is a measure often used as an indicator of the deprivation of an area. It is collected at the census unit of the Lower Super Output Area (LSOA) level. As GP catchments overlap fragments of many LSOAs, to be able to explore relative deprivation levels of GP surgeries we calculated a weighted score for each surgery. We followed the methodology devised by Zheng et al. [25] We combined data on the number of patients from each LSOA on each GP's list (available online from NHS Digital https://digital.nhs.uk/data-and-information/publications/statistical/patients-registered-at-a-gppractice/october-2015) with IMD data available from the gov.uk website (https://www.gov.uk/government/statistics/english-indices-of-deprivation-2015), creating a weighted index for each GP surgery. We also created a variable for the standard deviation of those weighted indices as a measure of the variability of the IMDs contributing to each GP surgery.

Data Preparation
The data were analysed using R Studio version 1.2.5019 . The separate datasets were joined using GP surgery codes. After joining, weighted IMD, standard deviation of the weighted IMD, and the straight line distance of each surgery from the clinic at RPH were calculated using the Ordnance Survey coordinate system.

Stage One: Modelling
The primary role of stage one was to identify which variables drive variation in referral numbers and to calculate expected referral rates from the GP surgeries included in the study. To account for the effect of different variables on headache referrals we used poisson loglinear models which are part of the class of models known as generalised linear models (GLMs). We used GLMs as they allowed us to include, and quantify, the effect of a number of different variables on the outcome of interest.
We chose to look at two separate models in stage one, as the effect on the number of headache referrals is unlikely to be uniform across the area of study. We conducted two analyses, one across the entire study area, and one concentrating on the area closest to the outpatient clinic. GPs and patients have an element of choice of outpatient clinic to which to refer/visit, and one of the factors influencing this choice is likely to be distance from the clinic. In general, the farther the GP surgery is from the clinic at RPH, the more likely a patient is to choose an appointment elsewhere. In addition, the influence of distance from the . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint clinic is likely to differ across the area and between CCGs as other hospitals offering neurology services are not uniformly distributed across the area.
Model A included data only from GPs in Greater Preston CCG, the CCG in which the outpatient neurology clinic is located. This represents the area in the study least likely to suffer from the influence of being referred to a clinic outside the study area, and therefore provides the most robust data.
Model B included data from GPs in the 7 CCGs which make up the larger catchment area for the clinic. Towards the outer edges of this area it is likely that larger proportions of patients are referred elsewhere for neurology outpatient care. We account for this by including both distance from the clinic at RPH and the number of other types of neurology referral made by each GP surgery.
The two models are separate and cannot be directly compared as the variable selection process resulted in different specifications for each of the models. For an explanation of the selection process used when deciding the models and included variables please see supplementary materials.
Stage Two: Examining the differences The primary purpose of stage two, and the overall purpose of this study, was to determine if it is possible to identify surgeries which refer fewer or more patients that expected. To achieve this, we extracted predicted values from each of the two models described above and compared them to the actual values observed during the study period. This gave us the differences between observed and expected referrals which can be visualised to determine if surgeries are referring as expected. We also examined potential spatial autocorrelation between the predicted and observed number of referrals using Moran's I. This index is similar in concept to a correlation coefficient and gives a value between -1 and 1. However, -1 indicates perfect clustering of dissimilar values, and 1 perfect clustering of similar values. A Moran's I of 0 indicates perfect randomness.

RESULTS
We analysed data from 202 GP surgeries across 7 CCGs (see Figure 1), which provided 388 headache referrals in total over the period of the study. Over the same period these surgeries referred 1371 patients for other suspected neurological disorders, thus headache accounts for 19% of all neurology referrals over this study period. The largest number of surgeries is in East Lancashire CCG, but the largest number of both headache and other neurology referrals was from the Greater Preston CCG. This may stem from the fact that GPs can refer patients to other neurology clinics if the patient prefers, and the farther away a GP surgery is from the outpatient clinic, the more likely they are to refer elsewhere. The characteristics of the GP surgeries in each CCG can be found in Table 1. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.   Within the CCGs GP surgeries referred differing numbers of patients for both headaches, and other types of neurological conditions. Figure 2 shows the relationship between the number of headache patients a GP surgery referred and the number of other neurology referrals. This relationship is shown separately for each CCG and demonstrates that there is a consistently positive relationship between headache and other neurology referrals, although this relationship appears to differ between CCGs.

Results from Model A -Greater Preston CCG Only
Model output . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint Supplementary Table 1. The model output shows that within the Grater Preston CCG the main driver of headache referrals was the number of other neurology referrals. This variable was highly correlated with list size, which was removed from the model, so it is likely that a part of the reason for the significance of the number of neurology referrals is due to the size of the surgery. This tells us that if a surgery refers many patients for any suspected neurological condition then it is likely that they also refer many patients for headache.

Examining the differences
Extracting standardised residuals from the model allows us to visualise whether some surgeries are referring unusual number of patients (Figure 3). This figure shows the standardised differences between observed and expected referrals for each GP surgery within the Greater Preston CCG. Values below -3 or above +3 can be considered statistically significant. Given the variables accounted for in the model, only one surgery referred a statistically significant unexpected number of patients for headache. The standardised difference between expected and observed numbers of referrals was tested for spatial correlation. Using Moran's I we found there to be no spatial autocorrelation for the GPs in Greater Preston CCG (Moran's I = -0.093, p = 0.179).

Model Output
Results from model B can be seen in Supplementary Table 2. Similarly to model A we see variation in referral is related to the number of other neurology referrals, but we also see that in model B the distance of a GP surgery from the clinic also has an impact on headache referrals. The results suggest that the further a surgery is from the clinic the fewer headache referrals are made.

Examining the differences
The standardised difference between the number of expected and observed referrals for each surgery within the seven CCGs is shown graphically in Figure 4. This figure shows a boxplot of the distribution of the overall differences, and a dot for each GP surgery colour coded by CCG. Values below -3 or above +3 can be considered statistically significant.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint We can see that when considering all the CCGs together in this model, there were no surgeries which referred fewer headache patients than expected, and 4 surgeries that referred statistically significantly more than expected, given the variables that were accounted for in the model. The distribution of differences is right skewed, meaning the median is less than 0, with a long tail to the right. A plot comparing the raw (non-standardised) values of observed and predicted referrals can be found in supplementary materials Figure 1.
Testing for spatial correlation in the differences between expected and observed referrals using Moran's I, we found there to be no spatial autocorrelation for the GPs across the 7 included CCGs (Moran's I = -0.013, p = 0.605).

Principal findings
This study shows that it is possible to identify GP surgeries which refer unexpected numbers of patients to an outpatient clinic. This is achieved by identifying a set of explanatory variables to include in a Generalised Linear Model, the results of which are then used to give predicted values for comparison against observed referral numbers. Once surgeries with unusual numbers of referrals are identified, further investigation can then be carried out to understand the circumstances leading to the unexpected referral numbers. This would allow support to be targeted to the places that need it, and lessons to be learnt, which could be shared across the CCGs.

Strengths and limitations
The basic methodology outlined in this paper could be modified and extended to other specialties. Although other specialties have their own drivers for referral patterns, they could be examined using the same methodology by adjusting the explanatory variables included in the initial GLM. The methodology could also be extended to cover larger geographical areas by including other referral centres.
As with all studies which include statistical modelling, if an informative variable has been excluded, either through unavailability of data or through not understanding the drivers of referral, then the results of the second stage of the process -identification of the unexpected referral rates -would be less accurate. This highlights the need for researchers to understand the drivers behind the type of referral under investigation, and to be able to access rigorous and valid data on which to build models. If understanding is limited, or if data is unavailable (or inconsistent) any assumptions drawn from modelling will be flawed.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) 1 1 The way that data is collected from GP surgeries can change over time. Surgeries can be closed, and new surgeries can be created from both amalgamation and splitting of previous surgeries. In this study we extracted data from the NHS Digital open access repositories for the dates at the start of the study as this allowed us to capture information for GP surgeries that have subsequently been closed or amalgamated. In addition, some GP surgeries are small single locations, whereas others comprise of a large hub surgery and several smaller branch surgeries affiliated to them which are included in their data returns. In this study we did not split branch surgeries from their parent location, as although this might allow for more detailed analysis, branches which come under a single surgery grouping are likely to have more in common than discernible differences. For example, surgeries which come under the same parent surgery are likely to share the same educational training and use the same referral guidelines.
This study has a relatively small sample size, in particular there are few surgeries which refer large numbers of headache patients. Expanding the study to a larger geographical area, including other clinics, or including a longer study period would help to alleviate these limitations. However, this expansion would be reliant on the availability of coded outpatient neurology diagnoses.
The methodology used in this study can be adapted to any type of referral, geographical location, and timescale by adjusting the explanatory variables used in the initial model. Although the results of the model used as a case study in this paper are not generalisable to other geographical locations or timescales, the methodology is generalisable. It would be possible to expand this analysis of headache referrals to a national level, but this would rely on the availability of consistent coding of neurology outpatient appointments.

Relation to previous studies
Previous studies have used statistical modelling to investigate the impact of different variables on referral rates from GPs to specialist services. [3,4] We have based the first stage of our study on this modelling process, and then extended the analysis to include the identification of GP surgeries which are referring unexpected numbers of patients for headache.
The NHS RightCare methodology seeks to identify areas of opportunity for improvement for CCGs, and although this study does not seek to replicate or replace the RightCare methodology, the intention behind it is similar -to allow CCGs to identify GP surgeries where interventions may be of use. However our methodology differs in a number of ways from RightCare which allows it to avoid the difficulties highlighted by Dropkin. [11] RightCare seeks to compare CCGs across large dislocated geographical distances, whereas we limit the study to a single contiguous geographical area, meaning that the CCGs are more likely to be similar in unmeasured ways. In addition the RightCare methodology has a fixed set of demographic variables against which the CCGs are measured regardless of the outcome . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint of interest, whereas we recommend that the explanatory variables used in the initial model are changed depending upon the outcome under investigation. It cannot be assumed that the explanatory variables used for headache referral would be relevant for orthopaedic referral, for example.

Meaning of the study
In a previous study we found that the majority of patients attending a neurology outpatient clinic had only one appointment. [16] Many of these appointments were for headache, with patients often discharged after only one appointment without investigation. Whilst we recognise that even a single consultation with a neurologist can make an important contribution to the patient's management, much of the advice given, particularly in relation to lifestyle factors and avoidance of medication overuse headache, could be delivered in primary care. Demand is rising and capacity, including in general practice, is of course limited. But it is recognised that consultant neurologist appointments are an especially limited resource, with an average of only one neurologist per 79,000 patients, and as little as one per 167,000 patients in some regions, [9] and we need to maximise access for those who need it most. Identifying which surgeries could potentially reduce referrals through alternative routes to treatment would be both more convenient for the patient and free up resources for other patients needing to access neurology consultant care.
It is recommended that the results of this study are interpreted with caution, as it provides only an indication of GP surgeries from which there may be unexpected referrals, it does not explain why those unexpected referrals may have occurred. However, identifying variation is the first step towards understanding it, and this methodology could be used by CCGs or outpatient clinics to understand where their patients are coming from, and to plan further targeted investigations.

Unanswered questions and future research
More research needs to be done to validate this methodology, and to extend it into other areas of referral. It could also be extended and refined to apply to larger geographies, or to other outpatient specialties. Expanding this research to larger geographical areas would require consistent coding of diagnoses resulting from outpatient neurology appointments, which is unfortunately not yet available. Further research is also needed to confirm the utility of conducting these types of analysis, in particular whether identifying unexpected referral rates leads to implementation of policies that improve patient care.

CONCLUSION
Identifying GP surgeries with unexpected numbers of referrals is a useful first step towards understanding the larger issue of variability in referral rates. Once identified, those GP surgeries with unexpected numbers of referral can be investigated further to help understand . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted February 8, 2022. ; https://doi.org/10.1101/2022.02.07.22270572 doi: medRxiv preprint why their referral rates differ from those expected, and if necessary, interventions can be targeted to where they are most needed. Using GLMs is an efficient way of including explanatory variables that are relevant to the type of referrals under investigation and variables can be changed to directly relate to any type of referral requiring investigation. This ensures that the methodology presented here is flexible enough to be applied to different types of referral or geographical area. Funding: Fran Biggin is funded by an EPSRC doctoral training partnership grant number EP/R513076/1. The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Declarations
Data Sharing statement: Due to patient data confidentiality and restrictions imposed by the HRA we are unable to directly share the data for this study. Anyone wishing to access this data must apply to do so through the HRA by completing an IRAS application. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Standardised difference between observed and predicted referrals