Wednesday, 31 July 2013

NHS Friends and Family Test

The NHS England's 'Board to Ward' patient feedback system with public transparency has gone public.

You can see the results for every Trust down to ward level on the Gov.UK website.

The calculation of the Friends and Family Test and guidance on how the results should be shared with staff and the public is also available. When trying to calculate the scores remember that the 'likely' response is included in the denominator but the 'don't know' is not.

The calculation is based on a net promoter mechanism (a customer satisfaction measure) popular in business but not without its criticism including for variability even in the industries where it is sometimes claimed to be superior to more robust survey methods.
"Managers have widely embraced and adopted the Net Promoter metric, which noted loyalty consultant Frederick Reichheld advocates as the single most reliable indicator of firm growth compared with other loyalty metrics, such as customer satisfaction and retention. ... the research fails to replicate his assertions regarding the 'clear superiority' of Net Promoter"
Having said that, it is quite a simple tool and arguably more practical for patients and relatives to complete on NHS wards. More importantly, it is what we currently have and it is here to stay. It is among the mandatory central information for Monitor. It was announced by Prime Minister David Cameron last year to identify the 'best performing hospitals'. If it has a use it is possibly better for detecting change and stress within a clinical area rather than comparing between clinical or geographical areas. A short stay on intensive care where your life is saved is very different to a long stay on a stroke rehabilitation ward where you are being urged to be as physically independent as you can when you are not used to being told what to do.

What has been the reaction in the first month of national publication of the figures?

99 per cent are happy with NHS hospital care (in a survey where only 10 per cent actually responded). Usefulness of data from post-Stafford research already called into question. Independent

Family and Friends test 'at best meaningless'. The Government's family and friends test for NHS hospitals has been condemned as "at best meaningless, at worst misleading" as critics warned the system is "open to gaming".  The Telegraph

Are you listening, Andy Burnham? NHS patients have been given a voice. The Friends and Family test improves transparency in the health service – and it's patients who will benefit the most. The Telegraph

I fear the gaming. Managers in NHS Trusts will put their own PR polish on their national results and not simply refer patients (and staff) to the transparent figures. They may add another layer of interpretation to the scores where none may exist. They may also try to shift the scores without making structural changes to the clinical areas themselves.

For example, a Trust may trumpet scores of 100 when the respondent numbers are very low, but then criticise a ward with scores nearer 50 when the respondent numbers are equally low. Low respondent numbers mean uncertain figures. Scores that conveniently support a particular Trust Board narrative may be selected for action in preference to others. Wards with low scores this month may have circulars to 'explain' the survey to patients and relatives so that 'the score can better reflect the true performance of the ward'.

Talk of ward scores should be responsibly done. I've been calculating the variance and margin of error on the ward scores using a discrete random variable approach. It gives an average margin of error of +/- 11.4 (range 1.1 to 43.8) for the Friends and Family score (based on the 68% of wards where variance can be estimated - the others are meaningless as numbers too small). [I've put the modified spreadsheet here ... column AA in the 'IP ward' sheet has the margin of error estimations where they are valid.]

Thursday, 25 July 2013

Learning analytics - need to do more

I've been thinking about learning analytics again lately and how to do more of it in the projects I'm involved in. has always been powered by analysis of activity - sequencing test items based on prior performance. However, not all learners are so fully engaged as they are when they are running through mock questions for a high-stakes exam. Although I had a great time helping to design the learning analytics for exam preparation, most learning is not so intensely focussed. It is more work-based, opportunistic and social.

In clinical medicine learning analytics would most benefit from objective measures of personal performance to help make recommendations and present the data. These outcomes are hard to capture and, even if you do, they are hard to analyse since there is so much variability. The easy things to measure aren't necessarily the most important to observe.

Using electronic health records would be good. Case mix, common diagnoses, common prescriptions, common investigations and findings, would be an ideal way to automatically design a syllabus. Heuristics could be defined to spot quality issues. But how to get in on that gig? Could work more with BMJ Informatica to link individual GP performance to bespoke learning I suppose. My background is secondary care however, and last time I tried asking to look at and explore solutions for individual physician performance at my local NHS Trust I hit a dead end.

I've worked with the background design of a learning tool which represents knowledge. Looking at patterns of errors with the quiz tools that can be created from this may help identify areas for focus for individuals, and areas for novices or experts to start. Have a look at this example tblable on MODY. I think this is too niche at the moment though. Too narrow a cognitive tool. I've got over a number of hurdles but it is a solution without a defined problem.

I also analysed millions of Tweets using tools in the GrabChat idea via the different Twitter APIs but not managed to glean anything particularly inspiring. There's a lot of guff about sentiment analysis of tweets (e.g. how it bombed in stock picking) but it does not transfer from anything other than the binary of emotionally positive or negative in particular communities and topics. Analysing tweets is great for finding interesting links and people but it still needs human filtering and a lot of spam gets in there. Nowhere near a tool that would be useful for learning analytics.

So, in all, I'm feeling a bit of a frustrated skunk worker. Will have to experiment some more.

Tatoo biosensor for lactate to enable extreme sporting performance

Obviously you have to put years of training in and be a pretty unique (and sporty) sort of individual to benefit from such a biosensor tatoo. This sensor detects lactate levels which climb with increasing intensity of exercise until the athlete 'hits the wall', or 'bonks'. Having been there and done that myself all judgement and rational thought have long disappeared before the moment occurs. (Also I'm not that sporty either).

So this biosensor could be the solution!

"The sensor can be applied to the human skin like a temporary tattoo that stays on and flexes with body movements.

In ACS' journal Analytical Chemistry, Joseph Wang and colleagues describe the first human tests of the sensor, which also could help soldiers and others who engage in intense exercise — and their trainers — monitor stamina and fitness." Source: Tatoo biosensor warns when athletes are about to 'hit the wall' (Kurzweil AI)
I worked for a while in Waikato Hospital and remember at the City Gym, in Hamilton, New Zealand, there were some crazy rowers who every now and then brought a portable lactate measuring machine into their training sessions. They would plot their heart rate against their lactate levels with a progressive exercise programme that took them basically to oblivion. They then knew what heart rate level and exercise they could maintain without hitting the wall. They used this for extreme performance in rowing and iron man competitions.

This method of anaerobic threshold training dates from work in the 1980s by Conconi and others [1] and it has enabled a generation of athletes to push the boundaries of their sport.

1. Conconi F, Ferrari M, Ziglio PG, Droghetti P, Codeca L. Determination of the anaerobic threshold by a noninvasive field test in runners. J Appl Physiol 1982 Apr;52(4):869–873. Available from:

Mortality rates, averages and the media

Now that it seems a 'mystery rise' of 600 people a week are dying more than average in the UK

there is a great opportunity for the media to start talking nonsense about things like excess deaths, older people more likely to die, why do we always seem to be above average half of the time, and why being above average is something to worry about. Maybe we'll get some daft MPs diving in too.

Hopefully, some sensible analysis will come out. It looks like Public Health England is considering whether it is a spike in respiratory illnesses earlier in the year. Of course it could just represent the nature of the real world where observed numbers tend to differ from those that were expected.

Friday, 19 July 2013

5 Ways the NHS has changed this week

Interesting piece from Nick Triggle BBC's Health Correspondent.

The gloves are off with the politics of the NHS, the care of the elderly matter, the new inspector of hospitals has a tough job, death rates and special measures are here to stay.

The NHS is certainly becoming a key election issue. Facts such as decreasing mortality in all NHS hospitals over the past 10 years also being conveniently not mentioned by some.