The Orthopaedic Data Evaluation Panel (ODEP) has recently introduced a listing of knee implant products that comply with the benchmarks set by NICE. Keith Tucker, Martin Pickford and Olga Taylor explain how the system has evolved from a procedure originally designed to evaluate hip implants
About four years ago, the demand for the Orthopaedic Data Evaluation Panel (ODEP) to introduce the ODEP process for knees gathered pace. The drive for change came from many of the knee manufacturers, the British Association for Surgery of the Knee (BASK), the British Orthopaedic Association (BOA) and from other sources.
ODEP now goes back 15 or so years. The panel was initiated and implemented by the Purchasing and Supply Agency (PASA) in response to a request by the National Institute for Health and Care Excellence (NICE) as part of their guidance, following the Capital Hip disaster. This was at about the same time as the National Joint Registry (NJR) was set up. As far as ODEP was concerned, the remit was to formulate a method to ensure that manufacturers would look at the performance of their products in a timely and consistent way, to enable us to inform surgeons about the strength of the evidence presented for a particular hip replacement. ODEP was never specifically set up to be a tool for any procurement process, and we have never had any dealings with price, although ODEP does of course recognise that the implants which survive longer will save money by not needing to be revised.
When we were introducing ODEP for knees we were fortunate in already having the experience of ODEP for hips on which to build. Some of us have also had experience working with the NJR, and particularly the Implant Performance Committee where poorly performing implants are identified.
ODEP for hips has been blessed by having some very experienced and highly respected hip surgeons who work with the core team, and for knees we have again been able to recruit some excellent surgeons, this time from BASK. They have all given freely of their time to put the process together, test it and share the responsibility of assessing the submissions. We would particularly like to thank Tim Wilton (President of BOA and past President of BASK), Richard Parkinson (President of BASK), Colin Esler and Andrew Porteous, as well as several members of ODEP.
When we started with ODEP for hips, there was little data for manufacturers to use in support of their submissions for the various benchmarks. We had to use KM (predictive) to get most of the manufacturers ‘over the line’ unless they were able to use data from one of the Scandinavian registries. Nowadays we have data from multiple sources and with the maturity of most of the world’s registries data is easier to access.
ODEP is not organised around the NJR; when we started there was no NJR and only for the past two years or so has there been NJR data available for 10-year ratings. We take data from multiple sources, including peer-reviewed data; RCTs, papers, presentations, etc. In addition to registry data we accept validated in-house data. These days much of the submitted data does come from NJR supplier feedback.
Supplier feedback
It must be said that one of the many outstanding features of the NJR is Supplier Feedback, which is unique and available to all manufacturers. There are some rules as to how Supplier Feedback data can be used and these include submission to ODEP, dealings with MHRA and internal audits and surveillance. Supplier Feedback is not listed as being available to another benchmarking organisation.
There are some other registries that do produce reports for manufacturers such as the Australian and New Zealand registries and we are very grateful to them.
Independence
There have always been some people who have criticised us and we have to accept that we will never please everybody. We strive to be honest and independent and we are not funded by those manufacturers whose data we have to scrutinise.
ODEP is a not-for-profit organisation and all the surgeons give freely of their time. Our office and expenses are funded by NHS Supply Chain. We think that any other system of funding will always call the quality of the ratings to be in doubt, which is one of the reasons we will probably remain independent.
Legal note
It’s important to remember that virtually all the products coming through ODEP are CE marked – the only ones that are not are those where a manufacturer has developed and is marketing a joint replacement in another country outside Europe and wishes to get an ODEP rating based on data from their local sources. This is perfectly in order and we are proud of the fact that the esteem which has been accorded to ODEP stretches far and wide.
One important legal caveat is that it would be illegal, under European law, for ODEP or any other body to stand in the way of a manufacturer trading their implant. We can only collect and present data. That is why we are the Orthopaedic Data Evaluation Panel and not the Device Evaluation Panel.
Common misunderstandings
We hear quite regularly of companies being refused access to a market because they do not have a 10A or a 10A* benchmark and they only have say a 5A or 5A*. This is not what ODEP is about. Eventually the 5A product may well turn out to be better than the 10A* that is currently being preferred by the procurement manager, and part of our philosophy is to support the good and promote innovation. The most important aspect about a less than 10-year benchmark is that the product must be followed-up carefully by the manufacturer so that it passes through the benchmarks as time goes by.
We have put together a table that provides a brief description of ODEP ratings, with examples of total knee systems that have recently achieved each respective benchmark. Further information can be found in the Products section of the ODEP website (odep.org.uk/Products.aspx).
The difference between knees and hips
When ODEP for hips was set up it was decided that cups and stems should be judged separately, which has been a good thing for hips. What was probably not a good thing was the fact that within some product ranges there were products that had not been performing as well as the rest of the brand. This is termed ‘camouflage’.
At present, but hopefully not for long, the NJR is unable to report on the different attributes contained in a brand. This is the case for all other registries to a greater or lesser degree. This does mean that, for hips, some ODEP ratings may not be completely comprehensive for a brand; this particularly applies to new materials in the bearing surface.
It does of course allow us to judge a cup as having 10A rating, with the stem that is being used with it rated as 3A or, for that matter, 3A*. We are of course aware that sometimes surgeons prefer to use components from more than one manufacturer in constructing a hip replacement.
ODEP for knees
Total knee replacement
In total knee replacement (TKR), because femoral components of knee replacements are only used with components within the same brand name, we elected to judge knees as one unit. Having said that, we all know that some knee brands have components of different vintages within the construct. This means that while a femoral component may have 10A* rating, the novel tibial plateau may only have a 3A rating. We realise this can make it seem complicated but we must be scrupulously clear with the benchmarks. It is quite common for a brand to have, for example, a 10A* rating for the fixed version on the tibial side and a 5A rating for the mobile bearing device which came into the market later.
Uni-condylar knee replacement
With uni-condylar knee replacements the situation is much more complex. As most people who will read this article will know, most uni-condylar knee replacements are not going to make a 10A rating with any comfort, and certainly not a 10A* if the same rigour in the benchmarking system is used as in hips.
It is not ODEP’s wish or indeed in its power to ban implants, and most of the ODEP knee panel members would not like to see all unis withdrawn from the market for good orthopaedic reasons.
At present we are trying to work around the following issues:
extension of disease;
poor surgery, with some surgeons only undertaking one or two uni operations per annum;
poor indication; and
patient-reported outcome measures (PROMs).
Perhaps we should unshackle ourselves from using crude revision rates and use patient time incidence rate (PTIR). We could compare the PTIR of various brands against the PTIR of the group. This could be applied in terms of standard deviations. The SDs could be related to the A*, A and B benchmarks. Alternatively, we could reduce the volumes required to meet the benchmark and increase the allowed percentage revision rate. Anyhow, we are working on it and would welcome comments.
The benchmarks
It has been suggested that ODEP does not follow ‘standard practice’ with its benchmarks, and we’d like to take the opportunity to clarify our position here. We developed benchmarks for hips so that patients could be sure that the performance of their hip replacement could be tracked. The original NICE guidelines recommended three- and 10-year benchmarks, and NICE thought that 10 years was long enough.
We felt strongly that implants must be seen to progress through the benchmarks and that manufacturers would want to check on the progress of their product every two years or so. That is why the five- and seven-year benchmarks were added. Nowadays, most hip replacements are lasting more than 10 years and with our population living longer this is very important. Already some manufacturers are asking for a 15-year benchmark. We are working on that but it is more likely that we will introduce a 13-year benchmark reasonably soon, to keep pace with the joint registries. Recently, we reclassified components coming through the Beyond Compliance system as Pre-entry A*. Thus while Pre-entry is not a benchmark, Pre-entry A* is.
It was decided early on in our discussions with the Association of British Healthcare Industries (ABHI) that wherever possible we should use the same benchmarking system for knees.
Submitting for benchmarks for knees and data sources
The overwhelming view from ABHI was that we should try and copy the hip benchmarks as closely as possible. The present submission form was designed with a lot of input from Mick Borroff from ABHI and will almost certainly evolve with time, just as ODEP has always done.
As has been noted many times, we welcome data in support of a submission from any reliable source. We make no apologies for asking for quite a lot of detail about death, loss to follow-up, reason for revision, and so on; we like to be assured that due diligence has been applied to the process.
As can be expected, this means that of the 150 or so knee submissions we have received so far, most of them have included several versions to be included against a particular brand name. Most well-known TKRs have several submissions that have included cruciate retaining, posterior stabilised, fixed platform, mobile platform, etc. Because it is thought that resurfacing the patella may have an effect on the outcome of TKR, we are at present requesting that manufacturers submit either ‘with patella resurfaced’ or ‘without patella resurfaced’.
Camouflage
One of the risks of ‘big data’ for revision analysis is that it might hide a small cohort of patients who have done badly. Here is an example:
If the ‘Brand X TKR’ has excellent results for the 20,000 patients who are through 10 years and the manufacturer introduced a ‘minor’ modification to one of the constituent parts (perhaps to be used in a certain group of patients) and they did poorly, aggregated big data could miss it. This is why we have decided to ask manufacturers to separate out the different options within a certain brand.
The new NJR database
Some of us have been working for NJR with the German National Joint Registry (Endoprothesenregister Deutschland or EPRD) and also with the International Consortium of Orthopaedic Registries (ICOR) group to upgrade the NJR database. The work has nearly finished and by the time this article is printed we will be well down the test drive track. The new database will make it possible for manufacturers to extract much more data about a particular product. This will make it much easier for them to produce reports about all the different versions of a particular brand. Watch this space!
Future developments and expansion
Nowadays most hip replacements are lasting over 10 years, and with an aging population this is very important. At ODEP, we have received enquiries from manufacturers asking us to extend the minimum follow-up period from 10 to 15 years. Although we are keen to introduce a longer follow-up period benchmark, we feel that a five-year gap would be too large, as a number of implants may fail sooner. We are therefore looking at the feasibility of a 13-year benchmark so we could keep pace with joint registries. The next benchmark after that will probably be at 16 years and doubtless that will come round more quickly than we expected.
ODEP has members representing the Netherlands benchmarking system; furthermore, the EPRD have been invited, and are expected to join us soon. We would welcome representatives from other groups who would like to be involved with our process and development.
Judging by the inquiries we receive, ODEP is being used worldwide. We are only a few steps away from ODEP for shoulders and a discussion regarding ODEP for revision devices is underway.
It has always been our intention to be a transparent, approachable, knowledgeable and, above all, independent organisation and we are determined to continue maintaining these values.
The authors would like to acknowledge help and support from all ODEP members. Please feel free to contact the authors at www.odep.org.uk
Authors
Keith Tucker is chairman of ODEP and Beyond Compliance, a consultant orthopaedic surgeon and a former member of the NJR steering committee.
Martin Pickford is an orthopaedic advisor to the NJR and an independent consultant.
Olga Taylor is a healthcare data analyst at Northgate Public Services.
