This page provides responses to the most frequently asked questions about the League Table project. If you do not find an answer to your question here, you are welcome to use the form below to contact us.
WHO PRODUCED THIS LEAGUE TABLE?
This project was conceived, developed and produced by Universities Allied for Essential Medicines (UAEM), an international nonprofit organization of graduate and undergraduate students in medicine, research, law, and related fields, and Medsin-UK, a student led charity focussed on global health advocacy, education and local action. Students from a wide range of UK institutions, including many of those evaluated by the GHRLT, contributed to the project. Research and analysis to produce the League Table was conducted over the course of late 2013 and 2014. Funding for the project was provided by the Scurrah Wainwright Charity.
WHERE DOES THE DATA USED IN THE LEAGUE TABLE COME FROM?
To promote fair evaluation and methodological rigour, we used standardized, authoritative, publicly accessible data sources for as many metrics as possible. Six of our 12 metrics rely entirely on publicly-available data sources, while another four are derived from a combination of publicly available and self-reported data.
Self-reported data was only sought on metrics for which public information was limited or inconsistent.
WHAT ARE THE MAIN PUBLICLY-AVAILABLE DATA SOURCES USED IN THE LEAGUE TABLE?
The most significant sources of publicly-available data used in this evaluation are:
- Department for International Development – Research for Development (R4D) database
- European Commission – Community Research and Development Information System (CORDIS)
- Bill and Melinda Gates Foundation Grants Database
- Research Councils UK – Gateway to Research Database
- Wellcome Trust – awarded grants spreadsheets
- ResIn – Research Investments in Global Health Database
- G-FINDER (Global Funding of Innovation for Neglected Diseases) Public Search Tool
- List of signatories to the Statement of Principles and Strategies for the Equitable Dissemination of Medical Technologies
- List of signatories to the Nine Points to Consider in Licensing University Technology
- University websites, technology transfer office websites
- PubMed and PubMed Central
- The Compact for Open-Access Publishing Equity
- The Registry of Open Access Repositories Mandatory Archiving Policies (ROARMAP)
WHY DID THE GHRLT CHOOSE THESE FIVE DATABASES?
The GHRLT focuses on research that is funded from public or philanthropic sources. We extracted grant data from the online databases of the largest funders for medical research in the UK, that are the primary sources of funding in global health research and provide complete and accurate data for analysis. The five funding bodies selected were:
a. Gateway to Research
- Department for International Development
- European Commission
- Wellcome Trust
- Bill and Melinda Gates Foundation
The Gateway to Research Database combines data of all research projects publicly funded by the 7 UK Research Councils (Medical Research Council, Economic and Social Research Council, Biotechnology and Biological Sciences Research Council, Engineering and Physical Sciences Research Council, Science and Technology Facilities Council, Arts and Humanities Research Council, and Natural Environment Research Council). It is a comprehensive database that provides details on the project title, description, funding, lead institution and collaborators. When examining the strategy of many of these councils there is a focus on delivering global research that “addresses the inequalities in health which arise particularly in developing countries.”(MRC website).
The Department for International Development (DfID) is a governmental organisation that funds global health research. Their Research for Development database is a comprehensive database that provides details on the project title, description, funding, lead institution and collaborators.
The European Commission database, Community Research and Development Information System (CORDIS), is a public repository of all EU funded studies. The strategic aims of the EU and their focus on “Health is wealth” and “Innovation” aligned with the themes within this project. CORDIS provided details on the project title, description, funding, lead institution and collaborators. Euros were converted to pounds based on a standard conversion rate (average of 2013).
The Wellcome Trust is a large philanthropic organisation, which heavily funds university health research in the UK. Data was extracted from annually publish spreadsheets of awarded grants. Unfortunately grants funded between 1/10/2013-31/12/2013 were not publicly available to evaluate, and were not included. The Wellcome Trust provides details on the name, description, funding, and lead institution of grants. Furthermore we excluded Awards for PhDs as details of PhD topics were often too sparse to determine whether the projects would be focussing on defined criteria.
The Bill and Melinda Gates Foundation Grants Database was used as they are a significant funder of global health research. The database provided information including project title, funding and lead institution. USD were converted to GBP based on a standard conversion rate (average 2013 conversion rate).
Accurate records of collaborators along with adequate descriptions of funding distribution within each project were not provided by any of the databases. It was therefore impossible to accurately distribute funds based on collaborator contribution. We experimented with arbitrary splits of funding (such as 50% of project total to lead institution and 50% split evenly between all other collaborators), which did alter the landscape of the results (particularly in large studies where many institutions had collaborated). We sought guidance from the Research Investments in Global Health Group (ResIn), a group utilising a similar methodology analysing funding data of UK infectious disease research Given the time requirements and high risk of discrepancies between databases, we were reassured that awarding funding to the lead institutions is scientifically robust.
WHY DOES GHRLT USE PUBMED AS A PUBLICATION DATABASE?
We chose to use PubMed as it is a comprehensive resource that is commonly used by researchers in the health field, together with its ease of use and features which allow us to easily download large amounts of data. Using more than one publication database would have meant duplication of work and greater room for error. Comparing publication sets between PubMed and PubMed Central allowed us to easily analyse how much of a university’s health related research output is freely available online.
HOW DID GHRLT COLLECT DATA FROM UNIVERSITIES FOR THE METRICS THAT RELIED PARTIALLY OR WHOLLY ON SELF-REPORTED INFORMATION?
For the Access section, an online questionnaire was developed on Survey Monkey, and e-mailed to the TTO officials best suited to provide the data. TTOs were contacted a minimum of 4 times by email and twice by telephone over a 12-week period beginning July 6, 2014. For those Universities that had not responded by the end of the 12-week period, a request for the information requested in the original online survey was made under the Freedom of Information Act 2000 on October 1, 2014.
Advance notice of the league table launch was sent to every university senior leadership office in advance of the public release, providing one more opportunity to submit updated or missing data for metrics that were based on self-reported information. Several institutions responded with additional information, which was included in the publicly released grades. Several more acknowledged receipt of the pre-release notice.
HOW DOES THE LEAGUE TABLE FAIRLY EVALUATE NON-RESPONDING UNIVERSITIES?
We took care to weight the League Table metric scores such that a non-reporting institution that received high marks on the public information-based metrics could still receive a competitive score. It is also important to note, however, that because transparency and disclosure are elements that we sought to emphasize in every aspect of this project, where a university did respond to the self-reported questionnaire for a given section they received a minimum credit for those metrics.
HOW DOES THE LEAGUE TABLE FAIRLY EVALUATE UNIVERSITIES WITH VARYING SIZES AND RESEARCH BUDGETS?
As universities selected for evaluation vary in significant ways (e.g. levels of funding, student body size), we designed League Table metrics and scoring systems to minimize the impact of such differences.
All quantitative metrics are “normalized” with respect to degree of institutional funding, total number of licenses executed, or another school-specific variable that serves as a proxy for university size. For example, rather than scoring a university on the absolute GBP amount of funding devoted to neglected disease research, or the absolute number of non-exclusive licenses executed in a given year, these numbers were divided by a relevant total for that school (total medical research funding from 5 funders or total licenses executed) to arrive at a percentage for each institution. All institutions with percentages falling in the same scoring range received the same score, regardless of absolute institutional size. This approach enabled meaningful comparison across institutions while minimizing or eliminating the impact of variations in size, budget, or resources.
For non-quantitative metrics, the League Table employs pre-defined sets of discrete categories by which all universities can be uniformly evaluated, and for which performance is again likely to be independent of variations in university size, funding, capacity or resources. For example, on the first question in the access section, public university commitments to socially responsible licensing were sorted into five pre-defined categories based on the specificity and details of the commitment each school had made. All universities falling into the same category received the same score.
HOW DOES THE LEAGUE TABLE DEFINE WHAT IS INCLUDED AS GLOBAL HEALTH RESEARCH?
In innovation, we assessed two different aspects of global health research. Firstly, we evaluated research on neglected diseases as defined by the G-FINDER report. Secondly, we evaluated research on health in developing countries.
These are areas of global health that universities are uniquely positioned to impact. These metrics can also be reliably evaluated due to availability of high-quality, consistent, publicly accessible data sources.
HOW HAVE YOU DEFINED UNIVERSITY RESEARCH ON NEGLECTED DISEASES?
The GHRLT used the G-FINDER report and the World Health Organization’s list of neglected tropical diseases to define the scope of neglected diseases. The G-FINDER report is a comprehensive survey of worldwide funding for research and development of innovative neglected disease treatments, medicines and health technologies, that establishes a specific, inclusive and empirically-grounded definition of “neglected diseases.” This definition and the specific diseases included are detailed here.
“Neglected diseases” (NDs) were defined as a set of diseases primarily affecting low- and lower middle-income countries. The scope of the research areas included was further focused in terms of subject matter and application. Notably, this definition of ND includes HIV, tuberculosis, malaria, diarrheal diseases, meningitis, and pneumonia; however, for several of the diseases there are substantial restrictions to include only aspects or subsets of these diseases that are truly neglected. For example, we did not include all research on HIV – only research pertaining to paediatric HIV, HIV diagnosis, diagnostics, microbicides, and vaccines.
HOW HAVE YOU DEFINED UNIVERSITY RESEARCH ON HEALTH IN DEVELOPING COUNTRIES?
Research on health in developing countries was defined as the range of health related research primarily affecting low- and lower-middle income countries.We felt these areas encompassed the most important areas of global health. Within our definition, low income and lower middle income were defined by World Bank criteria. The scope of the relevant research areas was based on the tentative typology outlined within document produced by the Global Health Forum, entitled ‘Monitoring Financial Flows for Health Research 2008’. By using this definition, our scope of global health research thereby acknowledges, in addition to biomedical research, research into health policy and systems research, behavioural and social science and operational research. We therefore include research ranging from social determinants of health, to mental health, where the focus is on low and lower middle income countries.
DOES THE LEAGUE TABLE MEASURE ALL POSSIBLE WAYS IN WHICH UNIVERSITIES CONTRIBUTE TO GLOBAL HEALTH?
The GHRLT aims to capture universities commitment and contribution to research that benefits global health. Our methodology does not assess all other areas where universities can have an impact on global health, including health partnerships, educational and training programmes and capacity building initiatives. We found that systematically quantifying, evaluating, and comparing broader global health activity beyond the global health research metrics included here is a much more challenging endeavor. We found there was a lack of data that captures the existence and quality of activities outside of global health research, that could be objectively compared across institutions. In future iterations of the GHRLT we hope to work towards expanding the scope of the project to include these areas, as better data and tools become available. We are actively seeking ways to more fully capture these additional contributions in the next iteration of the League Table, and welcome suggestions for doing so.
We acknowledge that, as with any assessment tool, our metrics are not able to capture every area of research that has some impact on global health. We believe our methodology does capture the majority of publications that would be defined as global health according to accepting definitions, and by focussing on the largest funders for global health research we capture most research funding institutions receive.
Our metrics only capture a snapshot in time – because of limitations in the time period of available data and the time required to compile and produce this evaluation, significant university initiatives launched within the past 12-18 months may not be captured.
We also recognize that many individuals and research groups within lower-ranked universities are doing ground-breaking and high impact work that may not be specifically highlighted or fully accounted for by our methodology, particularly for activities that do not fall within global health research. Our intention is that the League Table be viewed as an assessment of each institution as a whole in relation to its peers, and should in no way be seen as discrediting outstanding individual efforts.
We used a wide range of metrics to fairly capture significant global health research contributions. We believe that our metrics are rigorous and fair, providing a methodologically rigorous “snapshot” of university contributions to several of the most critical global health research domains.
HOW DOES THE LEAGUE TABLE ADDRESS HEALTH RESEARCH THAT IS NOT FOCUSSED ON DEVELOPING COUNTRIES, BUT MAY STILL PROVIDE BENEFITS TO THESE COUNTRIES? FOR EXAMPLE, DOES THE LEAGUE TABLE ASSESS WORK ON NON-COMMUNICABLE DISEASES (NCDs)?
Two of the four questions in the Innovation section assess research that focuses on health in Low- and Lower-Middle-Income countries. Where research on NCDs is focussed on developing countries this is captured in these metrics.
For health research that is not focussed on developing countries, such as developing new medicines to treat NCDs, the primary challenge is that medicines and diagnostics developed for these diseases are likely to come to market at prices that put them out of reach for people in developing countries. A crucial way in which universities can avoid this is by patenting and licensing their discoveries in a socially responsible manner. Universities can also ensure that research publications are freely available online, so they can accessed by researchers, health professionals and patients The Access section of the League Table evaluates university efforts in this area, including policies on patenting and licensing, and how successful they are at implementing these measures. These steps in university research are essential to ensuring that affordable medicines are available to fight the NCD epidemic, besides other diseases.
This is exactly the issue in the recent Indian court ruling on Novartis’ leukemia drug Glivec. The basic research to develop Glivec was conducted largely in academic laboratories, but ultimately transferred to the drug company Novartis, which sought to enforce exclusive intellectual property rights in India in order to reduce competition from more affordable generic alternatives. Today, NCD innovations regularly come to market at prices of tens of thousands of pounds per patient per year (in the UK, the price for Glivec is about £21,000 per patient per year). Such medicines and treatments won’t reach low-income developing world patients unless steps are taken to promote locally affordable versions.
While we celebrate universities’ extensive and important research on globally prevalent non-communicable diseases, institutions that are committed to impacting global health through biomedical research should use socially responsible licensing strategies to enable affordable generic production of the medicines they develop in developing countries.