The new National Strategic Plan on HIV, STIs and TB, 2012-2016 (NSP), like its predecessor, contains a large section on monitoring and evaluation (M&E). But despite ambitious targets in this field, poor M&E followed the last NSP. At the time, the Department of Health lacked the will and the staff to make a success of M&E.
Hopefully the new NSP will fare better. M&E in the new plan is better conceived than in the old one. The latest M&E section uses a more manageable set of indicators, a few of which contain plausible baseline data and describe feasible methods for measuring success both during and at the end of the NSP period.
The indicators are: (numbers rounded to avoid false accuracy)
- HIV prevalence in 15-to-24-year-olds, estimated at baseline to be 9% (HSRC Household Survey). The NSP targets a reduction to 4% in 2016.
- HIV prevalence in key populations. The NSP contains no baseline data for this, even though it proposes a 50% reduction.
- HIV incidence, estimated to be 1% in 2012 (ASSA). The goal is to reduce this by half by 2016.
- HIV mortality, estimated at over 40% of deaths at baseline (Stats SA). The aim is to reduce this percentage by half by 2016.
- TB incidence, estimated to be almost 1% in 2010 (WHO). South Africa aims to reduce it by half by 2016.
- TB mortality in HIV-negative people, estimated to be 50 per 100,000 in 2010 (WHO). The NSP targets a reduction to 25 per 100,000 in 2016.
- Rate of transmission to infants, estimated to be just under 4% at six weeks after birth in a study published last year by the MRC. South Africa aims to reduce this to less than 2% at six weeks and less than 5% at 18 months in 2016.
- Patients alive and on ART. No data is currently available for this indicator, but by 2016 the target is to have 94% retention in care by one year after commencing ART, 88% by two years, 82% by three years, 76% by four years and 70% by five years.
Some of these statistics should be treated with caution. The drafters of the NSP recognise this and have tasked the South African National AIDS Council (SANAC) secretariat with leading a process to calculate better baseline data.
For example, one indicator not mentioned above is called a stigma index. This is poorly defined and the NSP describes no way of measuring it. Nevertheless, most indicators in the plan are reasonable and can be used as the basis for a good M&E programme. The NSP also commits to annual programme reviews, a mid-term evaluation in 2014 and a final evaluation in 2016 that will also provide baseline values for the next NSP. This is ambitious. If it is to happen, there must be clear leadership and enough people to do this work.
Overall responsibility for M&E falls to the M&E Unit in the SANAC secretariat. The plan also notes that M&E units in provincial AIDS councils and sectors should co-ordinate M&E provincially. Exactly how this will all work, and who will lead the process, is unclear, as is the role of the Health Department. But wherever the M&E programme is located, it must have a substantial budget and quality human resources. The Director General of Health must take ultimate responsibility for ensuring the implementation of M&E.
Getting it right for treatment
How many people began antiretroviral treatment in public clinics and hospitals in South Africa? How many are still in treatment? How many have died? What regimens are people taking? How many have moved to second-line regimens? At what CD4 count does the average person start treatment? How many have suppressed viral load at one year? What is the average time that people have been on treatment?
A well-monitored antiretroviral rollout should be able to answer these questions. There should be reports every six months containing all these data and more. Unfortunately the monitoring of South Africa’s antiretroviral rollout does not yet provide this information consistently and accurately, although data collection and processing appear to be improving.
Occasionally the Department of Health releases a spreadsheet showing the numbers of people on treatment. The latest figures are for June 2011. According to the Department of Health, over 1.5 million people were on treatment in the South African public health system by the end of June.
Eastern Cape: 151,878, Free State: 79,204, Gauteng: 375,798
KwaZulu Natal: 481,809, Limpopo: 109,121, Mapumalanga: 121,516
North West: 130,901, Northen Cape: 17,407, Western Cape C: 101,812
There are problems with these data:
- The methodology used to collect them is unpublished. Therefore it is difficult to have confidence that robust systems to collect this information are in place. Most facilities get little support on how to track numbers and there is no standardised auditing system. Do health workers consistently record patient visits, viral loads and deaths? Probably not.
- Some provinces report the number of people who started treatment and others report the number of people on treatment.
- Even if these data were accurate and consistent, they would help answer only one of the above questions.
One of the best sources of information on South Africa’s rollout is the International epidemiologic Databases to Evaluate AIDS Southern Africa (IeDEA). This is a pooled database from multiple sites across southern Africa, including ten projects in South Africa. The group regularly publishes useful information about the rollout. It can reliably report number of patients, loss-to-follow-up and mortality. But although it covers a large number of patients, it is still only a fraction of the total population of patients on antiretroviral treatment in the country.
The Western Cape approach
The Western Cape, assisted by UCT’s Centre for Infectious Disease Epidemiology (CIDER), had the best provincial data collection system. The system allowed the provincial health department to produce a regular detailed report that helped answer many of the above questions. Although the system began in paper format it is slowly being converted to a digital system that emulates its paper-based predecessor.
However two factors have made the Western Cape system unwieldy in recent years. The province has been trying to integrate the HIV treatment system into its wider data systems—a complex task—but it has not invested in enough dedicated staff to maintain the system properly. If the number of people on ART were stable neither of these factors would be likely to present a problem. But coupled with the swift scale-up these factors have set back the province’s treatment data collection system.
In addition to the Western Cape, several donor-funded projects operating in the public health system have their own systems to monitor their subsets of the antiretroviral rollout. An audit showed there were at least 40 such systems, about 15 of which were properly functional. Most of these projects are funded by PEPFAR, which requires detailed treatment statistics so that it can report to the US government.
One consolidated system
The National Department of Health recognises that there are too many systems and that what is needed is one properly working consolidated system. It is encouraging all the various projects to merge their data collection into one national system. For this it has largely adopted the Western Cape system.
This national system recognises that there are three tiers of data collection that need to work together:
- Paper-based systems at small clinics.
- Standalone electronic data capturing systems at medium-sized clinics.
- Networked Electronic Medical Record systems at large facilities.
Clinton Health Foundation officials have been contracted to assist with the rollout of the system.
The government’s move towards a unified, rationally designed treatment monitoring system looks promising after many years of rudderless monitoring and evaluation. There is progress in some provinces, but getting M&E right will require herculean efforts by government and cooperation from donor-funded projects. The response from donor-funded projects is mixed. Some deserve praise for the way they are trying to merge their systems into the national one. However, others could do a lot better.
Technological improvements that will help
Several technological improvements and better management of technology will improve M&E systems. The issues discussed here are not exhaustive.
Networked computing is too expensive
Moving from paper to digital format is an important step forward. Computerised algorithms are far more accurate and faster at calculating the number of patients lost to follow-up than humans sifting through mounds of paper. When clinics computerise their records they usually find that they have underestimated loss-to-follow-up. One of the barriers to moving to networked computers is the State Information Technology Agency (SITA), from which network access must be purchased. SITA charges in the region of R20,000 per month for a Diginet line. Few clinics can afford, or need, such a sophisticated solution. Moreover it typically takes months to install such lines. Yet a simple wireless internet connection can be set up within days costing approximately R500 per month. While it is understandable that a central state agency should be responsible for this kind of infrastructure in order to prevent abuse, SITA’s exorbitantly priced solution is impractical and wasteful.
Free cell phone access to download viral load, CD4 and other results
Many health workers have cell phones with internet access. It should not be too technically challenging to build a system that allows them to download their patients’ blood results from the NHLS—obviously, with suitable security. It should also not be difficult to make this downloading free for nurses, i.e. if the cell phone companies redirected bandwidth charges for certain sites to a health department instead of to the health worker’s account.
Integration of information
South Africa has some very useful health data systems. Home Affairs tracks deaths. The NHLS has a database of viral load and CD4 results. Many pharmacies have implemented sophisticated dispensing systems. It would be ideal if these systems synchronised relevant data with the antiretroviral Electronic Medical Record system. So for example if a patient died and Home Affairs processed a death certificate, the medical system would be automatically updated; a loss-to-follow-up could be correctly recorded then as a death. Viral loads and CD4 counts could be automatically saved onto patient records. The patient record system could automatically send prescription requests (or vice versa) to pharmacy systems. This would be a complex undertaking but if our treatment monitoring system could get to this level of sophistication it would greatly improve the quality of information on the ART rollout.
Protection of equipment
A key challenge is maintaining computers in peripheral clinics that lack IT departments. Thefts and breakdowns can take weeks or months to rectify.
It is vital that the National Department of Health continue its effort to get the monitoring and evaluation systems for the ART programme up and running properly. To do this, donor-funded programmes and provincial governments must cooperate and work together, with the aim of creating one national integrated system that records all useful implementation data and reports it regularly.
by Nathan Geffen, Treasurer of the Treatment Action Campaign. Special thanks to Andrew Boulle for assistance with this article.