______________________________________________________________________________________

This case was written by ASP consultant Dr Ahsan Rana at the Lahore University of Management Sciences to
serve as basis for class discussion rather than to illustrate either effective or ineffective handling of an
administrative situation. This material may not be quoted, photocopied or reproduced in any form without the
prior written consent of the Lahore University of Management Sciences. This research was made possible
through support provided by the United States Agency for International Development. The opinions expressed
herein are those of the author(s) and do not necessarily reflect the views of the US Agency for International
Development or the US Government.

2013 Lahore University of Management Sciences



MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

In December 2012, Monitoring and Evaluation Assistants (MEAs) proceeded on a strike.
They were employees of the Punjab Education Department and in addition to their routine
work they also provided field visit information on healthcare facilities to the Department of
Health (DOH). For the last four years, they had been demanding a special allowance from the
DOH for such additional Monitoring and Evaluation (M&E) work. The department had been
negotiating with them on the justification and size of such an allowance. These prolonged
negotiations frustrated the MEAs. During the past one year, they had gradually suspended
work in several districts. The decision to stop working in the entire province was part of their
strategy to put pressure on the DOH to meet their demand without further delay.

Arif Nadeem had recently joined the DOH as its Secretary. In order to make an informed
decision on MEAs demand, he needed to learn more about their work, the process of data
collection and the indicators that they used. He wanted to find out if there were other means
of collecting the same data. His stance on the strike would depend in large measure on his
views on these issues. He was considering several options, ranging from agreeing with
MEAs demands to discontinuing the inspections altogether. He convened a meeting of his
senior aides.




30-122-2013-1
-2-

HEALTH PROVISION IN THE PUBLIC SECTOR

Punjab the most developed and populous province in Pakistan had an area of 205,345 sq.
km and a population of 96.7 million in 2012, of which 68% resided in rural areas
1
. The
literacy rate of the population, 10 years and above was estimated at 58.5 %
2
. Despite faring
better than other provinces, it had a long way to go before achieving its health related targets
for the Millennium Development Goals (MDGs). For example, in 2011-12, infant mortality
and under-five mortality rates were 82 and 104 per 1,000 live births, against a target of 40
and 52 respectively.

Network of Health Facilities

The DOH was responsible for providing preventive and curative healthcare services to this
largely rural and illiterate population through an extensive network of primary, secondary
and tertiary healthcare facilities. Consultation, diagnostic facilities and medicines were
provided without any charge to poor patients at highly subsidised rates.

The DOH network comprised more than 4,000 facilities of various types (Exhibit 1). These
facilities were divided into three tiers. The first tier consisted of Basic Health Units (BHUs)
and Rural Health Centres (RHCs). BHUs were small hospitals having one or two doctors and
a few paramedics. There was one BHU in each Union Council. The RHCs were larger
facilities with multiple specialties. There was one RHC for a group of Union Councils, called
a Markaz. This tier provided Primary Health Care (PHC) to the public at large in rural areas.
The PHC facilities also undertook outreach and community based activities, focusing on
immunisation, sanitation, malaria control, maternal and child health and family planning
services. The second tier comprised Secondary Healthcare (SHC) facilities, namely Tehsil
Headquarters Hospitals and District Headquarters Hospitals. This tier provided inpatient and
outpatient care in a variety of sub-disciplines. The third tier comprised tertiary care hospitals,
which were located in major cities and provided specialised treatment and care.




1
Punjab Bureau of Statistics 2012. Punjab Development Statistics. Lahore. Government of Punjab.
2
Pakistan Bureau of Statistics. 2011. "Pakistan Labour Force Survey 2010-11." Islamabad: Pakistan Bureau of
Statistics. Government of Pakistan.
30-122-2013-1
-3-

The Governance Structure

The Department of Health (DOH) was led by its Secretary, who was responsible for
managing health provision in the province and for providing supervision, oversight and
guidance to a very large workforce, comprising around 114,000 technical and non-technical
persons. He was responsible to the Minister and through him to the Chief Minister and the
provincial legislature. The Secretary was assisted by a team comprising senior civil servants
and public health specialists including a Special Secretary, the Director General Health
Services (DGHS), the Program Director (PD) of Punjab Health Services Reforms Program
(PHSRP) and four Additional Secretaries one each for Administration, Establishment,
Development and Technical wings (see the organogram in Exhibit 2).

The DOHs PHC and SHC facilities, i.e. the first two tiers of service delivery, were under the
supervision and control of district governments. Each district had an elaborate hierarchy of
officials to discharge functions assigned to districts under the current dispensation for local
governance. The district bureaucracy was led by the District Coordination Officer (DCO),
who was assisted by a number of Executive District Officers (EDOs), each responsible for a
set of departments at the district level. The EDO (Health) was the officer in charge of the
health department and was ipso facto responsible for managing human and fiscal resources at
the district level. He was assisted by District Officers (DOs), Deputy District Officer (DDOs)
and the Medical Superintendents of various Tehsil and District Headquarter hospitals
(Exhibit 2).

Data Collection and Analysis

The DOH collected substantial volumes of data every month from two streams. The first
stream comprised data sent by PHC and SHC facilities for the District Health Information
System (DHIS) and the second stream comprised data collected by the MEAs. These are
discussed below:

District Health Information System (DHIS)

The DHIS rolled out in Punjab during 2006-09 was based on data reported by the
facility staff. This data was derived from 24 registers that were maintained at the
health facilities and covered 79 and 83 clinical and non-clinical indicators for PHC
30-122-2013-1
-4-

and SHC facilities, respectively (Exhibit 3). Clinical indicators covered 43
communicable and non-communicable diseases, whereas non-clinical indicators
covered the availability of medicine, vacancy positions, budget, patients treated, etc.

The information was provided in a specific pro forma, usually filled by a non-
technical staff member and signed by the facility in-charge. The whole system was
paper based. The facility in charge was responsible for ensuring that the information
was recorded and reported accurately. However, the in-charge seldom had the time or
the incentive to do a thorough job in either case. This created the possibility of data
being misreported through acts of omission or commission.

To minimise this possibility, Lot Quality Assurance Sampling (LQAS) was
introduced as a quality control mechanism. LQAS involved drawing samples of
health facilities and indicators in each district and cross checking data for the sample.
Each district was required to randomly select 12 19 indicators reported by an equal
number of facilities and check the ground situation against the reported figures. The
District DHIS Coordinator in each district was responsible for carrying out this
activity through his team of Statistical Officers. The frequency of LQAS testing was
not specified. District DHIS Coordinators, therefore, could undertake this exercise as
many times in a year as they willed. But since the Coordinators were neither required
by the DGHS nor had an incentive to periodically undertake LQAS, it did not become
a regular feature of DHIS data collection.

Data Collection through MEAs

Rather than take measures to make LQAS a regular feature, the DOH launched a
parallel system of field inspections and data collection by MEAs, soon after the
DHIS was rolled out. The MEAs were initially fielded in four districts in 2004, as
part of the Chief Ministers Monitoring Program, in the education sector. They were
mostly retired junior commissioned officers of the armed forces, hired by the School
Education Department on 3-year contracts. Their exclusive responsibility was to
inspect schools and collect field-based data on selected indicators on the condition of
schools, enrolment and attendance. The program was extended to all districts in
Punjab in 2006. Each MEA was required to visit four schools per day. Since the
program was intended to inspect all the schools once each month, the number of
30-122-2013-1
-5-

MEAs deployed in a district varied. In all, 839 MEAs were working in various
districts throughout Punjab. Their school circles were rotated every month to ward
off the possibility of collusion with the staff that was being reported upon.

Starting in 2007, on a request from the DOH, the Education Department asked the
MEAs to also inspect PHC facilities within their school circles and to collect data on
a set of 14 indicators. These indicators covered the general upkeep of the facility,
staff attendance and vacancy, availability of medicine and use of outdoor and indoor
facilities (Exhibit 4). Each MEA visited 4 - 5 BHUs and RHCs every month, spent
around an hour on average inspecting the facility and initiated paper-based reports on
the prescribed pro forma (Exhibit 5). These reports were entered into a database at
the district level and consolidated reports were forwarded to the DCO concerned and
to the PHSRP. The latter prepared a monthly ranking of districts on the basis of the
performance of their PHC facilities for these indicators. District rankings were not
only placed on the PHSRP website, but were also sent to DCOs and EDOs for
appropriate action on staff absenteeism or medicine stock out.

During 2006-08, the PHSRP used to present district ranking based on MEA reports in
its periodic meetings with the Chief Secretary the chief bureaucrat in the province.
These meetings were attended by all the DCOs. A low ranking not only attracted the
Chief Secretarys ire, but also put the DCO concerned in an embarrassing position
before his peers. Gradually, however, the DOH started to develop skepticism towards
these mechanical district rankings. It was realised that a district may perform low due
to factors completely or partially out of its control, such as availability of medicine
and staff vacancy. Medicine was supplied by the DGHS and doctors were appointed
by the DOH. How a district could be then penalised for lacking on one of these.
Although district rankings continued unabated into 2012, these were not presented in
the Chief Secretarys meetings after 2008.

Since the MEAs were employed by the School Education Department, the DOH did
not make any additional allocation for the inspection of PHC facilities. When the
MEAs demanded additional pay for their services and resources to meet expenses on
visiting PHC facilities, the DOH did not have an allocated amount to meet the
demand. The demand was refused; but the MEAs persisted. In 2010-11, they
restricted their visits to health facilities located in the close vicinity of schools that
30-122-2013-1
-6-

they visited as part of their primary responsibilities with the School Education
Department. District rankings since then were based on a convenient sample of PHC
facilities, rather than on the inspection of all the facilities in a district. Gradually, the
districts started to stop the inspection regime. In December 2011, Multan dropped
out. By June 2012, eight districts had suspended the inspections regime and by the
year end (i.e. December 2012), only nine districts out of 36 districts in Punjab were
sending in their reports. The PHSRP was thus constrained to limit its district rankings
to only those districts that were still reporting.

INDICATORS AND THEIR QUALITY

The meeting was well-attended. In addition to senior managers from the Department
Special Secretary, DGHS, PD PHSRP and Additional Secretaries the Secretary had also
invited an EDO from the field. The Secretary was expecting an informed discussion and was
hoping to conclude the meeting with a clear departmental stance on data collection through
the MEAs.

The PD PHSRP was the first to speak. He presented a brief background of how the MEAs
were inducted into carrying health facility inspections in 2007, their demand for extra
payment and the history of DOH negotiations with them. He was sympathetic towards their
demands:

The MEAs have a point. They have been performing a useful service for the
Department. Why should the Department, then, not pay them for the work
they do? When we ask our own staff to inspect these facilities, they also ask
for travel and daily allowance.

In his view, data collected by the MEAs was more transparent and objective. After all,
Departmental officials the EDOs, DOs and DDOs belonged to the same professional and
personal networks as did the health practitioners they reported on. They had a reason to be
lenient and to look the other way if a delinquency took place. In comparison, the MEAs
belonged to a different cadre and were employed by a different Department. Ipso facto, they
were less likely to misreport. Attendance of doctors and paramedics has improved and so
has medicine availability since we launched this inspections regime four years ago,
observed the PD.
30-122-2013-1
-7-




The DG disagreed. He had a different perspective on almost all that the PD had said. He
asked:

Is there any evidence which suggests that overall attendance or medicine
availability is better today, than it was four years ago? Even in districts where
this seems to be the case at times, how safely can it be ascribed to MEA
inspections? So much is simultaneously happening in the micro and macro
environments that any one factor cannot be linked to any improvement or
deterioration.

The DG also spoke about duplication in data collection:

We have an excellent system already in place. The DHIS is now fully
operational. True, the LQAS testing has been less frequent in the past few
years than what should have been the case, but it is our fault and not the
DHISs. DHIS gives us monthly data on so many indicators. We should focus
on improving data accuracy.

The EDO questioned the notion that MEAs would report more objectively since they did not
share personal and professional networks with health practitioners. He pointed out that
MEAs mostly came from the same district and had a similarly longstanding assimilation into
the local milieu as anyone else. He asked, They are susceptible to the same pressures as
health managers and are similarly lured by pleasures of the flesh lust and greed. How else
could one understand the not-so-infrequent complaints of rent seeking or asking sexual
favors from female staff?

He was also critical of the process, MEAs do not visit all the facilities every month; they
take a convenient sample. Even for the ones they visit, all that we get is a snapshot on
selected indicators.

The Secretary asked about how adequate were the indicators used by MEAs in satisfying
health managers needs, especially at the district level. The EDO said that the MEAs data
30-122-2013-1
-8-

covered a small part of the working of a health facility. He agreed that staff attendance,
cleanliness, medicine availability, etc. were essential attributes of a health facility, but argued
that so were other things, such as the number of outpatients treated, types of diseases in the
area, surgeries and procedures performed, children immunised, etc.
The EDO observed;

Ranking health facilities on the basis of one small dataset amounts to sending
a message that the facility in charges do not have to worry about the other,
more numerous and comprehensive attributes. This seems quite reductionist.

The PD, however, was of the opinion that MEAs reported on the most important aspects of
the working of health facilities, especially from a management perspective. The PD
observed;

These data are of fundamental importance. Other things patients treated,
procedures performed and children immunised will depend upon whether or
not a doctor is present and medicines are available.

The Additional Secretary (Technical) made several suggestions about the quality of the
indicators;

Some of these are qualitative indicators and the response thereto is based on
the MEAs judgment, rather than an objective criterion. Perhaps we need to
reduce the frequency of data collection in such cases. This will give MEAs
more time to collect data on the remaining indicators. It will also allow the
addition of a few higher order indicators to the pro forma.

He was against worrying on who sent the data, as long as these were relevant and economic.
The former depended on the use of these data and the latter on the cost of collection.
Presently, he felt, MEA as well as DHIS data remained under-analysed. He suggested regular
time series analyses of performance patterns not only of individual facilities but also of
districts and the entire Department. Sometimes I feel that we collect more data than we can
handle.

30-122-2013-1
-9-

The Special Secretary asked if the data collected by MEAs was not self-reported in some
ways. Do the MEAs have to rely on data that is already recorded or what is reported to them
by the concerned officials in the facility, he enquired. The PD said that this was indeed the
case for some indicators, but even in these cases, the possibility of physical verification
makes misreporting by facility staff less likely than would be the case otherwise.

CONCLUSION

Arif Nadeem was happy on how the meeting had progressed as the participants had candidly
expressed their views. However, he felt that the question of giving extra remuneration to the
MEAs would arise only after the Department was convinced that the data they collected was
useful. He formed a three member Committee comprising DGHS, PD PHSRP and the
Additional Secretary (Technical) to examine, in detail, the indicators used by MEAs, keeping
in mind their quality and appropriateness. The Committee was to present a report on whether
or not the DOH still needed to collect data through MEA inspections. If it did, how could the
indicators currently in use be refined to make them more relevant for managers? He expected
the committee to finalise its recommendations in three days.


















30-122-2013-1
-10-

Exhibit 1
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Facilities and Human Resource

Number of public sector health facilities in various categories

Category Description Number
Basic Health Units Basic medical/ surgical care and referral 2,466
Rural Health Centers 10-20 inpatient beds for 100,000 people 297
Tehsil Headquarters Hospitals 40-150 beds with nine specialties 91
District Headquarter Hospitals >150-400 beds with 18 specialties 34
Teaching/ Tertiary Care Hospitals Large hospitals with multiple specialties 31
Others Dispensaries
TB Clinics and Hospitals
Mother and Child Health Centers
Specialized Hospitals
738
19
329
5
Total 4,010

Source: DOH 2012. DHIS Quarterly Report 4th Quarter 2012. Lahore. Directorate
General of Health Services, Government of Punjab.

Manpower Employed in the Health Sector (Public and Private)



Source: Ministry of Finance 2011. Pakistan Economic Survey 2011-12. Islamabad.
Ministry of Finance, Government of Pakistan
27,855
130,220
11,372
76,244
11,510
27,153
14,250
52,486
4,602
51,577
10,148
7,112
0
20000
40000
60000
80000
100000
120000
140000
Specialists MBBS BDS Nurses LHVs Midwifes
Pakistan
Punjab
30-122-2013-1
-11-

Exhibit 2
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Organogram Punjab Health Department


MS Medical Superintendent
CDCO Communicable Diseases Control Officer
DSI District Sanitary Inspector
DSV District Superintendent of Vaccination


Source: Case writers Notes
Secretary
DOH
Director
General
Health
Services
36 EDOs
MS, DHQ
Hospital
Principal
Nursing
School
MS, THQ
Hospital
DO (H)
CDCO,
Entomologist,
DSI, DSV
DDO (H)
(Tehsil)
District
Coordinator
National
Programs
District Drug
Inspector
PD District
Health
Development
Centre
Other
projects and
wings
Special
Secretary
DOH
Tertiary Care
Hospitals
and Teaching
Institutions
Additional
Secretaries
30-122-2013-1
-12-

Exhibit 3 (p1 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

DHIS indicators for PHC and SHC facilities

Sr. No. Indicator
Overall Performance
1 Daily Outpatient Department attendance
2 Full immunization coverage
3 Antenatal care coverage
4 Delivery coverage at facility
5 Tuberculosis-Directly Observed Treatment, short-course (TB-DOTS)
patients missing more than one week
6 Total visits for Family Planning
7 Obstetrics complications attended
8 C-Sections performed
9 Lab services utilization
10 Bed occupancy rate
11 LAMA
12 Hospital death rate
13 Monthly report data accuracy
Outpatients Attendance
14 New cases
15 Follow up
16 Number of cases of malnutrition < 5 yrs. children
17 Referred attended
Immunization and TB-DOTS
18 Children <12 months received 3rd Pentavalent vaccine
19 Children <12 received 1st Measles vaccine
20 Children <12 months fully immunized
21 Pregnant women received TT-2 vaccine
22 Intensive phase TB-DOTS patients
23 Intensive phase TB-DOTS patients missing treatment > 1 week


30-122-2013-1
-13-

Exhibit 3(p2 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH
DEPARTMENT


Family Planning Services
24 Total FP visits
25 Combined Oral Contraceptive (COC) cycles
26 Progestogen-only Pill (POP) cycles
27 Depot Medroxyprogesterone Acetate (DMPA) Inj.
28 Net-En Inj.
29 Condom Pieces
30 Intrauterine Contraceptive Devices (IUCD)
31 Tubal Ligation
32 Vasectomy
33 Implants
Maternal and Newborn Health
34 1st Antenatal Care (ANC) visits (ANC-1)
35 ANC-1 women with Hb < 10g/dl
36 Antenatal care revisit in the facility
37 1st Postnatal care visit in the facility
38 Normal vaginal delivery in facility
39 Vacuum/ forceps deliveries in facility
40 Cesarean Sections
41 Live births in the facility
42 Live births with Live Birth Weight (LBW) < 2.5 kg
43 Still births in the facility
44 Neonatal deaths in the facility
Community Data
45 Pregnant women newly registered by Lady Health Workers (LHWs)
46 Deliveries by skilled persons reported
47 Maternal deaths reported



30-122-2013-1
-14-

Exhibit 3 (p3 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH
DEPARTMENT


48 Infant deaths reported
49 Number of modern FP methods users
50 <5 years diarrhea cases reported
51 <5 years ARI cases reported
Community Meetings
52 Number of community meetings
53 Number of participants
Diagnostic Services
54 Total lab investigations
55 Total X-Rays
56 Total ultra sonographics
57 Total Electrocardiograms
Stock out
58 Stock out of drugs/vaccines
Indoor Services
59 Allocated beds
60 Admissions
61 Discharged not on same day of admission
62 Discharged on same day of admission
63 LAMA
64 Referred
65 Deaths
66 Total of daily patient count
67 Bed occupancy
68 Average length of stay





30-122-2013-1
-15-

Exhibit 3 (p4 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH
DEPARTMENT


Procedures
69 Operations under general anesthesia
70 Operations under spinal anesthesia
71 Operations under local anesthesia
72 Operations under other type of anesthesia
Human Resource Data
73 Sanctioned
74 Vacant
75 Contract
76 On general duty in facility
77 On general duty out of facility
Revenue Generated and Financial Report
78 Total receipts
79 Deposits
80 Total allocation for the fiscal year
81 Total budget released to-date
82 Total expenditure to-date
83 Balance to-date
Others
84 LHW pregnancy registering coverage
85 Total Homeo cases
86 Total Tibbi/Unani cases

Note: In all, data were collected on 79 and 83 of the above indicators for PHC and SHC facilities
respectively.


Source: DOH 2012. DHIS Quarterly Report 4th Quarter 2012. Lahore. Directorate
General of Health Services, Government of Punjab.
30-122-2013-1
-16-

Exhibit 4
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Indicators used for MEA Inspections

1. Cleanliness and general outlook of the facility
2. Display of signboard/direction board, organogram, maps, etc.
3. Availability and functionality of utilities
4. Disposal of hospital waste
5. Purchee fee deposited and OPD visits during last month
6. Attendance of doctors
7. Detail of absent staff other than doctors
8. Vacant posts
9. Inspection of the facility by District Government officers
10. Availability of medicines
11. Indoor patients and availability of MO and Nurses in evening and night shifts
12. public opinion regarding the following
a. Presence of doctors
b. Attitude of doctors towards patients
c. Waiting time
d. Free availability of medicines
13. Progress of development schemes/ provision of missing facilities
14. Availability and functionality of equipment

Note: Sr. No. 11 above was not applicable to BHUs.

Source: Constructed from M&E inspection pro forma


30-122-2013-1
-17-

Exhibit 5 (p1 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

MEA Inspection Pro Forma



30-122-2013-1
-18-

Exhibit 5 (p2 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT




30-122-2013-1
-19-

Exhibit 5 (p3 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT




30-122-2013-1
-20-

Exhibit 5 (p4 of 4)
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT




Source: Department of Health, Government of Punjab
30-122-2013-1
-21-

Appendix A
MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Acronyms:
BHU Basic Health Unit
DCO District Coordination Officer
DDO Deputy District Officer
DG Director General
DGHS Director General Health Services
DHIS District Health Information System
DO District Officer
DOH Department of Health
EDO Executive District Officer
LQAS Lot Quality Assurance Sample
M&E Monitoring and Evaluation
MEAs Monitoring and Evaluation Assistants
MO Medical Officer
PD Program Director
PHC Primary Healthcare
PHSRP Punjab Health Sector Reforms Program
RHC Rural Health Centre
SHC Secondary Healthcare