Development of an Intrastate Inspection Selection System (ISS) for Wisconsin

Professor Robert L. Smith, Jr. Department of Civil & Environmental Engineering University of Wisconsin-Madison 1415 Engineering Drive Madison, WI 53706 (608) 262-3649 smithrl@engr.wisc.edu Marcus H. Januario Graduate Research Assistant Department of Civil & Environmental Engineering University of Wisconsin-Madison Emil Juni Graduate Research Assistant Department of Civil & Environmental Engineering University of Wisconsin-Madison

Paper Number 01-3060

November 2000 Submitted to Transportation Research Board for publication and for presentation at the 2001 Annual Meeting

2
ABSTRACT

Roadside safety inspections have been one important component of federally supported programs to improve motor carrier safety. The Inspection Selection System (ISS) and the successor, ISS-2, were designed to help roadside inspectors target the vehicles of interstate carriers with the worst past safety performance. This research used the ISS as a starting point for developing an intrastate ISS for Wisconsin. Logistics regression was used to identify the relevant independent variables. The logit model estimates the probability that a vehicle to be inspected will be found to have an out-of-service (OOS) safety violation. An alternative to the logit model, a simple direct estimation technique using the total OOS rate for a motor carrier, was found to produce the highest success rates in predicting inspections that result in an OOS violation. The model can be

implemented both as a field inspection tool and as a means to generate a list of intrastate carriers ranked by their historical OOS violation record.

3 INTRODUCTION Improvements in motor carrier safety have long been a priority at the federal level. The federal Motor Carrier Safety Assistance Program (MCSAP) was established by the Surface Transportation Assistance Act of 1982. MCSAP provides federal funds to states with the goal of reducing the number and severity of accidents and hazardous materials incidents involving commercial motor vehicles. MCSAP covers funding for safety inspections of drivers and commercial vehicles, traffic enforcement, carrier safety compliance reviews, public education and awareness, and data collection. To help states improved the effectiveness of their roadside safety inspections of commercial vehicles, the Federal Highway Administration (FHWA) funded the development of the Inspection Selection System (ISS) during the early 1990s. The goals of ISS are to target the vehicles of carriers with the worst past safety performance and to increase the sampling of carriers with lower than average inspection rates for their size (1). In order to develop the Inspection Value that provides roadside inspectors with a recommendation regarding inspection of a particular vehicle, ISS uses data from the federal Motor Carrier Management Information System (MCMIS). The key to integrating multiple sources data from all of the states in MCMIS is the USDOT number. This number is required by federal regulations to be issued to all interstate motor carriers and the number must be displayed on the sides of all commercial vehicles operated by a carrier. Wisconsin began using ISS in 1996 as part of its MCSAP efforts relating to interstate carriers. Since Wisconsin does not require carriers that operate only in Wisconsin (intrastate carriers) to obtain a USDOT number, ISS cannot be used to enhance the roadside safety inspections of intrastate carriers in Wisconsin. Because of its interest in developing a more

4 comprehensive motor-carrier safety program in Wisconsin, the Wisconsin State Patrol proposed the development of an intrastate ISS for Wisconsin. A revised version of the interstate ISS, the ISS-2 methodology, has recently been incorporated into the Federal Motor Carrier Safety Administration s latest version of their Motor Carrier Safety Status Measurement System (SafeStat) (2). ISS-2 uses additional independent variables and a more complicated weighting procedure, but the overall results in terms of recommending vehicles that should be selected for inspection are consistent with the original ISS (3). In the interests of simplicity, the ISS-2 methodology is not used as the starting point for the intrastate ISS model developed for this research. RESEARCH OBJECTIVES The objectives of this research are to develop an Inspection Selection System (ISS) for intrastate carriers in Wisconsin using currently available data and to propose an implementation methodology. Development of an intrastate ISS for Wisconsin is complicated by the fact that Wisconsin does not have a formal intrastate motor carrier registration requirement. In addition, the registration of vehicles by motor carriers is not based on unique motor carrier names. Fortunately, a comprehensive database of intrastate roadside safety inspections has been developed by the Wisconsin State Patrol with a unique number assigned to motor carriers that have had one or more roadside inspection since the database was initiated. This database provides the starting point for the research. RESEARCH METHODOLOGY The interstate ISS process is based primarily on two sets of data: 1) historical data on carrier out-of-service (OOS) rates for vehicles and for drivers and 2) historical data on the carriers rates of roadside safety inspections per carrier power unit and per driver. Cumulative frequency

5 distributions of these data for carriers from the nationwide inspection database are developed. If a carrier has had three or more inspections in the past two years, then the percentile ranking of the OOS rate for an individual carrier is mapped to the ISS inspection value scale. The mapping uses a linear scale such that a 50th percentile ranking gives an ISS inspection value of 80 and a 100th percentile ranking a value of 100. If the rate of prior inspections per power unit or per driver fall below the 50th percentile ranking, then a linear scale is used to add points to the inspection value. The percentile rankings by carrier size are stratified by carrier size. The consideration of carrier size makes it more likely that carriers with low rates of inspection relative to their peers will be recommended for inspection. If a carrier has had two or fewer inspections during the past two years, then an inspection value ranging from 88 to 100 is assigned as a function of carrier size group. For Wisconsin the primary interest is in determining if a statistically reliable model can be developed to identify carriers that are more likely to generate a serious safety violation, as indicated by an out-of-service (OOS) violation, during a future roadside inspection. Consequently, in contrast to the interstate ISS, the analysis of OOS violations and the resulting recommendation for inspection will be separated from the question of the adequacy of the rate of inspections. Also, no initial assumption will be made about the minimum number of inspections per carrier needed for model development. The initial model to be tested for Wisconsin is that the probability of a roadside inspection resulting in an OOS violation can be estimated with a logistics regression model: Pr( OOS = 1) = exp( g(X) ) / (1 + exp( g(X) ) where g(X) = a0 + a1 * x1 + ... + an * xn (1)

6 and the xi are the independent variables such as carrier OOS rates, carrier safety violation rates and carrier size. The model is calibrated with base data for one or more years and then applied in a predictive mode to estimate the success rate of predicting actual OOS violations for various probability thresholds. DATA AVAILABLE The primary data available for this research are historical roadside inspection data for intrastate motor carriers in Wisconsin for the years 1996 through the third quarter of 1999. The data file covers 12,135 inspections of intrastate vehicles for the nearly four-year period. Since USDOT numbers are not available, in general, as an identifier for intrastate carriers in Wisconsin, each Wisconsin intrastate carrier that has had a roadside inspection is assigned a unique identifier, the Wisconsin S number. The inspection file includes the name of the carrier, the date of the inspection, the number of OOS violations and the number of non-OOS violations found, and other inspection related data items. No information is available on carrier size. Data on carrier size can potentially be derived from data on truck registrations in Wisconsin and the database of carrier names with the associated S number. For this research data on 53,628 trucks registered in Wisconsin were available together with a list of 14,699 intrastate carriers that had been assigned S numbers. Based on an initial count of unique carrier names in the truck registration database, about 24,000 intrastate carriers may operate in Wisconsin. Because the carrier names in the truck registration database are not unique, that is, the same carrier may have multiple names, matching the carrier names from the S number database with the number of trucks registered required some manual intervention. Consequently, a test sample of carriers with carrier names beginning with A or B was selected from the carrier names

7 database to obtain carrier size data (number of trucks registered with that or a similar carrier name). Carrier size data were generated for 636 intrastate carriers that had S numbers. MODEL DEVELOPMENT AND VALIDATION Logistics Regression Model The intrastate roadside inspection database has 10,166 inspections for the three years from 1996 to 1998 and 1,969 inspections for the first nine months of 1999. The data for the first three years are used for calculation of the basic carrier variables, which are the OOS violation rates and non-OOS violation rates per carrier, and for model parameter estimation. The data for 1999 are then used for model validation. The simplest logistics regression model involves only a single explanatory variable, the carrier OOS rate for 1996 to 1998. Here the carrier OOS rate is defined as the proportion of the carriers inspections for which the vehicle or driver was found to have a major safety violation and consequently was placed out-of-service. The model estimates the probability that a specific inspection resulted in an OOS violation. The dependent variable for input to the model, thus, is the binary variable for the result of the inspection (=1, if have one or more OOS violations and =0 otherwise). The maximum likelihood estimators for the constant term and the coefficient of the OOS rate were highly statistically significant and the model generated a high degree of association between the predicted probabilities and the actual OOS conditions with agreement 95% of the time. The application of the logit model based on carrier OOS rate to the 1999 inspection data involved simply estimating the probabilities of OOS violations and then calculating the success rate for various probability thresholds. The logit model was used to calculate the probability for each inspection that the inspection would result in an OOS violation. The model was applied to

8 the 914 vehicles that were actually inspected in 1999 and also had a prior inspection history (at least one inspection during 1996 to 1998). As shown in Table 1, probability thresholds of 0.8, 0.6 and 0.4 were applied to estimate the level of success in predicting which vehicles should be selected from the total available set of 914 vehicles. The overall OOS rate for the 914 vehicles was 28.4 percent. With a probability threshold of 0.8, the logit model identified 186 vehicles for inspection (predicted probability of OOS of 0.8 or more). Of these vehicles 81 or 43.6 percent were actually found to be OOS. For a lower threshold of 0.6 a total of 195 vehicles were selected with the same success rate of 43.6 percent, while for a threshold of 0.4, the 296 vehicles selected had a success rate of 39.9 percent. Thus, use of the simple logit model as a screening device would allow an inspector to target carriers for inspection that are more likely to have serious safety violations. The other simple model that could easily be tested on the 1996 to 1998 database was to add non-OOS violations as an independent variable. The problem here is that there was a high correlation of 0.79 between the OOS violation rate and the non-OOS violation rate. Thus, the two variables are not independent and, in general, should not be used in the same model. When the two variables were included in the same logit model, the coefficient for the non-OOS violation rate was not statistically significant and also had the wrong sign. One additional model that was tested for statistical significance involved carrier size as measured by the number of vehicles registered by a carrier. Because of the generation of the carrier size variable was, in part, a manual process, a carrier size database was developed for only a small sample of the intrastate carriers in Wisconsin. The result was carrier size data for 642 inspections during 1996 to 1998.

9 The OOS violation rates by carrier size for 1996 to 1998 are shown in Table 2. Except for the two largest vehicle size classes (carriers with 15 or more vehicles), which have substantially below average OOS rates, the carrier size data do not show a consistent pattern in OOS violation rates. Thus, there is little evidence that carrier size would be statistically significant as an independent variable in a logit model. Estimation of a logit model with OOS rate and carrier size as independent variables showed that the coefficient for carrier size was not statistically significant. Separate consideration of the two largest vehicle size classes is not warranted because together they only account for 7.5 percent of the inspections in the sample. Direct Estimation Model While the logistic regression model is useful to evaluate the potential contribution of alternative independent variables in predicting the probability of an OOS violation, a very simple direct estimation model can be used when the prior OOS record, the carrier OOS rate, is the only relevant independent variable. To apply the model, the carrier OOS rate for 1996 to 1998 is added to the records for the 914 inspections in 1999 for which the carrier OOS rate can be calculated. The carrier OOS rate then provides an estimate of the probability that a particular inspection will result in an OOS violation. The direct estimation model is applied by specifying a probability threshold for selecting vehicles for inspection. The success rates for the direct application model are shown in the top part of Table 3 (Binary OOS Rate). The success rates are essentially the same as for the logit model application with slightly different probability thresholds. For example, the logit model threshold of 0.4 gave a success rate of 39.9 percent (118 OOS from 296 inspections) while the direct model threshold of 0.5 gives essentially the same results.

10 An alternative independent variable for measuring OOS is based on the sum of OOS violations (total OOS) for each inspection. The variable of interest is then the average total OOS for each carrier (sum of total OOS for all of the carriers inspections divided by the number of inspections). The success rates for the direct application of the Total OOS Rate variable are shown in the bottom half of Table 3. This variable generates substantially higher success rates for at least a small proportion of the total inspections. For a Total OOS Rate threshold of 1.50 the model selects 105 inspections with a success rate of 53 percent which is much higher than the 45 percent maximum success rate for the binary OOS rate based model. Clearly, the sum of OOS violations provides additional useful information that an inspector can use to improve the selection of vehicles for roadside inspections. The Total OOS Rate model, however, is only marginally better than the Binary OOS Rate model and then only when applied very selectively using a high OOS threshold. On balance, the Total OOS Rate model is the recommended model. Both the Binary OOS Rate and the Total OOS Rate models can also be developed based on rates for different prior time periods. The success rates for the two models considering historical inspection data for the two-year period, 1997 to 1998, are shown in Table 4. In comparison with the initial models for the three-year period, 1996 to 1998, (see Table 3) the twoyear model has somewhat fewer cumulative inspections in 1999, but the overall cumulative success rate is the same. The trends in cumulative success rate for the two-year models are essentially the same as for the three-year models, but the cumulative success rates for the two-year models are consistently just slightly lower. These results are consistent with the historical data on OOS inspection results upon which the models are based. The overall OOS rate for the 1996 inspections was 38.1 percent compared with 30.2 percent for the 1997 to 1998 inspections. When data for 1996 are excluded from the models, only a small number of inspections are deleted from

11 the relevant inspections for 1999, but these are inspections that on average in 1996 had substantially higher OOS rates. Clearly, the three-year models are based on more information and at least for the 1996 to 1998 time period generate higher cumulative success rates. Adequacy of the Rate of Inspections For carriers that have the same OOS history, in general, inspectors should give priority to sampling carriers that have lower than average inspections per vehicle and inspections per driver. For interstate carriers, data on the number of power units (vehicles) and number of drivers are available from the carriers application for the USDOT number. No comparable data source is available in Wisconsin for intrastate carriers. Nevertheless, the sample data on carrier size (number of registered vehicles) for Wisconsin can provide an initial estimate of possible biases in Wisconsins current inspection program for intrastate carriers. The sample data on vehicles per intrastate carrier were merged with the 1996 to 1998 intrastate inspection data, which already had the number of inspections per carrier as a data item. The result was 642 inspections for which the inspections per vehicle could be calculated. The cumulative distribution of inspections per vehicle is shown in Table 5. The median number of inspections per vehicle is about 0.8 for the three year time period. This rate is substantially greater than the median rates for interstate carriers ranging from 0.27 to 0.38 that are reported in the ISS Final Report (1). The interstate rates should be increased by 50 percent since they only cover a two year time period. The adjusted interstate rates are still more than 25 percent below Wisconsins median rate. The overall distribution of Wisconsins inspections per vehicle rate, however, is reasonable and consistent with the distributions that are stratified by carrier size in the ISS Final Report. IMPLEMENTATION METHODOLOGY

12 The direct estimation model based on the Total OOS Rate can be implemented in two ways. First, the model can be used by inspectors in the field to identify vehicles that are more likely to generate OOS violations. Second, the model can be applied to the list of carriers that have an inspection history for the 1996 to 1998 time period to identify the carriers that have the worst safety record as measured by total OOS rates. The model is relatively simple to implement in the field. All that is needed is to provide inspectors with the ability to access the carrier record for a vehicle that is a candidate for a field safety inspection. The carrier record would contain the Total OOS Rate variable which the inspector would compare with the tabular values found in the bottom half of Table 3. The inspector would decide what probability of success in identifying an OOS violation was appropriate given the resources available for inspections for that day. Operationally, the inspector would enter the license plate number for a vehicle that is a candidate for a field inspection in the online computer system that is available in all inspection and weight facilities in Wisconsin. The license plate record would contain a link to the carrier inspection record and the associated Total OOS Rate. The tabular values for the Success Rate as a function of the Total OOS Rate can be updated quarterly or semiannually based on the most recent historical inspection data for intrastate carriers in Wisconsin. SUMMARY AND CONCLUSIONS Logistics regression (logit model) was used to identify the independent variables that are statistically significant in explaining the probability that a roadside safety inspection results in an out-of-service (OOS) safety violation. Models using either Binary OOS Rate or the Total OOS Rate for intrastate carriers as independent variables provided reasonable explanatory power with

13 nearly equal goodness-of-fit. The two other independent variables tested, non-OOS Violation Rate and carrier size were not found to be statistically significant. An alternative to the logit model, direct estimation, was found to provide success rates in predicting inspections that result in an OOS violation that were equal to the logit model results for the Binary OOS Rate independent variable. Direct estimation produced even higher success rates for the Total OSS Rate independent variable. The direct estimation model is also preferred because of the ease of application compared to the logit model. The Total OOS Rate direct estimation model is the preferred model although the advantage over the Binary OOS Rate model only applies to very highest OOS Rate classes. For 1999 these high OOS rate classes only accounted for 12 percent of the inspections. A full analysis of the adequacy of the rate of intrastate inspections in Wisconsin was not possible because of the lack of carrier size data for all carriers. Analysis of a small sample of carrier size data indicated that the median rate of inspections per vehicle for carriers in Wisconsin is somewhat higher than that for interstate carriers nationally, but the overall distribution is reasonable. This issue cannot be fully addressed in Wisconsin until a comprehensive intrastate carrier database is developed that would include carrier size. The direct estimation model can be implemented both as a field inspection tool and as a means to generate a list of intrastate carriers ranked by their historical OOS record. The model can easily be updated to include the latest historical OOS data. ACKNOWLEDGMENTS This research was funded by a grant from the Wisconsin Department of Transportation, Division of State Patrol. The authors would like to thank the staff of the Wisconsin State Patrol and the Wisconsin DOT for their support of the research. The analysis and results presented in

14 this report represent the views of the authors and do not necessarily represent the views or policies of the Wisconsin Department of Transportation and the Wisconsin State Patrol. REFERENCES 1. The Inspection Selection System (ISS). Final Report. The Upper Great Plains Transportation Institute, North Dakota State University. Fargo, North Dakota, September 1996. 2. SafeStat - Motor Carrier Safety Status Measurement System. Methodology: Version 8. John A. Volpe National Transportation Systems Center for the Federal Motor Carrier Safety Administration. Washington, D. C., April 2000. 3. Lantz, Brenda A.. ISS-2: The Integration of the Motor Carrier Safety Status Measurement System (SafeStat) into the Roadside Inspection Selection System (ISS). Final Report. The Upper Great Plains Transportation Institute, North Dakota State University. Fargo, North Dakota, January 2000.

15 TABLE 1 Results of Carrier OOS Rate Logit Model Application to 1999 Inspections Probability Threshold Number of Vehicles Selected for Inspection 914 296 195 186 Actual OOS 260 118 85 81 Actual OOS Rate for 1999 (Percent) 28.4 39.9 43.6 43.6

0.0 0.4 0.6 0.8

16 TABLE 2 Inspection Out-of-Service (OOS) Rates By Carrier Size Carrier Size 1 2 3 4 5 6 7 8 9 10 - 14 15 - 19 20+ All Number of Inspections 151 82 41 51 49 67 41 12 52 48 17 31 642 Cum. % 23.5 36.3 42.7 50.6 58.3 68.7 75.1 77.0 85.1 92.5 95.2 100.0 Inspection OOS Rate 0.36 0.37 0.29 0.35 0.39 0.30 0.32 0.17 0.38 0.46 0.24 0.13 0.34

17 TABLE 3 Results of the Direct Estimation Model based on the Binary OOS Rate Class and the Total OOS Rate Class OOS Rate Class 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 1.0 0.00 0.01 0.20 0.40 0.60 0.80 1.00 1.50 2.00 3.00 4.00 5.00 6.00 Count Number Cumulative of OOS Inspections Inspections Binary OOS Rate 439 99 914 33 7 475 56 14 442 58 12 386 33 10 328 101 33 295 27 12 194 13 4 167 3 3 154 151 66 151 Total OOS Rate 434 96 914 28 8 480 70 16 452 92 25 382 35 7 290 22 6 255 128 46 233 24 11 105 43 25 81 22 10 38 7 5 16 2 1 9 7 4 7 Cumulative OOS Inspection 260 161 154 140 128 118 85 73 69 66 260 164 156 140 115 108 102 56 45 20 10 5 4 Cumulative Success Rate 0.28 0.34 0.35 0.36 0.39 0.40 0.44 0.44 0.45 0.44 0.28 0.34 0.35 0.37 0.40 0.42 0.44 0.53 0.56 0.53 0.63 0.56 0.57

18

TABLE 4 Results of the Direct Estimation Model Using Inspection Results from 1997 and 1998 OOS Rate Class Count Number of Cumulative Cumulative Cumulative OOS Inspections OOS Success Inspections Inspection Rate

Binary OOS Rate 0.00 428 0.10 7 0.20 27 0.30 36 0.40 66 0.50 96 0.60 4 0.70 20 0.80 10 0.90 2 1.00 143 Total OOS Rate 0.00 428 0.20 21 0.40 64 0.60 76 0.80 22 1.00 131 1.50 18 2.00 27 3.00 33 4.00 11 5.00 4 6.00 4

101 2 5 5 19 26 4 9 7 0 58 101 6 13 17 11 38 8 15 17 7 1 2

839 411 404 377 341 275 179 175 155 145 143 839 411 390 326 250 228 97 79 52 19 8 4

236 135 133 128 123 104 78 74 65 58 58 236 135 129 116 99 88 50 42 27 10 3 2

0.28 0.33 0.33 0.34 0.36 0.38 0.44 0.42 0.42 0.40 0.41 0.28 0.33 0.33 0.36 0.40 0.39 0.52 0.53 0.52 0.53 0.38 0.50

19

TABLE 5 Cumulative Distribution of the Number of Inspections per Vehicle For Intrastate Carriers in Wisconsin

Number of Inspections Per Vehicle 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.2 1.3 1.5 1.7 1.9 2.0 2.1 2.2 2.5 3.0 4.0 5.0 All

Count of Inspections 2 16 58 67 29 71 28 29 17 28 124 6 5 15 5 17 34 19 13 5 36 8 10 642

Cumulative Percent 0.3 2.8 11.8 22.3 26.8 37.8 42.2 46.7 49.4 53.7 73.0 74.0 74.8 77.1 77.9 80.5 85.8 88.8 90.8 91.6 97.2 98.4 100.0