Multilevel confirmatory factor analysis: Evaluating its surplus value when constructing school process variables in secondary education

DHaenens,
1, Ellen

Van Damme,

1, Jan

& Onghena,

2 Patrick

(Ellen.DHaenens@ped.kuleuven.be) 1 Centre for Educational Effectiveness and Evaluation, K.U.Leuven, Belgium 2 Centre for Methodology of Educational Research, K.U.Leuven, Belgium

Problem statement => Research goal

Within educational effectiveness literature: Construction of school process variables to explain student achievement School process variables = Variables about the internal functioning of the school, for example gathered by means of a teacher questionnaire Construction of these variables (to date): 1. Standard factor analysis => scale or factor scores for each teacher 2. Aggregation => mean score per school Problem: Are the same variables that are relevant at the teacher level, also relevant at the school level?

Goal = to construct school process variables by means of: 1. Standard confirmatory factor analysis at the teacher level, without taking into account the nesting within cycles (CFA) 2. Multilevel confirmatory factor analysis at the teacher level and at the school group level (MCFA) Questions: Do we find different measures for school process variables, depending on the method of analysis used? Are different variables relevant at the different levels?

Method: Data
LOSO-project: Longitudinal Research in Secondary Education School Characteristics Questionnaire for Teachers 1550 teachers from 113 school groups 21 items (rated on a 4- or 5-point scale): Concerning school organization (cooperation among teachers + decision making process)

Results
Standard CFA = MCFA at the teacher level: 6 latent variables MCFA at the school group level: different competing models were tested (see Table: MODEL FIT) 3 latent variables at the school group level = best alternative, although improvement is still possible 6 latent variables at the school group level are NOT significantly better than 3 latent variables
MODEL FIT
Competing models at the school group level
T 6 LV's; SG 2 LV's T 6 LV's; SG 3 LV's T 6 LV's; SG 4 LV's T 6 LV's; SG 6 LV's T 6 LV's; SG 3 LV's + adjustments

1253.893 1052.954 1052.338 1035.932 1006.980

df
385 383 380 370 377

/df
3.257 2.749 2.769 2.800 2.671

p
-

RMSEA total
0.038 0.034 0.034 0.034 0.033

RMSEA school
0.167 0.128 0.130 0.133 0.120

SRMR teacher
0.042 0.041 0.041 0.041 0.041

SRMR school
0.263 0.192 0.189 0.169 0.178

409.244 <.001 0.614 11.183 ns ns

36.691* <.001

Note. T = Teacher; SG = School Group; LV = Latent Variable. Chi-square difference test in comparison with previous model, except *. * = Chi-square difference test in comparison with 3 LV's. ns = not significant.

Conclusion
Different latent variables are relevant at the different levels: More specific variables at the teacher level (CFA + MCFA) More global variables at the school group level (MCFA) Thus, different measures for school process variables, depending on the method used

Multilevel factor analysis = strongly advised when confronted with nested data!
BUT, exploratory variant would be a better alternative than confirmatory analysis

Acknowledgements: The study reported on this poster was made possible by Research Grant 3H051166 of the Research Foundation Flanders.