How robust are Structural Equation Models to model miss-specification? A simulation study

Structural Equation Models (SEMs) are routinely used in the analysis of empirical data by researchers spanning different scientific fields such as psychologists or economists. In some fields, such as in ecology, SEMs have only started recently to attract attention and thanks to dedicated software packages the use of SEMs has steadily increased. Yet, common analysis practices in such fields that might be transposed from other statistical techniques such as model acceptance or rejection based on p-value screening might be poorly fitted for SEMs especially when these models are used to confirm or reject hypotheses. In this simulation study, SEMs were fitted via two commonly used R packages: lavaan and piecewiseSEM. Five different data-generation scenarios were explored: (i) random, (ii) exact, (iii) shuffled, (iv) underspecified and (v) overspecified. In addition, sample size and model complexity were also varied to explore their impact on various global and local model fitness indices. The results showed that not one single model index should be used to decide on model fitness but rather a combination of different model fitness indices is needed. The global chi square test for lavaan or the Fisher's C statistic for piecewiseSEM are, in isolation, poor indicators of model fitness. Similarly, AIC and BIC values could not discriminate between overfitting, pure noise and exact models in piecewiseSEM. Combining the different metrics explored here provided few safeguards against model overfitting, this emphasizes the need to cautiously interpret the inferred (causal) relations from fitted SEMs. I provide, based on these results, a flowchart indicating how informations from different metrics may be combined to reveal model strength and weaknesses. Researchers in scientific fields with little experience in SEMs, such as in ecology, should consider and accept these limitations.