2.1 Free

Free parameters are estimated by the model according to the data. That is, they are estimated based by fitting the model to the data to achieve a criterion (e.g., OLS in regression).

2.2 Fixed

Fixed parameters are constants, often specified by you (the scientist).

2.3 Constrained

Constrained parameters specify a required relationship between two variables. For example, x = 2y. Or one could require that two factor loadings be equal. One can also specify that a parameter must fall within a certain range (e.g., -2 – 2) using inequality constraints.

grViz("digraph regression {
graph [ bgcolor=transparent]

forcelabels=true; splines=line;

node [shape = box, fontcolor=gray25 color=gray80]

node [fontname='Helvetica']
{rank=same; Anger_M [label=<Anger<sub>p1</sub>>]; node [fillcolor=gray90 style=filled] Sadness_M [label=<Sadness<sub>p1</sub>>]; }

{rank=same; Anger_F [label=<Anger<sub>p2</sub>>]; node [fillcolor=gray90 style=filled] Sadness_F [label=<Sadness<sub>p2</sub>>]; }

edge [color=gray50 style=filled]
{ rank=same; Anger_M -> Sadness_M [label='a'] }
{ rank=same; Anger_F -> Sadness_F [label='a'] }
Anger_M -> Sadness_F [label='b']
Anger_F -> Sadness_M [label='b']
}")

3.1 Endogenous

Endogenous variables have at least one cause (predictor) in the model. That is, their values depend on something that is modeled. In standard regression, we might call them dependent variables. In a path diagram, endogenous variables have at least one incoming arrow.

3.2 Exogenous

Exogenous variables are not caused (predicted) by other variables in the model.

4.1 Direct effect

Direct effects represent the relationship between a predictor and an outcome (endogenous variable) that are not mediated by any other variable (Bollen, 1987). This could mean either ignoring potential mediating variables (i.e., they are not included in the model) or including them. In causal mediation analysis, the direct effect represents the effect of an independent variable (e.g., treatment) on an outcome (e.g., depression) holding a candidate mediator (e.g., ruminative thoughts) constant at a level that would occur under the corresponding level of the treatment (Imai definition).

grViz("digraph regression {
graph [rankdir = LR bgcolor=transparent]

forcelabels=true;

node [shape = box, fontcolor=gray25 color=gray80]

node [fontname='Helvetica']
X;

node [fillcolor=gray90 style=filled]
Y;

edge [color=gray50 style=filled]
X -> Y [label='Direct']
}")

grViz("digraph regression {
graph [rankdir = LR bgcolor=transparent]

forcelabels=true;

node [shape = box, fontcolor=gray25 color=gray80]

node [fontname='Helvetica']
X; M;

node [fillcolor=gray90 style=filled]
Y;

edge [color=gray50 style=filled]
X -> Y [label='Direct']
X -> M [label='Indirect (a)']
M -> Y [label='Indirect (b)']
}")

4.2 Indirect effect

Indirect effects represent the effect of a predictor on an outcome via one or more intervening (mediating) variables. Thus, the idea is that the outcome is related to a predictor because the predictor influences a mediating variable, which in turn influences the outcome.

grViz("digraph regression {
graph [rankdir = LR bgcolor=transparent]

forcelabels=true;

node [shape = box, fontcolor=gray25 color=gray80]

node [fontname='Helvetica']
X; M;

node [fillcolor=gray90 style=filled]
Y;

edge [color=gray50 style=filled]
X -> Y [label='Direct']
X -> M
M -> Y
}")

grViz("digraph regression {
graph [rankdir = LR, bgcolor=transparent, layout=dot]

forcelabels=true;

node [shape = box, fontcolor=gray25 color=gray80]

node [fontname='Helvetica']
X; M1; M2;

node [fillcolor=gray90 style=filled]
Y;

edge [color=gray50 style=filled]
X -> Y [xlabel='e']
X -> M1 [label='a']
M1 -> M2 [label='b']
M2 -> Y [label='c']
M1 -> Y [label='d']
}")

4.3 Total effect

The total effect of \(X\) on \(Y\) is the sum of the direct effect and the total indirect effects. Note: Thus, the total indirect effect can also be conceptualized as the total effect minus the direct effect.

4.4 Common cause

Two variables can be related to each other because they were caused by a common third variable. For example, \(Y_1\) and \(Y_2\) may be related because they were both influenced by \(X\).

grViz("digraph regression {
graph [rankdir = LR bgcolor=transparent]

forcelabels=true;

node [shape = box, fontcolor=gray25 color=gray80]

node [fontname='Helvetica']
X;

node [fillcolor=gray90 style=filled]
Y1; Y2;

edge [color=gray50 style=filled]
X -> Y1;
X -> Y2;

}")

4.5 Common outcome

As we’ll return to when discuss causality, a variable can also be a common outcome of two (potentially independent) causes. In the causality world, this is also called an inverted fork.

grViz("
digraph g {
  # a 'graph' statement
  graph [overlap = true, fontsize = 12]

  # nodes for observed and latent
  node [shape = circle, fontname = Arial, width=1, height=1]
  Flu Lyme Fever

  edge [fontname = Arial]
  Flu -> Fever
  Lyme -> Fever
}")

5.1 RAM (graphical) notation

5.2 Simplified RAM notation

For simplicity, we can remove the distinction between exogenous variances and endogenous disturbances using a single inward-pointing arrow to denote both constructs:

5.3 LISREL all-y notation

In this class, we will typically adopt the ‘all-y’ LISREL notation in which the parameter matrices do not distinguish between exogenous and endogenous variables.

• \(\mathbf{y}_i\): A vector of observed values for the ith variable.
• \(\boldsymbol{\eta}_i\): A vector of estimated values of the ith latent variable. These are also referred to as factor scores.
• \(\boldsymbol{\Lambda}_y\) or \(\lambda_{1,1}\): Factor loadings of the observed variables \(\mathbf{Y}\) onto factors \(\boldsymbol{\eta}\). Also conceptualized as regression slopes (observed regressed on latent). Note that if expressed by specific subscripts (e.g., \(\lambda_{1,1}\)), the first subscript is the observed variable (\(y_{1...j}\)) and the second subscript is the factor (\(\eta_{1...k}\)). Thus, \(\mathbf{\Lambda}\) is \(j \times k\) in size.
• \(\mathbf{\Theta_\varepsilon}\) or \(\mathbf{\varepsilon}_i\): The variance-covariance matrix for all observed variables. Also includes covariances among errors. The diagonal will contain error variances for endogenous observed variables.
• \(\boldsymbol{\zeta}_i\): vector of disturbances in endogenous variables (i.e., residual variability due to unmeasured causes).
• \(\boldsymbol{\Psi}\): covariance matrix among latent variables. Note that the disturbances for endogenous variables fall along the diagonal.
• \(\mathbf{B}\): The matrix of regression coefficients.

5.4 lavaan notation for SEM matrices

The lavaan package largely uses LISREL notation in naming and structuring the underlying matrices in a SEM. More specifically, it uses the ‘all-Y’ notation in which there are not separate matrices for endogenous and exogenous variables (see Kline Appendix 10.A). Thus, for example, all structural regression coefficients are represented in the \(\mathbf{B}\) matrix, rather than separating it into exogenous (\(\mathbf{\Gamma}\)) and endogenous (\(\mathbf{B}\)) predictors. The parameter specification and notation of lavaan can be seen by examining the output of the inspect function on a fitted model object.

## The industrialization and Political Democracy Example 
## Bollen (1989), page 332
model <- ' 
  # latent variable definitions
     ind60 =~ x1 + x2 + x3
     dem60 =~ y1 + a*y2 + b*y3 + c*y4
     dem65 =~ y5 + a*y6 + b*y7 + c*y8

  # regressions
    dem60 ~ ind60
    dem65 ~ ind60 + dem60

  # residual correlations
    y1 ~~ y5
    y2 ~~ y4 + y6
    y3 ~~ y7
    y4 ~~ y8
    y6 ~~ y8

#regression of latent vars on observed
ind60 ~ y4
'

fit <- sem(model, data=PoliticalDemocracy)
#summary(fit, fit.measures=TRUE)
semPaths(fit)

inspect(fit)

## 
## Note: model contains equality constraints:
## 
##   lhs op rhs
## 1   3 ==   6
## 2   4 ==   7
## 3   5 ==   8
## 
## $lambda
##    ind60 dem60 dem65 y2 y4 y6 y8
## x1     0     0     0  0  0  0  0
## x2     1     0     0  0  0  0  0
## x3     2     0     0  0  0  0  0
## y1     0     0     0  0  0  0  0
## y2     0     0     0  0  0  0  0
## y3     0     4     0  0  0  0  0
## y4     0     0     0  0  0  0  0
## y5     0     0     0  0  0  0  0
## y6     0     0     0  0  0  0  0
## y7     0     0     7  0  0  0  0
## y8     0     0     0  0  0  0  0
## 
## $theta
##    x1 x2 x3 y1 y2 y3 y4 y5 y6 y7 y8
## x1 19                              
## x2  0 20                           
## x3  0  0 21                        
## y1  0  0  0 22                     
## y2  0  0  0  0  0                  
## y3  0  0  0  0  0 24               
## y4  0  0  0  0  0  0  0            
## y5  0  0  0 12  0  0  0 26         
## y6  0  0  0  0  0  0  0  0  0      
## y7  0  0  0  0  0 15  0  0  0 28   
## y8  0  0  0  0  0  0  0  0  0  0  0
## 
## $psi
##       ind60 dem60 dem65 y2 y4 y6 y8
## ind60 30                           
## dem60  0    31                     
## dem65  0     0    32               
## y2     0     0     0    23         
## y4     0     0     0    13 25      
## y6     0     0     0    14  0 27   
## y8     0     0     0     0 16 17 29
## 
## $beta
##       ind60 dem60 dem65 y2 y4 y6 y8
## ind60     0     0     0  0 18  0  0
## dem60     9     0     0  0  0  0  0
## dem65    10    11     0  0  0  0  0
## y2        0     3     0  0  0  0  0
## y4        0     5     0  0  0  0  0
## y6        0     0     6  0  0  0  0
## y8        0     0     8  0  0  0  0

7.1 Number of observations

The number of ‘observations’ in SEM refers to the number of unique values in the covariance matrix. We’ve seen this equation before:

\[ p = \frac{k(k+1)}{2} \]

where \(p\) is the number of variance-covariance parameters and \(k\) is the number of variables. The number of observations provides crucial information about how complex a SEM we can test given the number of observed variables.

The number of free parameters in a SEM is denoted \(q\), and the degrees of freedom for the overall model is the difference between \(p\) and \(q\):

\[ df_\textrm{M} = p - q \]

Thus, as we’ve discussed, the goal is to develop a parsimonious model with fewer free parameters than observations (covariance values). If two models fit the data equally well, the model with more degrees of freedom should be preferred as more parsimonious. We will return to this issue when considering relative model evidence.

7.2 Overidentified

A model with \(df_\textrm{M} > 0\) is called overidentified, meaning that there are more equations than unknowns. Thus, there are many parameter sets one could test, and all will be imperfect. Given a model parameterization (i.e., what matrices are included and what the free parameters are), the goal is to identify parameters that best fit the data. In an overidentified model, the idea is to adjudicate among parameter values based on a fit criterion. In SEM, this is usually the sample likelihood function, as mentioned above.

7.3 Underidentified

A model with \(df_\textrm{M} < 0\) is called underidentified, meaning that there are too man unknown parameters to solve for and no way to solve the equations.

\[ a + b = 6 \]

There are an infinite number of values for \(a\) and \(b\) that would solve this equation.

7.4 Just-identified

A model with \(df_\textrm{M} = 0\) is called just-identified, and by definition, it will fit the data perfectly. The idea is that there is one set of parameters in a just-identified model that reproduces the covariance matrix. But there are many alternative just-identified models one could test in a given dataset, but all fit the data equally well, making hypothesis testing difficult.

7.5 Aside: Nonrecursive Models

From Michael Clark

Recursive models have all unidirectional causal effects and disturbances are not correlated. A model is considered nonrecursive if there is a reciprocal relationship, feedback loop, or correlated disturbance in the model. Nonrecursive models are potentially problematic when there is not enough information to estimate the model (unidentified model), which is a common concern with them.

A classic example of a nonrecursive relationship is marital satisfaction: the more satisfied one partner is, the more satisfied the other, and vice versa (at least that’s the theory). This can be represented by a simple model (below).

Such models are notoriously difficult to specify in terms of identification, which we will talk more about later. For now, we can simply say the above model would not even be estimated as there are more parameters to estimate (two paths, two variances) than there is information in the data (two variances and one covariance).

8.1 Notation

Path analysis is largely an extension of linear regression with observed variables to the multivariate case. That is, in path analysis, there are typically multiple predictors and multiple outcome variables. Furthermore, chained relationships (e.g., \(X \rightarrow M \rightarrow Y\)) lead to the possibility of indirect effects and the potential for a given variable to be endogenous (i.e., its values depend on other variables in the model) and to predict another endogenous variable.

Conceptually, we can think of path analysis in terms of a set of regression equations (Kline Appendix 6.A):

\[ \begin{align} Y_1 &= \gamma_{11} X_1 + \beta_{12} Y_2 + \zeta_1 \\ Y_2 &= \gamma_{22} X_2 + \beta_{12} Y_1 + \zeta_2 \end{align} \] We could re-express this in matrix form as:

\[ \mathbf{Y} = \mathbf{\Gamma X} + \mathbf{B Y} + \boldsymbol{\zeta} \]

In addition, there may be associations among exogenous variables (in \(\boldsymbol{\Phi}\)) or endogenous variables (in \(\boldsymbol{\Psi}\)).

\(\boldsymbol{\Phi}\) contains correlations among measured exogenous variables.

\[ \boldsymbol{\Phi}=\begin{bmatrix} \phi_{11} & \\ \phi_{21} & \phi_{22} \end{bmatrix} \]

\(\boldsymbol{\Psi}\) contains correlations among the disturbances for endogenous variables.

\[ \boldsymbol{\Psi}=\begin{bmatrix} \psi_{11} & \\ \psi_{21} & \psi_{22} \end{bmatrix} \]

Recall that in LISREL all-y notation, we eliminate the need to distinguish between exogenous and endogenous variables in the formal matrix representation, though these are nevertheless very important conceptual distinctions.

\[ \boldsymbol{\Phi}=\begin{bmatrix} \phi_{11} & \\ \phi_{21} & \phi_{22} \end{bmatrix} \]