1 Calculating ROM effect size

The following R code calculates two effect sizes (Hedges’ g and ROM), compares their distributions and heterogeneity, and performs meta-analyses on the dat_ROM dataset which includes measurements of target-like use (see Foster & Wigglesworth, 2016) as a percentage-based measure of accuracy (e.g., of using past tense) in learners’ writing tasks between two groups (treatment and control) at a post-test across experimental studies.

As we shall see further below, ROM is preferred over Hedges’ g due to its less skewed distribution and smaller heterogeneity ($I^2$). A subsequent meta-analysis with ROM will then reveal that the treatment group shows a $43.9\%$ improvement over the control group when interpreted via $ROM\%$ (see manuscript). The following code describes all these steps.

1.1 Software, Data, and Effect Size Calculation

The following R code loads the metafor package, a tool for conducting meta-analyses in R. It provides functions for calculating effect sizes (escalc) and performing statistical analyses:

source("https://t.ly/TVmIg")

The next code calculates two types of effect sizes for each row in dat_ROM dataset that comes with the software. The output will include natural-log-transformed Ratio of Means ("ROM") and Hedges’ g ("SMD"). Both effect sizes use means of treatment and control groups (mT, mC), standard deviations (sdT, sdC), and sample sizes (n) here being equal for the treatment and control groups:

dat <- escalc(
  data = dat_ROM, measure = "ROM", var.names = c("ROM", "v_ROM"),
  m1i = mT, 
  m2i = mC, 
  sd1i = sdT, 
  sd2i = sdC,
  n1i = n, 
  n2i = n)

dat <- escalc(
  data = dat, measure = "SMD", var.names = c("g", "v_g"),
  m1i = mT, 
  m2i = mC, 
  sd1i = sdT, 
  sd2i = sdC,
  n1i = n, 
  n2i = n)

The var.names parameter assigns new column names for the calculated effect sizes (ROM, g) and their sampling variances (v_ROM, v_g).

1.2 Visualize Distributions of Effect Sizes

To understand the characteristics of the calculated effect sizes, histograms are created for both Hedges’ g and ROM. These plots show the distributions and help assess their suitability:

hist(dat$g, main = "Histogram of Hedges' g", xlab = "Hedges' g")

hist(dat$ROM, main = "Histogram of ROM", xlab = "ROM")

As shown above, the histogram for Hedges’ g shows much more skewness. But ROM tends to be less skewed, indicating a more stable and interpretable measure.

1.3 Perform a Diagnostic Meta-Analysis

The next code performs random-effects meta-analyses on both Hedges’ g and ROM to calculate pooled effect sizes and heterogeneity ($I^2$):

# Meta-analysis for Hedges' g
g_meta <- rma(g, vi = v_g, data = dat)

# Meta-analysis for ROM
ROM_meta <- rma(ROM, vi = v_ROM, data = dat)


g_meta$I2 # I2 for Hedges' g

## [1] 74.95044

ROM_meta$I2 # I2 for ROM

## [1] 60.18319

As can be seen, the heterogeneity measure $I^2$ (expressed in percentage) is larger for Hedges’ g. This supports the choice of ROM as the preferred effect size. Because when choice of effect size alone (a methodological decision) changes the amount of hetrogenity, such heterogeneity is likely artifactual rather than substantive.

1.4 Interpret ROM as Percentage Difference

The pooled ROM is interpreted as the percentage difference between the treatment and control groups using a transformation function (pct_dif_tran).

post_rma(ROM_meta, tran = pct_dif_tran, type = "response")

##         1 Response    SE   Df  Lower  Upper      z p-value Sig.
## 1 overall   43.880 2.325  Inf 39.395 48.511 22.512   0.000  ***

The post_rma() function (see Norouzian & Bui, 2024) applies the transformation (pct_dif_tran) to the ROM effect size (see manuscript). This converts the pooled ROM into a percentage difference, making it easier to interpret. Based on the output, the treatment groups performed about $43.9\%$ better than the control groups.

2 Calculating DiD effect sizes

The next R code is calculating within-group standardized mean change effect sizes for two groups (treatment and control), then combining these to compute the difference in effect sizes between the groups for use in a meta-analytic model. Here’s what each section of the code does.

2.1 Load Required R programs

source("https://t.ly/TVmIg")

Loads the metafor package, a tool for conducting meta-analyses in R. It provides functions for calculating effect sizes (escalc) and performing statistical analyses. It also loads dat_quasi (see manuscript), a dataset used for demonstrating DiD effect size.

2.2 Calculate Effect Sizes for the Treatment Group

datT <- escalc("SMCRP", 
               ni=n, 
               m1i=mpost, 
               m2i=mpre, 
               sd1i=sdpre, 
               sd2i=sdpost, 
               ri=r, 
               data=subset(dat_quasi, Group=="T"))

Purpose: Computes the within-group standardized mean change effect size (SMCRP) for the treatment group.
ni: Sample size of the group.
m1i: Mean at post-test.
m2i: Mean at pre-test.
sd1i: Standard deviation at pre-test.
sd2i: Standard deviation at post-test.
ri: Correlation between pre-test and post-test scores.
data=subset(dat_quasi, Group=="T"): Filters dat_quasi to include only rows where the Group is “T” (treatment group).

The result (datT) includes two key outputs: yi: The calculated effect size for each study. vi: The variance of the effect size.

2.3 Calculate Effect Sizes for the Control Group

datC <- escalc("SMCRP", 
               ni=n, 
               m1i=mpost, 
               m2i=mpre, 
               sd1i=sdpre, 
               sd2i=sdpost, 
               ri=r, 
               data=subset(dat_quasi, Group=="C"))

Purpose: Same as above, but for the control group. The subset is filtered for Group == "C" (control group).

2.4 Compute DiD Effect Sizes

dat <- data.frame(DiD = datT$yi - datC$yi, 
                  vi = datT$vi + datC$vi, 
                  time_interval = datT$time)

Purpose: Creates a new data frame that combines the treatment and control effect sizes to calculate the DiD effect sizes. DiD = datT$yi - datC$yi: Computes the difference in within-group effect sizes between the treatment and control groups. vi = datT$vi + datC$vi: Adds the variances of the treatment and control groups to compute the variance of the between-group effect size. time_interval = datT$time: Copies the time interval information from the treatment group.

3 Intact class adjustment

The last R code adjusts the bias in effect size estimates (Hedges’ g) and specifically their sampling variability in studies that used assignment by intact classes, but failed to account for its effect in their analysis. Here’s a step-by-step explanation of the code.

3.1 Load the Adjustment Functions

source("https://t.ly/TVmIg")

The line above loads the custom functions g_cluster() and g_vi_cluster() from Norouzian & Bui (2024) 3m R package. These functions adjust:

g_cluster(): Hedges’ g effect size for intact class assignment.
g_vi_cluster(): The sampling variance of Hedges’ g for intact class assignment.

3.2 Define Adjustment Parameters

Each adjustment function requires:
g: Hedges’ g effect size estimate from the study.
n_cluster: Total number of clusters (e.g., classrooms).
Nt: Number of subjects in the treatment classes.
Nc: Number of subjects in the control classes.
icc: Intraclass correlation coefficient, defaulting to 0.15. This parameter reflects the similarity of outcomes within clusters (e.g., students in the same classroom).

3.3 Load Example Data

The dataset dat_note (see manuscript) contains meta-analysis data with:

Effect size (g) and its variance (v_g).
Assignment type (assign_type): whether groups were assigned by student (individuals) or class (clusters).
Number of clusters (n_class) and
Number of participants in treatment (Nt) and control (Nc) classes.

3.4 The Adjustment

group_by(dat_note, study) %>% 
  mutate(
        g2 = ifelse(assign_type=="class", g_cluster(g, n_class, Nt, Nc), g),
      v_g2 = ifelse(assign_type=="class", g_vi_cluster(g, n_class, Nt, Nc), v_g)
  ) %>% ungroup()

group_by(dat_note, study): Groups the data by study for easier application of adjustments within each study.
mutate(): Creates two new variables:
g2: Adjusted Hedges’ g for studies with intact classes (assign_type == "class"). For studies with individual randomization, the original g is retained.
v_g2: Adjusted variance for g if clustering was ignored.
ifelse Conditions:
For assign_type == "class", apply g_cluster() and g_vi_cluster() for adjustments. Otherwise, retain the original values (g and v_g).
ungroup(): Removes grouping to return the final adjusted dataset.

The adjustments are most noticeable in the sampling variances (v_g2), which become larger compared to the original values (v_g). This reflects the reduced precision due to the clustering effect.

Table of Contents

Supplement to ‘Advancing meta-analysis in applied linguistics: Coding and effect size considerations for improved rigor’

© Reza Norouzian

Citation: Norouzian, R. (2025). Advancing meta-analysis in applied linguistics: Coding and effect size considerations for improved rigor. Research Synthesis in Applied Linguistics, 3, 1-21.