You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- P1 CS/SA covariance-source deviation: the R7 edit said current
`pretrends.py` "computes Wald power/MDV from the pre-period variance-
covariance block", but only the `MultiPeriodDiDResults` branch extracts
the full pre-period sub-VCV (pretrends.py:592-601). The CS branch
(pretrends.py:609-652) and SA branch (pretrends.py:660-687) hard-code
`vcov = np.diag(ses^2)`, dropping the pre-period correlations the
paper's Propositions rely on. Added an explicit "Note (deviation in
current covariance-source)" bullet in the "Relation to Existing
diff-diff Estimators" section documenting the per-result-type split and
cross-referencing the conservative-variance framing in REPORTING.md.
The follow-up audit will decide whether to extend full-sub-VCV
extraction to CS/SA or keep the diag fallback as a deliberate variance-
conservatism choice.
- P1 Assumption 1 model-vs-estimated Sigma: the R2 edit said
"Assumption 1 is a condition on the *estimated covariance matrix*
Sigma". Roth's Section II.B states Assumption 1 on the *model*
covariance Sigma (the population VCV in Equation 1). Software can only
inspect the *estimated* Sigma_hat, so any direct numerical check is a
HEURISTIC implementation proxy, not the paper's assumption itself.
Rewrote the Prop 2 Implementation-use cell to (a) name Sigma as the
model covariance, (b) acknowledge Sigma_hat as the implementation
surrogate, (c) require any sharper warning based on Sigma_hat be
labeled heuristic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/methodology/papers/roth-2022-review.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -218,6 +218,7 @@ see "Library design choices" above for the proposed `method` / `n_sim` knobs.
218
218
### Relation to Existing diff-diff Estimators
219
219
-**Pre-existing `diff_diff/pretrends.py`** (1133 lines) — implements a Wald-test-based pre-trends MDV/power workflow framed around Roth (2022); the current code path computes Wald power/MDV from the pre-period variance-covariance block rather than the full arbitrary-Sigma Proposition 1 / Proposition 3 / Proposition 4 conditional-moment computations. This paper review's main use is to audit the existing surface against the paper's exact equations and identify which Roth-2022 quantities are missing.
220
220
-**Currently composes with** (per the shipped `compute_pretrends_power` adapter in `diff_diff/pretrends.py`): `MultiPeriodDiDResults`, `CallawaySantAnnaResults`, `SunAbrahamResults`. The adapter raises `TypeError` for other result types. Theoretical compatibility extends to any estimator producing an event-study coefficient vector and a consistent variance estimator (e.g., `TwoWayFixedEffects`), but adapters for additional result families are a follow-up audit decision.
221
+
-**Note (deviation in current covariance-source):** the shipped adapter uses different sources for the pre-period covariance depending on the result type. For `MultiPeriodDiDResults` it extracts the full pre-period sub-block from `results.vcov` when `interaction_indices` is populated, falling back to `diag(ses^2)` otherwise (`pretrends.py:592-601`). For `CallawaySantAnnaResults` and `SunAbrahamResults` it hard-codes `vcov = diag(ses^2)` even when an event-study VCV is available (`pretrends.py:609-652` for CS, `pretrends.py:660-687` for SA). The diagonal fallback drops the pre-period correlations the paper's Propositions rely on; the documented deviation is captured in `docs/methodology/REPORTING.md` as a conservative variance source for staggered adapters. The follow-up audit should decide whether to extend full-sub-VCV extraction to CS/SA (matching the MultiPeriodDiD path) or keep the staggered diag fallback as a deliberate variance-conservatism choice.
221
222
-**Complement to `HonestDiD` (Rambachan-Roth 2023)**: Roth 2022 asks "what bias survives a pretest under linear violations?"; Rambachan-Roth 2023 asks "what is the identified set of tau_post under bounded violations?" Both use the same (beta_hat, Sigma_hat) input contract — the library should expose a unified entry-point that can produce both Roth-2022 and HonestDiD reports from one event-study result object.
222
223
-**Shares zero-anticipation convention with HonestDiD**: tau_pre = 0, so beta_pre = delta_pre. Cross-reference the existing `diff_diff/honest_did.py` for the contract.
223
224
@@ -228,7 +229,7 @@ see "Library design choices" above for the proposed `method` / `n_sim` knobs.
228
229
| # | Statement | Implementation use |
229
230
|---|-----------|---------------------|
230
231
|**Proposition 1**| For any B(Sigma): E[beta_hat_post | beta_hat_pre in B] = tau_post + delta_post + Sigma_{12} Sigma_{22}^{-1} (E[beta_hat_pre | beta_hat_pre in B] - beta_pre) | The main bias decomposition formula. Drives the conditional-bias computation in step 4 of the algorithm. |
231
-
|**Proposition 2**| Under Assumption 1 (homoskedastic-equicorrelated Sigma) and monotone trend (delta_pre < 0, delta_post > 0): E[beta_hat_post | beta_hat_pre in B_NIS] > beta_post > tau_post | Justifies a WARN that conditional bias is worse than unconditional bias under monotone trends — applicable in many but not all empirical settings. Assumption 1 is a condition on the *estimated covariance matrix* Sigma, not on design metadata; any sharper warning must therefore be triggered by a *direct numerical check* of Sigma (approximately-constant diagonal entries + approximately-constant positive off-diagonal entries below the diagonal). Without such a check, the library should issue only the generic caveat that the sign-of-bias result is ambiguous outside Assumption 1. |
232
+
|**Proposition 2**| Under Assumption 1 (homoskedastic-equicorrelated Sigma) and monotone trend (delta_pre < 0, delta_post > 0): E[beta_hat_post | beta_hat_pre in B_NIS] > beta_post > tau_post | Justifies a WARN that conditional bias is worse than unconditional bias under monotone trends — applicable in many but not all empirical settings. Assumption 1 in the paper is a condition on the *model* covariance Σ (the population variance-covariance of β̂ in Equation 1), not on design metadata. Software can only inspect the *estimated* Σ̂, so any direct numerical check (e.g., approximately-constant diagonal entries + approximately-constant positive off-diagonal entries below the diagonal) is a heuristic implementation aid, not the paper's assumption itself. A library that surfaces a sharper warning based on Σ̂ should label it as a heuristic; without such a check, the library should issue only the generic caveat that the sign-of-bias result is ambiguous outside Assumption 1. |
232
233
|**Proposition 3**| Var[beta_hat_post | beta_hat_pre in B] = Var[beta_hat_post] + (Sigma_{12} Sigma_{22}^{-1}) (Var[beta_hat_pre | beta_hat_pre in B] - Var[beta_hat_pre]) (Sigma_{12} Sigma_{22}^{-1})' | The conditional-variance formula; drives the over/under-coverage analysis. |
233
234
|**Proposition 4**| If B(Sigma) is convex: Var[beta_hat_post | beta_hat_pre in B] <= Var[beta_hat_post] (variance-reduction guarantee, conditional on convex B only). | Justifies the "do not interpret a wide CI as ample power" warning. Implication for CI coverage (Section II.C paragraph after Prop 4): CIs based on unconditional Sigma tend to OVER-cover under parallel trends with symmetric B; under violations they tend to UNDER-cover *only if the bias is sufficiently large* to outweigh the variance reduction — the under-coverage direction is contingent on bias magnitude, not universal. |
0 commit comments