Skip to content

Commit 8f14d73

Browse files
igerberclaude
andcommitted
Address R9 review (2 P1) on roth-2022-review.md
- P1 CS/SA covariance-source deviation: the R7 edit said current `pretrends.py` "computes Wald power/MDV from the pre-period variance- covariance block", but only the `MultiPeriodDiDResults` branch extracts the full pre-period sub-VCV (pretrends.py:592-601). The CS branch (pretrends.py:609-652) and SA branch (pretrends.py:660-687) hard-code `vcov = np.diag(ses^2)`, dropping the pre-period correlations the paper's Propositions rely on. Added an explicit "Note (deviation in current covariance-source)" bullet in the "Relation to Existing diff-diff Estimators" section documenting the per-result-type split and cross-referencing the conservative-variance framing in REPORTING.md. The follow-up audit will decide whether to extend full-sub-VCV extraction to CS/SA or keep the diag fallback as a deliberate variance- conservatism choice. - P1 Assumption 1 model-vs-estimated Sigma: the R2 edit said "Assumption 1 is a condition on the *estimated covariance matrix* Sigma". Roth's Section II.B states Assumption 1 on the *model* covariance Sigma (the population VCV in Equation 1). Software can only inspect the *estimated* Sigma_hat, so any direct numerical check is a HEURISTIC implementation proxy, not the paper's assumption itself. Rewrote the Prop 2 Implementation-use cell to (a) name Sigma as the model covariance, (b) acknowledge Sigma_hat as the implementation surrogate, (c) require any sharper warning based on Sigma_hat be labeled heuristic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 1e4a2d9 commit 8f14d73

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

docs/methodology/papers/roth-2022-review.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,7 @@ see "Library design choices" above for the proposed `method` / `n_sim` knobs.
218218
### Relation to Existing diff-diff Estimators
219219
- **Pre-existing `diff_diff/pretrends.py`** (1133 lines) — implements a Wald-test-based pre-trends MDV/power workflow framed around Roth (2022); the current code path computes Wald power/MDV from the pre-period variance-covariance block rather than the full arbitrary-Sigma Proposition 1 / Proposition 3 / Proposition 4 conditional-moment computations. This paper review's main use is to audit the existing surface against the paper's exact equations and identify which Roth-2022 quantities are missing.
220220
- **Currently composes with** (per the shipped `compute_pretrends_power` adapter in `diff_diff/pretrends.py`): `MultiPeriodDiDResults`, `CallawaySantAnnaResults`, `SunAbrahamResults`. The adapter raises `TypeError` for other result types. Theoretical compatibility extends to any estimator producing an event-study coefficient vector and a consistent variance estimator (e.g., `TwoWayFixedEffects`), but adapters for additional result families are a follow-up audit decision.
221+
- **Note (deviation in current covariance-source):** the shipped adapter uses different sources for the pre-period covariance depending on the result type. For `MultiPeriodDiDResults` it extracts the full pre-period sub-block from `results.vcov` when `interaction_indices` is populated, falling back to `diag(ses^2)` otherwise (`pretrends.py:592-601`). For `CallawaySantAnnaResults` and `SunAbrahamResults` it hard-codes `vcov = diag(ses^2)` even when an event-study VCV is available (`pretrends.py:609-652` for CS, `pretrends.py:660-687` for SA). The diagonal fallback drops the pre-period correlations the paper's Propositions rely on; the documented deviation is captured in `docs/methodology/REPORTING.md` as a conservative variance source for staggered adapters. The follow-up audit should decide whether to extend full-sub-VCV extraction to CS/SA (matching the MultiPeriodDiD path) or keep the staggered diag fallback as a deliberate variance-conservatism choice.
221222
- **Complement to `HonestDiD` (Rambachan-Roth 2023)**: Roth 2022 asks "what bias survives a pretest under linear violations?"; Rambachan-Roth 2023 asks "what is the identified set of tau_post under bounded violations?" Both use the same (beta_hat, Sigma_hat) input contract — the library should expose a unified entry-point that can produce both Roth-2022 and HonestDiD reports from one event-study result object.
222223
- **Shares zero-anticipation convention with HonestDiD**: tau_pre = 0, so beta_pre = delta_pre. Cross-reference the existing `diff_diff/honest_did.py` for the contract.
223224

@@ -228,7 +229,7 @@ see "Library design choices" above for the proposed `method` / `n_sim` knobs.
228229
| # | Statement | Implementation use |
229230
|---|-----------|---------------------|
230231
| **Proposition 1** | For any B(Sigma): E[beta_hat_post | beta_hat_pre in B] = tau_post + delta_post + Sigma_{12} Sigma_{22}^{-1} (E[beta_hat_pre | beta_hat_pre in B] - beta_pre) | The main bias decomposition formula. Drives the conditional-bias computation in step 4 of the algorithm. |
231-
| **Proposition 2** | Under Assumption 1 (homoskedastic-equicorrelated Sigma) and monotone trend (delta_pre < 0, delta_post > 0): E[beta_hat_post | beta_hat_pre in B_NIS] > beta_post > tau_post | Justifies a WARN that conditional bias is worse than unconditional bias under monotone trends — applicable in many but not all empirical settings. Assumption 1 is a condition on the *estimated covariance matrix* Sigma, not on design metadata; any sharper warning must therefore be triggered by a *direct numerical check* of Sigma (approximately-constant diagonal entries + approximately-constant positive off-diagonal entries below the diagonal). Without such a check, the library should issue only the generic caveat that the sign-of-bias result is ambiguous outside Assumption 1. |
232+
| **Proposition 2** | Under Assumption 1 (homoskedastic-equicorrelated Sigma) and monotone trend (delta_pre < 0, delta_post > 0): E[beta_hat_post | beta_hat_pre in B_NIS] > beta_post > tau_post | Justifies a WARN that conditional bias is worse than unconditional bias under monotone trends — applicable in many but not all empirical settings. Assumption 1 in the paper is a condition on the *model* covariance Σ (the population variance-covariance of β̂ in Equation 1), not on design metadata. Software can only inspect the *estimated* Σ̂, so any direct numerical check (e.g., approximately-constant diagonal entries + approximately-constant positive off-diagonal entries below the diagonal) is a heuristic implementation aid, not the paper's assumption itself. A library that surfaces a sharper warning based on Σ̂ should label it as a heuristic; without such a check, the library should issue only the generic caveat that the sign-of-bias result is ambiguous outside Assumption 1. |
232233
| **Proposition 3** | Var[beta_hat_post | beta_hat_pre in B] = Var[beta_hat_post] + (Sigma_{12} Sigma_{22}^{-1}) (Var[beta_hat_pre | beta_hat_pre in B] - Var[beta_hat_pre]) (Sigma_{12} Sigma_{22}^{-1})' | The conditional-variance formula; drives the over/under-coverage analysis. |
233234
| **Proposition 4** | If B(Sigma) is convex: Var[beta_hat_post | beta_hat_pre in B] <= Var[beta_hat_post] (variance-reduction guarantee, conditional on convex B only). | Justifies the "do not interpret a wide CI as ample power" warning. Implication for CI coverage (Section II.C paragraph after Prop 4): CIs based on unconditional Sigma tend to OVER-cover under parallel trends with symmetric B; under violations they tend to UNDER-cover *only if the bias is sufficiently large* to outweigh the variance reduction — the under-coverage direction is contingent on bias magnitude, not universal. |
234235

0 commit comments

Comments
 (0)