What is being validated
ivreg2r is designed to reproduce the output of Stata’s
ivreg2 command (Baum, Schaffer & Stillman, 2003, 2007).
That is a strong claim, and the package treats it as one that has to be
earned test by test rather than asserted once in the documentation.
Every statistical quantity the package reports — coefficients, standard
errors, the full variance-covariance matrix, the weak-identification and
overidentification statistics, degrees of freedom, observation counts,
cluster counts — is checked against a value that Stata itself produced.
This article explains how that check is constructed, what standard it
holds the package to, and how a reader with a copy of Stata can verify
any part of it independently.
The short version is that the ground truth is machine-generated by Stata, written to disk by Stata, and read back by the R test suite without a human ever retyping a number. There is no step in the chain where a value could drift from what Stata computed to what the test expects.
The fixture chain
The ground truth lives in tests/stata-benchmarks/. That
directory holds a set of Stata do-files, one per feature area, named
generate-*.do — for example
generate-fixtures.do for the core estimators,
generate-hac-fixtures.do for the HAC and kernel machinery,
generate-cue-fixtures.do for the continuously-updated
estimator, and so on across weights, clustering, partialling, centering,
first-stage diagnostics, and the rest of the option surface. Each
do-file loads a dataset — usually one the package itself bundles, in a
few cases data simulated inside the script from a fixed seed — runs
Stata’s ivreg2 with a particular combination of options,
and writes the results to comma-separated fixture files under
tests/stata-benchmarks/fixtures/.
The writing is done with Stata’s file write, one numeric
field at a time, so the fixture files are a direct transcription of the
contents of Stata’s returned results into CSV. Coefficients, the full
covariance matrix, the diagnostic statistics and their p-values, and the
integer bookkeeping are all recorded this way. Nothing is copied out of
a Stata log or a printed table by hand. This matters because it makes
every expected value in the test suite traceable: for any number the R
tests compare against, you can open the corresponding do-file and read
the exact ivreg2 command that produced it. The expectation
is not “a value someone once observed”; it is “the value this command
returns.”
The testthat suite on the R side reads those CSV files back with the
helpers in tests/testthat/helper-fixtures.R, fits the same
model with ivreg2r, and compares. The fixture directory is
the single authority for what “correct” means here.
What is compared, and at what standard
The tolerances for every comparison are defined in one place,
tests/testthat/helper-tolerances.R, and the test files draw
from it rather than choosing their own thresholds — with a small number
of documented per-site exceptions, each carrying its rationale in a
comment at the call site. The standard tolerances are:
| Quantity | Tolerance | Type |
|---|---|---|
| Coefficients, standard errors, VCV elements | 1e-6 | relative |
| Test statistics (F, chi-squared, LM, J) | 1e-4 | relative |
| p-values | 1e-4 | absolute |
| Degrees of freedom, N, cluster counts | exact | integer |
The philosophy behind these numbers is worth stating plainly, because it is the opposite of the usual temptation. A tolerance here is set at the level where exceeding it would indicate a likely bug — a wrong formula, a wrong scaling factor, a transposed matrix — and it is emphatically not calibrated up to whatever discrepancy happened to be observed. If a comparison breaches its tolerance, the rule is to find the root cause before touching the threshold, never to widen the tolerance so a test will pass.
The gap that remains at 1e-6 for coefficients is not
implementation error. The three independent R packages that compute
these quantities — ivreg, estimatr, and
ivreg2r — agree with each other to near machine precision,
and all three differ from Stata by the same small amount, typically on
the order of 1e-8, attributable to different LAPACK builds
and floating-point accumulation order rather than to any package getting
the arithmetic wrong. The hardest cells sit closer to the bound: for
CUE, where the reported coefficients are a numerical optimizer’s
stopping point, the measured worst case approaches 1e-6
itself, which is part of why the tolerance is set where it is. The
package does not claim bit-for-bit identity with Stata; it claims
agreement to a tolerance close to the floor of what cross-implementation
floating-point noise allows. For the test statistics and p-values the
measured discrepancies sit far inside their thresholds, with a margin of
two orders of magnitude or more.
The scale of the coverage
The benchmark suite is large. There are hundreds of fixture files and, across all the test files that read them, thousands of individual assertions. Rather than quote counts that would go stale, it is more useful to describe the shape of the coverage, which is meant to be exhaustive over the features an applied user reaches for.
Every estimator is exercised: OLS, two-stage least squares, LIML,
Fuller and the k-class family, two-step efficient GMM, and CUE. Every
variance estimator is exercised: classical i.i.d.,
heteroskedasticity-robust, one-way and two-way clustering, HAC and
autocorrelation-consistent errors across all eight kernels, the Kiefer
and Driscoll-Kraay panel estimators, and cluster-plus-kernel
combinations. Every automatic diagnostic — weak identification,
underidentification, overidentification, endogeneity, first-stage F and
partial R-squared, the weak-instrument-robust statistics, orthogonality,
and redundancy — is checked against its Stata counterpart. And the
option surface around the estimators is covered as well: all four Stata
weight types, partialling-out, centering of the moment conditions, the
dofminus degrees-of-freedom adjustments, user-supplied
weighting matrices, factor variables, and time-series operators.
The audit layer
Passing the tolerances tells you the package is correct to within the
stated thresholds; it does not by itself tell you how much headroom
there is. That is the job of tests/audit-tolerances.R (in
the source repository), a script that re-measures the actual worst-case
R-versus-Stata discrepancy across a representative set of configurations
— 2SLS, LIML, Fuller, CUE, GMM, weighted, centered, partialled, and
dofminus specifications — and reports the largest observed
relative error in each category, flagging any that come within a factor
of five of their threshold. The point is to keep the distance between
observed accuracy and the test tolerances a known quantity rather than
an assumed one, so that a slow numerical drift would be visible long
before it broke a test.
Reproducing it yourself
None of this has to be taken on faith. The fixture CSVs and the test
suite ship with the package itself; the generate-*.do
scripts live in the package’s source repository at https://github.com/restatr/ivreg2r under
tests/stata-benchmarks/, and the audit script one level up
at tests/audit-tolerances.R. A reader with Stata and the
ivreg2, ranktest, and bcuse
packages (all available from SSC — bcuse fetches the
datasets) can open any generate-*.do file, run it, and
rebuild the fixtures it produces; the CSV output is the same file the
test suite reads, so a rebuilt fixture that matches the shipped one
confirms the ground truth was recorded faithfully. A reader without
Stata can still open the do-file and read the exact ivreg2
command behind any expected value, then confirm that
ivreg2r reproduces it. The chain from a Stata command to a
test expectation is short and fully written down.
Intentional differences
Not every difference from Stata is a bug to be driven below
1e-6; a few are deliberate choices about the R interface.
Those are cataloged in the “Intentional differences from Stata” table in
vignette("introduction"), which covers the API-level
departures (the formula interface, the naming of options, and the like)
together with the one place where a different numerical path is used,
the CUE optimizer (?ivreg2 carries the note on its
behavior). On the statistical conventions that matter for parity — how
sigma is normalized, which finite-sample corrections apply, when a t
distribution is used in place of a normal — the package follows Stata
rather than R’s lm(), and those choices are documented at
?ivreg2r-conventions.