# Reproduce

These steps rebuild every published number from the public TSE files.

## 1. What you need

- Linux. The reference run used Linux x86_64. macOS should work but was not tested.
- Python 3.13.5 (`.python-version`). Another 3.11+ version runs, but output hashes may differ.
- `pdftotext`, from poppler-utils (`apt install poppler-utils`, or `brew install poppler`).
- `curl` is not needed. The download step uses Python only.
- Network access to `cdn.tse.jus.br`, `servicodados.ibge.gov.br`, `geoftp.ibge.gov.br`, `geoserver.funai.gov.br`, `certificacao.incra.gov.br`, `crt.sectigo.com` and the Python package index.
- Disk: about 12 GB in the data folder: 3.4 GB of TSE and IBGE downloads, 1.5 GB of outside registers (mostly the IBGE census-sector file), and intermediate tables.
- Memory: the largest step peaked at 5.5 GB in the reference run (measured, step 11 included).

## 2. Time

- Download: about 4.9 GB, sequential. It depends on your connection.
- Rebuild: about 17 minutes after the download on the reference machine (1,035 s measured with step 11. Steps 1 to 10 alone took 541 s). It runs mostly on one core.

## 3. Run

```
cd zero-vote-sections   # this folder
./run_all.sh
```

To keep the data somewhere else:

```
ZV_DATA_DIR=/path/with/10GB ./run_all.sh
```

`run_all.sh` does everything, in order:
1. Creates `.venv` and installs `requirements.lock` with `--require-hashes`. Every package file must match the SHA-256 published by PyPI.
2. `01_download.py` downloads each file in `sources.lock.tsv` and checks its size and SHA-256. It stops if the TSE has changed a file since the reference download (see section 6).
3. Steps `02` to `05` rebuild the section tables, models and consistency checks. Step `11` then compares the place categories with the outside registers (method fixed in `EXTERNAL-CHECK-PLAN.md`). Steps `06` to `09` follow, because `06` and `08` read the register categories from step `11`.
4. `10_verify.py` compares the outputs with `checksums/results.sha256`, and tests the privacy rule on every published file.

## 4. Settings

| Variable | Default | Meaning |
|---|---|---|
| `ZV_DATA_DIR` | `./data` | Where raw, intermediate and private files go. |
| `ZV_LABELS_DIR` | `$ZV_DATA_DIR/private/labels` | Where the analyst labels are read from, if you have them. |
| `ZV_REGISTERS_DIR` | `$ZV_DATA_DIR/registers` | Where the outside registers for step 11 go. |
| `ZV_ALLOW_CHANGED_SOURCES` | unset | Set to `1` to continue when a TSE file differs from the reference. |
| `ZV_ACCEPT_NEW_REGISTERS` | unset | Set to `1` to continue when an outside register differs from the archived reference copy. |

## 5. Verify

The last lines of the run say how many published files match the reference, and whether the privacy rule holds. To check the key tables by hand:

```
sha256sum results/observed_vs_expected.csv results/u2_history_check.csv results/model_scores.csv \
  results/consistency_summary.csv results/scale.csv results/m4_*.csv
```

Expected (reference run, Python 3.13.5, Linux x86_64):

```
66abeabb42a14110dcc890471594778e4bc7b9861e58c9960fb05bc838c0cae1  results/observed_vs_expected.csv
20a6057ea616a314425c241800756edb7326a33e7b41e9fa699187b0baa48015  results/u2_history_check.csv
5aea13aa62458986cc159aa92f20394cecb5cfe1f23c893c49e9668089126458  results/model_scores.csv
ddfe1ab0daf5b0b2cf50dd0d833111a2192434492abae399b6e89d68c6d3159f  results/consistency_summary.csv
ea4c347351132334169b612a55e7697000aedacb16c2c286c0fb5c8ba623d97c  results/scale.csv
9edc64c03f5de90138688e7ae9061245b13de2b338534b44e945fa05bff94be3  results/m4_counterfactual_reweighting.csv
33426d267044a08c93d7403a950080276563b699bc5a2ae73cd8f168e2729e29  results/m4_flag_components_by_year.csv
17df1b67dafb9ea55826fbdd689a46764c2dac05c7f58350a0e41f82c426eb3b  results/m4_place_matching_across_years.csv
8adb68ffbd4e196f2bf068e65689b9251417fb3095747010128fec5c0cb70666  results/m4_stable_panel_zero_rates.csv
```

The `m5b_*` checksums and the register rows of `interactive/zero_sections_by_category.csv` and `results/m4_counterfactual_reweighting.csv` can be matched only with the archived register copies, which are not distributed (section 6).

Floats are written with 10 significant digits, so the same package versions should match byte for byte. On another platform a last digit can change. If a file differs, compare it numerically with the published copy before drawing any conclusion.

## 6. If a source file has changed

**TSE files.** The TSE sometimes regenerates published files. Step 1 then stops and names the file. `sources.tsv` records what you downloaded, and `sources.lock.tsv` records the reference. Results built from a changed file may differ from the published ones. That is a data revision, not a reproduction failure. Set `ZV_ALLOW_CHANGED_SOURCES=1` to continue and compare.

**Outside registers.** Step 11 uses eight files from IBGE, FUNAI and INCRA, plus the IBGE municipal mesh. `sources.lock.tsv` pins the SHA-256 of the archived copies used for the published tables (kinds `register` and `ibge_api`). INCRA regenerates its quilombola file every night, and FUNAI and the IBGE mesh API build their files on request, so a fresh download will usually differ. Step 1 then stops and names the files. Set `ZV_ACCEPT_NEW_REGISTERS=1` to continue with the newer files. Only the step-11 tables (`results/m5b_*`) can then differ from the published ones, and `10_verify.py` will report them as differing.

**Licence of the registers.** The IBGE, FUNAI and INCRA download pages state no licence. Brazilian federal open-data policy (Decreto 8.777/2016) applies (inference). The registers are cited, not redistributed, so the archived copies are not part of this package.

## 7. The analyst labels

Two inputs are labels, not code:
- the category of each of the 110 zero sections of 2026,
- the labels of a random sample of 200 sections flagged as indigenous.

They were written by the analyst who wrote the classifier rules, an AI agent. The analyst read only the TSE text fields and the official place type, could see the rules' output, and checked nothing against an outside register. So the labels show whether the text means what the rules assume. They do not show that a place is in fact an indigenous or quilombola community.

The labels identify individual small communities and prison sections, so they are not published. Without them, step 9 skips those checks, and `10_verify.py` reports the files that depend on them as skipped. Everything else reproduces without them. The sample to label is written to `data/private/precision_sample_2026_to_label.csv`, so you can label it yourself and compare.

## 8. Output

- `results/`: every published table. `FINDINGS-v2.md` names the file behind each number.
- `interactive/`: privacy-safe data files. The rule is at the top of each file.
- `data/private/`: lists of individual sections. Never publish these.
- `checksums/results.sha256`: SHA-256 of each published file from the reference run.

## 9. Reproduction test

Run on 2026-10-29, with step 11 included, to test these steps:
- A new folder holding only the files in this package (no `.venv`, no outputs, no labels).
- A new, empty data folder. Its `raw/` and `registers/` files were linked to an earlier download of the same files instead of downloaded again. Step 1 re-hashed all 79 of them, and all 79 matched `sources.lock.tsv`.
- The TSE had republished one file since the reference download: `votacao_candidato_munzona_2022.zip` (29 Oct 2026 07:18 GMT, 582,280,209 bytes, against 620,464,618 for the reference). Step 1 noted it and used the reference copy. A fresh download today would stop at step 1 and ask for `ZV_ALLOW_CHANGED_SOURCES=1`.
- `./run_all.sh` created `.venv` from `requirements.lock` with `--require-hashes` on Python 3.13.5, Linux x86_64.
- The rebuild took 1,202 s, while a second rebuild ran on the same machine.
- `10_verify.py`: 55 published files match, 0 differ, 0 missing. 7 files that need the private analyst labels were skipped, as expected without them. The privacy rule holds.
- All nine key tables in section 5 have exactly the SHA-256 values shown there. So do the step-11 tables listed in `checksums/results.sha256`.
- A second clean run in the reference folder, with the analyst labels, matched all 62 published files.
- After the third recheck (step 11 moved before steps 6 and 8, two tables added), two more clean runs in the reference folder: the first produced the new tables and checksums, the second matched all 63 published files. The privacy rule holds.

- After the third recheck, a fresh-folder run of the final package with no labels took 593 s: 56 published files match, 0 differ, 7 skipped (they need the private labels). The privacy rule holds.

Not tested: a fresh download from the sources, macOS, or another Python version.