# Human Evaluation Protocol

Small, reproducible human study of the canine vocalization demo's semantic
output. It measures whether the system's per-clip predictions
(**environment / activity / intention / interpretation**) match what a person
sees in the corresponding video.

## Design

- **Stimuli:** N = 30 short dog video clips, sampled for diversity across breed
  and duration and drawn from distinct source videos (see `_candidates.json`).
  Selection is seeded (seed 42) and reproducible via `../` build scripts.
- **System under test:** the demo's full pipeline
  (denoise → SED → HuBERT phone transcription → semantics). For each clip the
  environment/activity/intention predictions are the top dominant themes in each
  category from `models/categorized_word_meaning.json`; the interpretation is the
  demo's `overall_meaning` sentence. Predictions are frozen in `predictions.csv`
  / `predictions.json`.
- **Raters:** two annotators (A, B) score every clip **independently**.
- **The video is ground truth** — raters judge predictions against the video.

## Scales

- **environment / activity / intention:** 0 = Incorrect, 1 = Plausible,
  2 = Correct. Ties resolve to 1.
- **interpretation:** 1 (nonsensical) … 5 (accurate & informative).

## Metrics (`score_human_eval.py`)

- **Accuracy** — fraction of attribute ratings equal to 2 (Correct), averaged
  over annotators. (We use accuracy averaged over annotators rather than
  majority vote, which is degenerate with only two raters — any disagreement is
  a tie.)
- **Acceptability** — fraction of attribute ratings ≥ 1 (Plausible or Correct).
- **Agreement %** — fraction of clips where both annotators gave the same score;
  reported alongside κ because it is more intuitive with two raters.
- **Fleiss' κ** — chance-corrected inter-annotator agreement (2 raters, 3
  categories). *Caveat:* with two raters and heavily skewed scores (e.g. almost
  everything "Correct"), κ can look low or negative — a known artifact of the
  statistic, not necessarily poor labeling. Read it together with Agreement %.
- **Interpretation quality** — mean ± std of the 1–5 ratings.

## Running it

1. Give each rater `tester_instructions.md`, their own `sheet.csv`, the `clips/`
   folder, and access to the predictions (the `/evaluation` page or
   `predictions.csv`).
2. Collect both completed sheets and concatenate the rows into one
   `merged.csv` (same header, both annotators' rows).
3. Run:

   ```bash
   python3 human_eval/score_human_eval.py merged.csv
   ```

4. Paste the printed LaTeX block over the red `XX` cells in `evaluation.tex`
   (Table 1) and replace the red takeaway note with one sentence.

## Files

| File | Purpose |
|------|---------|
| `clips/` | the 30 video clips (`clip01.mp4` … `clip30.mp4`) |
| `predictions.csv` / `.json` | frozen system predictions per clip |
| `sheet.csv` | blank 30×2 scoring sheet (60 rows, A & B pre-filled) |
| `tester_instructions.md` | one-page rater briefing |
| `score_human_eval.py` | metrics + LaTeX generator |
| `manifest.csv` | clip_id → source clip / breed / duration (provenance) |
| `build_eval_set.py` | regenerates `clips/`, predictions, sheet from `_results.json` |
| `batch_process.py` | runs the demo pipeline over the candidate clips |
| `_candidates.json`, `_results.json` | selection + raw pipeline output (provenance) |
