# human_eval — Canine Demo Human Evaluation

A self-contained kit for a 2-rater human study of the canine vocalization demo.
Each rater watches 30 dog clips and scores the system's predictions
(**environment / activity / intention / interpretation**) against the video.

## Contents

```
human_eval/
├── clips/                 30 video clips: clip01.mp4 … clip30.mp4
├── predictions.csv        the system's 4 predictions per clip (rater-facing)
├── predictions.json       full per-clip detail (drives the /evaluation page)
├── sheet.csv              blank 30×2 scoring sheet (60 rows, A & B pre-filled)
├── manifest.csv           clip_id → source clip / breed / duration (provenance)
├── tester_instructions.md one-page briefing to hand each rater
├── protocol.md            full study design + metric definitions
├── score_human_eval.py    computes metrics + a LaTeX block for the paper
├── evaluation.tex         drop-in paper section + Table 1 (red XX placeholders)
├── build_eval_set.py      rebuilds clips/ + csvs from _results.json
├── batch_process.py       runs the demo pipeline over the candidate clips
├── _candidates.json       the 36 sampled candidate clips (seed 42)
└── _results.json          raw pipeline output for every processed clip
```

## Quick start (running the study)

1. **Give each rater:** `tester_instructions.md`, their own copy of `sheet.csv`,
   the `clips/` folder, and the predictions — either the live **`/evaluation`**
   page (video + prediction + inline scoring + one-click CSV export) or
   `predictions.csv`.
2. **Collect** both completed sheets and concatenate the rows into one
   `merged.csv` (same header; A's rows then B's rows).
3. **Score:**
   ```bash
   python3 human_eval/score_human_eval.py merged.csv
   ```
   It prints Accuracy, Acceptability, Agreement %, Fleiss' κ, the interpretation
   mean ± std, and a LaTeX block.
4. **Paper:** paste the LaTeX block over the red `XX` cells in `evaluation.tex`
   and replace the red takeaway note with one sentence.

## How the predictions were produced

The 30 clips are sampled (seed 42) from `/data/bigdrive/_assets/dog_sentence_clips2`
for diversity across breed, duration, and source video. Each clip is run through
the demo's **actual pipeline** (denoise → SED → HuBERT phone transcription →
semantics). For each clip:

- **environment / activity / intention** = the top dominant theme in each
  category, from `models/categorized_word_meaning.json` (the same knowledge base
  the results page uses);
- **interpretation** = the demo's `overall_meaning` sentence.

To regenerate everything from scratch:

```bash
python3 human_eval/batch_process.py      # runs the pipeline (~25 s/clip)
python3 human_eval/build_eval_set.py     # assembles clips/ + csvs + sheet
```

## The `/evaluation` page

A route was added to the demo (`canine.evaluation`, linked in the top nav). It
shows each clip's video beside its prediction, lets a rater pick their letter
(A/B), click 0/1/2 and 1–5 scores (saved in the browser as they go), and
download their `sheet_A.csv` / `sheet_B.csv`. Clips are served from
`human_eval/clips/` via `canine.evaluation_clip`.
