# Canine Vocalization Demo — Human Evaluation (Paper Notes)

Paper-ready notes for the human study of the canine vocalization semantics demo.
Everything here is meant to be lifted into a paper's evaluation section. Numbers
reported as `XX` are filled in by running `score_human_eval.py` on the collected
rater sheets (see **Reproducing** at the end).

---

## 1. Motivation

The demo maps a dog's vocalization to four human-readable predictions —
**environment**, **activity**, **intention**, and a one-sentence
**interpretation**. Automatic metrics can't tell us whether those predictions
actually match reality, because the "reality" of a bark is what a human sees in
the accompanying video. We therefore run a small, controlled human study in which
annotators score the system's predictions against the ground-truth video.

## 2. Data

- **N = 30** dog video clips.
- Sampled (fixed seed 42) from the project's sentence-clip corpus
  (`dog_sentence_clips2`, ~100k clips) for diversity across:
  - **breed** — husky (6), shiba (6), labrador (6), chihuahua (4), pitbull (4),
    german shepherd (4);
  - **duration** — 1.5–21.8 s;
  - **source video** — all 30 clips come from **distinct** source videos (no two
    clips from the same recording).
- Clips were **not** cherry-picked for easy cases; selection is breed/duration
  stratified and otherwise random under the seed.

## 3. System under test

Each clip is run through the demo's full pipeline:

1. **Denoising** (AudioSep, "Dog" query)
2. **Sound-Event Detection** (ATST-SED) → sentence boundaries
3. **Phone transcription** (HuBERT, 250-cluster k-means) → phone-unit sequence
4. **Semantic inference** — matched vocal "words" are looked up in a
   frequency-weighted knowledge base (`categorized_word_meaning.json`) that
   distributes each pattern over environment / activity / intention themes.

For each clip we freeze four predictions:

- **environment / activity / intention** — the top dominant theme in each
  category (the same theme ranking shown on the demo's results page);
- **interpretation** — the demo's `overall_meaning` sentence.

Predictions are stored in `predictions.csv` / `predictions.json` and are held
constant during rating.

> **Implementation note / limitation.** In the evaluated configuration the
> optional LLM (SmolLM3-3B) that phrases the interpretation could not be loaded
> (the environment ships PyTorch 2.1 while the installed Transformers requires
> ≥2.4), so the system used its deterministic template for the interpretation
> sentence. The theme knowledge base is also heavily skewed toward a few
> globally frequent labels, so the `activity` and `intention` predictions are
> nearly constant across clips ("Movement & Locomotion", "Calm & Relaxed"). The
> study therefore largely measures how often these dominant labels actually hold,
> which is itself an informative result about the demo's current behavior.

## 4. Protocol

- **Two annotators (A, B)** score **every** clip **independently**; they do not
  confer until both finish (~45–60 min each).
- The **video is the ground truth.** Raters judge each prediction against what
  they see and hear.
- **Scales.**
  - environment / activity / intention: **0** = Incorrect, **1** = Plausible,
    **2** = Correct. Ties resolve to 1.
  - interpretation: **1** (nonsensical) … **5** (accurate & informative).
- Ratings were entered either through the demo's `/evaluation` page (video +
  prediction + inline scoring) or on a CSV sheet; both produce the same
  `clip_id,annotator,environment,activity,intention,interpretation,notes` rows.

## 5. Metrics

For the three attributes, pooled and per-attribute:

- **Accuracy** — fraction of ratings equal to *Correct* (2), averaged over
  annotators. We use accuracy averaged over annotators rather than majority
  vote, which is degenerate with only two raters (any disagreement is a tie).
- **Acceptability** — fraction of ratings that are at least *Plausible* (≥ 1).
- **Agreement** — percentage of clips on which both annotators gave the same
  score. Reported next to κ because it is more intuitive with two raters.
- **Fleiss' κ** — chance-corrected inter-annotator agreement (2 raters, 3
  categories).
  - *Caveat:* with two raters and a heavily skewed score distribution (e.g.
    almost everything "Correct"), κ can be low or even negative — a known
    artifact of the statistic, not necessarily poor labeling. Interpret it
    together with % Agreement.

For the interpretation sentence: **mean ± std** of the 1–5 ratings.

## 6. Results

Filled in by `score_human_eval.py merged.csv`.

**Table 1. Human evaluation (N = 30 clips, two annotators).**

| Attribute   | Accuracy | Acceptability | Agreement | Fleiss' κ |
|-------------|:--------:|:-------------:|:---------:|:---------:|
| Environment |   XX%    |      XX%      |    XX%    |    XX     |
| Activity    |   XX%    |      XX%      |    XX%    |    XX     |
| Intention   |   XX%    |      XX%      |    XX%    |    XX     |
| **Overall** | **XX%**  |    **XX%**    |  **XX%**  |  **XX**   |

Interpretation quality: **XX ± XX** on the 1–5 scale.

**Takeaway (replace with one sentence once numbers are in):** _e.g., "Environment
predictions were acceptable on the majority of clips, while the near-constant
activity/intention labels were correct only when the dog happened to be moving /
calm, indicating the knowledge base's skew is the main limitation."_

## 7. Reproducing

```bash
# 1. (re)generate the 30 clips + predictions from the pipeline
python3 human_eval/batch_process.py       # runs denoise→SED→HuBERT→semantics
python3 human_eval/build_eval_set.py       # writes clips/, predictions.*, sheet.csv

# 2. after both raters return sheet_A.csv and sheet_B.csv
python3 human_eval/merge_sheets.py sheet_A.csv sheet_B.csv   # -> merged.csv (+ prints scores)
#    or, if you already have a combined file:
python3 human_eval/score_human_eval.py merged.csv
```

The printed LaTeX block drops straight into `evaluation.tex` (Table 1).

## 8. Files (provenance)

`_candidates.json` (the 36 seeded candidates), `_results.json` (raw pipeline
output per clip, incl. detected words and full theme ranking), `manifest.csv`
(clip_id → source clip / breed / duration). These make the selection and the
frozen predictions fully auditable.
