# Data Sources

HSK data is generated by `scripts/fetch-hsk.mjs` from `drkameleon/complete-hsk-vocabulary`, MIT License.

Generated bundle:

- Merged unique entries: 11,470
- Legacy HSK 2.0 exclusive levels 1-6: 4,991 rows
- New HSK exclusive levels 1-6 plus the repository's 7-9 aggregate: 10,969 rows

Current gameplay scope is intentionally capped to HSK 1-5 for the AP Chinese Lang study goal. The broader rows remain in the generated source bundle only so the data can be refreshed later without changing the ingestion script.

The app uses marked pinyin from the source data as the primary display value. Numeric pinyin is retained internally for tolerant answer matching and as a fallback input to `markedPinyinFromNumeric` when a future source row lacks marked pinyin.

Stroke-order data is vendored from `hanzi-writer` and `hanzi-writer-data`. Hanzi Writer is MIT licensed. Hanzi Writer Data is derived from Make Me A Hanzi / Arphic font data and is redistributable under the Arphic Public License; attribution is included in `NOTICE.md`.

Run `npm run fetch:data` to regenerate `src/hsk-data.js` and `NOTICE.md`.

## Sentence Practice

The Sentences view uses the text-only HSK 1-5 subset of
[`no7z/hsk-sentences-audio`](https://github.com/no7z/hsk-sentences-audio), pinned at
revision `857dfbab91027ebd97df01260e91b817cb6b4854`. The upstream dataset is licensed
CC BY-SA 4.0; its attribution identifies CC-CEDICT as the source for pinyin and gloss data.
The bundled derivative is also shared under CC BY-SA 4.0.

The resource keeps 3,312 sentence records: HSK 1 (281), HSK 2 (538), HSK 3 (727), HSK 4
(801), and HSK 5 (965). It retains each source ID, sentence, traditional form, English
translation, pinyin, topic, and level, and omits audio and token-level glosses. The source
grades its levels against GF0025-2021; the Sentences view labels that grading basis so it is
not confused with the separate official syllabus vocabulary bundle used elsewhere in the app.
The downloaded source snapshot SHA-256 is
`8f8ffd34eee29f395724ce060cad22511eeeabb7af731d5b5879dc0548e01660`.

## Studio Pronunciation Audio (website)

Every website word, slow word, practice sentence and grammar sentence is pre-rendered by
`tools/build_web_audio.py` with the Microsoft neural voice `zh-CN-YunyangNeural` (male,
Mainland standard Putonghua, newsreader delivery), using the open-source `edge-tts` client.

- Format: 24 kHz mono MP3 at 96 kbit/s, the highest-quality format the service accepts
  (48 kHz, PCM and Ogg/Opus requests are refused). Each file's first frame is checked so
  lower-bitrate audio can never be mixed in.
- Speaking rate: -8% for normal takes; vocabulary words also get a separate natively slowed
  take (-35%) so slow mode never time-stretches.
- The service pads every clip with about 1.4 s of silence. `ffmpeg` cuts each clip 250 ms
  after the voice decays below -45 dB, using stream copy (no re-encoding).
- Word-boundary timings are stored in `audio/yunyang-v1/marks/<hash prefix>.json` and drive
  word highlighting, tap-a-word playback, word-by-word slow mode and shadowing.
- Polyphones: when a card's reading differs from the default reading of a character, the
  synthesizer is given an unambiguous homophone for that syllable (e.g. 长 cháng → 常).
  一 and 不 are never substituted, so their natural tone changes stay intact.
- Files are content-addressed (`FNV-1a` hash pair of the item key, shared by Python and
  `src/audio-engine.js`) and live on the orphan `web-audio` branch, checked out at
  `public/audio` locally and by the Pages workflow.

Regenerate or extend (resumable): `python tools/build_web_audio.py` (needs `edge-tts`,
`pypinyin` and `ffmpeg`), then commit inside `public/audio` and push the `web-audio` branch.

## Native Speaker Word Recordings (website, optional)

Settings → Pronunciation → "Native speaker recordings where available (Wei Gao, Beijing)",
off by default, plays a human recording for every word that has one. Slow mode, sentences,
grammar and all other words stay on Yunyang.

- Source: the Shtooka Project collection `chi-balm-hsk1`, read by Wei Gao (male, born in
  Beijing in 1979) and recorded by Nicolas Vion, as published on Wikimedia Commons
  (`insource:"chi-balm-hsk1"`, 926 files on 2026-11-09).
  Credit: **Wei Gao & Nicolas Vion, Shtooka Project, CC BY 2.0 FR**
  ([license](https://creativecommons.org/licenses/by/2.0/fr/)). shtooka.net itself no longer
  hosts the project and is not used.
- Speaker check: a file is used only when its current Commons upload is the 2007 bulk import
  of Wei Gao's recording or a reviewed later upload of the same recording (a mouth-noise trim
  of 咖啡; the 哪/哪儿 swap). Excluded: 练习 and 审 (now a female speaker from Liaoning), 为
  (the original said wèi but was labelled wéi; since replaced by another speaker) and 准备 (the
  original said 装备; replaced by an unidentified speaker). An automated pitch and spectral
  comparison with the Yunyang clips found no clear further mislabels, but it is not precise
  enough to rule them out, so nothing else was excluded on its evidence.
- Matching: a recording is used for a word only when its hanzi (from the file description,
  e.g. `(觉得)`) and its pinyin (from the file name, e.g. `Zh-juéde.ogg`) both match the
  reading the website shows (`src/hsk-data.js` plus `src/hsk-primary-forms.json`). Tones must
  agree, except 一/不 tone sandhi and a full-versus-neutral tone on a later syllable
  (`shàngbiān` for 上边 shàngbian). That tolerance is refused when the recorded reading is
  itself another dictionary reading of the word, so 东西 dōngxi never gets dōngxī and 精神
  jīngshen never gets jīngshén. A polyphone recorded in its other reading (长 zhǎng, 得 děi,
  数 shǔ, 了 liǎo) is not used.
- Processing (`tools/build_human_audio.py`, `ffmpeg`): gain to the median integrated loudness
  of the Yunyang clips for the same words (EBU R128, true peak held at or below -1 dBTP);
  tail cut 250 ms after the voice decays below -45 dB, as for Yunyang; lead-in silence
  shortened to Yunyang's median; a 4 ms fade-in; 24 kHz mono MP3 at 96 kbit/s.
- Files live at `audio/human-weigao-v1/<hash prefix>/<hash>.mp3` with the same hash keys as
  Yunyang (`w:<hanzi>|<numeric pinyin>`) and duration shards in `marks/`.
  `audio/human-weigao-v1/sources.json` lists every clip's Commons file, recorded pinyin,
  match type, gain and trims, plus the coverage table.

Coverage: 876 of the website's 4,649 HSK 1–5 words (851 exact matches, 25 that differ only by a
later neutral tone).

| Level | HSK 2.0 list | HSK 3.0 list |
| --- | --- | --- |
| HSK 1 | 124 / 150 (83%) | 343 / 506 (68%) |
| HSK 2 | 117 / 147 (80%) | 296 / 750 (39%) |
| HSK 3 | 184 / 298 (62%) | 157 / 953 (16%) |
| HSK 4 | 174 / 598 (29%) | 42 / 972 (4%) |
| HSK 5 | 78 / 1298 (6%) | 18 / 1059 (2%) |

Regenerate: `python tools/build_human_audio.py --source <empty folder>` (downloads at no more
than one request per second, verifies each file's SHA-1, then converts), then commit inside
`public/audio` and push the `web-audio` branch before `main`.

## Reading Corrections

The vocabulary source lists every CC-CEDICT reading of a word, and its first reading is often
a surname or literary form (都 Dū, 看 kān, 行 háng, 便宜 biànyí). `tools/choose_hsk_forms.py`
scores each reading against the official HSK syllabus pinyin bundled with the desktop app
(tie-broken by HSK level, then pypinyin's common reading) and writes
`src/hsk-primary-forms.json`. The website shows and pronounces the chosen reading and moves
any saved progress from the old reading's ID.

## Grammar Lessons

`src/grammar/hsk1.json` … `hsk5.json` hold 174 original lessons written for this project
(HSK 1: 23, HSK 2: 31, HSK 3: 38, HSK 4: 41, HSK 5: 41) covering the grammar points of the
HSK 2.0 syllabus levels 1–5. Each lesson has a pattern, explanation, pitfalls, 4–5 example
sentences with chunk-by-chunk breakdowns, and 3–4 drills. `tools/validate_grammar.py` checks
structure, breakdowns, drill integrity and pinyin letters against `pypinyin`; tone marks were
audited separately against every dictionary reading of each character.
