1What the method is for
To place a speaker on the Tokyo–Kansai gradient as a coordinate, without relying on a listener's judgement.
Comparing Japanese pitch accent across speakers is harder than it looks. Two speakers of the same Tokyo type can differ entirely in absolute \(F_0\); a fast speaker and a slow one place the same pattern at different points along the time axis; phonation differs from person to person. Measure a distance without removing these, and what is measured is the speaker, not the pattern.
What this study builds is not a classifier but an instrument. The aim is to place a speaker on the gradient of accent varieties documented in the literature, stably and interpretably. That the goal is instrument design also shapes how it is evaluated (section 6).
The speakers are from Kashiwabara in Maibara City. In dialectological description, Kashiwabara lies in the Tarui-type zone, intermediate between the Kansai and Tokyo types. In other words, a prediction of where these speakers should fall is supplied from outside, by prior dialectological description, independently of the data collected here. Being able to use that external prediction as the evaluation criterion is the reason for choosing this location.
2Removing the speaker
Normalization removes differences, but removing too much also removes what is to be compared.
An \(F_0\) track is extracted from the audio and normalized in two stages.
Rather than hertz, this takes the logarithm of the ratio to the speaker's own median. High-pitched and low-pitched speakers alike are placed on the same scale of "how far above my own centre". The factor 12 makes the unit a semitone, so that an octave — a factor of two — comes out as exactly 12.
Taking the logarithm of a ratio is not an arbitrary choice. Perceived pitch is governed by frequency ratios, so the quantity that behaves additively is the logarithm, not the hertz value.
The track is then divided by a measure of spread. MAD (median absolute deviation) is used here rather than the standard deviation. \(F_0\) estimation can fail by whole octaves rather than semitones, and a single such outlier inflates a standard deviation considerably. MAD is median-based and is not dragged by a few extreme values.
These two steps align pitch height (location) and pitch range (scale) across speakers. What remains is only the shape — where the rise occurs and where the fall — and it is this that carries the accent pattern.
| Preprocessing | Setting |
|---|---|
| Voice activity detection (VAD) | top_db = 25, minimum segment 120 ms |
| \(F_0\) estimation | YIN (pYIN as fallback), \(f_{\min}=50\,\mathrm{Hz}\), \(f_{\max}=700\,\mathrm{Hz}\) |
| Smoothing | Savitzky–Golay filter (window 11, order 3) |
| Recording | 16 kHz / 16 bit, indoors |
Normalization is a design decision about what to discard
Subtracting the median and dividing by MAD amounts to deciding that a speaker's average pitch height and pitch range carry no information. That is defensible for comparing patterns, but if, say, Kansai-type speakers were characteristically wider in range, that fact has been discarded at this point. What was kept and what was thrown away continues to bear on how the later results should be read.
3Aligning while allowing time to stretch
Matching a fast speaker's frames against a slow speaker's, one by one, means nothing.
Two sequences \(a = (a_1,\dots,a_n)\) and \(b = (b_1,\dots,b_m)\) are to be compared without being made equal in length. DTW (dynamic time warping) considers every way of putting the points into correspondence and selects the alignment whose total cost is smallest.
\(D(i,j)\) is the minimum cost of having aligned the first \(i\) elements of \(a\) with the first \(j\) of \(b\). Moving diagonally means "align these two"; moving down or across means advancing one side alone, which is what stretching time amounts to. The initial condition is \(D(0,0)=0\), with the remaining boundary set to \(\infty\).
The implementation uses \(c_{\mathrm{sub}} = 1.0\) and \(c_{\mathrm{ins}} = c_{\mathrm{del}} = 1.1\). A small penalty is placed on stretching, which discourages forcing a match by warping without limit.
Used raw, the total cost grows with the length of the utterance. Dividing by \(n+m\) gives an average cost per step, which makes utterances of different lengths comparable.
The Sakoe–Chiba band: how far apart may points be aligned
If alignment is left entirely free, points far apart in time end up joined and the judgement of similarity breaks down. An upper bound is therefore placed on \(|i-j|\). Here the band width is 0.15 times the sequence length (band_ratio = 0.15). The cost also drops from \(O(nm)\) to \(O(n \cdot \text{band})\).
As section 7 shows, this band width turns out to be the parameter that moves the results most.
DTW is computed not only on \(z(t)\) itself (where the pitch is) but also on its difference \(\Delta z(t)\) (whether it is rising or falling), and the two are combined with weights. An accent pattern is determined less by height than by the location of the fall, so change is entered as a separate term.
The difference is computed by central differences, skipping unvoiced stretches where \(F_0\) is unavailable. If either \(\mathrm{dtw}(z)\) or \(\mathrm{dtw}(\Delta z)\) cannot be computed, the weights are renormalized over whichever term remains.
4Anchoring the ends of the axis outside the data
When one says "closer to Tokyo", where exactly is Tokyo?
This is the most consequential design decision in the study. A distance matrix computed among the speakers alone yields an axis that expresses only relative position within the 68 people recorded. Record elsewhere and the axis means something else; results cannot be compared across studies.
Four reference points — Tokyo, Kansai, Tarui and Kagoshima types — are therefore synthesized from phonological descriptions rather than from recordings. The L / H / R symbol strings found in the descriptions of Uwano (1999), Kibe (2010) and others are converted into numerical contours that serve as virtual reference utterances.
Declination — the tendency of the whole contour to drift downward as an utterance proceeds — is added with a slope between \(-0.05\) and \(-0.10\), and the result is smoothed with the same Savitzky–Golay filter (window 11, order 3) used on the real audio. Passing the references through the same smoothing as the speakers ensures that the same processing is applied to both sequences being compared.
Each speaker is thereby represented as a vector of distances to four reference points. Because the ends of the axis are defined by external descriptions, the coordinates do not close over this dataset. Without this design, the evaluation in the next section could not be carried out at all.
The virtual references are idealized types, not real speakers
A contour synthesized from a symbol string is a phonological description rendered numerically. It is not an utterance by an actual speaker of the Tokyo dialect. What is being measured is therefore distance from a described type, not distance from a real Tokyo speaker. Assigning \(-1.0\) / \(+1.0\) / \(+0.5\) to L / H / R is itself a design decision, and whether placing R at \(+0.5\) rather than midway between H and L is appropriate cannot be tested from inside this framework.
5What it was compared against
Presenting the proposed method alone would say nothing about whether it is any good. Three families of baselines are used.
- Mean \(F_0\) — the speaker's average pitch and nothing else. The crudest baseline, carrying no temporal information at all.
- Histogram EMD — the Earth Mover's Distance between distributions of \(z\) (30 bins over \([-3, 3]\)). It sees how long the speaker spent at each height but not the order in which.
- wav2vec 2.0 (XLSR-53) — a self-supervised speech representation, in three configurations: 3a mean pooling of the final layer (L24) with cosine distance; 3b frame-level DTW on the final layer; 3c frame-level DTW on an intermediate layer (L9). The intermediate layer in 3c follows prior findings that prosodic information is better preserved there (Pasad et al. 2021, among others).
6Evaluating by gradient preservation, not by accuracy
With no ground-truth labels, how is an instrument judged?
No speaker carries a label saying "this person is Tarui type". Any attempt to assign one would come down to somebody's auditory judgement, which returns the problem to subjectivity. The study therefore uses a criterion that is not classification accuracy.
Kashiwabara is described dialectologically as lying in the zone intermediate between the Kansai and Tokyo types. A good instrument should therefore place the 68 speakers in the intermediate region of the Kansai–Tokyo axis. Three indicators describe how they are placed.
\(d_T\) is the distance to the Tokyo reference and \(d_K\) to the Kansai reference. \(r\) is a balance ratio: the closer to 1, the more symmetric the axis is with respect to its two poles. \(s\) is a symmetry index: the closer to 0, the less the placement leans toward either pole. Without \(r \approx 1\) and \(s \approx 0\), a high Inter.% means only that the axis has collapsed to one side.
| Method | Inter.% | \(r\) | \(s\) | Reading |
|---|---|---|---|---|
| Mean \(F_0\) | 1.5% | 1.42 | −0.05 | Gradient collapses |
| Histogram EMD | 85.3% | 0.83 | +0.12 | Gradient over-flattened |
| wav2vec 2.0 (3a: mean pool, L24) | 0% | 4.46 | −0.61 | Collapses to a pole |
| wav2vec 2.0 (3b: DTW, L24) | 0% | 1.55 | −0.21 | Collapses to a pole |
| wav2vec 2.0 (3c: DTW, L9) | 22.1% | 0.99 | +0.01 | Off-axis |
| DTW (proposed) | 57.4% | 1.01 | −0.01 | Gradient preserved |
Converted to counts out of 68 speakers: 1.5% = 1, 22.1% = 15, 57.4% = 39, 85.3% = 58.
No single number ranks these methods
On Inter.% alone, histogram EMD's 85.3% beats the proposed method's 57.4%. It is nonetheless judged over-flattened, because \(r = 0.83\) and \(s = +0.12\) show the axis leaning to one side. An instrument that places almost everyone in the middle has lost the ability to distinguish intermediate from non-intermediate. The claim takes the form that only the proposed method satisfies all three indicators at once.
That is a defensible evaluation design, but it also means a judgement about how to weigh the indicators has entered the conclusion. The ranking does not fall out mechanically from a single score.
The gap to wav2vec 3c is not a gap in accuracy
Configuration 3c matches the proposed method on symmetry (\(r = 0.99\), \(s = +0.01\)) but reaches only 22.1% on Inter.%. The repository reads this as measuring a different axis: the self-supervised instrument does have an internal axis, but it is skewed relative to the linguistically defined Kansai–Tokyo gradient. This is a distinction from a difference in accuracy along a shared axis.
7What happens when the parameters move
Agreement of the auxiliary hard-label diagnostic as band and α are swept.
The primary output is the continuous four-pole distance vector, but a two-stage hard-label classifier is run alongside it as a diagnostic (stage 1: Tokyo / Kagoshima / Kansai block; stage 2: Kansai block → Kansai / Tarui). Its agreement rate across Sakoe–Chiba band widths and values of \(\alpha\) is as follows.
| band | α | Agreement | Withheld | Withheld % |
|---|---|---|---|---|
| 0.12 | 0.5 | 77.9% | 30 | 44.1% |
| 0.12 | 0.7 | 75.0% | 28 | 41.2% |
| 0.12 | 0.9 | 72.1% | 26 | 38.2% |
| 0.15 | 0.5 | 98.5% | 14 | 20.6% |
| 0.15 | 0.7 | 100.0% | 15 | 22.1% |
| 0.15 | 0.9 | 97.1% | 15 | 22.1% |
| 0.2 | 0.5 | 64.7% | 10 | 14.7% |
| 0.2 | 0.7 | 61.8% | 10 | 14.7% |
| 0.2 | 0.9 | 52.9% | 11 | 16.2% |
Agreement is measured against the adopted setting (band = 0.15, α = 0.7), which is why that row reads 100% by definition. "Withheld" counts the speakers caught by the ambiguity thresholds (\(m_{\mathrm{abs}} < 0.011\), \(m_{\mathrm{rel}} < 0.035\)).
This table needs reading with care
An agreement of 100% says only that the adopted setting agrees with itself; it is not an accuracy. What the table shows is the size of the sensitivity — that moving the band to 0.12 or 0.2 shifts the assignments substantially — not that 0.15 is correct.
The striking feature is rather that widening the band to 0.2 cuts the withheld cases to 10 while agreement falls to 52.9%. It hesitates less but answers differently — the behaviour one expects when free alignment produces forced matches and, with them, unwarranted confidence. Whether the choice of band = 0.15 rests on grounds outside the Inter.% of section 6 cannot be determined from the repository's materials.
A check by unsupervised clustering
Ward clustering on the 68×68 distance matrix (FastDTW) gives silhouette coefficients of 0.660 at \(k=2\), 0.466 at \(k=3\), 0.357 at \(k=4\), 0.343 at \(k=5\) and 0.331 at \(k=6\). The value falls monotonically with \(k\), with no clear elbow or trough. This is no evidence that the Kashiwabara speakers fall into discrete types; it is consistent instead with the study's treatment of them as a continuous gradient.
8Computational cost
To be usable as an instrument, the method has to run in a realistic amount of time. Measured times per pair are as follows.
| Method | Time (ms) | Relative | Notes |
|---|---|---|---|
| Mean \(F_0\) | 0.04 | 1× | No temporal information |
| Histogram EMD | 0.20 | 5× | Distribution only |
| DTW (static \(z\) only) | 0.26 | 7× | Sakoe–Chiba band |
| DTW (proposed, full) | 0.99 | 25× | \(z(t)\) + \(\Delta z(t)\), \(\alpha=0.7\) |
| wav2vec 2.0 | approx. 450 | approx. 11000× | GPU recommended |
With Numba JIT acceleration, the full set of 68 speakers × 25 phrases × 4 references completes in under 30 seconds. Requiring no GPU matters for something meant to be used as a working tool.
9Limits
The evaluation criterion is a dialectological prediction
Inter.% takes the prior description of Kashiwabara as an intermediate zone to be correct and measures how many speakers are placed there. If the description is wrong, so is the evaluation. The criterion also carries a direction — more in the middle is better — so if some speakers genuinely are not intermediate, it cannot detect them. There is still no ground truth for any individual speaker.
One location, 68 speakers
Only Kashiwabara is covered; no validation at other locations is included. Anchoring the poles in external descriptions should in principle transfer, but whether the same properties hold when it is transferred has not been checked. The indicator values are computed on a sample of 68, and no credible intervals are given.
Parameters were chosen on the same data the performance is measured on
The band width of 0.15, \(\alpha = 0.7\) and the ambiguity thresholds were settled while looking at these 68 speakers, and the indicators of section 6 are computed on the same 68. The data used to choose the settings is not separated from the data used to measure performance, so the reported figures may be optimistic as estimates of performance on independent data.
DTW alignment knows nothing of linguistic units
The recursion operates on a sequence of frames, not on morae or syllables. Warping is decided purely by cost, with no constraint saying "align these because they are the same mora". Alignments that are linguistically impossible can therefore come out as minimum-cost. The band constraint mitigates this but does not remove it.
The audio is not public
The recordings are withheld for privacy reasons. The per-speaker de-identified metadata is likewise excluded from the public repository, because the combination of hometown and school-name fields exceeds the aggregate-distribution scope of the ethics consent; it is available under restricted access on reasonable request. The code and the symbolic pattern table are public, but a third party cannot at present fully reproduce the main results of this study — only aggregate-level consistency checks are possible.
10Sources and code
The code is released under the MIT licence, structured so that normalization, DTW, synthesis of the virtual references, the baselines, the sensitivity sweep and the figures all run as one pipeline. The manuscript is under review.
- Onohara, A. “Speaker-normalized Acoustic Distance for Japanese Pitch Accent: Design Principles and Evaluation”. github.com/aonoa68/pitch-accent-dtw2025 (v1.0.0, 2026-07-08). Licence: MIT
- ORCID: 0009-0004-3082-3267
- The symbolic accent patterns (L/H/R) are public as
data/accent_types_20251027.xlsx. The audio is not.
Structure of the implementation
src/extractor.py — \(F_0\) extraction and normalization (VAD, YIN, semitone, MAD-z, smoothing)src/dtw_distance.py — DTW under the Sakoe–Chiba constraint, in a Numba version and a pure-Python onesrc/virtual_reference.py — synthesis of virtual reference contours from symbolic descriptions (L/H/R)src/baselines.py — mean \(F_0\), histogram EMDsrc/wav2vec_enhanced.py — wav2vec 2.0 variants 3a / 3b / 3c and the layer sweepsrc/classifier.py — the two-stage hard-label assignment (diagnostic)src/unsupervised.py — the 68×68 distance matrix, MDS, Ward clusteringsensitivity_sweep.py — the sweep over band width and \(\alpha\)benchmark_timing.py — the timing measurements
References for the method
- Sakoe, H. & Chiba, S. (1978) Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1), 43–49.
- de Cheveigné, A. & Kawahara, H. (2002) YIN, a fundamental frequency estimator for speech and music. Journal of the Acoustical Society of America, 111(4), 1917–1930.
- Conneau, A. et al. (2021) Unsupervised cross-lingual representation learning for speech recognition (XLSR-53). Interspeech 2021.
- Pasad, A., Chou, J.-C., & Livescu, K. (2021) Layer-wise analysis of a self-supervised speech representation model. ASRU 2021.
- Phonological descriptions of the accent varieties: Uwano (1999), Kibe (2010), among others.
Which perspective this method belongs to
What DTW minimizes is not a sum of squares but a total cost over an alignment. In that sense it is not least squares. It sits under Perspective 1 nonetheless, because it shares the stance of fixing a distance first and minimizing under it — and because everything downstream lands squarely in the least-squares world. The MDS that projects the 68×68 matrix onto a plane, and the Ward clustering, both minimize sums of squares. It occupies the distance-building end of Perspective 1 (geometry) in Seven Perspectives in Equations.
As an exercise in building a distance, it pairs with Gower's distance (a distance from nominal-scale data) and normalized compression distance (a distance from compressibility). The objects differ — an acoustic time series, symbol agreement, a byte string — but the idea of designing the distance itself to fit what is to be compared is the same.
This page sets out, for readers in the same field, a method Onohara uses in actual research. Every figure is quoted from the published repository's output files and README; no estimated or approximate values are given. The manuscript is under review, so the conclusions should be read as provisional.
The whole picture of where each method sits is in Seven Perspectives in Equations, and which teaching session falls under which perspective is in Methods & Mathematics.