Principal Component Analysis — Ayaka Onohara

1What the method is for

With many variables, what is discarded and what is kept.

What is at hand is an \(n \times p\) table of numbers \(X\). Rows are cases (settlements, papers), columns are variables (words, tallies), and every cell holds a number. With \(p\) equal to 4 one might get by simply looking at the table, but at 24 or 37 the relations among the columns can no longer be followed in the head.

So the \(p\) variables are bundled together into a small number of new ones. There are endlessly many ways to bundle them, so a criterion is needed. The criterion principal component analysis adopts is variance. Project the data onto a single line; the direction in which the projected values spread most is taken as the first principal component, and among the directions orthogonal to it, the one with the next largest spread as the second. And so on.

What is being discarded is the directions of small variance. The assumption is that small spread means the direction does not account for differences among cases, and only the directions of large spread are kept. Whether that assumption holds is a question outside the method, taken up in section 8.

In Onohara’s research this same operation is applied to three tables of quite different shape: 21 settlements × 4 variables, 23 cases × 37 variables, and 30 papers × 24 words.

2Formulation: from maximizing variance to an eigenvalue problem

Look for “the direction that maximizes variance”, and an eigenvector of the covariance matrix comes out.

Take a \(p\)-dimensional random vector \(x\), with the mean already subtracted (\(\mathbb{E}[x] = 0\)). Write the covariance matrix as \(\Sigma = \mathbb{E}[x x^{\top}]\). The variance of the value \(w^{\top}x\) obtained by projecting onto the direction of a unit vector \(w\) is a quadratic form in \(\Sigma\):

Variance of the projected value

\[ \mathrm{Var}\bigl[w^{\top}x\bigr] \;=\; \mathbb{E}\bigl[(w^{\top}x)^{2}\bigr] \;=\; w^{\top}\,\mathbb{E}\bigl[x x^{\top}\bigr]\,w \;=\; w^{\top}\Sigma\, w \]

\(w\) … the direction projected onto. Unless its length is fixed at 1, \(w\) could be scaled up without limit and the variance made arbitrarily large
\(\Sigma\) … the covariance matrix; symmetric and positive semi-definite

The first principal component is therefore defined as the solution of the following constrained maximization.

Definition of the first principal component

\[ \max_{\lVert w \rVert = 1}\; \mathrm{Var}\bigl[w^{\top}x\bigr] \;=\; \max_{w^{\top}w = 1}\; w^{\top}\Sigma\, w \]

Solving with a Lagrange multiplier

Bring the constraint \(w^{\top}w = 1\) in with a Lagrange multiplier \(\lambda\).

\[ L(w,\lambda) \;=\; w^{\top}\Sigma\, w \;-\; \lambda\bigl(w^{\top}w - 1\bigr) \]

Since \(\Sigma\) is symmetric, \(\partial (w^{\top}\Sigma w)/\partial w = 2\Sigma w\) and \(\partial (w^{\top}w)/\partial w = 2w\).

Stationarity condition

\[ \frac{\partial L}{\partial w} \;=\; 2\Sigma w \;-\; 2\lambda w \;=\; 0 \qquad\Longleftrightarrow\qquad \boxed{\;\Sigma w \;=\; \lambda w\;} \]

That is, the direction \(w\) maximizing the variance must be an eigenvector of the covariance matrix \(\Sigma\). The Lagrange multiplier \(\lambda\) is itself the eigenvalue.

Evaluating the objective at this \(w\),

\[ w^{\top}\Sigma\, w \;=\; w^{\top}(\lambda w) \;=\; \lambda\, w^{\top}w \;=\; \lambda \]

so that the eigenvalue is itself the variance of the projection onto that direction. To maximize the variance one therefore takes the largest eigenvalue \(\lambda_1\), and the direction \(w_1\) of the first principal component is the eigenvector corresponding to \(\lambda_1\).

The second component onward

For the second principal component, the same problem is solved with the added constraint of orthogonality to \(w_1\). Since the eigenvectors of a symmetric matrix can be taken mutually orthogonal, the answer is the eigenvector \(w_2\) of the second eigenvalue \(\lambda_2\). Continuing in the same way,

\[ \Sigma \;=\; \sum_{k=1}^{p} \lambda_k\, w_k w_k^{\top}, \qquad \lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p \ge 0 \]

the eigendecomposition of the covariance matrix gives the sequence of principal components directly. The components are mutually orthogonal, so the new variables are uncorrelated with one another. When the definition of Furusawa and Haga (1992), quoted by Onohara and Yano (2014), adds “with the composite indices bearing no relation to one another”, it is this orthogonality that is meant.

In a finite sample

In practice \(\Sigma\) is unknown, so a sample covariance matrix is formed from the centred \(n \times p\) data matrix \(X\).

\[ S \;=\; \frac{1}{n-1}X^{\top}X, \qquad z_{ik} \;=\; w_k^{\top}x_i \]

Here \(z_{ik}\) is the \(k\)th component score of case \(i\). Computationally one usually obtains these not from an eigendecomposition of \(S\) but from a singular value decomposition of \(X\): in \(X = U\Sigma_{\mathrm{sv}}V^{\top}\), the columns of \(V\) are the \(w_k\), and the squared singular values correspond to \((n-1)\lambda_k\).

3What the explained proportion of variance is

A ratio of eigenvalues. To be read cumulatively.

Because the components are mutually orthogonal, the total variance of the original data equals the sum of the eigenvalues.

Decomposition of the variance

\[ \mathrm{tr}(\Sigma) \;=\; \sum_{j=1}^{p}\mathrm{Var}[x_j] \;=\; \sum_{k=1}^{p}\lambda_k \]

The left-hand side is the total spread of the original variables, the right-hand side the total spread across the components. The two agree because they are the same quantity counted in different coordinate systems.

Explained proportion of the \(k\)th component, and the cumulative proportion

\[ c_k \;=\; \frac{\lambda_k}{\sum_{l=1}^{p}\lambda_l}, \qquad C_K \;=\; \sum_{k=1}^{K} c_k \]

\(c_k\) is the proportion of variance explained, \(C_K\) the cumulative proportion. These are the percentages that R’s prcomp and FactoMineR’s PCA print on the axis labels of a figure.

Cautions in reading it

What the explained proportion answers is one thing only: what share of the spread that was in the original table this axis has picked up.

It does not say whether the axis has captured a meaningful structure. It does not say whether the analysis succeeded. To look at a two-dimensional figure is to discard everything outside \(C_2\), and a claim read off the figure is guaranteed only within \(C_2\).

And the explained proportion depends heavily on the number of variables and on the correlation structure. With a correlation matrix, \(\mathrm{tr} = p\), so with only 4 variables \(c_1\) tends to come out large, while with 37 variables the same structure gives a smaller \(c_1\). The explained proportions of different studies cannot be compared directly. This point is taken up again in section 8.

4Component scores and component loadings

Scores place the cases, loadings place the variables. These are two different figures.

Principal component analysis has two kinds of output. Because both are drawn on axes labelled “Dim 1 / Dim 2”, they are easily confused, but the points plotted are of different kinds.

Principal component score

\[ z_{ik} \;=\; w_k^{\top}x_i \qquad (i = 1,\dots,n) \]

The value obtained by projecting case \(i\) onto the direction of the component. The number of points is \(n\), the number of cases. One point in the figure is one settlement, or one paper.

Principal component loading

\[ \ell_{jk} \;=\; \mathrm{Corr}\bigl(x_j,\, z_k\bigr) \;=\; w_{jk}\,\frac{\sqrt{\lambda_k}}{\sqrt{\mathrm{Var}[x_j]}} \qquad (j = 1,\dots,p) \]

The correlation between variable \(x_j\) and the \(k\)th component score. The number of points (or arrows) is \(p\), the number of variables. One point in the figure is one word, or one tally.
Note that the term “loading” is used in two ways in the literature: for the components \(w_{jk}\) of the eigenvector themselves, and for the correlation above.

How to tell them apart

In practice, count the points. In “Types and geographical distribution of vocabulary in the upper Ibi River basin”, Figure 8 has 23 cases and Figure 9 has 37 variables. The former carries settlement names together with “Tokyo” and “Kyoto”; the latter carries variable names such as Tfish and Kcorrect.

The other way to tell is by the scale. When the data have been standardized before the analysis, loadings are correlation coefficients and therefore necessarily lie between \(-1\) and \(1\), falling inside a circle of radius 1 in a two-dimensional figure. That circle is drawn in Figure 9, “Variables factor map (PCA)”, and in Figure 4 of Onohara and Yano (2014). The score plot (Figure 8, “Individuals factor map (PCA)”) has no such circle, and its scale runs beyond \(-5\) to \(5\).

“Close” means different things

Two scores being close means that those cases take similar values across the variables as a whole. Onohara and Yano (2014) write that “papers plotted in close positions use similar nouns”.

Two loadings being close means that those variables move in similar ways across the cases. The same paper states that “nouns plotted in close positions occur at similar frequencies within a paper”.

And the distance between a score and a loading has no direct meaning. What can be read from a figure that superimposes the two (a biplot) is not distance but direction. If the point for a case lies far out in the same direction as the arrow for a variable, that case is read as having a large value on that variable. Figure 4 of Onohara and Yano (2014) and the figure in the presentation on the area around the old Tokuyama Village are both of this superimposed kind.

When “Types and geographical distribution of vocabulary in the upper Ibi River basin” prefaces its conclusion with “looking at the relation between the component scores and the component loadings”, it is pointing to this way of reading the two figures against each other. Only by looking at both — where the words cluster in Figure 9, and which settlements lie in that direction in Figure 8 — can the statement be made.

5Correlation matrix, or covariance matrix?

When variables are on different scales, failing to standardize lets the high-variance variable monopolize the axis.

The formulation in section 2 used the covariance matrix \(\Sigma\). But the eigenvectors of \(\Sigma\) depend on the scale of the variables. Merely converting one variable from centimetres to millimetres multiplies its variance by 100, and the first principal component is pulled toward that variable.

Standardizing before the analysis = decomposing the correlation matrix

\[ \tilde{x}_j \;=\; \frac{x_j - \bar{x}_j}{s_j}, \qquad \Sigma_{\tilde{x}} \;=\; R \;=\; \bigl(\mathrm{Corr}(x_j, x_{j'})\bigr)_{j,j'} \]

The covariance matrix of the standardized variables equals the correlation matrix \(R\) of the original ones. Since \(\mathrm{tr}(R) = p\), every variable enters with the same share.

When variables measured in different units are mixed (height and weight, frequency and years), the rule is to use the correlation matrix. Conversely, when all variables are measured in the same unit with the same meaning and the size of the variance itself is informative, there is a reason to use the covariance matrix.

Which was used in this research

In all three cases, the text does not state whether a correlation matrix or a covariance matrix was used. The following, however, can be said from the materials.

The presentation on the area around the old Tokuyama Village prints the original data (21 settlements × 4 variables) in full. Running principal component analysis on that table with the correlation matrix gives explained proportions of 83.92% on the first component and 15.03% on the second, which agree with the values printed in the material (a check carried out for this page). Analysing the same table with the covariance matrix does not agree. The analysis can therefore be judged to have been a standardized principal component analysis.

Figure 9 of “Types and geographical distribution of vocabulary in the upper Ibi River basin” and Figure 4 of Onohara and Yano (2014) are both correlation circles, drawn with a circle of radius 1. This is consistent with the output of a standardized principal component analysis. The former paper lists the R package FactoMineR among its references, and the figure titles “Individuals factor map (PCA)” and “Variables factor map (PCA)” match the output of that package’s PCA function, whose default is to standardize. But the body of the paper says nothing to that effect, so the claim is kept to what the figures show.

For the first two, moreover, the variables are all vocabulary data coded as 0 / 0.5 / 1, so the scales are already comparable and standardization may well not change the result much. That point, too, cannot be verified from the materials.

6Three applications

The same method, applied to tables of entirely different shape.

Study	Shape of the data	1st	2nd	3rd	4th	Source
(a) Vocabulary of the upper Ibi River basin	23 cases × 37 variables	23.07%	14.73%	—	—	Figs. 8 and 9
(b) Vocabulary around the old Tokuyama Village	21 settlements × 4 variables	83.92%	15.03%	—	—	Figure on the results slide
(c) Submissions to Kōdō Keiryōgaku	30 papers × 24 words	20.54%	12.1%	10.27%	9.38%	Figs. 4 and 5

All figures are values printed in the respective sources. “—” indicates that the source does not report the explained proportion from the third component onward. Only (c) gives the third and fourth in its text.

(a) Checking a random forest result from another angle

In “Types and geographical distribution of vocabulary in the upper Ibi River basin”, principal component analysis is not listed in section 4, “Analytical procedure and methods”. That section lists three: phylogenetic estimation (NeighborNet), correspondence analysis, and random forest. Principal component analysis appears in the results section (5.3), as an auxiliary check on the random forest result.

The original data The basic vocabulary surveyed by Yamada (1978, 1980) at 21 sites in the upper Ibi River basin. For each word, the record is + if the core morpheme corresponds to the Tokyo or the Kyoto dialect form, − if it does not, ○ if doubtful, and △ where there are two equivalent forms of which only one corresponds.
Coding and narrowing For the random forest, words that were + for both Tokyo and Kyoto were dropped, and any word containing ○ or NR in even one settlement was deleted column-wise. The remaining variables take the form “word × Tokyo/Kyoto”, as in Tfish and Kcorrect, with values corresponding to 1 / 0.5 / 0.
Size of the data The header of Figure 7 prints rf. 23cases, 37vars, loop=100. The 23 cases are the 21 settlements plus Tokyo and Kyoto.
Principal component analysis The paper states that “Figures 8 and 9 are the result of a principal component analysis of the data used for the random forest”.

What was read from it is the following. The words grease, good, bite, fish, correct, they, he, sit and lie, which random forest ranked highest as characteristic words, cluster at the circled position in the loading plot (Figure 9). And from the score plot (Figure 8), those words show a distinctive pattern in settlements such as Hisaka, Higashi-Tsugumi, Nishi-Tsugumi, Higashi-Yokoyama, Sakamoto and Kinomoto. Reading scores and loadings together, the paper concludes that these are vocabulary items that separate Tokyo-leaning from Kyoto-leaning dialect forms, and that lean to the Kyoto side.

Principal component analysis is thus used here to confirm by eye, in a different coordinate system, an answer produced by another method. Two different criteria — importance for classification (random forest) and direction of variance (principal component analysis) — are applied to the same data, and the question is whether the same set of words emerges.

(b) Four tallies to see differences among settlements — an example the reader can work through

The presentation “A re-examination of the vocabulary of the area around the old Tokuyama Village from a mathematical standpoint” prints the original data in full: four tallies for each of 21 settlements. Since it is an example that can be reproduced at hand, it is transcribed below.

Definition of the four variables (as stated in the material)

Ep … the number of vocabulary items corresponding to the Tokyo vocabulary
En … the number of items not corresponding (to the Tokyo vocabulary)
Wp … the number of vocabulary items corresponding to the Kyoto vocabulary
Wn … the number of items not corresponding (to the Kyoto vocabulary)

The material further gives, in a table, the correspondence between the four variables and the combinations of + / − against Tokyo and against Kyoto. Words that are + for Tokyo — both “similar to Tokyo and to Kyoto” (++) and “similar to Tokyo” (+−) — go into Ep; words that are − for Tokyo — both “similar to Kyoto” (−+) and “similar to neither” (−−) — go into En. In the same way, words that are + for Kyoto (++ and −+) go into Wp, and words that are − for Kyoto (+− and −−) into Wn.
That is, E stands for East (Tokyo), W for West (Kyoto), and p / n for whether the item corresponds or not; Ep and En, like Wp and Wn, are the same 200 words counted over from the two opposite sides.

Settlement	District / village	Ep	En	Wp	Wn
Kadonyū	Tokuyama	168	27	170	27
Tonyū	Tokuyama	168	27	166	30
Tsuka	Tokuyama	168	27	169	27
Hazehara	Tokuyama	170	25	167	29
Yamate	Tokuyama	168	26	167	28
Kami-Kaiden	Tokuyama	168	28	167	29
Hongō	Tokuyama	168	28	170	27
Shimo-Kaiden	Tokuyama	168	27	168	29
Kawakami	Sakauchi	170	25	172	24
Hirose	Sakauchi	172	22	172	23
Sakamoto	Sakauchi	173	22	173	23
Tsurumi	Fujihashi	172	24	171	24
Higashi-Yokoyama	Fujihashi	176	19	177	20
Nishi-Tsugumi	Kuze	176	18	175	21
Higashi-Tsugumi	Kuze	173	22	177	18
Ozu	Kuze	173	24	174	23
Hisaka	Kuze	175	21	177	20
Soto-Tsugumi	Kuze	174	23	176	22
Imajō	Nanjō	166	31	175	23
Ikeda	Imadate	167	30	172	26
Kinomoto	Ōno	176	21	185	13

Transcribed as printed on the slides of the presentation. The “district / village” column is labelled “村 / mura” in the material. Kinomoto is given as Ōno there, but in a figure in the same material Kinomoto is colour-coded as Ika, and Table 1 of “Types and geographical distribution of vocabulary in the upper Ibi River basin” likewise places the town of Kinomoto in Ika District, Shiga Prefecture. On this one point the material is inconsistent with itself.
The sum of Ep and En falls between 194 and 197, and that of Wp and Wn between 195 and 198; neither matches the 200 items of the basic vocabulary. The material gives no explanation for the difference.

The original data The material describes “the vocabulary called ‘basic vocabulary’ — 200 words such as ‘I’, ‘you’, ‘head’ that may be supposed to occur in any region (the 200 selected by Hattori (1955) from the Swadesh List (Swadesh, 1950))”, and uses the tallies of Yamada (1978, 1980).
Variables and cases As stated in the material, “in order to bring out the differences among settlements, a principal component analysis was carried out with Ep, En, Wp, Wn as variables and the settlements as cases”.
Result On the plane of Dim 1 (83.92%) and Dim 2 (15.03%), the settlement scores and the vectors of the four variables are drawn superimposed. Ep and Wp point to the right, En and Wn to the left.
What was read The settlements of Tokuyama Village cluster on the left of the figure, while Kinomoto sits alone at the right edge. The discussion in the material says of Tokuyama Village that it is “even less similar to the Tokyo and Kyoto vocabulary than the settlements of the other villages”.

The text of the discussion slide reads “skewed toward the Wn, Pn side”, but there is no variable Pn in the figure; it can be read as a slip for En.

(c) Following change over time in the content of submitted papers

Onohara and Yano (2014) also apply principal component analysis to the same “30 papers × 24 nouns” data that they put through correspondence analysis. The stated motive is “to consider the data from another angle”, and, following Furusawa et al. (1992), the paper defines principal component analysis as “an analysis that, for a set of samples each carrying two or more attributes, constructs composite indices taking the relations among the attributes into account (with the composite indices bearing no relation to one another), and on that basis shows the relation between the attributes and the samples”.

Figure 4 Plots simultaneously the component scores (the tendency of paper content, based on the noun information) and the component loadings (shown as arrows: the tendency in how frequent nouns are used within papers). The explained proportion is 20.54% on the first component and 12.1% on the second, followed by 10.27% on the third and 9.38% on the fourth.
Figure 5, left The first-component score of each paper as a bar chart, re-sorted by date with older papers at the top and newer ones at the bottom. From the 2000s onward, the scores change from negative to positive.
Figure 5, right The first-component loadings. The nouns characteristic of papers with positive scores are, in decreasing order of loading, “evaluation”, “nature”, “scoring”, “task”, “person”. Those characteristic of papers with negative scores are “word”, “phoneme”, “vocabulary”, “speech”, “language”.
The second component and below The paper states explicitly that “the temporal change seen on the first component was not seen on the second component or below”.

What does the work here is the operation of re-sorting the component scores by external information (year of publication). The analysis itself knows nothing of dates. The structure of the reading is that when cases are arranged on an axis built in ignorance of the dates, they turn out to line up with them. That the paper stops at “there is a possibility that papers on education are appearing” and “there is a possibility that papers on themes related to the economy are appearing” — in both cases only a possibility — can be read as an acknowledgement of the after-the-fact character of such an interpretation.

7The difference from correspondence analysis

Both place points on a plane, but the quantity being minimized differs.

Principal component analysis and correspondence analysis both minimize the squared error of a low-rank approximation, and both are solved by the singular value decomposition. The figures look much alike too. The difference lies in what goes into the matrix, and in which distance it is measured.

The quantities being minimized, side by side

\[ \text{PCA:}\quad \max_{\lVert w \rVert = 1} w^{\top}\Sigma\, w \qquad\Longleftrightarrow\qquad \min_{\mathrm{rank}\,\le\, K}\bigl\lVert X - \hat{X} \bigr\rVert_F^{2} \] \[ \text{CA:}\quad \min_{\mathrm{rank}\,\le\, K}\bigl\lVert D_r^{-1/2}\bigl(P - \mathbf{r}\mathbf{c}^{\top}\bigr)D_c^{-1/2} - \hat{S} \bigr\rVert_F^{2} \]

What principal component analysis decomposes is the data matrix itself (centred, and standardized if required), under Euclidean distance. What correspondence analysis decomposes is the departure from independence, weighted by the marginals, under chi-square distance.

	Principal component analysis	Correspondence analysis
Input	A table of quantitative variables. Negative values and standardized values are both admissible	A table of non-negative counts (a contingency table), or 0/1 binary data
Distance measured	Euclidean distance (which changes with standardization)	Chi-square distance, weighted by the column marginal proportion \(c_j\)
Maximized / minimized	Maximizes the variance of the projection	Maximizes the weighted correlation between the quantified values of rows and columns
Size of a row	Enters the distance as it is (though centring is applied)	Disappears through profiling
What goes in the figure	Scores (cases) and loadings (variables) are separate figures; superimposing them makes a biplot	Placing rows and columns on the same plane at once is the definition
Index for an axis	Explained proportion of variance \(\lambda_k / \sum \lambda_l\)	Explained proportion of inertia \(\sigma_k^2 / \sum \sigma_l^2\); total inertia is \(\chi^2/n\)

What happens when both are applied

Onohara and Yano (2014) apply both to the same 30 papers × 24 words data. As a result, what became visible differed. In the correspondence analysis figure the papers cluster by genre; in the principal component analysis a temporal change appeared in the first-component scores.

This difference, however, is not due to the methods alone. The correspondence analysis result was read with genre information overlaid, and the principal component result was read after re-sorting the scores by year. Which external information the low-dimensional coordinates are matched against also changes what becomes visible. The difference between methods and the difference between ways of reading need to be kept apart.

“Types and geographical distribution of vocabulary in the upper Ibi River basin” also uses both, but there the inputs differ. Correspondence analysis is applied to a table of vocabulary against the settlements other than Tokyo and Kyoto; principal component analysis to the 23 cases × 37 variables table prepared for the random forest.

8Limits

A high explained proportion is not a certificate of success.

An explained proportion of 83.92% does not mean the analysis “worked”

The first component in (b) explains 83.92%, and together with the second, 98.95%. On the numbers alone this is ideal, but it is also an expression of the fact that the four variables were strongly correlated with one another.

Indeed, Ep and En are the same 200 words counted from the two opposite sides, and summing them over the published table gives values in the range 194 to 197. Wp and Wn likewise give 195 to 198. Once one is fixed the other is all but fixed, so the effective degrees of freedom of the four variables are roughly two. That two dimensions account for nearly 100% is close to inevitable.

A high explained proportion can be read as “the data summarized well” or as “there was little independent information in the variables to begin with”. Which of the two it is cannot be told from the number. The only way is to go back to how the variables were constructed. Conversely, the 20.54% of (c) is not a failure: it means that the 24 words were not much correlated with one another. There is no point in setting the explained proportions of different studies side by side and arguing which is better.

It looks only for linear directions

What principal component analysis looks for is straight-line directions, and nothing else. If the data lie along a curved surface, or if the cases separate in a non-linear way, the method projects them onto a line all the same. A plane minimizing the squared error is found, but there is no guarantee that this plane preserves the structure.

For a quantity that acts monotonically, such as a count of vocabulary agreements, the linear assumption is not a stretch. But for vocabulary data coded as 0 / 0.5 / 1 ((a)), or for frequencies divided by paper length ((c)), whether the relation between the variables and the latent structure is linear has not been verified from the materials.

Because variance is maximized, outliers decide the axes

Since the objective function is variance, a single point far from the mean enters squared. A few outlying cases can end up all but determining the direction of the first principal component.

In the figure of (b), Kinomoto sits alone near 4 on Dim 1, away from the other 20 settlements. In the published table, too, Kinomoto’s Wp is 185, eight larger than the next largest value. How much this single point contributes to the direction of the first component cannot be judged, because the material presents no recomputation with Kinomoto removed. In Figure 8 of (a), likewise, “Tokyo” lies apart from the other cases.

In general this needs to be checked either by reporting the result with outliers removed alongside, or by using a robust principal component analysis. In none of the three cases was that check carried out.

Every interpretation of a component is made after the fact

What the method returns is only “orthogonal directions ordered by variance”; it says not a word about what the first component represents. Naming an axis is the analyst’s doing.

When (c) states that “on the positive side of the first component axis there is a possibility that papers on education are appearing”, that is an interpretation given after looking at the nouns with large loadings (“evaluation”, “nature”, “scoring”, “task”, “person”). The paper’s consistent reservation, “there is a possibility”, is the right way to write it, but it remains hard for a reader to see that an axis label is an interpretation and not an output of the analysis.

In particular, the procedure of naming an axis after looking at the top few loadings has no criterion for how far down to look. (c) takes the top five, and there is no ground for that number five.

There are many variables relative to the sample size

(a) is 23 cases × 37 variables, (c) 30 papers × 24 words. In both, the number of cases is close to, or below, the number of variables. With \(n\) cases and \(p\) variables, the rank of the sample covariance matrix is at most \(n-1\), so in (a) only 22 of the 37 dimensions carry any meaning.

The more fundamental problem is that with small \(n\) the estimate of the sample covariance matrix is itself unstable. Eigenvalues are known to be overestimated at the top and underestimated at the bottom, so explained proportions tend to be inflated as well. How far the 23.07% and 20.54% printed here reflect the structure of the population is not guaranteed at this sample size.

In fact, in (c) the first four components together come to only 52.29%, and the eigenvalues are not clearly separated. When eigenvalues lie close together, the corresponding component directions can swap places under a slight change of sample. Whether the first and second components drawn in the two-dimensional figure are stable directions cannot be judged from the materials.

Which variables are included changes the components

Principal components are defined relative to the set of variables put in. Add one variable and every component changes.

(c) selects 24 words from 6,096 distinct words, in decreasing order of the standard deviation of frequency normalized by paper length. The number 24 was chosen by eye, at the point where the rate of decrease in the standard deviation begins to stabilize. (a) arrives at 37 variables by deleting the columns for words that are + for both Tokyo and Kyoto and for words containing ○ or NR. Both are reasonable preprocessing steps, but a different criterion would yield different components. The explained proportion is a value for the selected set of variables, not for the original data as a whole.

A two-dimensional figure says nothing about the discarded dimensions

What Figures 8 and 9 of (a) explain is 23.07 + 14.73 = 37.80%. Figure 4 of (c) is no more than 20.54 + 12.1 = 32.64%. The remaining sixty per cent and more is not drawn on the plane.

So two points being close in a figure means no more than “close in these two dimensions”. They may always be far apart on a discarded axis. Where, as in (b), the first two components explain 98.95%, this worry is small. But if that high explained proportion derives, as described above, from redundancy among the variables, then another question remains in its place: what the four variables were measuring in the first place.

9Sources and data

(a) Vocabulary of the upper Ibi River basin

Onohara, A. Types and geographical distribution of vocabulary in the upper Ibi River basin. Research paper. The journal title, volume and year of publication could not be confirmed from the copy at hand, so the bibliographic entry is kept to the title.
The principal component analysis is in section 5.3, Figure 8 “Principal component analysis (component scores)” and Figure 9 “Principal component analysis (component loadings)”. The figure titles are “Individuals factor map (PCA)” and “Variables factor map (PCA)” respectively, with axes Dim 1 (23.07%) and Dim 2 (14.73%).
The original data are Tatsuya Yamada (1978) On the basic vocabulary of the upper Ibi River valley. Nagoya City University, Bulletin of the Faculty of Liberal Arts, Studies in Humanities and Social Sciences, 22, 33–66; and Tatsuya Yamada (1980) On the basic vocabulary of the Imajō and Ikeda dialects of Fukui Prefecture and the Kinomoto dialect of Shiga Prefecture, in relation to the Tokuyama dialect, ibid. 24, 23–34. The survey items are the 200 items of Hattori, S. (1973) Japanese dialects, Current Trends in Linguistics 11, 368–400.
Package ‘FactoMineR’ (2012) is listed among the references of that paper (its use is stated explicitly for the correspondence analysis).

(b) Vocabulary around the old Tokuyama Village

小野原彩香「旧徳山村周辺地域の語彙に関する数理的側面からの再検討――徳山村とその周辺には差が見られるか――」（同志社大学大学院文化情報学研究科文化情報学専攻博士後期課程1年当時）発表資料。
Onohara, A. A re-examination of the vocabulary of the area around the old Tokuyama Village from a mathematical standpoint: are differences to be seen between Tokuyama Village and its surroundings? Presentation material (then a first-year doctoral student, Graduate School of Culture and Information Science, Doshisha University). The file at hand records neither the venue nor the date of presentation, so the bibliographic entry is kept to the title and the affiliation.
The original data are Yamada (1978, 1980). The material states that the 200 items of basic vocabulary are those selected from the Swadesh List (Swadesh, 1950) by 服部四郎 (1955)「『言語年代学』即ち『語彙統計学』の方法について――日本祖語の年代――」季刊民族學研究 19(1), 100–101 (Shirō Hattori (1955) On the method of “glottochronology”, that is, “lexicostatistics”: the date of Proto-Japanese. Japanese Journal of Ethnology, 19(1), 100–101).
The 21 settlements × 4 variables table transcribed on this page holds the values printed on the “About the data used” and “Method of analysis” slides of that material.

(c) The analysis of submissions to Kōdō Keiryōgaku

小野原彩香・矢野環 (2014)「雑誌『行動計量学』における言語に関する投稿論文の傾向――投稿論文の時系列変化と『計量国語学』論文との比較」岸江信介・田畑智司編『テキストマイニングによる言語研究』ひつじ研究叢書〈言語編〉第121巻、pp. 1–20、ひつじ書房。ISBN 978-4-89476-695-2
Onohara, A. & Yano, T. (2014) Trends in language-related submissions to the journal Kōdō Keiryōgaku: temporal change and a comparison with Keiryō Kokugogaku. In S. Kishie & T. Tabata (eds.) Text Mining and Its Applications in Language Research, Hituzi Linguistics in English No. 121, pp. 1–20. Tokyo: Hituzi Syobo.
The principal component analysis is in section 4.1.2 (pp. 11–13). Figure 4 “Principal component analysis” (Variables factor map (PCA), Dim 1 (20.54%), Dim 2 (12.1%)); Figure 5 “First-component scores (left) and first-component loadings (right)”.
古沢正三・芳賀敏郎 (1992)『多変量解析事例集第1集』日科技連出版社
Furusawa, S. & Haga, T. (1992) Casebook of Multivariate Analysis, vol. 1. Tokyo: JUSE Press. (The work that paper uses for its definition of principal component analysis.)

References on the method

Pearson, K. (1901) On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2(11), 559–572. doi:10.1080/14786440109462720 (the formulation as minimization of the squared error of a projection)
Hotelling, H. (1933) Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), 417–441. doi:10.1037/h0071325 (the formulation as maximization of variance)
Eckart, C. & Young, G. (1936) The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211–218. doi:10.1007/BF02288367 (that the optimal low-rank approximation is given by the singular value decomposition)
足立浩平・村上隆 (2011)『非計量多変量解析法：主成分分析から多重対応分析へ』シリーズ〈行動計量の科学〉9、朝倉書店。ISBN 978-4-254-12829-1
Adachi, K. & Murakami, T. (2011) Nonmetric Multivariate Analysis: From Principal Component Analysis to Multiple Correspondence Analysis. Tokyo: Asakura Publishing.

Pearson (1901), Hotelling (1933) and Eckart & Young (1936) are general references cited here to situate the method on this page; they are not cited in the materials (a), (b) and (c) above.

Which perspective this method belongs to

Principal component analysis sets up no probability model. No likelihood, no prior appears. What appears is only a question of geometry: onto which plane should the data be projected so that their shape is best preserved?

And there are two ways of putting that question. One is the maximization of variance solved in section 2; the other is the minimization of the squared error of the projection. Project centred data \(x_i\) onto the line spanned by a unit vector \(w\), and the residual sum of squares is \(\sum_i \lVert x_i - w w^{\top}x_i \rVert^{2} = \sum_i \lVert x_i \rVert^{2} - \sum_i (w^{\top}x_i)^{2}\). The first term does not depend on \(w\), so minimizing the squared error and maximizing the sum of squares of the projected values (that is, the variance) are the same problem. Pearson (1901) reached the method from the first, Hotelling (1933) from the second.

This framework — a low-rank approximation minimizing squared error — is shared with correspondence analysis and with k-means. All that differs is what goes into the matrix and with which weights it is measured. This equivalence sits at the centre of Perspective 1 (geometry, least squares) in the seven perspectives in equations.

This page sets out, for readers in the same field, a method Onohara uses in her own research. All figures such as explained proportions are quoted from values printed in the sources. The checks mentioned in sections 5 and 8 (reproducing the explained proportions with the correlation matrix, and the range of the sums of Ep and En) were computed from the table published in the presentation material, and are identified as such. No estimates or approximations have been added.
For a method applying a different criterion to the same data, see random forest; for other methods that place points on a plane, see correspondence analysis and Neighbor-Net. The overall placement of methods is in the seven perspectives in equations, and which course sessions fall under which perspective is in Methods & Mathematics.