I just read The New Psychometrics. Science, Psychology and Measurement from Paul Kline (Routledge, 1998) who made well-acknowledge contributions to a better understanding of standardized measurement in psychological constructs.

Although I somehow disagree with his claims regarding the usefulness of “modern” psychometrics (IRT), I was pleased to read his clear account of Factor Analysis (Chapter 3). Here is a recap' of his “criteria for sound factor analysis” (pp. 62-64):

1. Sampling of subjects. It is essential in an exploratory factor analysis that a full range of subjects is sampled. If the sample is homogeneous for a factor, that factor will have little variance and thus not take up its proper place in the factor analysis. If, for example, in a study of personality, ann the subjects were bar staff or salespersons, who are likely to be highly extraverted, extraversion would be greatly attenuated.
2. Sampling of variables. In an exploratory analysis it is also essential that the full range of variables is sampled. Thus in the field of abilities it is certain that many factors have yet to be identified, especially those of a practical kind, not easily testable with standard, current psychometric tests. Examples of these would be rapport with horses (as in jockeys) and a mason’s skill at splitting bricks with a single blow.
3. Sample size. Standard textbooks vary on this point. However, there are two issues to be borne in mind. Since one of the aims of a factor analysis is to be able to reproduce from the factor loadings the original correlations it is essential that these are as free of statistical error as possible. For this reason alone, a sample size of 100 subjects is a desirable minimum. However, it is also agreed that the ratio of subjects to variables is important. If there are more variables than subjects the factor analysis is meaningless. There must always be more subjects than variables. However, textbooks vary in their recommendations concerning the ideal ratio, running from 10 to 1 (Nunnally, 1978) to 2 or 1 (Guilford, 1956). Barret and Kline (1981b) carried out an empirical study of this problem with a large sample of subjects using the EPQ test. They found that with a sample as low as 2 to 1 the main factors were retrieved and beyond 3 to 1 there were no improvements. A more recent study by Arrindel and van de Ende (1985) claimed that the critical factor was the number of subjects to factors, a ratio of 20 to 1 being required for stability. There is no doubt that the greater the number of subjects, the more stable and reliable the results are likely to be.
4. Principal factor rather than principal component analysis. Although Harman (1976) showed that in large matrices there was little difference between component and factor analysis, on theoretical statistical grounds factor analysis is to be preferred. This is because, by the estimating the diagonals in the correlation matrix, error variance is eliminated from the factor matrix. Furthermore, since the factors are estimates rather than the actual factors in the particular matrix, they are more likely to be generalizable to other matrices. There is some argument as to how these diagonals should be computed, and readers are referred to Harman (1976), Gorsuch (1974) or Catell (1978) for excellent discussions of this point.
5. Number of factors to rotate. Principal component and principal axis factor analysis produce a series of factors of decreasing size. as each one is extracted it accounts for less variance than the preceding factor. Factors were defined as the linear sums of variables, and since each variable has an eigenvalue of 1, any factor with a smaller eigenvalue than this must be of no significance. Thus some statistical packages choose for rotation all factors with eigenvalues greater than 1. However, it has been shown by Cattell (1978) that such a procedure can, in some instances, overestimate the number of factors. This criterion, therefore, becomes the lowest bound. It is certainly true that no factors should be rotated with eigenvalues less than 1. Cattell (1978) has always argued that selecting the right number of factors is crucial for obtaining simple structure. Rotate too many factors and factors split. Rotate too few and factors run together.
6. Selecting the right number of factors. If eigenvalues greater than 1 cannot be relied upon to select the right number of factors, what should be done? Barrett and Kline (1982) investigated various methods of factor extraction with reference to obtaining the correct number of factors. They found that two methods seemed to find the target: Cattell’s (1966b) Scree test and the Velicer (1976) method. Although the Scree test can be automated, it is a subjective method, essentially an algorithm which works effectively, and it is sensible to use both methods as a check and, if different, to rotate both sets of factors. Many factor analysts prefer the genuine statistical selection of the number of factors offered by factoring using the maximum likelihood method. However, attractive as this option appears, it is not without its own problems. The main one is that without a large sample the maximum likelihood estimates are not reliable (Krzanowski, 1988).
7. Choice of rotational procedure. Cattell (1978) regards this, together with the selection of the right number of factors, as the crucial element in obtaining simple structure. The need to rotate factors to simple structure arises from the fact that there is an infinity of equivalent mathematical solutions once the preliminary factor analysis has been computed. Orthogonal rotations yield uncorrelated factors and many factor analysts regard these solutions as the best, the uncorrelated set being necessarily simpler than any others. There is no doubt that the best procedure for orthogonal rotation is Varimax rotation (Kaiser, 1958), and this is contained in most computer packages for factor analysis. Cattell (1978) has been one of the main advocates of oblique factors on the grounds than in real life it would be unlikely that causal determinants, as he believes factors to be, would be uncorrelated. Furthermore, in allowing factors to take up any position relative to each other, each factor can be made as simple as possible, using the criteria of simple structure. There are many different structures for oblique rotation but studies of different methods in relation to simple structure tend to show that Direct Oblimin (Jenrich and Sampson, 1966) is highly effective (see Harman, 1976). Cattell favoured his own methods – Maxplane followed by Rotoplot – but these need considerable skill to operate and have been shown by Harman (1976) to be no better than Direct Oblimin, which is now the preferred method of rotation for most factor analysis.

A good discussion on the same points highlighted above is mirrored on PARE by Costello and Osborne (2005). A critical review of FA applied to psychological studies can be found in Ford et al. (1986).

Point 3 is also addressed in Falissard’s book about subjective measures in Health (Masson, 2008, 2nd ed.), chapter 4 p. 54: For a given instrument with five dimensions, a sample of 300 subjects allow to estimated principal factors with a 95% CIs of 0.1 (each tail). Obviously, multivariate normality must hold for this to be correct and confidence intervals for PCA eigenvalues is known to yield unreliable results with low or large sample size. MacCallum et al. (1999, 2001) provide insights into the importance of model error in sample size determination.

About Points 5 and 6: SPSS rotates all factors according to Kayser’s rule, unless otherwise specified. This is very sad because if the researcher is inclined to trust default option from SPSS, he will end up with an overestimated number of “true” factors in his data set. All FA-related functions in R call for the number of factors to estimate or rotate, which is a safer choice for the end user. To my opinion, the better way to handle this problem would be to rely on a close inspection of the scree test with simulated noise data superimposed on the real data (aka, Parallel analysis), as implemented in the psy (B. Falissard) and psych (W. Revelle) packages. Note that the same consideration applies to Cronbach’s alpha: Since it is only a lowest bound for reliability, it should not be reported alone. With a very large sample (say, N>800), using ML method will almost always lead to reject all factor solutions.

Point 7: To my knowledge, very simple structure is only available in the R psych package.

References cited in the above text are listed at the end of this article.

References

1. Nunnally, J.O. (1978). Psychometric Theory. New York, McGraw-Hill.
2. Barrett, P. and Kline, P. (1981). The observation to variable ratio in factor analysis. Journal of Personality and Group Behavior, 1, 23-33.
3. Guilford, J.P. (1956). Psychometric Methods. New York, McGraw-Hill.
4. Harman, H.H. (1976). Modern Factor Analysis. Chicago, University of Chicago Press.
5. Arrindel, W.J. and van de Ende, J. (1985). An empirical test of the utility of the observations to variables ratio in factor and components analysis. Applied Psychological Measurement, 9, 165-178.
6. Gorsuch, R.L. (1974). Factor Analysis. Philadelphia, Saunders.
7. Cattell, R.B. (1978). The Scientific Use of Factor Analysis. New York, Plenum.
8. Falissard, B (1998). Mesurer la subjectivité en santé. Perspective méthodologique et statistique (2nd ed.). Masson.
9. Barrett, P. and Kline, P. (1982). Factor extraction: an examination of three methods. Journal of Personality and Group Behavior, 2, 94-98.
10. Cattell, R.B. (1966). The Scree test for the number of factors. Multivariate Behavioral Research, 1, 140-161.
11. Velicer, W.F. (1976). Determining the number of components from the number of partial correlations. Multivariate Behavioral Research, 12, 3-32. A MATLAB implementation is available on B.P. O’Connor’s website.
12. Krzanowski, W.J. (1988). Principles of Multivariate Analysis. Oxford, Clarendon Press.
13. Kaiser, H.F. (1958). The Varimax criterion for analytic rotation in factor analysis. Psychometrika, 23, 187-200.
14. Jenrich, C.I. and Sampson, C.F. (1966). Rotation for simple loadings. Psychometrika, 31, 313-323.
15. Costello, A.B. and Osborne, J.W. (2005). Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most From Your Analysis. Practical Assessment Research and Evaluation, 10(7).
16. Ford, J.K., MacCallum, R.C., and Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291-314.
17. MacCallum, R.C., Widaman, K.F., Zhang, S., and Hong, S. (1999). Sample size in Factor Analysis. Psychological Methods, 4(1), 84-99.
18. MacCallum, R.C., Widaman, K.F., Preacher, K.J., and Hong, S. (2001). Sample Size in Factor Analysis: The Role of Model Error. Multivariate Behavioral Research, 36(4), 611-637.