Spss and sas programs for generalizability theory analyses. The goal of the book is to present different aspects and applications of generalizability theory in a way that allows the reader to apply the theory. This book introduces the theory and practice of measurement in education and psychology, with applications in the statistical software r using opensource data and functions. Morris, and murat kilinc university of iowa abstract although widely recognized as a comprehensive framework for representing score reliability, general. How elitelevel soccer dynamics has evolved over the last. Issues in combining generalizability theory and item response. However, the stability of scores from depression measures has generally been estimated using only classical test score theory, rather than the more powerful generalizability theory. The essential features of univariate generalizability theory were largely completed with technical reports in 1960. Although multivariate generalizability theory was introduced more than 30 years ago, its use has been relatively limited. The nice thing about using for g theory is that you can choose functions for many different packages to fully apply g theory.
Generalizability theory was developed as an overarching structure with which to evaluate the dependability of psychological measurements. Generalizability g theory, which was developed by cronbach, gleser, nanda and rajaratnam 1972 as an alternative to the classical test theory ctt, is a statistical theory for evaluating the dependability or reliability of behavioral measurements brennan, 2001a. Shavelson is an educational psychologist who has published over 100 research articles and books in the fields of educational assessment, cognitive psychology, and science education. A multivariate generalizability analysis of data from a.
Generalizability theory subsumes and extends classical test. A recent extended treatment of the theory is the 2001 book by robert l. Oconnor lakehead university, thunder bay, ontario, canada the identification and reduction of measurement errors is a major challenge in psychological testing. We also applied generalizability theory analysis brennan, 2001.
General medical council gmc in the uk has emphasized the importance of internal consistency for students assess. Because it is possible to have unbalanced data sets while conducting a study, the researchers have devised an easy solution, other than deleting data, to balance the design to cope with this situation. Reliability and therapeutic decision through generalizability theory. The reliability and validity of big five inventory scores with african american college students. Generalizability theory rasch measurement software.
The fundamental differences between classical test theory and generalizability theory have been stated by shavelson, webb, and rowley 1989, p. Validity studies the childrens depression inventory. Contexts in which such measurement prevails include, but are not limited to, performancebased assessments, standard settings, and content validation studies. Variance components in generalizability theory springerlink. Introduction to educational and psychological measurement using r. This study investigated the dependability of scores from the childrens depression inventory using both generalizability and classical test score analyses. Issues in combining generalizability theory and item response theory in this presentation, i compare historical, procedural, technical and practical aspects of irt and gt and discuss the issues in the combination of the two theories. Generalizability theory g theory has been presented as a way to refine the designs of measurement procedures in an attempt to yield reliable data. A primer, which made the basics of g theory more understandable and the theory more applicable for relevant research studies. A modification of the schutte emotional intelligence scale. Oct 10, 2012 ive been using synthetic data from brennans book and shavelson and webb s book to ensure that my results match theirs.
The concept of rel iabil ity, so fundamental to classical o 8 eric l l theory, is replaced by the broader and more flexible notion of general izabil ity. Introduction to educational and psychological measurement. Yet despite the strong interest in g theory within the measurement community, applications of it are still rare when reporting results for measures of individual differences. In this volume, shavelson and webb offer an intuitive develo. Assessment of the intrarater and interrater reliability of an. Generalizability theory of behavioral measurements. The aim of this paper is to introduce a software that is.
G theory demands judging designs for which the variance terms are estimable. Analysis of observational data in speech and language. One reason for the common neglect of generalizability theory is the absence analytic facilities for this purpose in. Facets demands partial crossing, such as by having raters at all test centers rate the same videotaped performances. Generalizability theory gt provides a flexible, practical framework for examining the dependability of behavioral measurements. Application of generalizability theory to the big five. Pdf spss and sas programs for generalizability theory analyses. Oconnorspss and sas programs for generalizability theory analyses behavior research methods, 38 2006, pp.
Ppt generalizability theory powerpoint presentation free. Applications of generalizability theory and their relations to classical test theory and structural equation modeling walter p. Coping with unbalanced designs of generalizability theory. It is also meant to provide a unifying theory for evaluating the psychometric soundness of a test. Application of generalizability theory to the big five inventory. Document resume ed 377 218 tm 022 386 author thompson, bruce. Pdf generalizability theory gt provides a flexible, practical. This article provides a brief introduction to generalizability theory, describes easy to use spss, sas, and matlab programs for conducting the recommended analyses, and provides an.
He is an emeritus professor in the stanford graduate school of education, a former dean of the stanford graduate school of education, and a past. In soccer settings, generalizability theory has been applied to investigate the in. Pdf generalizability g theory is a statistical theory for evaluating the. Page 4 of 7 encyclopedia of social science research methods. Gtheory demands judging designs for which the variance terms are estimable. Multivariate ge neralizability theory mgt, in contrast with univariate generalizability theory ugt, was developed to meet the challenge rajaratnam, 1965.
Concepts central to generalizability theory are discussed, and a hypothetical data set illustrates the usefulness of generalizability measurements in speech and language research. Later in 1991, shavelson and webb published their book generalizability theory. A primer measurement methods for the social science book 1 kindle edition by shavelson, richard j. Brennan 1992 developed gt into univariate generalizability theory and, from this, proposed multivariate generalizability theory mgt, in which analysis of measurement errors is performed in greater depth, so that more accurate results are obtained in terms of scale reliability. One reason for the common neglect of generalizability theory is the absence of analytic facilities for this purpose in popular statistical software packages. It provides the background on g theory, but if youre hoping to conduct your own g study, youll also need brennans generalizability theory and the genova software that is free and available on the web.
A primer measurement methods for the social science book 1 ebook. Fortran programs for g theory analyses have been provided by. In this volume, shavelson and webb offer an intuitive development of generalizability theory and cover a wide variety of. Most investigators rely solely on classical test theory for assessing reliability, whereas most experts have long recommended using generalizability theory instead. Generalizability and transferability the wac clearinghouse. It is meant as an alternative to the correlational analyses which are typically used in classical test theory. Comparing the effectiveness of spss and edug using different. After the appropriate design is identified, the gtheory analysis may proceed. This is a wellwritten, easytodigest book that i was able to read in two sittings over one weekend. Applications of generalizability theory and their relations. Brennan2 1medical education unit, the university of nottingham, uk 2centre for advanced studies in measurement and assessment, the university of iowa, usa. Ive been using synthetic data from brennans book and shavelson and webb s book to ensure that my results match theirs. A collection of articles on generalizability theory. The aim of this paper is to introduce a software that is appropriate for the generalizability theory for not only balanced but also unbalanced data sets.
Reliability coefficients and generalizability theory. Application of generalizability theory in estimating the. Comparing the effectiveness of spss and edug using. A multivariate generalizability analysis of student style. Shavelson and webb 1991 provide example code for the bmdp software, which is no longer available. A detailed description of generalizability theory is beyond the scope of the current investigation. Spss and sas programs for generalizability theory analyses christopher mushquash and brian p. By using detailed illustrations and examples, shavelson and webb clearly describe the logic underlying major concepts in generalizability theory to enable readers to apply these methods when investigating the consistency of their own measurements. Facets accepts deliberate and accidental missing data. There are two elements to gtheory, generalizability g studies and decision d studies. The present discussion of the benefits of generalizability theory will be brief, given space limitations. By using detailed illustrations and examples, shavelson and webb clearly describe the logic underlying major concepts in generalizability theory to enable readers to apply these methods when. Also, shavelson and webb 1991 have provided a primer on generalizability theory. The advantage of g theory lies in the fact that researchers can estimate what proportion of the total variance in the results is due to the individual factors that often vary in assessment, such as setting, time, items, and raters.
1425 774 1370 76 1285 1375 397 1321 1254 84 418 672 714 930 1345 1541 1296 257 835 348 288 564 1495 229 942 497 1034 117 616 500 184 944 1656 1079 621 191 32 1276 251 129 387 5 421 565 500 303 791 1290