Scale Development and Psychometrics
Researchers in the academic and private sectors often need to measure some aspect of people’s psychology be it their attitudes, satisfaction, motivation, or intentions. We assume that the numbers these scales, questionnaires, tests, and surveys produce are meaningful: that someone with a higher satisfaction score is in fact more satisfied than someone with a lower score. Because scale scores are used to make decisions like how to measure critical outcomes in a research study, develop a product, or admit a student or promote an employee, researchers need to thoroughly evaluate their validity. This short course will cover how to develop, evaluate, and refine scales using modern psychometric methods.
In this course, you will learn how to apply modern validity theory and psychometric methods to appropriately develop and use scales measuring psychological attributes.
- Overview of construct validity theory and types of validity evidence
- Item writing
- Item content review and think-aloud protocol
- Executing and interpreting item analysis
- Overview of types of factor analysis
- Executing and interpreting exploratory factor analysis
- Executing and interpreting reliability analysis
- Interpreting and evaluating validity evidence for scale selection and use
TIME COMMITMENT AND COURSE DELIVERY
The course will be delivered in person at the University of Calgary’s downtown campus (906 8th Avenue SW, Calgary, Alberta, Canada). The course will run on December 9 and 10, 2022 from 9:00am - 5:00pm both days.
The course will focus on scale development and refinement with psychometric methods that can be implemented in many statistical software packages. Because this is a hands-on course, learners are encouraged to bring a laptop to class with a copy of R or SPSS installed. However, instruction will focus on demonstrating the statistical techniques and interpreting the most common outputs in various software programs. Provided materials and examples will include analysis scripts with annotated output from both SPSS and R.
WHO WILL BENEFIT?
This course will be helpful for researchers in any field —including psychology, sociology, education, business, human development, social work, public health, communication and others that rely on social science methodology —who want to develop and use scales to measure psychological attributes. Learners should have background knowledge in introductory statistics topics such as univariate statistical tests, descriptive statistics, and correlation. Ideally learners should be comfortable with multiple regression techniques. Though proficiency in a specific software isn’t required, ideally participants will have some familiarity with running analyses using some type of statistical software (e.g., R, SPSS, SAS, STATA).
Upon completing this course, you will
- Be able to define construct validity and describe different forms of validity evidence
- Evaluate scale items for poor, confusing, or problematic wording
- Use descriptive statistics to quantitatively evaluate item properties
- Use qualitative approaches to review item content
- Compare different approaches to factor analysis
- Compare different approaches to quantifying reliability
- Interpret an exploratory factor analysis
- Interpret a reliability analysis
- Evaluate multiple sources of validity evidence to select a scale
- Evaluate multiple sources of validity evidence to develop or refine a scale
Dr. Flake is an Assistant Professor of Quantitative Psychology and Modelling at McGill University. She received an MA in quantitative psychology from James Madison University and a PhD in Measurement, Evaluation, and Assessment from the University of Connecticut. Her research develops and applies latent variable models for use in psychological research with an emphasis on improving measure development and use. Her work is highly cited and published in top methodological and substantive outlets such as Nature: Human Behavior, Psychological Methods, Advances in Methods and Practices in Psychological Science, Structural Equation Modeling, Psychological Science, and the Journal of Personality and Social Psychology. She was named an Association of Psychological Science Rising Star in 2021 and received a Society for the Improvement of Psychological Science Commendation in 2020 for her research into questionable measurement practices.
Her work focuses on technical and applied aspects of psychological measurement including scale development, psychometric modelling, and scale use and replicability. She is a top-rated professor in the Department of Psychology at McGill University, regularly teaching measurement and statistics courses as well as workshops at international conferences. Further, she routinely works in applied psychometrics as a technical advisory panel member for the Enrollment management Association, a non-profit that develops educational assessments, and serves as the Assistant Director for Methods at the Psychological Science Accelerator, a laboratory network that conducts large-scale studies. More information about her work can be found at https://www.jessicakayflake.com/