HSK 844 - CCRAM: Scale Development and Psychometrics

Haskayne Executive Education

Course Description

Scale Development and Psychometrics

Researchers in the academic and private sectors often need to measure some aspect of people’s psychology be it their attitudes, satisfaction, motivation, or intentions. We assume that the numbers these scales, questionnaires, tests, and surveys produce are meaningful: that someone with a higher satisfaction score is in fact more satisfied than someone with a lower score. Because scale scores are used to make decisions like how to measure critical outcomes in a research study, develop a product, or admit a student or promote an employee, researchers need to thoroughly evaluate their validity. This short course will cover how to develop, evaluate, and refine scales using modern psychometric methods.

Course Details

LEARNING STATEMENT

In this course, you will learn how to apply modern validity theory and psychometric methods to appropriately develop and use scales measuring psychological attributes.

Overview of construct validity theory and types of validity evidence
Item writing
Item content review and think-aloud protocol
Executing and interpreting item analysis
Overview of types of factor analysis
Executing and interpreting exploratory factor analysis
Executing and interpreting reliability analysis
Interpreting and evaluating validity evidence for scale selection and use

TIME COMMITMENT AND COURSE DELIVERY

The course will be delivered in person at the University of Calgary’s downtown campus (906 8^th Avenue SW, Calgary, Alberta, Canada). The course will run on December 9 and 10, 2022 from 9:00am - 5:00pm both days.

COMPUTING

The course will focus on scale development and refinement with psychometric methods that can be implemented in many statistical software packages. Because this is a hands-on course, learners are encouraged to bring a laptop to class with a copy of R or SPSS installed. However, instruction will focus on demonstrating the statistical techniques and interpreting the most common outputs in various software programs. Provided materials and examples will include analysis scripts with annotated output from both SPSS and R.

WHO WILL BENEFIT?

This course will be helpful for researchers in any field —including psychology, sociology, education, business, human development, social work, public health, communication and others that rely on social science methodology —who want to develop and use scales to measure psychological attributes. Learners should have background knowledge in introductory statistics topics such as univariate statistical tests, descriptive statistics, and correlation. Ideally learners should be comfortable with multiple regression techniques. Though proficiency in a specific software isn’t required, ideally participants will have some familiarity with running analyses using some type of statistical software (e.g., R, SPSS, SAS, STATA).

LEARNING OUTCOMES

Upon completing this course, you will

Be able to define construct validity and describe different forms of validity evidence
Evaluate scale items for poor, confusing, or problematic wording
Use descriptive statistics to quantitatively evaluate item properties
Use qualitative approaches to review item content
Compare different approaches to factor analysis
Compare different approaches to quantifying reliability
Interpret an exploratory factor analysis
Interpret a reliability analysis
Evaluate multiple sources of validity evidence to select a scale
Evaluate multiple sources of validity evidence to develop or refine a scale

INSTRUCTOR BIO

Dr. Flake is an Assistant Professor of Quantitative Psychology and Modelling at McGill University. She received an MA in quantitative psychology from James Madison University and a PhD in Measurement, Evaluation, and Assessment from the University of Connecticut. Her research develops and applies latent variable models for use in psychological research with an emphasis on improving measure development and use. Her work is highly cited and published in top methodological and substantive outlets such as Nature: Human Behavior, Psychological Methods, Advances in Methods and Practices in Psychological Science, Structural Equation Modeling, Psychological Science, and the Journal of Personality and Social Psychology. She was named an Association of Psychological Science Rising Star in 2021 and received a Society for the Improvement of Psychological Science Commendation in 2020 for her research into questionable measurement practices.

Her work focuses on technical and applied aspects of psychological measurement including scale development, psychometric modelling, and scale use and replicability. She is a top-rated professor in the Department of Psychology at McGill University, regularly teaching measurement and statistics courses as well as workshops at international conferences. Further, she routinely works in applied psychometrics as a technical advisory panel member for the Enrollment management Association, a non-profit that develops educational assessments, and serves as the Assistant Director for Methods at the Psychological Science Accelerator, a laboratory network that conducts large-scale studies. More information about her work can be found at https://www.jessicakayflake.com/