Lee, "Reliability and validity of problem-solving assessments for Taiwanese high school students," 1999

Charlotte Yeh Lee, Ph.D

Abstract (Summary)

There is a call in the education field that our students need higher order thinking skills. Problem Solving is viewed by many as a higher order thinking skill, and was identified as a critical academic skill (foundation skill) by the SCANS study. Thus there is no doubt of the need, but the he major question is how can we know the students have learned problem solving skills, i.e., how do we measure problem solving. Fortunately, a method of assessment of problem solving has been developed by CRESST (National Center for Research on Evaluation, Standards, and Student Testing).

The three components of the CRESST problem solving assessment are defined as content understanding, problem solving strategies, and self-regulation. The approaches used for measurement of problem solving include knowledge maps for content understanding, explanation essays for problem solving strategies, and a four-scale self-regulation questionnaire for self-regulation. However, there is not enough reliability and validity evidence for this problem solving assessment approach. Moreover, there is no evidence for its cross cultural generalizability.

This study estimated the reliability and validity of the CRESST problem solving assessment with Taiwanese high school students. Generalizability theory was used to investigate the reliability of the problem solving assessment. Two pilot studies were completed in Taiwan, fall 1997. The first pilot study ( N = 302) tested the reliability and validity of the 32-item Chinese version of the self-regulation questionnaire. The second pilot study ( N = 152) investigated the other two problem solving assessment components, i.e., knowledge maps and problem solving strategy questions, with Chinese high school students. The main study ( N = 253) was also completed in Taiwan, during the spring 1998. The three-scale, 24-item. revised self-regulation questionnaire was used. At the same time, two knowledge maps and six problem solving strategy questions were tested on Taiwanese high school students.

Data analysis involved several statistical techniques, such as the SPSS' repeated measure design for comparing the variables differences, EQS/Windows was used for confirmatory factor analysis, and the ITRS was used for generalizability analysis.

Results of the main study showed there were high correlations among these three components of problem solving assessment, although the generalizability coefficients were not as high as expected. Gender differences exist in this study with boys performed better than girls. The format of the map had an impact on how well the students respond to the problem solving strategy questions. In addition, the counterbalancing had effects on students according to the sequence of the task. The detail of the summary, conclusion, results and a discussion of the finding concluded this dissertation.

Advisor: O'Neil, Harold, Jr.