Assessment of General Studies
The Academic Profile
To assess student learning in the Concord University General Studies Program for 2003-04, the Academic Profile Short Form was administered; the Academic Profile is a test of general academic knowledge and skills. It is designed for use by colleges and universities in assessing the outcomes of their general education programs for the purposes of improving the quality of instruction and learning. The College Board and Educational Testing Service jointly sponsor the Academic Profile.
The Test focuses on the academic skills students develop through general education courses rather than specific knowledge acquired about subject matter taught in General Studies courses. The Test measures skills in college-level reading, writing, critical thinking, and mathematics in the contexts of humanities, social sciences, and natural sciences. The Short Form consists of 36 multiple-choice questions to be answered in 40 minutes. Included in the questions are reading selections, graphs, tables, and artistic designs concerned with issues, themes, and ideas from the humanities, social sciences, and natural sciences. Scores are norm referenced for comparison with similar post-secondary students taking the same Test.
Comparative data for 2003-04 institutions will not be posted by Educational Testing Service, Incorporated on-line until Fall 2004. Instead, Concord University scores were compared with the scores of colleges and universities in the Baccalaureate, Liberal Arts category from the 2002-03 group. The twenty comparative institutions for the 2002-03 sample were:
- Albertus Magnus College, CT
- Mississippi Valley State University, MS
- Albright College, PA
- Missouri Baptist College, MO
- Buena Vista University, IA
- Morris College, SC
- Claflin University, SC
- Mount Vernon Nazarene University, OH
- College of Saint Benedict, MN
- Shawnee State University, OH
- Fisk University, TN
- Shorter College, GA
- Florida Southern College, FL
- Spelman College, GA
- Johnson C. Smith University, NC
- Trinity Christian College, IL
- Martin Methodist College, TN
- University South Carolina/Spartanburg, SC
- Metropolitan State College, CO
- Western State College, CO
The Concord University Sample
Faculty members for classes in which eligible students were enrolled distributed a letter from the Director of Assessment. Students were informed that they had been selected to participate in the assessment of the General Studies Program in order to assist the University in improving student learning. Students were asked to report for the testing at one of the dates and times listed.
A total of 70 Concord University students volunteered to participate in this assessment activity. They were identified by the Computer Center as having completed between 30 and 60 credit hours and successfully completed one Mathematics course, MATH 101, or higher. Sample students self-rated themselves as 3 freshman, 64 sophomores, and 3 juniors; a total of 70 student volunteers participated.
Analysis by Subgroup and Demographics
Mean scores were provided by age, core requirements completed, hours worked, enrollment status, ethnicity, gender, GPA, credit hours completed, language, program, hours transferred, and major. For further information on mean scores by demographic subgroups, contact Dr. Thomas Brewster, Director of Assessment. All but one participating Concord University students were full-time. National demographic data is given within parentheses.
The Concord University data provides a profile of the local sample.
- 33% (33%) under 20; 68% (55%) 20-29
- 46% (26%) male; 54% (74%) female
- 50% (31%) not employed
- 26% (29%) working 1-15 hours per week
- 19% (20%) working 16-30 hours per week
- 6% (20%) working more than 30 hours per week
- 99% White (37%)
- 1% African-American (55%).
- 93% (79%) not a transfer
- 4% (5%) transferred 0-15 hours
- 3% (6%) transferred 16-30 hours
The Concord University subgroup mean comparisons to the national samples subgroup means are as follows. Concord University’s older students, (20-29, N=46), achieved the highest mean score in all categories. Students working 16-30 hours per week (N=13) achieved the highest mean scores in three of seven categories (Mathematics, Humanities, and Natural Science) with those not employed (N=35) scoring highest in two categories (Critical Thinking and Social Science). Students working 1-15 hours per week (N=18) and students working more than 30 hours per week (N=4) achieved the highest mean scores in one category each.
Males scored higher in three areas; Critical Thinking, Mathematics, and Social Science with Females scoring higher in four areas; Reading, Writing, Humanities, and the Natural Sciences. Overall, Males had a higher total mean score (N=32, 452.13) than the Females (N=38, 451.32) taking the test. Students with 3.5-4.0 GPAs (N=24) achieved the highest mean score in all categories. Scores for GPA ranges with fewer than three students are not reported here. Sophomores (N=64) scored equal to or higher than the freshmen (N=3) and juniors (N=3) in two categories (Writing and Social Sciences). Those students completing about 75 percent (N=43) of the core requirements scored the highest total mean score, 452.26. For further information on success of Concord University students by demographic indicators see Appendix A.
The summary mean scores for the Concord University sample will be reported and compared to comparable mean scores achieved at the 2002-03 participating institutions. Comparative scores for the 2003-04 participating institutions will be reported in Fall 2004. The national data from the 2002-03 sophomore sample were used to determine the success rate of Concord University’s sample.
The Concord University students’ (N=70) total mean score was 451.69, SD 16.37. Last year, the students’ (N=65) total mean score was 451.81, SD 18.37. The possible range for the Test is 400-500. The range of Concord University’s individual mean scores was 419-489. The 95% Confidence Limit for the total mean was 448 to 456. “These confidence limits were computed by a procedure that has a 95 percent probability of producing upper and lower limits that will surround the true population mean.” The 451.69 mean score places Concord University’s participants near the top when compared to other Baccalaureate Liberal Arts colleges. In fact, only two other of the other 20 institutions scored at a higher level than Concord University. The mean of the 21 institutions (N=2,456) was 440.97, SD 9.54. In 2001-2002, the mean score of the 13 comparable institutions (N=929) was 437.80, SD 10.52. Concord University mean score is higher than the mean score achieved at Master’s Colleges and Universities I and II, 442.73, SD 6.95; Concord’s total mean 451.69.
|Concord University||National Sample|
|Concord University||National Sample|
As can be noted from the data, Concord University students had higher mean scores than the National Sample in every subscore category. In Reading and Mathematics, one institution scored higher. In Humanities, one institution scored at the same level as Concord and one institution scored higher. In the Social Sciences, five institutions scored at the same level as Concord and one institution scored higher. In the area of Critical Thinking, four institutions scored higher. In Writing, two institutions scored at the same level as Concord and seven institutions scored higher. In Natural Science, five institutions scored higher.
Proficiency ratings are criterion references for Reading and Critical Thinking, Writing, and Mathematics. The ratings are noted as Proficient, Marginal, or Not Proficient. Ratings for the Concord students were as noted below.
|Reading, Level 2||50%||19%||31%|
|Reading, Level 1||86||10||4|
|Writing, Level 2||9||54||37|
|Writing, Level 1||76||20||4|
|Math, Level 2||40||37||23|
|Math, Level 1||74||19||7|
The three lowest proficiency ratings achieved by Concord University students were at Level 3 for Critical Thinking, Mathematics, and Writing. Thus, it was at the highest testing levels that Concord University students achieved their lowest proficient scores. See Attachment A for criterion for the higher levels.
|Writing, Level 3||4||26||70|
Scores for majors with fewer than three students are not reported here. Of the student-identified majors reported, the 3 Mathematical Sciences students averaged the highest mean score at 471.67, SD 15.31.
Mean Scores by Major and Standard Deviations are as follows.
|Major Not Listed||5||451.60||19.32|
The essay portion of the assessment protocol is designed to measure students’ ability to write an extended response to a specified topic in the humanities, the social sciences, or the natural sciences. Students were given 10 minutes to read selections on all three topics and choose one of interest. They had 45 minutes to plan and write an essay on the selected topic. The essay examination is designed to test writing skills plus a student’s familiarity with terms, examples, and issues from general education college-level courses.
The Computer Center generated a list of sample students who had completed between 30-60 hours and were enrolled in ENGL 203 and 204. Faculty members, for the ENGL 203 and 204 sections in which eligible students were enrolled, distributed letters from the Director of Assessment. Students were advised that they would be taking the Essay Exam during a class period.
The essay portion was supervised and evaluated by four faculty members in Literature and Languages: Dr. Martha Shrewsbury, chair; Dr. Liz Roth; Ms. Cheryl Mays; and Ms. Rosalie Peck. A two and a half hour orientation session for the evaluators was conducted prior to the evaluation of the 122 writing samples. The faculty members listed above administered the Essay Examination during regularly scheduled class periods.
The Essay Examination assesses how well students have mastered the content and discourse of a general education curriculum and how they can express knowledge gained in an analytic essay. Each question presents a quotation and a writing task based on that quotation. The quotation and the writing tasks are meant to reflect the level of sophistication to present issues that a student encounters in a general education curriculum in humanities, social sciences, or natural sciences.
The Academic Profile Essay is scored with a holistic scoring procedure and a four-point scale defined by the testing company. See Attachment B for listing of the scoring standards. When evaluating student writing, one looks at the level of discourse, discrete errors and accomplishments at the word and phrase levels, such as language choice, grammar, spelling and punctuation. The evaluator also examines the students’ ability to articulate and develop an idea, marshall appropriate evidence in support of an argument, maintain a consistent tone, and demonstrate an awareness of audience.
Concord University student achievements on The Academic Profile Essay fell into the following groupings with 4 being high and 1 low. The data indicates that 35 percent achieved at the 3-4 level and 64 percent achieved at the 1-2 level.
Summary and Conclusions
Assessment of Concord University student learning in the General Studies Program was conducted Spring 2004 using The Academic Profile Short Form. The 2002-03 comparative sample of 20 similar institutions indicates that only two other institutions scored at a higher level than Concord University. The Concord University total mean score on the standardized test is higher than those achieved at Master’s Colleges and Universities I and II. On the Essay Exam, locally evaluated, 35% of Concord University students placed in highest levels 3-4.
Criterion for the Higher Proficiency Levels
Criterion for the Higher Proficiency Level, Level 3
Level 3—Critical Thinking
In addition to performing Level 1 and 2 skills successfully, students at Level 3 can also evaluate and analyze arguments and within an academic field handle interpretation, inductive generalizations, or causal explanations.
Level 3 skills are differentiated within those areas as follows:
- Humanities: Evaluate views and interpretations
- Social Sciences: Evaluate claims, disputes, and inductive generalizations
- Natural Sciences: Evaluate explanatory hypotheses and draw conclusions
In addition to performing Level 1 and 2 skills successfully a student at Level 3 also can generalize and apply mathematical knowledge and skills in nonroutine situations, and demonstrates real comprehension of exponents, variables, geometry, and measurements. A student at this level can solve multi-step and nonroutine problems involving a range of reasoning skills.
In addition to performing Level 1 and 2 skills successfully, a student at Level 3 also can identify logical statements and comparisons and is able to solve difficulty or subtle writing problems such as appropriate use of parallelism. These students can make fine distinctions among closely related root words and grammatical structures characteristic of a mature writing style.
Academic Profile Essay Scoring Standards
Academic Profile Essay Scoring Standards
A paper scored at 1:
Fails to address the task presented in the topic in one or more of the following ways:
- fails to demonstrate understanding of the quotation and/or the task presented by the topic
- is so incoherent that the paper cannot be followed
- depends on feelings, beliefs, or clichés to develop the essay rather than knowledge of relevant coursework
- displays writing deficiencies so severe that the essay does not convey information
A paper scored at 2:
Demonstrates an understanding of the quotation but fails to address the task in one or more of the following ways:
- depends on poorly selected or inaccurate examples from coursework
- fails to develop examples adequately
- merely lists (phrases, theories, authors, concepts)
- provides abstractions and generalizations related to the discipline or topic, but fails to develop, explain, or effectively incorporate them into the essay
- addresses only one part of the task
- provides well-developed examples but does not relate them to the topic
A paper scored at 3:
- Demonstrates the ability to use the discourse and analysis appropriate to academic discipline
- Displays a clear understanding of the quotation and the task presented in the topic
- sustains a focused discussion
- Uses evidence to support a point, e.g., uses a single well-developed example or presents several pertinent, though not thoroughly developed, examples
- displays a level of writing skill that does not interfere with the conveying of information
A paper scored at 4:
- demonstrates that ability to use the discourse and analysis appropriate to the academic discipline
- displays a clear understanding of the quotation and the task presented in the topic
- sustains a focused discussion
- uses evidence to support a point, e.g., uses consistently well-developed, well-chosen examples
- demonstrates an awareness of or insight into the complexities implied in the quotation
- avoids sweeping generalizations, clichés, and unsupported assertions
- displays a level of writing skill that supports and enhances the discussion