The SAT III?
January 18, 2004
By CECILIA CAPUZZI SIMON
As any parent will attest, predicting the behavior of an
evolving, impressionable 17-year-old is a crapshoot. Will
he show up at an 8 a.m. class, or party too much? Will he
stay in college in the face of personal or financial
challenges, or even persevere through a difficult course?
''To stick with college shows remarkable resilience and
initiative,'' says Wayne Camara, vice president of research
at the College Board, which administers the SAT. ''They are
different skills than cognitive abilities.'' The SAT, the
admissions exam used by 80 percent of American colleges,
addresses only a student's cognitive abilities, or academic
aptitude. It does not tell the whole story.
Colleges roll the dice every year around this time, as
applications flood admissions offices. Small elite colleges
can delve the inner soul with essays, interviews and
intuition, but most large universities must rely on cold
hard numbers: grade point average and admissions test
scores.
To better predict college readiness, Dr. Camara says, ''we
need to look at noncognitive factors -- personality,
temperament, flexibility, proclivity to learn, ability to
adjust, to get along with a roommate, to make appropriate
decisions about studying.''
To the tune of almost $2 million so far, the organization
is backing two outside research projects to identify and
quantify such traits, much as employers do in personnel
testing and hiring, and develop a standardized test that
might supplement the SAT. Two prominent psychologists head
the projects: Neal Schmitt, chairman of the department of
psychology at Michigan State University, and Robert J.
Sternberg, a professor at Yale whose term just ended as
president of the American Psychological Association (where
Dr. Camara headed testing before joining the College Board
in 1994). Both projects have recently completed initial
phases with promising results, but the tests would be at
least six years off, and how they would be used has not
been determined. While cautious about the noncognitive
future, Gaston Caperton, the College Board president, says
the board is ''serious about this.''
''We know the value of a test of analytic ability,'' Mr.
Caperton says. ''But we also recognize that there are other
kinds of intelligence, and that schools would want to know
more about them because they are important to students'
performance, and to their careers.''
Dr. Schmitt, an expert in personnel selection, borrowed
concepts from conventional employment tests that present
situations and ask test takers to choose a reaction. Dr.
Sternberg's test has similar elements but includes a more
controversial component: cartoon captioning and short-story
writing to quantify creativity.
Dr. Sternberg says the goal of his research, which he calls
the Rainbow Project, is twofold: to increase a college's
chance of identifying students who will profit from its
environment and to counteract the pronounced gap on SAT
scores of white and non-Asian minority applicants. His test
challenges traditional ideas about intelligence. In his
view, there are three separate intelligences: analytic,
creative and practical. Analytic intelligence -- what the
SAT measures -- lends itself to memorization and analyzing
information; people express creative intelligence by
applying knowledge in a novel way; practical intelligence
reveals itself in everyday situations. Dr. Sternberg says
that analytic intelligence may not be enough to excel in
college or in life. A person who is strong in creative or
practical skills may thrive in an academic setting even if
he doesn't excel in the analytic skills tested by the SAT.
That student, however, is derailed by the test.
Perhaps the most elusive aspect of Dr. Sternberg's test is
his effort to measure creativity. In Phase 1, students were
asked to write short stories based on a title -- say, ''The
Octopus's Sneakers.'' Others dictated stories into tape
recorders based on a page of loosely linked drawings.
Students also wrote captions for old New Yorker cartoons.
The test was given to 973 students at 13 colleges and
graded by trained readers. Dr. Sternberg says the test is
twice as likely as SAT scores to predict students'
first-year grades. It also narrowed the gap between
minority and white scores. In Phase 2's longitudinal study,
more than 5,000 students will be followed through four
years of college. The University of Illinois at
Urbana-Champaign, the University of Indiana at Bloomington
and Yale have committed to participate.
For his study, Dr. Schmitt searched the mission statements
of 35 four-year colleges looking for keywords and goals
describing ideals and personality traits the colleges
valued. He boiled them down to 12 ''dimensions.'' Among
them: motivation to continue learning beyond the classroom,
an appreciation of art and music, multicultural tolerance,
ability to communicate effectively and get along with
others, leadership, social responsibility, perseverance,
ethics and adaptability. From those, he developed
situational questions. For example: What would you do if
you discovered that your roommate was writing essays for
other students for a fee? There is even a question about
the ''freshman 15'': What if you're already overweight and
add 15 pounds? Another part of the test asks biographical
questions to tease out leadership traits, community
mindedness and desire to learn beyond the classroom.
The test was given to 654 freshmen at Michigan State. Dr.
Schmitt found that his test, when combined with ACT and SAT
scores, improved the chances of predicting freshman grades
by 6 percentage points. The test outperformed the SAT and
ACT in predicting absenteeism, and outperformed peer
ratings and self-ratings in the 12 dimensions. Dr. Schmitt
expects to administer a revised version at about a dozen
campuses.
A spokesman for ACT Inc. says it is not developing a
similar test and would not comment about the College
Board's efforts. But Dr. Schmitt and Dr. Sternberg are not
the only educators studying noncognitive measures to gauge
college readiness.
William E. Sedlacek, professor of education at the
University of Maryland, uses dimensions similar to Dr.
Schmitt's but also measures factors like self-image, family
support and how a student copes with racism.
A consortium of universities -- North Carolina State
University, Appalachian State University, North Carolina
A&T State University, University of North Carolina-Pembroke
and the University of Maryland -- has applied for a
$400,000 federal grant to help create a test based on Dr.
Sedlacek's research that would ''more accurately assess
university applicants'' than admission exams can do alone,
says Raymond Ting, associate professor of counselor
education at North Carolina State. Dr. Ting, who
coordinated the grant application, says efforts to
standardize psychological and personal measures are the new
wave, especially as higher education's minority population
grows. ''Students have to be considered more fully as
individuals, not just test scores,'' he adds.
Dr. Sedlacek finds it pointless to revisit traditional
tests of academic readiness. ''We focus too much on how to
improve what we already have, like the SAT,'' he says.
''But that's the wrong question. We don't know how to do
the verbal or quantitative testing any better than we do.''
Oregon State University is using Dr. Sedlacek's measures to
assess applicants' potential, to provide direction once the
student is on campus, and in making financial aid
decisions. The measures are also used by the Bill and
Melinda Gates Foundation and the National Action Council
for Minorities in Engineering to select students for
scholarships. Dundee Holt, the council's vice president for
development and communications, says the council finds such
measures a better ''indicator of success than the SAT for
minority students'' and doesn't consider entrance exam
scores. ''We've had enough students who've earned their
doctorate with 900's on their SAT's to know that there are
other things going on,'' Mr. Holt says.
SO which student will make it through four years of
college? According to a study by the Department of
Education, the most significant precollege factor in
whether a student graduates is the intensity of high school
curriculum. Curriculum ''reflects 41 percent of the
academic resources students bring to higher education,''
the report states. Class rank and grade point average --
less reliable benchmarks because of disparities in school
quality and standards nationwide -- reflect 29 percent and
test scores just 30 percent.
With the SAT under fire from several quarters, including a
handful of elite colleges that have recently made scores
optional, it's not surprising that the College Board wants
to increase its ability to assess students. Complaints
about the exam are well documented: It favors whites and
males, and an overemphasis on scores has turned the
admissions process into the ''educational equivalent of a
nuclear arms race,'' as Richard Atkinson, former president
of the University of California, once put it. Dr.
Atkinson's threat to drop the SAT as an application
requirement jarred the College Board into revamping the
exam to better reflect high school curriculum. The changes
are due in 2005.
In addition, the Supreme Court ruling last June in a
University of Michigan case -- which upheld a university's
right to consider race in admissions but outlawed quotas --
has admissions officials scrambling for ways to identify
promising minority students.
Educators have a new buzzword to describe the admissions
process: ''holistic.''
Jerome Lucido, vice provost for enrollment and management
at the University of North Carolina at Chapel Hill, says
his first thought when hearing a presentation on Dr.
Sternberg's test was: ''That's exactly the information
we're trying to extrapolate when we read an applicant's
portfolio.''
At competitive schools, academic achievement is a given in
any application, making entrance exam scores almost beside
the point in admissions decisions. ''This generation of
students is so good we don't have to worry about academic
performance,'' Dr. Lucido says. ''What we need to look at
is whether they are good citizens; will they challenge the
faculty, will they engage in research?''
MOST large public universities don't have the resources to
review portfolios. Penn State, for example, received just
over 51,000 applications for its freshman class of 2003.
''We're not 'reading' every application,'' says Randall
Deike, its director of research for enrollment management
and administration. For such universities, which rely on
test scores and grade point average, a test that measures
motivation, leadership and good judgment would seem just
the ticket.
But the tests face hurdles not only in design but also in
institutional and public acceptance. ''I'm a great admirer
of their goals,'' says Paul Sackett, professor of
psychology at the University of Minnesota and a member of
the College Board research advisory committee. ''The real
question is: Will they be able to develop a test that holds
up under scrutiny?'' Foremost, as Dr. Sackett points out,
is controlling for fakery. You can't bluff a math problem.
But looking at sample questions, it's not hard to figure
out socially acceptable responses. While fabricators may be
flagged by tricks embedded in the tests -- asking follow-up
questions or multiple questions designed to illicit the
same response -- such tests are not prep-proof. Coaching
can teach how to write a creative caption or answer
questions in a way that reflects desirable qualities.
Educators say they are intrigued by the research but have
reservations about using such measures competitively. Dr.
Lucido expresses concern about test questions that could
eliminate students ''because of some disability that's
mental in nature.'' Patte Barth, senior associate with the
Education Trust, an advocacy group for poor schoolchildren,
calls the whole thing ''squirrelly.'' Even in the SAT
reasoning test, Ms. Barth says, there is room for
interpretation. ''You take something even more difficult to
measure, and I don't think they could hone this accurately
for any meaning.''
Robert Schaeffer, public education director of the National
Center for Fair and Open Testing, a watchdog group that
advocates for testing reform, says efforts to move the SAT
beyond academics ''is like painting lipstick on a pig.''
''FairTest agrees with Sternberg's underlying thesis that
college readiness is far broader than what the SAT
measures,'' Mr. Schaeffer says, ''but no test -- neither
the SAT, a tweaked SAT or an alternative -- is necessary in
the college admissions process.''
Both Dr. Schmitt and Dr. Sternberg have presented their
findings to college officials nationwide, Dr. Sternberg
says, to ultimately interest them in using such a test to
assess applicants. Convincing colleges (not to mention
parents and students) of the need for another test could be
a challenge.
''I think for most colleges, what they already have seems
to be working fine,'' says Robert L. Linn, co-director of
the Center for Research on Evaluation, Standards and
Student Testing at the University of Colorado, in defense
of the SAT. ''There's been a lot of research over the years
that's been barren in turning up anything better.''
But, in fact, Dr. Camara cites something that predicts
success better than entrance exams or high school
curriculum. The No. 1 predictor of whether a student
graduates, he says, is which college a student attends.
Ninety-eight percent of Harvard's 1996-97 freshman class,
for example, graduated within six years; Chapel Hill and
Penn State had 80 percent rates; less competitive public
institutions had rates no higher than the mid-50's.
William G. Bowen and Derek Bok's ''Shape of the River,'' in
which the authors followed minority students with moderate
SAT scores through four years of Ivy League college and
beyond, also suggests that campus environment is crucial.
Such students, they found, were more likely to succeed at
an elite university than at a community college. That begs
a question, Mr. Schaeffer of FairTest says: Does the SAT
measure talent or does it simply provide entree into the
colleges more likely to nurture and sustain their students?
''It's a difficult puzzle,'' he says.
That, in essence, is what Dr. Sternberg and Dr. Schmitt are
trying to piece together. How do you understand the mind
and motivations of an adolescent whose world is about to
open up before him? Why do some succeed and others, even
the academically gifted, struggle?
Sometimes, explains Dr. Linn, it's as simple, and
inexplicable, as stumbling on an activity or subject that
grabs the imagination. ''I've seen students who were
floundering, then all of a sudden something sparked their
interest and they became very good students,'' Dr. Linn
says. ''Until that time, they were not.''
Dr. Camara does not doubt the complexities of the task
undertaken. ''We're trying to predict human behavior,'' he
says. ''We're not predicting the economy or the price of
gold in six months.''