Section 5, Chapter 45:
How to Evaluate Your Present Spelling Program
Do we really need a different spelling program? How can we determine whether or not Brand X Spelling by Famous Publishers is as good as its hype? Does any spelling program really work? These and a few other questions should automatically come to mind if you are a principal, a member of a spelling curriculum committee, or a teacher concerned about helping your students become adequate spellers.
You can read the latest studies and you can read all the official data put out by publishers, but will all this reading really give you the answers to the important questions? We at AVKO don’t think so. We have read the studies. We have looked at all kinds of spelling programs. We have read their propaganda sheets. And because we are a non-profit organization dedicated to finding solutions to educational problems, we have designed a method that any school can use to test the effectiveness of their current spelling program or to compare various spelling programs being used within their system.
We can do it for you. But we’d have to charge an arm and a leg and perhaps a finger and toe on top of that. So, I’m sure you’d rather have a system you could follow that would answer those crucial questions.
It’s cost? Two sheets of paper per student, one sheet of paper with the spelling test for each teacher, and about two minutes of some reliable spelling checker’s time per student tested, another minute per student for a data entry person to compile the data and perhaps four hours tops for the person in charge to write the report, plus perhaps eight to twenty hours for someone to design the test and prepare the teachers involved in how to administer it.
Before we tell you how it should be done we would like to point out the pitfalls of standard methods of evaluation of spelling.
1. Spelling tests too often measure students’ short term memories and not their permanent memories.
2. Standardized spelling tests are not designed to discriminate between incidental learning of spellings and the learning of spellings that can be directly attributed to the teaching of spelling. By incidental learning of spelling we mean such learning that may take place just from reading, from having spelling errors in papers corrected, by using the dictionary, by asking for correct spellings, etc.
To overcome the first problem, we believe that the best way to test students’ spellings of words is to have the pretest done on the first day of school in September and the posttest done on the first day of school in September of the following year.
To overcome the second problem of being able to determine the actual amount of learning that can be attributed to the spelling program as compared to incidental learning, we recommend using the paired word approach. We take two sets of words. Each word in Set A is paired with a word of exactly the same level of difficulty in Set B. The difference between them will be that only the words in Set A were directly taught as part of the spelling program. The words in Set B were not taught.
Without the paired word approach many spelling programs will show false positive gains. For example, Brand X shows statistical evidence claiming that 90% of the students tested on the words ship, little, and show could spell these words as the result of their third grade spelling program. What they are not telling you is that if the school was just an average school 49% of the students already knew ship before they started the 3rd grade, 65% already could spell little, and 58% could spell show. And if the school is just an average school, 71% of the students would be able to spell ship at the end of the 3rd grade without any special instruction on that word and 86% would be able to spell little and show.
Now this might be significant to a statistician, but it certainly is NOT significant in terms of instruction because with or without any special instruction 99% will eventually learn all three of these words ship, little and show before they get out of middle school anyway.
Before a school switches to Brand X Spelling (or even to AVKO’s Sequential Spelling Series)[1], a school should determine several things.
1. The definition of an Effective Spelling Program (ESP) should be agreed upon. Teachers could decide that for a spelling program to be effective a certain percentage of the students who did not know how to spell a given word at the beginning of the program should, as a result of direct instruction, be able to spell that word one year later. That certain percentage could be: (a) 10% (b) 25% (c) 33% (d) 50% (e) 67% (f) 75% (g) 90%. It is crucial to determine what constitutes an effective program first and then to test it. You should never test the program first and then establish your effectiveness criteria.
2. Does the spelling program improve a student’s ability to read? Remember, the decoding processing is merely the flip side of the encoding process. Pretests and posttests using standardized measures and control groups can determine this. Your local statistician should know how to set it up; if not, your nearest university will be able to help you.
3. Is the spelling program designed to help improve students’ handwriting? Is the handwriting program designed to improve spelling? Is the composition program designed to help improve both spelling and handwriting?
Thanks to the most complete study of the spellings of school children by Harry A. Greene called the New Iowa Spelling Scale,[2] we can use “baseline” words to determine the amount (if any) of improvement in spelling that can be attributed to the classroom study of spelling words.
For example, we know from this study that we can expect in a mythically average school that in September only 16% of the average 5th grade students can spell the word marriage. We can expect 27% to correctly spell marriage in September of the 6th grade. We can expect 52% by the 7th grade and 63% by the 8th. If the word marriage were to have been taught in your school’s spelling series during the 5th grade, you would expect that far more than 27% should be able to spell it in the 6th grade. If not, then the old learn-for-the-test-on-Friday and forget-by-Monday syndrome is working.
We have rearranged The New Iowa Spelling Scale[3] so that any local curriculum coordinator or statistician can construct a test to determine how effective your spelling series is. The test would use two sets of words. One set of words would be words that have been studied. The other set of words would be words of exactly the same statistical level of difficulty that had not been studied. If your spelling program is truly effective then there should be a readily recognizable difference between the scores, not just merely a statistically significant difference. Statisticians may believe an increase of 5% over an expected 10% gain is significant, but it would hardly justify the expense of most spelling programs. However, the amount of gain you expect to receive is something you should determine BEFORE you give the test.
Sample Construction of the Test
A 20 matched pair word test should be sufficient. The odd numbered words should be those words that are to be studied in the coming year. The even numbered words should be words that have not been studied in any previous year and are not scheduled to be studied in the coming year. They can, of course, be slated for study in following years. Obviously each grade should have a separate pretest. The pretests should be given to every student from grades 3 through 7 on the first day of school. The posttests will be exactly the same but given to grades 4 through 8 on the first day of the following school year.
Words 1-4 should be very easy words. That is, they should be correctly spelled by 96-100% of all 8th graders.
Words 5-8 should be relatively easy words. That is, they should be correctly spelled by 90-95% of all 8th graders.
Words 9-12 should be easy words known by 80-89% of all 8th graders.
Words 13-16 should be fairly hard words known only by 61-79% of all 8th graders.
Words 17-20 should be difficult words known only by 40-60% of all 8th graders.
There isn’t anything sacred about the percentages shown above. What is sacred is choosing matched pairs of words from different levels of difficulty and always progressing from the easiest to the most difficult. The following is a sample test we constructed for a school system to test the effectiveness of their spelling program..
Column A Column B
Words Studied Words not studied
1. sport (98) 2. race (98)
3. wire (97) 4. wool (98)
5. member (93) `6. lesson (93)
7. jail (92) 8. leap (92)
9. torn (87) 9. sort (87)
11. regular (86) 12. regulations (86)
13. producing (84) 14. limb (84)
15. quiet (74) 16. minute (74)
17. instant (70) 18. neglect (70)
19. journal (53) 20. particulars (53)
Certainly if the spelling program is working, we would expect many more words in Column A would be correctly spelled than column B. We would think that an ordinary spelling program of moderate effectiveness would result in 25% more correct spellings of words studied than those not studied. A truly Effective Spelling Program (ESP) should have at least 75% more correct spellings of words studied than those not studied. The instrument we give teachers contains the word and a sentence in which it occurs, and then the word repeated one more time. Directions are that teachers may repeat the word, the sentence and the word one more time only. If a teacher wants to check her/his own spelling program first before trying to convince the school system to do a system or school wide testing, she/he should give the pretest on the first day of school with the words being taught coming from the first two months of the school year and the posttest being given late in the spring.
So, use any grouping of the following paired words to devise your own test of your spelling system’s effectiveness. The paired words in this pamphlet are arranged from “easiest” to “hardest.” We used the statistics from The New Iowa Spelling Scale.[4] The only real change we made was to also add the percentage figures from the 5th through 8th grades for a total of the number out of an average 400 students who could correctly spell the words. The pairs were “picked” by the “computer” but they do seem to indicate something about how spelling is and isn’t taught across the country.
Steps to follow (Summarized)
Step 1. Let teachers determine what percentage of students who don’t know a word on pretest should be able to spell the word a year later if that word is directly taught. In other words, what percentage of learning constitutes an Effective Spelling Program (ESP). See sample survey questtionaire below..
Step 2. Let local spelling committee create Pretest using paired words of equal difficulty. See sample pretest on p. 243
Step 3. Have teachers give pretest on 1st day of school.
Step 4. Have designated scorer (not the teachers) correct tests and complete Pretest score sheet. See sample score sheet on p. 244.
Step 5. Have designated analyst determine (a) expected gains from incidental learning and (b) expected gains from direction instruction of an Effective Spelling Program (ESP).
Step 6. Have teachers use their spelling program for the year.
Step 7. Have teachers give the Posttest on 1st day of school (the following year)
Step 8. Have designated scorer (not the teachers) correct tests and complete Posttest score sheet.
Step 9. Have designated analyst determine gains and compare real gains to the minimum score projected for an Effective Spelling Program (ESP). See p. 245.
Step 10. Have analyst present findings to Spelling and Curriculum Committees and teachers.
Steps to follow (in greater detail)
Step #1. Let teachers determine what constitutes effective learning.
Questionnaire for Teachers:
For you to consider a spelling program to be effective, what percentage of students who were not able to spell a specific spelling program word at the beginning of the year should be able to demonstrate that they have learned to spell that same word correctly the following year? Circle one: A—10%. B—20%. C—25%. D—33% E—50%. F—67%. G—75%. H—80%. I—90%
Tabulate your teachers’ responses. Pick the percentage that appears to be the concensus. Use this figure to determine the effectiveness of the program. For purposes of explanation, we assume 75% would be the passing grade selected by most teachers for a spelling program developed by experts.
Goals to be expected by Effective Spelling Program (ESP). ________
Step #2. Create Pretest using paired words of equal difficulty. Use odd numbers for words that are being studied. Use even numbers for words that are not being studied. Write directions for the teachers. Include in the directions the words and the sentences using the words as in the sample below.
Step #3. Give the pretest.
Sample Teacher’s Sheet (the pretest)[5]
Hand out the student answer sheets and tell the students that they are going to help their school test their spelling books. Tell them to do their best but not to worry. They are not going to be graded on this test. Read the word. Read the sentence and read the word one more time. If a student needs to have it repeated, you may repeat the word, the sentence and the word but only repeat once.
1. time It’s not time to go yet. time.
2. love I just love taking a test that’s easy. love.
3. milk You should drink some milk everyday. milk.
4. May The month of May follows April. May.
5. toe I stubbed my toe against the step. toe.
6. singing I enjoy singing folk songs. singing.
7. learn This year you will learn many things. learn
8. rented We rented an apartment for a year. rented.
9. tie I hope you know how to tie your shoelaces. tie.
10. helped A boy scout would have helped them. helped.
11. porch I enjoy sitting on a porch. porch.
12. pretty I think kittens are pretty cute. pretty.
13. thinking What were you thinking about? thinking.
14. sank The Titanic sank when it hit an iceberg. sank.
15. victory It was a great victory for our team. victory.
16. serve Who is going to serve the meal? serve.
17. passage We read a short passage from the book. passage.
18. ironing Do you have an ironing board at home? ironing.
19. thrill We got a thrill out of going on that ride. thrill.
20. hundreds It cost us hundreds of dollars to go there. hundreds.
Sentence dictation
Students are to write the entire sentence.
Repeat each sentence three times. Use normal “sloppy” speech.
Do NOT over articulate. Read going to as “gonna”!
· It’s too bad their team is going to lose the big game.
· They’re going to learn that it wasn’t Coach Johnson’s fault.
· Many students often find studying difficult.[6]
Step #4. Scorer completes Pretest Check Sheet.
Sample Scorer’s Pretest Check Sheets (Partially Completed).
Based on one small elementary school.
Grade 4 (68 students) Grade 5 (67 students)
# Correct Divided by Percent #Correct Divided by Percent
Spellings # Students Known Spellings # Students Known
1. time _______[7] _______ _______ _______ _______ _______
2. love _______ _______ _______ _______ _______ _______
3. milk _______ _______ _______ _______ _______ _______
4. May _______ _______ _______ _______ _______ _______
.....
.....
17. passage 14 ¸ 68 = 21% 28 ¸ 67 = 42%
18. ironing 15 ¸ 68 = 22% 29 ¸ 67 = 45%
19. thrill _______ _______ _______ _______ _______ _______
20. hundreds _______ _______ _______ _______ _______ _______
It’s _______ _______ _______ _______ _______ _______
too _______ _______ _______ _______ _______ _______
bad _______ _______ _______ _______ _______ _______
their _______ _______ _______ _______ _______ _______
team _______ _______ _______ _______ _______ _______
is _______ _______ _______ _______ _______ _______
going to _______ _______ _______ _______ _______ _______
lose _______ _______ _______ _______ _______ _______
the _______ _______ _______ _______ _______ _______
big _______ _______ _______ _______ _______ _______
game. _______ _______ _______ _______ _______ _______
They’re _______ _______ _______ _______ _______ _______
going to _______ _______ _______ _______ _______ _______
learn _______ _______ _______ _______ _______ _______
that _______ _______ _______ _______ _______ _______
it _______ _______ _______ _______ _______ _______
wasn’t _______ _______ _______ _______ _______ _______
Coach _______ _______ _______ _______ _______ _______
Johnson’s _______ _______ _______ _______ _______ _______
fault. _______ _______ _______ _______ _______ _______
Many _______ _______ _______ _______ _______ _______
students _______ _______ _______ _______ _______ _______
often _______ _______ _______ _______ _______ _______
find _______ _______ _______ _______ _______ _______
studying _______ _______ _______ _______ _______ _______
difficult _______ _______ _______ _______ _______ _______
Step 5. Analyst compares school’s scores to national averages for each grade. If close as in our example, analyst uses the scale. If not, the analyst uses appropriate statistical interpolation to determine expected average gains from incidental learning.
Predicted Percentage Scores from National Averages See pp. 248 & 248
Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
17. passage 21 45 68 79 88
18. ironing 23 46 66 79 88
In this case, to determine the expected gain from the target word[8] passage, analyst subtracts the percentage correct in the 4th grade (21) from the percentage correct in the 5th grade (45) to arrive at the expected gain for the target word passage of 24. The same procedure is used to determine the expected gain from incidental learning for the control word[9], ironing. See Predicted Percentage Scores bottom of page 247.
Step 7. Analyst determines what the score for each word should be on the posttest if the spelling program is effective. Formula he uses is:
Pretest score + minimum ESP Gain expected = minimum expected posttest score.
Example using statistics for the word passage
100% 100
minus beginning 4th Grade percentage (21%) -21
equals percent who haven’t yet learned 79
minus expected incidental 45
learning gain score -21 -24
equals percent who need to be taught 55
times effective teaching percent x.75
Minimum Gain ABOVE expected 41
Minimum Percentage Score for
a 75% effective spelling program for
the word passage studied in the 4th grade
in the posttest given in grade 5 should be:
Score in 4th grade 21
Plus Incidental learning gain 24
Plus Minimum Gain 41 + 65
Minimum Score for 5th 86
for 75% effective spelling
program.
Step 8. The posttest is given the following September. As long as the pretests have been collected and never returned to the teacher, the same test can be used. Same directions.
Step 9. Analyst subtracts real score for each of the target words from the Minimum Score Projected for an Effective Spelling Program (ESP). See Sample Scorer’s Posttest Check Sheets on p. 247.
Step 10. Analyst presents findings to the school’s curriculum spelling subcommittee, the curriculum committee, and the teaching staff.
The chart,Word Pairs of Equal Difficulty, starting on page 248 contains a good sampling of equivalent word pairs of equal statistical difficulty.
If you would like to develop your own, you can find 5,508 words whose difficulty levels have been determined and in their rank order in the Reading Teacher’s List of Basic Spelling Words which is available from the AVKO Foundation.
1. No matter how expensive your current spelling program is, it will be judged as ineffective by your teachers and your spelling committee if you use our simple test.
2. If your system uses the AVKO Sequential Spelling program that does not require any books for the students, the test results will satisfy your teachers and your spelling committee.
3. If your system uses the AVKO Sequential Spelling method and your teachers develop their own materials, the test results will more than satisfy your teachers and your spelling committee.
Sample Scorer’s Posttest Check Sheets for just the 4th Grade based on one small middle school. Notice the pretest scores are for the lower of the two grades. The posttest scores are for the higher of the two. Posttest is taken 1st day of Grade 5.
Grade 4 Grade 5
® Pretest Number + Expected ESP Actual - Expected Target
correct Gain Score Correct Score +/-
1. time _____ _____ _____ _____ _____ _____
2. love _____ _____ _____ _____ _____ _____
3. milk _____ _____ _____ _____ _____ _____
4. May _____ _____ _____ _____ _____ _____
5. toe _____ _____ _____ _____ _____ _____
6. singing _____ _____ _____ _____ _____ _____
7. learn _____ _____ _____ _____ _____ _____
8. rented _____ _____ _____ _____ _____ _____
9. tie _____ _____ _____ _____ _____ _____
10. helped _____ _____ _____ _____ _____ _____
11. porch _____ _____ _____ _____ _____ _____
12. pretty _____ _____ _____ _____ _____ _____
13. thinking _____ _____ _____ _____ _____ _____
14. sank _____ _____ _____ _____ _____ _____
15. victory _____ _____ _____ _____ _____ _____
16. serve _____ _____ _____ _____ _____ _____
17. passage 21 65 86 46 86 -40
18. ironing 29 14 43 44 43 +1
19. thrill _____ _____ _____ _____ _____ _____
20. hundreds _____ _____ _____ _____ _____ _____
It’s _______ _______ _______ _______ _______ _______
too _______ _______ _______ _______ _______ _______
bad _______ _______ _______ _______ _______ _______
their _______ _______ _______ _______ _______ _______
team _______ _______ _______ _______ _______ _______
is _______ _______ _______ _______ _______ _______
going to _______ _______ _______ _______ _______ _______
lose _______ _______ _______ _______ _______ _______
the _______ _______ _______ _______ _______ _______
big _______ _______ _______ _______ _______ _______
game. _______ _______ _______ _______ _______ _______
They’re _______ _______ _______ _______ _______ _______
going to _______ _______ _______ _______ _______ _______
learn _______ _______ _______ _______ _______ _______
that _______ _______ _______ _______ _______ _______
it _______ _______ _______ _______ _______ _______
wasn’t _______ _______ _______ _______ _______ _______
Coach _______ _______ _______ _______ _______ _______
Johnson’s _______ _______ _______ _______ _______ _______
fault. _______ _______ _______ _______ _______ _______
Many _______ _______ _______ _______ _______ _______
students _______ _______ _______ _______ _______ _______
often _______ _______ _______ _______ _______ _______
find _______ _______ _______ _______ _______ _______
studying _______ _______ _______ _______ _______ _______
difficult _______ _______ _______ _______ _______ _______
[1]This we humbly suggest is better than any spelling series on the market, but not nearly as good as a spelling program that can be designed by each teacher using the principles of AVKO’s Sequential Spelling in conjunction with an overall curriculum program designed to fit the needs of the students.
[2]This has recently been revised by Bradley M. Loomer.
[3]Greene, Harry. Revised by Bradley M. Loomer. The New Iowa Spelling Scale. Iowa City: University of Iowa, Project Spelling, 1978
[4]Greene, Harry. Revised by Bradley M. Loomer. The New Iowa Spelling Scale. Iowa City: University of Iowa, Project Spelling, 1978.
[5]This test can be used also as the posttest the following year.
[6]Those students who badly misspell this sentence should be tested for learning disabilities, specifically dyslexia and/or dysgraphia. An example of such a bad misspelling would be: Mene ãutend oìen fine ãudin difcaulgt
[7]Number of Correct Spellings divided by total students in grade=Percent correct
[8]The target word will be the word that will be studied sometime during the year.
[9]The control word is the word that is almost identical in statistical difficulty but which will not be taught during the year.
Forward to Section 6 Chapter 46 Word Pairs of Equal Difficulty