First, the new secretary of education was chosen, Education Week: Obama: Duncan ‘Doesn’t Blink’ on Tough Decisions, but I hadn’t commented on it. I thought the “Doesn’t Blink” comment was a bit “tin-eared” after the Palin fiasco. Mr. Duncan seems more genial and cooperative than Spellings, or the other “top choices” that were not Linda Darling-Hammond, but he still loves bubble tests. Looks like there is little hope for portfolio-based assessment, ala Doug Christensen.

Then, I saw, Why Assess? at Chris Lehnamann’s Practical Theory where he shares this from Gary Stager, who is not afraid to be out on the ragged edge:

I’ll be outrageous and say that all assessment is an interruption to the learning process.

I’ve been thinking about assessment as I and my school practice it lately, not just the required “summative” assessments, which my prior posts indicate, I’m not as crazy about. We’re in a transition that I’ve experienced before. As the Reading First grant is winding down, we are not having to do as much of the required assessment (weekly assessments with the language arts text, a unit assessment from the County Office of Education). Both of those assessment are lousy the farther up in grade levels that you get, the worse it is. The weekly text assessment is not aligned well with either the standard or the difficulty level found on the state tests that they are ultimately given. The unit assessment attempts to fix that but it is a bastardized instrument.

Since these are no longer required, some teachers seem not to be doing any assessment. I went through some of the same in my last year teaching a self-contained class when our administrator decided to forgo Reading First money. The problem that I ran across is that you miss a lot in a class of 27-33 kids when you aren’t doing some form of assessment. Gary is right that assessment takes time, but in my opinion if you aren’t doing some assessment, you are operating in the dark, which is not where we want to be. Now, my site needs to figure out what assessment instruments will be useful. Much of it will likely be observational, but it can’t be nothing.

Someone wasn’t doing their job where this man taught. We’re both in California. That means that you have formal observations every other year for at least the first 10 years you teach, and every year when you are untenured (the first two years in a district). These include a number of informal observations as well. I have been formally observed EVERY year of my teaching career because I changed districts or school sites, making me untenured or subject to revaluation a lot. The years I was tenured, or didn’t change sites, my “number” came up for a Stull Act evaluation. In addition to doing two formal evaluations there are a number of informal walk-throughs done for all teachers (both at my current and former school site). I already have 5 of them for this year (keep in mind my Asst. Principal who is doing my formal evaluation was out of the office for about 5 weeks between his mother’s funeral, and covering in a class with a vacancy). I would say this varies and to say all urban schools fall into the practice of not doing regular walk-throughs is inaccurate.

I don’t want to seem harsh, but I think a lot of the hand-wringing about “lack of accountability” in teaching from younger teachers is because they are unaware of how unaccountable evaluations can be in any professional job. You are often measuring something that is unmeasurable. If your job experience up to entering teaching is centered on making sandwiches, working in sales, or customer service, those are all business where you get pretty instant feedback. Fast food places track how quickly customers are served and work those numbers. It’s obvious when your sales are not meeting a target. Customer service is full of metrics like response time, number of problems resolved, and customer feedback, BUT even those systems can be gamed. I remember a recent conversation with a friend about an early job she had at an insurance company where they needed to re-file customer files. They were rated on how many piles they eliminated. Some workers would “stuff” the files anywhere to get rid of them. Some would “cherry pick” the piles. But, these are not professional jobs, what about those?

Generally in a business you have two types of departments, cost centers (they cost money to run, and don’t have a direct revenue stream), and revenue centers. Folks in revenue centers get measured on dollars. For instance, when I was at Wells Fargo they looked at the ROA, return on assets, and ROE, return on equity, that departments and individuals generated. Most of what I worked in were cost centers. They tried to make us more efficient by having us “bill out” to the departments we supported, but really we weren’t on subject to the numbers near as much as the revenue folks. Teaching is like working in a cost center, and the evaluations there are pretty darn subjective. If Killian and others are complaining about a lack of accountability in teaching, it’s endemic to jobs that don’t involve taking in money.

Here is my discussion with my BIL who is a manager at a large computer software company that would be easily recognized if I shared the name:

ME: Do you have an evaluation instrument? Does it have a numeric scale?

BIL: Yes, it goes from 1 to 5 with one as the lowest value.

ME: Do you have guidelines for each value?

BIL: Yes, there are examples, and the “higher-up” you are in job grade, the higher the expectations, also the employee can add comments to what I put down and that is included in their HR file.

ME: Well, those are all the same in the forms used in my evaluations, and the contract specifies that I can add comments too. Is this an objective measure?

BIL: No, it is subjective. Really, how well you are doing has a lot to do with your manager and how well you get along with them. I spend a lot of time on these evaluations (about 10 hours per employee for a total of 100 hours a year). First, the subjects evaluate themselves, and we agree on objectives in advance that are SMART (specific, measurable, realistic and timely), but how I see their work is not always the same as how someone else would see it.

So this looks a lot like the world I am evaluated in, except that I also get periodic walk-throughs whereas BIL only talks to employees once a year. The only area that is different is that you can go up in job grades, and your objectives go up with that. I don’t know if merit pay would settle this issue, but I would not rule it out. I think with a lot of these things, who you are working with is a big deal. BIL felt (based on my experience, and the experience of my sister as teachers, and numerous other friends in the business) that having a bad manager (principal) in teaching was worse than in a large business because it is harder to “move around” and escape.

I will share that the time in my professional life when I felt the most isolated, and felt like I was not being adequately managed/supervised did not occur when I was teaching, but when I was working as the only person doing my job at a division of a large financial company in the middle of a merger. I was a low priority, was shuffled around on the organizational chart, and basically was left alone too much. The job was also at odds with a lot of my personal beliefs. Happily, that and getting my retirement vested, led to my making the break and looking for a new career that led to the teaching profession. My point, you can have Killian’s career crisis an just about any job. I do not believe there is anything endemic about that feeling of isolation in teaching. He appeared to be working in one site that had become dysfunctional for most or all of his career as a classroom teacher. If I stayed in the site I used to work in like that, I’d feel the same way. The question is do you still feel that way when you move to a new place in the same company or business (as I did in banking)?

My own feeling is that contract rules, labor law, and traditions in education worsen the situation because it discourages leaving a site. As someone who has moved around a lot, it’s not as easy as it was in the business world, and at a certain point, I will be “punished” (forced to take a pay cut) if I switch to a different district. In a few more years, I will be “stuck” in my district unless I want to take a cut, because other districts often don’t recognize more than 7 or 10 years of service on the pay scale. I think we need to think about job mobility, but that often involves trade-offs in job security which can be hard to give up.

_{DONE FOREVER from mr. nightshades photostream}

Testing, it is the only objective criteria for assessment isn’t it? How can you be sure that students have “gotten” it if they can spit it back up on a test three months later? How can a test be fair if kids use aids like a calculator, how do we know that they “really” know how math if they need “help”? I could just go down the line shooting these and other “myths” away, but what takes its place, and why is that better? Okay, enough questions, let’s try for some answers, NOW!
What is the goal of education? Most of us want our students to be some blend of good and productive citizens which translates crudely into being able to hold down a job. Others may have a more beneficent vision of education, but this is the most functional goal, and testing still fails it. Let’s pull out a case study. I was discussing this recent post with my bil who is a manager at a software firm. I told him about how students in the study who showed mastery based on the formative data, did not always have mastery based on either of the summative assessments (a standardized bubble test, and a report they had to write summarizing their findings). We discussed whether mastery was in completing a task (the formative data), or passing a test. BIL thought that the written report was probably the assessment instrument that was most like work projects. Now this is where the conversation got interesting. He and his co-workers have gone to some of the best schools in the country. Many, like BIL, graduated with honors, but the biggest problem he has is that they can do the task part of their job very well (they’d pass a formative assessment), and they do well on tests (he guessed many had 800s on their Math SATs), but they can’t explain what they are doing when requests for information come from management.
My conclusion from discussions with BIL and others and based on my 7+ years in analytical work, you will rarely be given a bubble test as part of your professional life, you will be asked to solve problems, and to communicate with others. Bubble tests do not accurately assess whether someone can perform a task in context. They also are not a great way to assess how clearly folks can communicate about those tasks and projects. I hope this story illustrates both why bubble tests are not useful, but I hope that it also points the way at how we should be assessing, and why those types of assessments are called authentic. What can it look like in education? Here is an interesting article on what Nebraska tried to do. Interestingly, part of why it died was because scores on tests (NAEP) did not match how students performed on their portfolios, and it was assumed that the test was “accurate” and grade inflation was occurring in the portfolios. It never seemed to occur to folks that it might be the bubble tests that were missing something, instead of blaming the students and teachers.
I’ve blogged before about the romance of the quantifiable, and paired with that, a distrust of measures seen as subjective. Here is my suggestion, lets let Nebraska do their portfolios. Since it’s the first time we’ll be doing this under NCLB, do some audits on them. Have outsiders look at it, auditors without a stake in the outcome. They couldn’t do any worse than KPMG has lately.

Links:

• Living in Dialogue: Doug Christensen: He Fought the Law…

• Romancing the quantifiable | Reflections on Teaching

Brian Crosby at In Practice recently discussed the “safety” that educators can find in using set instructional programs. It got me thinking about how we love our numbers. Recent financial events show where an uncritical romance with numbers can led to. Much of what was done looked great on paper, but took on a less rosy hue when it met with time and reality. Numbers can be nice, and there is a a natural human tendency to trust numbers because they are solid, and unambiguous looking, but numbers are like all information and have to be approached critically. What is being measured? How were the numbers assigned? Do the numbers that you are measuring match the goal or standard you are trying to achieve?

from Dr. Chris Dede, Harvard University in “New Strategies for Educational Assessment” at ILC 2008 Conference.

Dr. Dede had one of the more interesting presentations for me at ILC. He’s worked on assessment theory on a federal panel. His conclusions, if you do formative assessment really well, summative assessment becomes superfluous. His presentation was a case-study of a project using an immersive environment, called River City, to teach students life sciences and problem solving. The project had the students gather facts, and going through the interaction to try to discover the cause for a mysterious illness in a turn of the century town. This was an unusual learning situation because rather than too little, they almost have too much information. What they found was that traditional testing measures missed mastery that was shown through the observational data.

Had most of us world enough and time, we could gather tons of observational information about our students. When you’re at Harvard, you get grad students to do it, so they combed the log files from River City, and coded the students activities. In addition to activities that students did trying to “solve the mystery”, they also had to do a “final project” writing up their findings in a letter to the mayor. They were also given a standardized test. This is where things got interesting. They found that students in the study who showed mastery based on the logs, did not always show mastery on the final project or the standardized test.

So the question is, what are we preparing students to do take a test, or solve problems with others? You could make the argument that they should show mastery on the report to the mayor, because being able to communicate what they’ve done is an important part of being a scientist, but you’ll never fill out a multiple choice grid to show what you’ve discovered.