I used to practice, and teach, medicine. Time after time, some young gun would run a willy-nilly battery of needless tests, then come running to me with the "diagnosis," completely off base.
If you do not have a good grasp of how tests depend on the prevalence of the condition being tested, you will be misled by even good tests. Lots of decent doctors are lousy at metrics.
Suppose Arne's Army developed a test that could separate the good teachers from the bad. Never mind what "good" or "bad" means, the test itself defines competency.
For simplicity, let's make it a urine dipstick--if you get a "+" you're in the club, a "-" and your license is revoked.
Suppose the test was so well designed that if you were competent, the test would be "+" 95% of the time, and if you were incompetent, it would be "-" 95% of the time. Not bad, eh?
So what happens if you get a "-"--does that mean you're likely incompetent? Are you 95% confident in the result?
(I'll pay the Jeopardy theme while you come up with an answer....)
No way to tell--the test's usefulness depends on how many teachers are actually, in fact, incompetent.
Let's say only 5% of teachers are incompetent--in that case, a positive test reflects a truly incompetent teacher only half the time.
Here's the math: if a 100 teachers are tested, the 5 who are incompetent would likely be picked up. Of the remaining 95, however, there is a 5% false positive rate, and 5% of 95 is about 5. Only half of those who test positive would indeed be incompetent.
It's counterintuitive, but it's real; it's called Bayes theorem, and it reveals a practical problem with any type of binary testing--the accuracy of the results depends on the frequency of the condition you are testing for.
If 90% of teachers are incompetent, and some of the public might even believe that, then the chance that a test is a true positive exceeds 99%--same test, different population.
I have no problem with metrics, but I am opposed to what knuckleheads can do with numbers. Numbers don't kill careers, people do.
Most docs are reasonably bright people with a generous dose of ambition, and a lot of them can't grasp this--what hope do we have that Arne will?