What the Luck?

I just finished the book, What the Luck - a book by Gary Smith that goes in depth into how [regression to the mean] affects us in so many parts of our lives.

Regression is a package of pitfalls and opportunities. Avoid the pitfalls and seize the opportunities.

One human folly is in "[the law of small numbers]" - when we overreact to limited data, we make generalizations without good reason to do so. This happens with coin flips, imagined patterns, sports, etc.
We have [selective recall] - "We remember things that support our beliefs and discount evidence to the contrary"
[Gambler's Fallacy] = The idea that a streak of good luck makes bad luck more likely. This is also called the fallacious law of averages. The chance of good or bad luck happening does not change based on past history.
The [paradox of Luck and Skill] - It is interesting that in athletics, the higher the level of playing, the more skilled the players are, and the more likely that the winner is determined by luck and not skill.
[Post hoc reasoning] - post hoc ergo propter hoc (after this, therefore because of this) - Just because something followed an event, does not mean the event caused it.
[Observational data trap] - "Anecdotes aren't evidence and data is not the plural of anecdote." "If we use data to invent the theory, then of course the data support the theory!" To test a theory, you need new data - watch out for chance correlations!
"Regression creates a [bias for change]" in organizations. Those doing well will not likely do change, and those who need to improve will change - this gives the appearance that change itself is beneficial.
"Something extremely unlikely is not unlikely at all if it has already happened." - [Feynman trap]
[Peter Principle] - when people are promoted when they are good at what they do until they are not good - "managers rise to the level of the incompetence" (a great example of regression)
[Sunk cost traps] - ignore the regrettable decisions that cannot be undone - do not act recklessly after a big loss in stocks, poker - regression demonstrates patience is better than a Hail Mary.

He gives some thoughts on predictions. Predictions are wrong, positive ones tend to be overestimates and negatives underestimates. Large predictions of changes are more likely to be overestimates. A way to adjust predictions to be more likely is using the Kelley equation:

Estimated ability = R/(Performance) + (1 - R)/(Average group performance)

R = reliability - the extent to which performances are consistent - If correlation between events in 0.80 for a person, then R = 80%

This helps take in a person's ability and buffer some of the luck involved.

"Good luck cannot be counted on to continue indefinitely, and neither can exceptional success."

The author discussed one of my concerns over medical research (and science in general). We use a p \< 0.05 as a mark of significance. Still, that means a significant event has a 1 in 20 chance of happening by chance. He gave a good illustration with a nice table if 10% of all treatments are truly effective (a very optimistic scenario) and to keep things simple a 10% false negative rate:

                          Significant       Not significant    Total

Effective Treatment           90                  10\           100
                                           (false negatives)

Ineffective Treatment 495 (0.05*9900)\ 9,405 9,900 (false positives)

       Total                  585                9,415         10,000

Note that while 90% of all effective treatments are significant, but only 15% (90/585) of all significant treatments are effective - another way of saying that 85% of positive significance results are actually chance. This is a good point to think about whenever reading scientific literature.

Another concern with scientific literature is publication bias. Because of this, "treatments that are reported to be beneficial tend to be less beneficial than reported." Another example of regression to the mean. The problem is, we only see studies with positive results and thus overestimate benefit.

Recognize situations that fall into the [Random Walk Hypothesis] - the idea that a drunkard's steps are unrelated to previous steps - similar to coin flips, stock price, etc.

My distractions

What the Luck?

Tags: