I just finished the book, What the Luck - a book by Gary Smith that goes in depth into how [regression to the mean] affects us in so many parts of our lives.
Regression is a package of pitfalls and opportunities. Avoid the pitfalls and seize the opportunities.
He gives some thoughts on predictions. Predictions are wrong, positive ones tend to be overestimates and negatives underestimates. Large predictions of changes are more likely to be overestimates. A way to adjust predictions to be more likely is using the Kelley equation:
R = reliability - the extent to which performances are consistent - If correlation between events in 0.80 for a person, then R = 80%
This helps take in a person's ability and buffer some of the luck involved.
"Good luck cannot be counted on to continue indefinitely, and neither can exceptional success."
The author discussed one of my concerns over medical research (and science in general). We use a p \< 0.05 as a mark of significance. Still, that means a significant event has a 1 in 20 chance of happening by chance. He gave a good illustration with a nice table if 10% of all treatments are truly effective (a very optimistic scenario) and to keep things simple a 10% false negative rate:
Significant Not significant Total
Effective Treatment 90 10\ 100 (false negatives)
Ineffective Treatment 495 (0.05*9900)\ 9,405 9,900 (false positives)
Total 585 9,415 10,000
Note that while 90% of all effective treatments are significant, but only 15% (90/585) of all significant treatments are effective - another way of saying that 85% of positive significance results are actually chance. This is a good point to think about whenever reading scientific literature.
Another concern with scientific literature is publication bias. Because of this, "treatments that are reported to be beneficial tend to be less beneficial than reported." Another example of regression to the mean. The problem is, we only see studies with positive results and thus overestimate benefit.
Recognize situations that fall into the [Random Walk Hypothesis] - the idea that a drunkard's steps are unrelated to previous steps - similar to coin flips, stock price, etc.