Hypothesis Testing: Fishing for Trouble
Introduction “Can you check if this is significant?” It was a seemingly innocuous question from a dangerous source: a semi data-literate scientist. The kind who believed, deep in his heart, that small p-values were “good” and large p-values were “erroneous”. On this day, the man in question had come forth with a large, complex multivariate dataset. He’d manually combed the data, visually inspected it, and hand-picked a hypothesis. “Can you check if this is significant?
