Over at Marginal Revolution, they've got a post up about the meaning of statistical significance , with links to otherposts on the issue. The post discusses the flaws of relying too heavily on alpha levels for determining significance, the reasons why this is a flawed approach, and the particular problems we have in the social sciences with respect to this issue. I remember discussing this in graduate school to some degree, including discussions about broadening one's knowledge of statistics to utilize other measures such as effect size to gain a better sense of the true significance of a finding. I also recall more than a few issues where there might have been statistical significance found with respect to a study, but the clinical significance was negligible. Here is what I wrote about that issue in the comments section:
We see some of this in psychological testing. For example, in IQ testing, we may assess someone with a gap of, say, 10 points between their verbal IQ and their performance (i.e. nonverbal) IQ. While this gap may be statistically significant (happening with a lack of frequency that, statistically, it reaches significance), it doesn't mean much in terms of someone's intellectual functioning. What is of far more concern is whether a gap between the scores approaches what is called clinical significance - at that point, we may see certain issues resulting from a gap that large (say, roughly, 20+ points difference off the top of my head).
In other words, just because something reaches statistical significance (at the .05 level) doesn't mean it is actually telling you anything useful, even if it is infrequent.
I think you sometimes run up against this as you gets tons of data on a particular issue, where the effect size gets bigger and bigger: you may get statistically significant differences for groups that really don't tell you much. As an example (and to stay with IQ), I remember that men outscored women on the nonverbal section by a couple points (on average), while women outscored men on the verbal section by a couple of points. Given the thousands of data sets that were utilized in conducteing such large studies, a difference of 103 versus 100 on Verbal IQ came out as statistically significant, even though a difference of three points is clinically meaningless.
If you are interested in statistics and research, I recommend checking out the links...