In class

“...An observation is judged significant, if it would rarely have been produced, in the absence of a real cause of the kind we are seeking. It is a common practice to judge a result significant, if it is of such a magnitude that it would have been produced by chance not more frequently than once in twenty trials. This is an arbitrary, but convenient, level of significance for the practical investigator, but it does not mean that he allows himself to be deceived once in every twenty experiments. The test of significance only tells him what to ignore, namely all experiments in which significant results are not obtained. He should only claim that a phenomenon is experimentally demonstrable when he knows how to design an experiment so that it will rarely fail to give a significant result. Consequently, isolated significant results which he does not know how to reproduce are left in suspense pending further investigation.”
Q:  Why do so many colleges and grad schools teach p = .05?

A:  Because that's still what the scientific community and journal editors use.


Q:  Why do so many people still use p = 0.05?

A:  Because that's what they were taught in college or grad school.
With the banning of the NHSTP (null hypothesis significance testing procedures) from BASP, what are the implications for authors? 

Question 3. Are any inferential statistical procedures required? 

Answer to Question 3. No, because the state of the art remains uncertain. However, BASP will require strong descriptive statistics, including effect sizes. We also encourage the presentation of frequency or distributional data when this is feasible. Finally, we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem. However, we will stop short of requiring particular sample sizes, because it is possible to imagine circumstances where more typical sample sizes might be justifiable.

http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503

http://www.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108

1. P-values can indicate how incompatible the data are with a specified statistical model.  

2. P-values do not measure the probability that the studied hypothesis is true, or the
probability that the data were produced by random chance alone.  

3. Scientific conclusions and business or policy decisions should not be based only on whether 
a p-value passes a specific threshold.  

4. Proper inference requires full reporting and transparency.  

5. A p-value, or statistical significance, does not measure the size of an effect or the
importance of a result.  

6. By itself, a p-value does not provide a good measure of evidence regarding a model or
hypothesis.

https://www.youtube.com/watch?v=5OL1RqHrZQ8

https://www.openintro.org/stat/why05.php?stat_book=os

To Turn In

  1. Researchers have conjectured that the use of the word “forbid” is more off-putting than the word “allow” (in affecting people’s responses to survey questions). In particular, the suggestion is that people do not like to “forbid” anything. Students in an introductory statistics class were randomly assigned to answer one of the following questions:
    • Should your college allow speeches on campus that might incite violence?
    • Should your college forbid speeches on campus that might incite violence?

Of the 14 students who received the first question, 8 responded yes. Of the 15 students who received the second question, 13 said no.
a. Find a way to analyze the data such that you can say something “significant”.
b. Find a way to analyze the data such that your result is not “significant”.
c. Give a complete conclusion to the data analysis (that is, conclude what you think is most appropriate). Provide what you believe is the most accurate significance result, a sense of who (what population, if any) the results can be applied to, and whether or not causation can be concluded.

  1. What is your personal level of significance? (Just report the number.) https://www.openintro.org/stat/why05.php?stat_book=os

  2. Read the ASA’s statement on p-values. http://www.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108
    Choose two different principles and explain each (separately) as if to a peer in a science class who is making conclusions about a recent study.

  3. Why use p-values at all? That is, what is benefit of having a p-value (as opposed to simply descriptive statistics or graphs of the data)?

If you are still curious about the ideas in this lab (not part of the assignment):

  1. Read Ioannidis (2005), “Why Most Published Research Findings are False” http://www.plosmedicine.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pmed.0020124&representation=PDF

  1. Consider table 1. Suppose that the level of significance is taken to be 0.05 and the power is 0.8. Also, set R (the number of true to not true relationships) to be 2 (for every 3 experiments, one is null). What percent of research findings (i.e., “significant” findings) are actually true (i.e., Ha is true)? [hint: for ease of calculation, you can set c to be something like 10,000.]

  2. Consider table 1. Suppose that the level of significance is taken to be 0.05 and the power is 0.3. Also, set R (the number of true to not true relationships) to be 0.1 (for every 11 experiments, 10 are null). What percent of research findings (i.e., “significant” findings) are actually true (i.e., Ha is true)? [hint: for ease of calculation, you can set c to be something like 10,000.]