Qfyilyi

I'm a sociology undegrad working on an essay for a methods class. I'm also planning on submitting it as a sample for my application to grad school. I don't want to be too specific, but I believe that this work is quite original and my hypothesis would confirm previous literature, and all in all I think it would would make a good impression on the admissions committee.

So basically I've run the tests and I'm getting conflicting results. Using one dataset (which has more observations) gives me very significant results, while using another one (which would arguably be more accurate) doesn't give me anything. So here I am at a crossroads, and I've come up with three possible options as to what to do:

1. Only show the significant results. After all, this is just a ten-page essay, it's not supposed to be publishable or anything, right?




2. Only use the better dataset and admit that there just isn't much there - maybe blaming it on the small sample size or on the not-so-good dependent variable. Hopefully the committee would appreciate the honesty and the relatively advanced methods that I used.




3. Show results from both datasets, suggesting that the differences might be due to the sample size or maybe to chance.

As I type this I'm leaning more towards option 3, but I'd like to hear from people with more experience in academia. What should I do?

In research, you don't set out to prove that something is true. You set out to discover whether or not it is true. This would be knowledge. The other is just propaganda.

Negative results are not a failure. They give you evidence just as do positive results. If you ignore, or obscure, results you are lying to yourself and others. If you design an "experiment" so that it is guaranteed a priori to produce positive results, it isn't research.

Hoping that something is true isn't evidence. Many researchers start out with that idea. I think this is true. I really want it to be true. But if it is false, it is just as valuable (possibly more so) to know that and to be able to investigate why.

Report all your results. Try to explain why different aspects lead you in different directions. Only then can your learning begin.

Omitting negative findings and selectively reporting only the positive findings would be a breach of research ethics. As a researcher you are supposed to uncover knowledge,* not to obscure it. Findings are often contradictory and in need of interpretation. By explaining how you obtained these contradictory results (i.e. your methods), you help others to avoid dead ends in the future and to make sense of what looks confusing today.

_{*Interestingly, the knowledge that research creates often takes the form of higher-level confusion rather than ultimate certainty.}

Are your significant results a large effect size, or just a tiny change that is significant because of the large sample size?

Are your non-significant results similar in direction and magnitude to the significant results from the other dataset?

Consider how much the size of the dataset is impacting what you are seeing - you may be able to frame one study as confirming the results of the other if they are in agreement apart from significance. Look at more than just the p-values, especially if they are coming from a very large dataset.