Final Report Summary - ACADIS (Academic Dishonesty: the Non-Survey Approach)
The project was inspired by signals of growing threat of some forms of violations of academic integrity. Its main objective was to use field studies and experiments to validate findings from surveys, which may be plagued with social desirability bias and other sources of unreliability.
Four of the five initially planned studies have been carried out. Research on plagiarism had to be postponed due to data unavailability. Instead, the remainder of the balance of the research project has been modified and extended.
In the study published in Accountability in Research (with Ernesto Reuben, Columbia) we have tried to establish whether authors of economic papers are willing to deliver on their promises to send any interested reader supplementary material, such as alternative econometric specifications, 'upon request'. The requests to the authors were sent either by a researcher affiliated at Columbia University, New York, United States of America (USA) or the University of Warsaw, Poland; furthermore, the authors' position (assistant professor) was specified in half the e-mails only. Overall, 64 % of the approached authors responded to our message, of which two thirds (44 % of the entire sample) delivered the requested materials. The frequency and speed of responding and delivering were very weakly affected by the position and affiliation of the sender. We have also found that authors of published papers were much more likely to share than authors of working papers.
In the study now resubmitted to the American Statistician, I have investigated the use and misuse of p values in papers published in five (high-ranked) journals in experimental psychology. I used a data set of over 135 000 p values from more than five thousand papers. I inspected:
(1) the way in which the p values are reported; and
(2) their distribution.
The main findings are following: first, the authors choose the mode of reporting their results in an arbitrary and strategic way, that is, to make them seem more statistically significant than they really are (which is well known to improve the chances for publication). Specifically, they frequently report p values 'just above' significance thresholds directly, whereas other values are reported by means of inequalities (e.g. p < 0.1) they round the p values down more eagerly than up and choose between the significance thresholds and between one- and two-sided tests after seeing the data. Second, about 9.2 % of reported p values are inconsistent with their underlying statistics (e.g. F or t) and it appears that there are 'too many just significant' values, suggesting use of data or model manipulation techniques to bring the p value to the right side of the threshold.
The third study focused on behaviour of academics dealt with the question whether calculations and derivations omitted in economic papers are correct. A Google scholar search of papers in economics / business containing the phrase 'tedious but straightforward' was performed. Of these, we took a random sample of 50 and attempted to verify the correctness thereof. For the 27 papers that we believed we could deal with, Mathematica code has been developed in the attempt to bridge the gap in the paper. The authors' assertion concerning the skipped part of derivation was found to be correct in 19 cases. In 8 remaining cases (29 %), mistakes or omissions were identified. The relevant authors have been notified. In the time of writing, we are still in the process of clarifying one of these cases.
The remaining studies dealt with students' behaviour. In a laboratory study submitted to Organisational Behavior and Human Decision Processes, I investigated factors affecting propensity to cheat. I found that making outcomes depend on random event rather than own performance made subjects behave more dishonestly, but only when procedural unfairness of this procedure is made salient.
A study submitted to the Journal of Higher Education reports a field experiment investigating effectiveness of moral appeal in discouraging exam cheating. In all conditions, substantial level of cheating was identified using an index of test answers similarity, contrasted with low self-reports. The treatment manipulation made an impact on self-reported but not observed frequency of cheating. Hypothesised gender difference, whereby males took but not gave more illicit information than females was also found.
Finally, in a study (paper is in preparation) on Polish internet websites offering illicit academic writing services ('paper mills') we have found that a substantial fraction of college students purchases and submits ghost-written thesis as their own. These pieces are usually of relatively low quality and are available for as little as equivalent of approximately EUR 500.
The studies cover selected issues within the very broad field of research on academic integrity. Nevertheless, taken together they seem to consistently yield following general conclusions. First, there is often a discrepancy between self-reports and practice of academic integrity, thus non-survey studies are a must. Second, some forms of unethical behaviour can be committed by nearly anyone, rather than but a few 'bad apples'. Third, material incentives and organisational arrangement have profound impact on prevalence of questionable practices.
Four of the five initially planned studies have been carried out. Research on plagiarism had to be postponed due to data unavailability. Instead, the remainder of the balance of the research project has been modified and extended.
In the study published in Accountability in Research (with Ernesto Reuben, Columbia) we have tried to establish whether authors of economic papers are willing to deliver on their promises to send any interested reader supplementary material, such as alternative econometric specifications, 'upon request'. The requests to the authors were sent either by a researcher affiliated at Columbia University, New York, United States of America (USA) or the University of Warsaw, Poland; furthermore, the authors' position (assistant professor) was specified in half the e-mails only. Overall, 64 % of the approached authors responded to our message, of which two thirds (44 % of the entire sample) delivered the requested materials. The frequency and speed of responding and delivering were very weakly affected by the position and affiliation of the sender. We have also found that authors of published papers were much more likely to share than authors of working papers.
In the study now resubmitted to the American Statistician, I have investigated the use and misuse of p values in papers published in five (high-ranked) journals in experimental psychology. I used a data set of over 135 000 p values from more than five thousand papers. I inspected:
(1) the way in which the p values are reported; and
(2) their distribution.
The main findings are following: first, the authors choose the mode of reporting their results in an arbitrary and strategic way, that is, to make them seem more statistically significant than they really are (which is well known to improve the chances for publication). Specifically, they frequently report p values 'just above' significance thresholds directly, whereas other values are reported by means of inequalities (e.g. p < 0.1) they round the p values down more eagerly than up and choose between the significance thresholds and between one- and two-sided tests after seeing the data. Second, about 9.2 % of reported p values are inconsistent with their underlying statistics (e.g. F or t) and it appears that there are 'too many just significant' values, suggesting use of data or model manipulation techniques to bring the p value to the right side of the threshold.
The third study focused on behaviour of academics dealt with the question whether calculations and derivations omitted in economic papers are correct. A Google scholar search of papers in economics / business containing the phrase 'tedious but straightforward' was performed. Of these, we took a random sample of 50 and attempted to verify the correctness thereof. For the 27 papers that we believed we could deal with, Mathematica code has been developed in the attempt to bridge the gap in the paper. The authors' assertion concerning the skipped part of derivation was found to be correct in 19 cases. In 8 remaining cases (29 %), mistakes or omissions were identified. The relevant authors have been notified. In the time of writing, we are still in the process of clarifying one of these cases.
The remaining studies dealt with students' behaviour. In a laboratory study submitted to Organisational Behavior and Human Decision Processes, I investigated factors affecting propensity to cheat. I found that making outcomes depend on random event rather than own performance made subjects behave more dishonestly, but only when procedural unfairness of this procedure is made salient.
A study submitted to the Journal of Higher Education reports a field experiment investigating effectiveness of moral appeal in discouraging exam cheating. In all conditions, substantial level of cheating was identified using an index of test answers similarity, contrasted with low self-reports. The treatment manipulation made an impact on self-reported but not observed frequency of cheating. Hypothesised gender difference, whereby males took but not gave more illicit information than females was also found.
Finally, in a study (paper is in preparation) on Polish internet websites offering illicit academic writing services ('paper mills') we have found that a substantial fraction of college students purchases and submits ghost-written thesis as their own. These pieces are usually of relatively low quality and are available for as little as equivalent of approximately EUR 500.
The studies cover selected issues within the very broad field of research on academic integrity. Nevertheless, taken together they seem to consistently yield following general conclusions. First, there is often a discrepancy between self-reports and practice of academic integrity, thus non-survey studies are a must. Second, some forms of unethical behaviour can be committed by nearly anyone, rather than but a few 'bad apples'. Third, material incentives and organisational arrangement have profound impact on prevalence of questionable practices.