SCIENTIFIC RIGOUR AND TRANSPARENCY
We are committed to improving methodological rigour and transparency in our work.
Open Data and Materials
Increased Statistical Power
Hormone Measurement Validity
Pre-Registration
Replication
Registered Reports
Statistics and Results Interpretation: Moving Away from Significant/Not Significant Binary Thinking
Pre-Prints
Peer Review
Teaching
Publications from our lab relevant to reproducibility, methods, and open science.
Chopik, W.J., Sundin, Z.W., Welker, K.M., Ascigil, E., Brandes, C.M., Chin, K.A., Ketay, S., Knight, E.L., Kordsmeyer, T.L, McLarney-Vesotski, A.R., Prasad, S., Reese, Z.A., Roy, A.R.K, Sim, L., Stern J., Carré, J.M., Edelstein, R.S., Mehta, P.H., Penke, L., Slatcher, R.B. & Tackett, J.L. (Stage 2 registered report; under review). Estimating the associations between big five personality traits, testosterone, and cortisol.
Knight, E.L., McShane, B.B., Kutlikova, H.H., Morales, P.J., Christian, C.B., Harbaugh, W.T. Mayr, U. Ortiz, T.L., Gilbert, K. Ma-Kellams, C.. Riečanský, I., Watson, N.V., Eisenegger, C., Lamm, C., Mehta, P.H., & Carré, J.M. (2020). Weak and variable effects of exogenous testosterone on cognitive reflection test performance in three experiments: Commentary on Nave et al. (2017). Psychological Science.
Prasad, S, Lassetter, B., Welker, K. M., & Mehta, P.H. (2019). Unstable correspondence between salivary testosterone measured with enzyme immunoassays and tandem mass spectrometry. Psychoneuroendocrinology, 109, 104373.
Schultheiss, O.C. & Mehta, P. H. (2019). Reproducibility in social neuroendocrinology: Past, present, and future. In O.C. Schultheiss and P.H. Mehta (Eds.). The International Handbook of Social Neuroendocrinology. Routledge Press.
Schultheiss, O.C., Dlugash, G., & Mehta, P. H. (2019). Hormone measurement in social neuroendocrinology: A comparison of immunoassay and mass spectrometry methods. In O.C. Schultheiss and P.H. Mehta (Eds.). The International Handbook of Social Neuroendocrinology. Routledge Press.
Welker, K. M., Lassetter, B., Brandes, C. M., Prasad, S., Koop, D. R., & Mehta, P. H. (2016). A comparison of salivary testosterone measurement using immunoassays and tandem mass spectrometry. Psychoneuroendocrinology, 71, 180-188.
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Open Data and Materials
- Several of our publications have open datasets and open materials. Links are posted on the Publications page.
- New studies are being designed with the aim of making datasets open, e.g., seeking ethics approval for open datasets.
Increased Statistical Power
- We are boosting sample sizes in new studies, as time and resources allow.
- We are engaging in multi-site collaborations, as a useful way to increase sample sizes and estimate heterogeneity in effects (e.g. Knight et al., 2020, Psychological Science).
Hormone Measurement Validity
- We have conducted methods studies to estimate the validity of immunoassays for the measurement of steroid hormones. Our recent work points to the possibility that saliva handling and storage factors within a researcher's control could improve the validity of immunoassays for steroid hormone measurement (Prasad et al., 2019). However, more research is needed to test this hypothesis directly.
- We have written several methodological papers to improve the methodological foundation of social neuroendocrinology research (e.g., Schultheiss, Dlugash, & Mehta, 2019).
Pre-Registration
- We are pre-registering many of our new studies.
Replication
- We are conducting replication studies, with relatively high statistical power (compared to many prior studies) achieved through multi-site collaboration, and working toward publishing these studies independent of outcome (e.g. Knight et al., 2020, Psychological Science).
- Our lab participated in one of the many replication studies reported in this paper.
Registered Reports
- We love the idea of registered reports, which we discuss in Chapter 3 of Schultheiss & Mehta, 2019.
- We currently have a registered report in progress with a large group of collaborators.
Statistics and Results Interpretation: Moving Away from Significant/Not Significant Binary Thinking
- We are gradually moving away from the null hypothesis significance testing framework (significant/not significant) to draw conclusions and moving towards adopting estimation techniques. For example, for an effect of interest, we are reporting (i) a point estimate of the direction and magnitude of the effect, and (ii) a confidence or credible interval to capture uncertainty in our estimate. We are drawing conclusions based on the point estimate and interval.
Pre-Prints
- We are now posting pre-prints of our working papers (e.g. https://psyarxiv.com/) and updating these pre-prints as our papers are revised during peer review. Links to pre-prints can be found on our Publications page.
Peer Review
- We are doing our best to evaluate research based on scientific rigour, independent of the results.
Teaching
- We are teaching about methodological rigour and transparency in classes, workshops, and lab meetings.
Publications from our lab relevant to reproducibility, methods, and open science.
Chopik, W.J., Sundin, Z.W., Welker, K.M., Ascigil, E., Brandes, C.M., Chin, K.A., Ketay, S., Knight, E.L., Kordsmeyer, T.L, McLarney-Vesotski, A.R., Prasad, S., Reese, Z.A., Roy, A.R.K, Sim, L., Stern J., Carré, J.M., Edelstein, R.S., Mehta, P.H., Penke, L., Slatcher, R.B. & Tackett, J.L. (Stage 2 registered report; under review). Estimating the associations between big five personality traits, testosterone, and cortisol.
Knight, E.L., McShane, B.B., Kutlikova, H.H., Morales, P.J., Christian, C.B., Harbaugh, W.T. Mayr, U. Ortiz, T.L., Gilbert, K. Ma-Kellams, C.. Riečanský, I., Watson, N.V., Eisenegger, C., Lamm, C., Mehta, P.H., & Carré, J.M. (2020). Weak and variable effects of exogenous testosterone on cognitive reflection test performance in three experiments: Commentary on Nave et al. (2017). Psychological Science.
Prasad, S, Lassetter, B., Welker, K. M., & Mehta, P.H. (2019). Unstable correspondence between salivary testosterone measured with enzyme immunoassays and tandem mass spectrometry. Psychoneuroendocrinology, 109, 104373.
Schultheiss, O.C. & Mehta, P. H. (2019). Reproducibility in social neuroendocrinology: Past, present, and future. In O.C. Schultheiss and P.H. Mehta (Eds.). The International Handbook of Social Neuroendocrinology. Routledge Press.
Schultheiss, O.C., Dlugash, G., & Mehta, P. H. (2019). Hormone measurement in social neuroendocrinology: A comparison of immunoassay and mass spectrometry methods. In O.C. Schultheiss and P.H. Mehta (Eds.). The International Handbook of Social Neuroendocrinology. Routledge Press.
Welker, K. M., Lassetter, B., Brandes, C. M., Prasad, S., Koop, D. R., & Mehta, P. H. (2016). A comparison of salivary testosterone measurement using immunoassays and tandem mass spectrometry. Psychoneuroendocrinology, 71, 180-188.
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.