SCIENTIFIC RIGOUR AND TRANSPARENCY

The movement to improve methodological rigour and transparency is gaining momentum in many scientific fields. The SPELab regards this movement as crucial for the future of science.  We are actively revamping our efforts to align ourselves with these improved standards and practices. We are also prioritising methods studies to strengthen the foundation of social neuroendocrinology research. Here are some of the changes we are implementing.

Open Data and Materials
  • Several of our publications have open datasets and open materials. Links are posted on the Publications page.
  • New studies are being designed with the aim of making datasets open, e.g., seeking ethics approval for open datasets.

Increased Statistical Power
  • We are boosting sample sizes in new studies, as time and resources allow.
  • We are engaging in multi-site collaborations, as a useful way to increase sample sizes and estimate heterogeneity in effects (e.g. Knight et al., in press, Psychological Science).

Hormone Measurement Validity: A Gradual Move Toward Mass Spectrometry-Based Methods
  • The most common technique to measure steroid hormone concentrations within psychology is with immunoassay. Although psychologists have generally assumed that this technique is valid, our recent research points to lower measurement validity than we would hope as well as unstable measurement validity (Welker et al., 2016, Prasad et al., 2019). Our recent papers have documented these limitations of salivary testosterone enzyeimmunoassays, but this general pattern is expected to generalise to other hormones that are present in low concentrations in saliva (see Schultheiss, Dlugash, & Mehta, 2019). We also note that salivary cortisol immunoassays seem to be more valid than salivary testosterone immunoassays, likely because cortisol is present at higher concentrations in saliva compared to testosterone (Welker et al., 2016).
  • We have written several methodological papers to improve the methodological foundation of social neuroendocrinology research (e.g., Chapters 2 and 3 in Schultheiss & Mehta, 2019).
  • When feasible, we recommend that researchers employ mass-spectrometry based methods for measuring steroid hormone concentrations. Our new papers explain why we are making this recommendation (Welker et al., 2016; Schultheiss, Dlugash, & Mehta, 2019, Prasad et al., 2019).

Pre-Registration
  • We are pre-registering many of our new studies.

Replication
  • We are conducting replication studies, with relatively high statistical power (compared to prior studies) achieved through multi-site collaboration, and working toward publishing these studies independent of outcome (e.g. Knight et al., in press, Psychological Science).
  • Our lab participated in one of the many replication studies reported in this Science paper

Registered Reports
  • We love the idea of registered reports, which we discuss in Chapter 3 of Schultheiss & Mehta, 2019.
  • We currently have a registered report under review with a large group of collaborators.

Statistics and Results Interpretation: Moving Away from Significant/Not Significant Binary Thinking
  • We are moving away from the null hypothesis significance testing framework (significant/not significant) to draw conclusions and moving towards adopting estimation techniques. For example, for an effect of interest, we are reporting (i) a point estimate of the direction and magnitude of the effect, and (ii) a confidence interval to capture uncertainty in our estimate. We are drawing conclusions based on the point estimate and confidence interval.

Pre-Prints
  • We are now posting pre-prints of our working papers (e.g. https://psyarxiv.com/) and updating these pre-prints as our papers are revised during peer review. Links to pre-prints can be found on our Publications page.

Peer Review
  • We are doing our best to evaluate research based on scientific rigour, independent of the results.

Teaching
  • We are teaching about methodological rigour and transparency in classes, workshops, and lab meetings.

Publications from our lab relevant to reproducibility, methods, and open science.

Sundin, Z.W., Chopik, W.J., Welker, K.M., Ascigil, E., Brandes, C.M., Chin, K.A., Ketay, S., Knight, E.L., Kordsmeyer, T.L, McLarney-Vesotski, A.R., Prasad, S., Reese, Z.A., Roy, A.R.K, Sim, L., Stern J., Carré, J.M., Edelstein, R.S., Mehta, P.H., Penke, L., Slatcher, R.B. & Tackett, J.L. (Stage 1 registered report; data analyses in progress). Estimating the associations between big five personality traits, testosterone, and cortisol.

Knight, E.L., McShane, B.B., Kutlikova, H.H., Morales, P.J., Christian, C.B., Harbaugh, W.T. Mayr, U. Ortiz, T.L., Gilbert, K. Ma-Kellams, C.. Riečanský, I., Watson, N.V., Eisenegger, C., Lamm, C., Mehta, P.H., & Carré, J.M. (in press).  Weak and variable effects of exogenous testosterone on cognitive reflection in three experiments: Commentary on Nave et al. (2017). Psychological Science.

Prasad, S, Lassetter, B., Welker, K. M., & Mehta, P.H. (2019). Unstable correspondence between salivary testosterone measured with enzyme immunoassays and tandem mass spectrometry. Psychoneuroendocrinology, 109, 104373.

Schultheiss, O.C. & Mehta, P. H. (2019). Reproducibility in social neuroendocrinology: Past, present, and future. In O.C. Schultheiss and P.H. Mehta (Eds.).  The International Handbook of Social Neuroendocrinology. Routledge Press.

Schultheiss, O.C., Dlugash, G., & Mehta, P. H. (2019). Hormone measurement in social neuroendocrinology: A comparison of immunoassay and mass spectrometry methods. In O.C. Schultheiss and P.H. Mehta (Eds.).  The International Handbook of Social Neuroendocrinology. Routledge Press.

Welker, K. M., Lassetter, B., Brandes, C. M., Prasad, S., Koop, D. R., & Mehta, P. H. (2016). A comparison of salivary testosterone measurement using immunoassays and tandem mass spectrometry. Psychoneuroendocrinology, 71, 180-188.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.