Developing a Critical Eye: Why You Shouldn’t Always Believe the Data
Humans introduce bias into all research (scientific, economic, political, geographical…). Survey respondents for a study are more likely to remember some types of events than others (recall bias), research samples can under-represent particular people or groups, and over–represent others (selection bias), bias is common in both peer review and publishing. Then there’s observation bias, confirmation bias, social desirability bias, question-order bias, and the list goes on. Even the use of AI can encode human biases.
This Curious Minds digest aims to help you develop a critical eye when analysing data and research studies - when reading articles around your subject, it is important to hone your ability to critically appraise and challenge assumptions in research. If you are approaching university interviews, this will be particularly useful as you may be given an article to read as pre-reading or data to analyse during the interview itself. This digest will pose a range of interview-style questions and suggest tasks to get you thinking about bias in research, as well as recommend a number of further reading resources.
Question 1:
Bias is any trend or deviation from the truth in data collection, data analysis, interpretation and publication which can cause false conclusions (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900086/) . It can occur at each stage of a systematic investigation, either consciously or unconsciously:
Gathering the data and running an experiment can go awry
Researchers can introduce bias in the study when collecting data for research. For example, selection bias occurs when a sample may not reflect the general population - it could be too small or some patients may be more likely to enter the study than others, meaning that some groups will be under-represented and others, over-represented.
Experimental error and failure to take into account all possible variables can also account for some bias in research. For example, a Type IV error in scientific research refers to the incorrect interpretation of a null hypothesis that has been correctly rejected. This could be due to an error in analysis, bias or playing with the data to arrive at incorrect conclusions
Methods chosen to analyse the data can influence the results
The recent article, “Methods and Madness”, in the Jul 31st 2021 edition of The Economist, provides an illuminating example as to why we need to look into the specifics of an experiment, not only in terms of the sampling strategy and data collection, but also study methodology, data analysis, and conclusions drawn from this analysis.
The article refers to a new paper from a team of 49 researchers headed by Martin Schweinsberg, a psychologist at the European School of Technology in Berlin. Each was handed a dataset of 3.9m words of text from 8,000 comments made on Edge.org, an online forum for intellectuals. Schweinsberg hypothesised that high-status participants would talk more than their low-status counterparts. Each researcher was asked to post their methods to a website called DataExplained. No researchers achieved the same results, or used the same methods. 29% of analysts reported that high-status participants did indeed participate more, whilst 21% thought the opposite. Schweinsberg found that the range of statistical techniques used, as well as the researchers’ choice of definition of ‘high-status’, impacted the results, leading them to draw differing conclusions.
Question 3: What can be done to mitigate bias in research?
Try to brainstorm an answer to this question:
Begin by defining “bias”
You could discuss how bias can be introduced at each stage of a systematic investigation
Research solutions e.g. Internal vs. External Validity, the importance of critical appraisal (see this article to find out how to critically appraise a paper), the use of Artificial Intelligence (see below) etc.
Question 4:
The use of AI has been lauded by some as a way to mitigate against human’s introduction of bias into research, as a move away from ethnocentric and androcentric views of the world. However, there is increasing concern that machine learning tools also embed bias in their operations and outputs. To explore this question further and work out where you stand on the debate, visit The Naked Scientists Podcast site for some great resources on the issue:
Final Tips for Pre-Reading at Interviews
Recognising the assumptions/ bias that underpin most research, as well as understanding the process of critical appraisal, will stand you in great stead for prospective top university/ Oxbridge interviews. To practise for the eventuality that you are given a study/ article as pre-reading, select an article e.g. a short review article from Nature - attempt the following:
(i) Offer a very clear introduction to the article;
(ii) Aim for good use of signposting/structuring of response to guide the interviewer through the article;
(iii) Offer a thoughtful and comprehensive critique of the study’s limitations.
For further reading, try Ben Goldacre’s book Bad Science - you’ll find a great review by University College, Oxford here. The Royal College of Surgeon’s article linked in Question 3: “Dissecting the literature: the importance of critical appraisal”, also suggests a range of further resources including a checklist to use when reading a paper.
Are you applying to a top UK University/ Oxbridge and keen to put your new critical appraisal skills to the test? We offer interview preparation & host three mock interview days to build confidence ahead of the real thing:
Visit our University Applications Page on our on-curriculum division, U2 Tuition for full details. Our Oxbridge programmes aim to advance students to the level required for success at interview through regular tutorial sessions and specific admissions preparation (for personal statement, admissions test & interview). In sessions, students are expected to orally communicate, defend, analyse & critique ideas or solve problems, in conversation with the tutor (an Oxbridge graduate in the student’s course of application), as in an Oxbridge tutorial. This readies students for the challenging entry process and provides them with the tools needed to stand out amongst other strong candidates.
1-1 Sessions From £70/h.
We also host three Oxbridge/ Medical School Online Mock Interview Days in the lead up to interviews.