From the course: The Data Science of Experimental Design

Features of an experiment

From the course: The Data Science of Experimental Design

Start my 1-month free trial

Features of an experiment

“

- [Instructor] Now we are going to look at specific features of experiments. This study design hierarchy diagram is one that I've shown you before. Just as a reminder, the experiment is counterpoised against or basically the opposite of an observational study. In an observational study, the researcher collects data about humans but does not assign them to do anything they would not normally do. In contrast, in an experiment, the researcher actually makes choices for the research participant and controls their choices in the study. So that is the first feature I want you to remember about experiments, and that is in an experiment, the researcher makes choices for the participants. Not the participant. Let's get into the details of what I mean by choices. Here is what the researcher assigns in an experiment, the conditions. Let's talk about the word conditions. In experimental research, the word condition roughly means the environment in which the study or research occurred. Think of a study testing a new uniform for army soldiers. One condition might be desert and one might be rainy. So another feature of an experiment is that you need at least two conditions and the goal is to compare the conditions. Let's sum what we have gone over so far as features of experiments, but let me put them in a slightly different way. We just talked about how experiments need to have at least two conditions, so you can compare them and tell which is better. Earlier, we talked about another feature of an experiment, which is that the researcher makes choices for the participants. Actually, the technical term for participants is experimental units, which means the type of entity on which you are doing your experiment. So the two features of experimental design we have talked about so far are that the experiment must have at least two comparable conditions and that the researcher must be the one to assign the participants to the conditions, not the participants themselves. So why would you ever do an experiment? As I showed you earlier, you could just do an observational study. That way, you would not need any conditions and you would not need to assign anyone to them. But experimental design is really what you need if you want to see if something new is better without completely converting over to the new way. In an experiment, the old way can be one condition and the new way can be the other condition, and you can compare them. That way, you have the information to choose whether or not to convert to the new way or stay with the old way. So up to now, experiments might sound like a pretty good deal but a lot of people do not appreciate how much work they are. Imagine an app developer who develops her app with a blue color scheme and wonders if users would respond more positively to a green color scheme. To do an experiment, she would basically have to build two versions of the app, one blue and one green. This would be a lot of work, especially if it was found at the end of the experiment that the blue-themed app was preferable. And it is also effort intensive and time consuming to keep all these assignments straight and to collect two different buckets of data, one for each condition. This is why you will find that observational studies are much more common than experiments. They are expensive and more work but on the other hand, experiments can be very valuable. Remember how I talked about all the bias involved in observational studies? That is what all the effort in experiments is going towards, reducing the bias so you can see which condition is really better, the old way or the new way. Experiments can provide uniquely valuable information that cannot be provided by other study designs. So under the right circumstances, the extra expense and time are worth it.

Contents