From the course: The Data Science of Experimental Design

Experiment is a type of study

From the course: The Data Science of Experimental Design

Start my 1-month free trial

Experiment is a type of study

- [Instructor] I'm so pleased you chose this data science course about experiments. As you may already know, an experiment is a type of study. Whenever we choose to analyze data, whether it's data already in a database or data we collected ourselves, we always have to have some sort of study design behind it. An experiment is just one type of study design. To better understand the features of experimental study designs, it's helpful to understand the other study designs first. I'm going to explain them to you using terminology from my field, which is public health. But please understand that the same study designs might be called something different in another field, such as psychology. But lucky for you, in every field, we all agree on what an experiment is. In epidemiology, I find it helpful to think of the study designs not as equals but as a hierarchy, so I'm going to show you this helpful diagram I call the study design hierarchy. It starts at the top with human research, which is epidemiologists do, and if you are studying user behavior in your experiment, you will be researching humans as well. Let's start going down the hierarchy. Our first stop on the left side of the diagram is called observational studies. These are where we collect data about humans by simply recording what they do naturally. Perhaps I enroll a group of people who suffer migraines in a study, and ask them to record what medications they take when they get sick. If I do that, I'm just observing their behavior. This seems like a nice study design, right? Well, it is nice, but there are caveats. Let's move down the hierarchy from observational and you will see that it splits. On one side we have descriptive studies. Descriptive study designs do not test a hypothesis, but simply try to describe the data. You will learn more about this in this course actually because it is necessary to do some descriptive analysis as part of preparation for an experiment. Here, you will see the other way that observational studies split is called analytic. Those study designs are intended to test hypotheses. And trust me, they are really challenging. Remember, when you have a hypothesis, you are trying to figure out a cause, and in an analytic study, you are trying to test a hypothesis, but the data are very biased. I'll give you some examples. I'm vegetarian and I also do yoga, and if you look in the world, you'll find a lot of people who are vegetarian and do yoga. Many video game fans are also comic books fans, and many history buffs like to both read about history and visit historical museums. So these observational biases, while interesting, can interfere with being able to tell what causes what in the data. So if I find out in my observational study that vegetarians are healthier than everyone else, is it because they are vegetarian, or because a lot of them do yoga? Or could it be caused by some other healthy behavior they do that we don't even know about? So we have this problem with being able to tell what caused what. This is where experiment comes in. Let's go back to the study design hierarchy. You will see I added experiment as a type of human research, opposite to observational studies. In experiment, you are not simply observing what people are doing. You are actually controlling what they are doing. Well, only to some extent. There are boundaries and limitations. In fact, you will learn about those later in the course. But that basic idea that the researcher controls the choices, not the research participant, is the hallmark of an experiment. It is what makes an experiment different from an observational study. So, although we will do some descriptive analysis in this course to prepare for our experiment, the focus of this course will be on the specific steps to take when conducting a data science experiment.

Contents