We use data from experiments to understand differences.

Differences come in many forms:

• Performance
• Preferences

Experiments can be:

• Exploratory
• Hypothesis driven

Statistical analysis is about creating confidence from evidence

Size of experiment

For most experiments, more is better.

If things we are comparing are similar, we need more subjects.

With enough people though, we can always find differences, so we must be careful.

Every experiment has 4 main considerations:

• Partecipants
• Apparatus
• Experiment Procedure
• Design and Analysis

Partecipants

Sampling: how to select subjects for our study from broader population, from which we want to draw conclusions (inferences).

There are two main types of sampling: probability/non probability sampling.

• probability: select randomly
• non probability: uses other approaches to acquire participants (convenience/snowball/purpose sampling)

What criteria do we use for our study to select partecipants? Exclusion criteria, etc

Apparatus

what do we need in terms of equipment and resources to run the test?

How will data be captured?

• Log files.
• human observation
• video recoding

Procedure

what do partecipants do? Do they perform tasks? How many? Do they get to practice?

Design and Analysis

what is the formal design that we are using?

Considerations

Informed consent is vital. Set clear expectations. Debrief partecipants at end of study.

References

• https://www.coursera.org/learn/designexperiments

Next -> test-of-proportions