Tips On Conducting Experiments
From PsychWiki - A Collaborative Psychology Wiki
Revision as of 00:16, 23 August 2006 by Monroe
Lessons learned from conducting research...
- Use this link for Internet Research.
- Each of us has learned the hard way the many "do's" and "dont's" of conducting research that you can only find out about by hands-on experience. Here is a chance to pass on what you have learned and the tips/tricks that will make it easier for others who are starting to conduct their own research projects...
what other categories can you think to add here...
- A pilot test is usually not published in journal articles.
- Can run less Ss with pilot test than regular study.
- Less pilot testing is needed when using established measures and established procedures.
- If purpose of pilot test is to test the effect of newly created manipulations and the manipulation checks are not accurate/effective in the pilot test, you have to ask yourself whether there is flawed manipulation check and/or flawed manipulation.
- The position of multiple measures before a key outcome measure may reduce or eliminate the effect on the outcome measure, and in such cases, it may make sense to run separate pilot studies with separate manipulation checks, so that when you run the real study you have increased chances of having the manipulation check accurately ascertaining the manipulation.
- Random assignment is always required and you should adhere to the rules of random assignment as strictly as you can to prevent inaccurate data, and worse yet, reviewers which reject your paper for failing to ensure random assignment.
- The Research Randomizer is a great tool for random assignment.
- If separate experimenters, each experimenter has own set of random assignment procedures.
- Replace suspicious Ss and ‘botched implementation’ Ss (e.g., bizarre, etc) with the next person in random assignment order.
- Try to decide after running each Ss whether to include or not.
- Decide for each person without looking at data and/or outcome measure(s).
- Have a “toss” variable in data analysis, e.g., assign each Ss a code as 'yes', 'no', or 'maybe' toss.
- In data analysis, code and analyze for these 'toss Ss' to ensure they are not different (and possible confound) than any other Ss in the study.
- When manipulating a variable, you can choose to manipulate “yes versus no” (e.g., manipulate the presence or absence of the variable), or you can choose to manipulate “high versus low” (e.g., the variable is always present, but you are manipulating the strength). In some situations, when doing “high versus low”, you need the “no treatment” control condition to see which side(s) is the source of the effect -- example of positive/negative mood manipulation where without “no treatment” control condition, unsure of whether the effects are due to the positive mood or negative mood.
- Reviewers will criticize that you can't identify the source of the effect if don’t have a “no treatment” control condition.
- There are different types of "no treatment" control conditions -- Imagine a study where the manipulation is an essay that is positively valenced, so the "no treatment" control condition would be an essay that is neutrally valenced. Compare this to a situation where the manipulation is the essay itself (e.g., reading an essay produces some predicted outcome) in which case "no essay" would be the control condition, but you have to ensure that the "no essay" condition matches the other conditions in terms of time/effort to hold constant possible confounds.
- Can pilot test control conditions to ascertain characteristics of manipulations and whether they are truly working and/or the control condition is truly not confounding.
- Depending on the circumstances, it may make sense to start a research program looking at “yes versus no” before moving on to “high versus low”.
- Two types of manipulation checks are (1) to ask the Ss if they remember what you asked them to do in the study, or (2) instead measure the underlying construct or manipulation. Recommended to put both into the study.
- Manipulation checks can be used as mediators. You can use the manipulation check as the “manipulation check” in that part of the results section, and then later use the same measure as a mediator.
- Only the second type of manipulation check can be used as a mediator.
- The manipulation check is necessary when: If your argument is that the manipulation is doing something specific, or you are trying to argue why the manipulation is producing a result on the DV, then you need the manipulation check to prove this. Thus, when you are arguing that X, and not Y, is producing the result, you need the manipulation check to prove this. For example, if your argument is that anger produces an effect on the DV, then after a mood manipulation you need to prove that anger, and not sadness or general affect, is producing the result.
- The manipulation check is not necessary when: If your argument is simply that the manipulation is producing a result, irrespective of whatever it is specifically that the manipulation may be producing, then you do not need a manipulation check because you are NOT arguing why or how the manipulation is producing the result. For example, if your argument is simply that manipulating mortality salience within terror management theory produces an effect on the DV, and you are not arguing why or how mortality salience produces this result, then a manipulation check is not essential, but can still be informative if included in the study.
- When critiquing or reviewing articles, you only criticize a study for not having a manipulation check when: 1) The argument in the paper relates to why or how the manipulation is producing an effect on the DV, and 2) you don’t have an alternative reason why or how the manipulation is producing an effect on the DV.
- If you adhere to the information above about random assignment and discarding subjects, you should have reasonable equal cell sizes. If you lose too many subjects from one cell, this may be an issue because that cell may have issues/problems related to design of study.
- If unequal cell sizes, then reduces power.
- More confidence in generalization to population when cell sizes more equal.
- When running subjects, how do you know when to stop running subjects? Sometimes you can estimate from prior studies in that same topic/field how many Ss you need to achieve a reasonable good/high level of power.
- General rule is that you don’t run stats on the data while running Ss and stop when you reach significance.
- Some people adhere to a heuristic of 15-20 Ss in a cell, but the number varies across studies.
- If you conceptually replicate a finding, then it bolsters your confidence in your findings, even if those studies don’t always have high cell sizes.
Mining your data for ideas
- Mining your data (e.g., going beyond just the standard statistical tests and looking at all possible relationships in your study even if they are not directly related to your purpose for running the study) is one way to ascertain other possible relationships amongst the data and research ideas.
- This information may not go into the manuscript, but it is a good source of ideas for you for future research.
- One idea is to look at “pre-measure” measures for your studies, and/or putting in personality measures into the pre-measure to determine whether there are personality variables that interact/influence variables in your study.
- If you collect demographic data in the study, this is another source of information for possible future research.
- One reason to collect demographics at beginning of the study is if one of the demographic variable is an a priori variable in the study so you want to measure it before the manipulation or if you need to know that demographic information in order to assign Ss to conditions.
- You should run your own studies first for a few replications instead of letting an R.A. run all the subjects because then you can pass along what you learned to your R.A. when they start running the study.
- Occasionally interrogate/debrief your Ss and your RAs over the course of the experiment to ensure they are doing what you told them to do.
- Run RAs through the experiment as a Ss so that they can learn about the study from the Ss point of view.
- Also, run R.A.s and graduate students through the study as if they are Ss to help you with double-checking that the instructions, questions, and other materials are clear and error-free.
- If multiple RA’s are running subjects, each RA should have their own set of random assignment conditions, and each RA should do full replication before moving on to next randomly assigned conditions, and each RA should run the same amount of Ss.
- Implementation of experimenter blind procedures is the best approach, but sometimes difficult to achieve when you are running the Ss yourself, but even in that case you can protect yourself from knowing which condition the Ss is in if the manipulation is simply presenting information (e.g., survey or form you hand to them) by preparing packets of materials ahead of time so that you are unaware of which packet is going to which subject.
- When its impossible to be blind to the procedures, you want to code who was the experimenter to show/prove that when R.A. is a blind experimenter the data are the same as when you ran the Ss (and was not blind).
- Positioning items non-randomly is not recommended, such as presenting each scale in its entirety before presenting next scale in its entirety.
- This goal (of random assignment of all items) is the gold standard for non-experimental studies, but is less practical/relevant for experimental studies because we want pure/accurate measurement of the key dependent variables, which may be reduced/eliminated if they are thrown into a mix and presented after other items.
- Run data analysis relatively early in the implementation of the study to ensure that its not a waste of your time to continue, and/or to go check/ensure R.A.s are running subjects properly.
- What is the best way to assess suspicion in participants? Funnel debriefing -- if the study is an in-person experiment. Funnel debriefing is when you start with the most abstract and open-ended questions and then funnel down to the most specific and closed-ended questions. If not in-person, such as field studies and online studies, can try using written questions such as asking them what they think the purpose of the study was, or what was your impression of [the person or events] depicted in the study.
► Back to Research Tools mainpage