Refine the Research Questions

There are a lot of evaluation questions for which we’d like to have answers. However, not every question can be asked because of the issues just below. Refining your questions to ask and answer means dealing with these factors, so that what your inquiry pursues is reasonable, cost-effective, and the answers will really help you improve your program. Some questions you COULD ask will not fit those criteria, so figure that out and don’t waste your and others’ time.

Another aspect to this issue is whether the use of data from these assessments is a high stakes use. The higher the stakes, the more these factors must be addressed. For example, whether someone is rehired or not IS high stakes and holds many ramifications, so assessment that inform that decision need to be high quality.

  1. Measurability -Can what you want to know actually be reasonably measured? If there is no assessment available for it, can you develop your own?
  2. Feasibility – Would that require such time, skill, expense, etc. as to make it unreasonable to do it?
  3. Clarity – Sometimes what we want to know seems so clear, but measuring it or using the tool designed to measure it is confusing. If so, can the use be made more user-friendly, more straight forward, more obvious?
  4. Validity – Something is valid if it measures what we expect it to assess. Experts in a disciple can tell you if your assessment is valid or not. certainly, using it in a pilot with just a few folks and getting weird answers will tell you too.
  5. Reliability – This is the extent to which repeated use of a measure, or use by different persons, gives similar and, therefore, believable answers or scores. If this becomes an issue, you may need some expert guidance to deal with this challenge. High stakes? Get high quality advice.
  6. Fairness – This is am measure of whether persons being assessed have an equal opportunity to succeed on the assessment. A task group of people of different backgrounds, categories reflective of your employees, your students’ diversity, etc, is what is needed. They can tell you if they feel their constituents would have equal opportiunity. This is not as evident as you might think – exactly because you or your evaluation team may have made assumptions about which you are not aware. Check it to avoid problems.
  7. Consensus – This is simple, but not always easy. The more a range of people support it, the more likely it is good and needed. The consensus of a diverse group is important evidence that you have and plan something worth doing. THAT addresses many of the above issues – soooo, make sure your evaluation team is diverse, representative, and reaches consensus on what the plans are.