Steps 4 & 5: Data Collection Tools and Methods and Collecting Data

Although deciding on data collection tools and methods and actually collecting data are actually two different steps in the evaluation cycle, they are inextricably linked and we thus combine them as part of the tutorial.

Data are a collection of information, evidence, or facts from which you can draw conclusions .

Data can be quantitative - numeric information - or it can be qualitative - involving stories, observations, etc.
Data are important because they offer evidence for the outcomes your program is producing and allow you to confidently make the case for your programs.

The data that you want to collect will be driven from the questions you want answered (see the Theory of Change section for more information on developing evaluation questions).

There are a variety of different types of data collection methods. Some of the more common types are:

Activity Logs/Skill Sheets	Written documentation of participant's attendance, achievement or acquisition of skills	Good for what and how many questions
Document review	Review of written documents such as performance ratings, program logs, tally sheets, and other existing indicators	Good for what and how many questions
Focus groups	Moderated discussions on a particular topic or issue	Good for what, how, and why questions
Interviews	Data collection through oral conversations	Good for what and why questions
Observation	Watching people engaged in activities and recording what occurs	Good for how, what, and how many questions
Questionnaires	Written responses to clearly defined questions	Good for what and how many questions.

You can also use alternative documentation approaches including having program participants keep written, audio, or video journals of both their experiences and their thoughts, feelings, etc.

Any of the above methods can also be implemented through pre/post tests in which people are tested both before the program has been implemented and after the program has been implemented, and changes in attitudes, skills, behaviors, etc. are measured.

If possible, you should use more than one method to collect data as it enhances credibility of the data.

There are many existing data collection tools available (see Tools section below) but sometimes it is necessary to create your own data collection tool. To do this you should implement the following steps:

Finalize your evaluation questions.
Decide what information you already have and what information you will need to collect.
Based on the type of data you need, choose an appropriate data collection method.
Develop tools to collect the data.

While it is impossible to collect data without any bias (e.g., the questions you are interested in answering lead you to want to collect certain data), it is possible to minimize bias. You can do this by gathering as many perspectives as possible, using a variety of methods, checking your interpretations with the program stakeholders, and ensuring that data collectors aren't biasing responses by asking leading questions or incentivizing the respondent to answer in a particular way.

Creating Culturally Appropriate Evaluation Methods: Examples from Practice

It is important to consider the cultural context in which you are conducting your evaluation and to make sure that the data collection methods you use are sensitive to the cultural context and that you consider language and literacy barriers of some youth.

As an example, the Director of the Tohono O'odham Community Action (TOCA) in Sells, AZ explains that Native American youth and adults negatively associate surveys with a history of being researched and judged. Thus, when conducting their evaluation research they engage youth and elders in story telling and capture the changes in skills, behaviors, etc. through this more accepted method.

Once you have chosen your data collection method(s) you are ready to collect the data. The following are some guidelines for collecting data:

No matter what type of data collection tools you decide to use, you should test them with a small sample of volunteers before you implement data collection. This will allow you to determine whether the tools are easy to understand, whether the questions are clear, relevant, and unbiased, if the data collection could be completed in a reasonable amount of time, and whether there are any improvements that could be made to the data collection tool.
Establish a firm schedule for data collection.
Ensure the data collectors have both the training and the tools they need for collecting data.
Make sure data is carefully stored and that confidentiality is maintained. This does not mean that you can not report the data, but rather that you do not associate any of the data with a particular person.

Tools for Data Collection Tools and Methods and Collecting Data:

Innovation Center for Community and Youth Development - Reflect and Improve:

The Reflect and Improve tool kit is a resource for community-based organizations looking to engage youth and adults in the evaluation of community and youth development initiative. The resources in this section of the tool kit are designed to help organizations examine various methods for collecting data and, upon determining the questions they wish to answer, help them choose the most appropriate data collection method to use.

Also see http://theinnovationcenter.org

Overview of Methods to Collect Information by Carter McNamara, PhD; last revision Feb 16, 1998)

This table provides an overview of the major methods used for collecting data during evaluation.

List of Survey Methods

This list, created by Gene Shackman, Applied Sociologist and partially supported by the International Consortium for the Advancement of Academic Publication includes a list of scholarly resources for methods in evaluation and Social Resources

SPR Tools for Data Collection Methods

These tools offer guidance for collecting data using different methods including document review, interviews, and focus groups. They were created and/or adapted by Social Policy Research Associates for Innovation Center for Community and Youth Development.

Shelterforce Online
Getting at Impact: A Beginners Guide
This article "Getting at Impact: A Beginner's Guide" written by Dr. Paul Clements from Western Michigan University gives information on how to measure program impact as well as a link to information about collecting data.

Observations: An Underused Method to Collect Data
Program Evaluation Tipsheet #37 (PDF)
This Evaluation Tip Sheet created by Pennsylvania State Cooperative Extension gives information about conducting observations as an evaluation method.

A Field Guide for Designing Quantitative Instruments to Measure Program Impact
A Field Guide for Designing Quantitative Instruments to Measure Program Impact (PDF)
This guide developed by Dr. Susan Barkman offers guidance for developing quantitative survey instruments for use in evaluation.

On-line Survey Development
Online Survey Tools (PDF)
This document from NPowerNY offers guidance for nonprofits interested in using technology to create surveys. The author reviews 8 different online survey creation tools as well as offering information about what online surveys are and how to use them.

SPR Evaluation Consent Form
The Evaluation Consent Form (PDF)
This worksheet created by Social Policy Research Associates for Innovation Center for Community and Youth Development presents guidelines for developing consent forms for data collection.

Creating Valid Answer Categories
Program Evaluation Tipsheet #59 (PDF)
This evaluation tip sheet created by Pennsylvania State Cooperative Extension gives tips and examples for creating answer categories for close-ended survey questions.

Project Star: Data Collection
Common Data Collection Methods (PDF)
Collecting Performance Measurement Data (PDF)
These worksheets from Project STAR, a Corporation for National and Community Service sponsored technical assistance provider in performance measurement, offers information and tips for data collection, including information on various methods for data collection.

W.K. Kellogg Foundation Evaluation Handbook
The W.K. Kellogg Foundation Evaluation Handbook (PDF)
This document from the W.K. Kellogg Foundation is a guide for non-profit organizations interested in conducing self-evaluations. Information about data collection methods and implementation begins on page 69.

Connect with others

Talk with others in the field who are doing similar work or who have experiences to share. Share your experiences in Data Collections Tools and Methods and Collecting Data through the Practitioner Database