Understanding the ATE Annual Survey, An Interview with EvaluATE’s Lori Wingate and Arlen Gullickson

AThis month, ATE Central asked Lori Wingate and Arlen Gullickson of EvaluATE to share a bit about the history, practical purposes, and impacts of the ATE Annual Survey. The 2016 Survey will be sent out to PIs later in February and we thought it might benefit all of us in the ATE community to understand a little bit more about the survey and its roots.

Can you tell me a bit about the history of EvaluATE's Annual ATE Survey? How did it get started and who participates?

The annual survey of ATE principal investigators began in 2000 as part of an overall program evaluation conducted by The Evaluation Center at Western Michigan University (WMU). The program evaluation also included site visits, case studies, and more than 60 published reports on the status and effects of the ATE program. The reports crossed a broad array of ATE activities and impacts addressing topics such as recruitment and retention and business and industry’s perspectives of the value added by the ATE program. In addition to book and journal publications, many of these reports are currently available at http://www.evalu-ate.org/ate-program-evaluation-research/.

Initially the survey findings were intended primarily for use by NSF’s Education and Human Resource directorate and ATE program officers in planning program activities, assessing program progress, and preparing annual testimony and reports. In the survey’s early years, the survey findings were timed to be available for use by NSF staff in preparing its annual GPRA report to Congress. The program evaluation ended in 2006, but we (the WMU Evaluation Center under the auspices of EvaluATE) have continued to conduct the survey as a means to monitor the program and have focused on service to ATE grantees. Every ATE PI receives the survey, and more than 90 percent participate each year.

Are there different uses/purposes of the survey results for prospective vs current ATE grantees? What is the best way for members new to the ATE Community to use the survey results and does this differ from the uses of veteran projects and centers?

The survey results may be used by prospective grantees to find out about program activities and to determine how a proposal might address a need in the program. For example, we know from last year’s survey results that less than 20 percent of students in ATE programs in the areas of energy and manufacturing are women—evidence of a need for efforts to recruit and retain women in these areas (see the Women in ATE data snapshot at http://www.evalu-ate.org/annual_survey/).

The survey data may be used by current grantees for benchmarking and comparing their results with the program overall. To illustrate this use, we created a short video that demonstrates use of survey data in combination with other national data (see related newsletter article with link to video at http://www.evalu-ate.org/newsletter/2015-winter-rqra/).

How has the survey evolved since its first iteration?

The survey started before there were sophisticated online survey services available commercially, so the Web-based survey was designed and programmed by computer science students under the direction of a WMU computer science professor. Later we employed a Web programmer. Commercial services eventually caught up with what we could do with our custom system, so we switched to Qualtrics as our survey platform in 2015.

Perhaps the greatest difference is that the survey initially played a major role in the external evaluation of the ATE program and served NSF’s needs to determine the program’s quality and value to technician education. Now the survey provides an annual update (status) of the program that projects and other interested parties use to gain understanding of what the program does and is accomplishing.

Also, the original survey was much, much longer!

What are the most interesting findings from last year's results? Do you anticipate any trends with the upcoming survey?

There has been considerable expansion of the ATE program into secondary school settings. Two-thirds of the ATE-supported programs developed or modified in 2014 were for secondary school students, compared with 47 percent in 2010.

In addition to the complete 2015 ATE Annual Survey Report, your site offers Data Snapshots, Survey Fact Sheets, and graphics. Are these free for readers to use and integrate into their own projects? Have you seen any really creative/helpful uses?

We have made the graphics available free for use as stand-alone files in response to a suggestion by a panel of ATE PIs that we met with at the 2015 HI-TEC conference. These graphics may be downloaded and used without contacting us for permission. While that assists those who want to use the materials, we cannot track actual use. We would welcome feedback from users both to know how these materials are used and their value to those who use them.

What are the Data Snapshots and what are the best ways to use them?

The data snapshots provide a more in-depth, yet succinct look at certain variables from the survey. Because of NSF’s priorities around engaging women and underrepresented minorities in STEM, we always create snapshots that show how well represented these groups are in the ATE program and by discipline within the program. These snapshots clearly show that while the program has done well in terms of engaging minority students, involving women has been more of a persistent challenge (as it has been throughout STEM education, not just ATE).

What's the most common question people ask you and your team about the survey?

The common question used to be, “can’t you just get the information from our annual reports to NSF?” This is among other questions we answer in our Survey FAQ document, and the answer is that although there is some overlap between the information required for the survey and ATE project/center annual reports to NSF, our survey is much more specific about ATE activities and outcomes. Moreover, there is no way to aggregate the largely narrative information in annual reports into a coherent summary report about the overall ATE program.

We also used to get many queries about the period about which we ask for information. Different grants have different start and end dates, and different institutions have different fiscal years. Many years ago, it was decided that the survey answers should be about the prior calendar year. This was problematic for many people because the calendar year often spanned two different fiscal years or grant periods. As a remedy to this persistent problem, this year’s survey asks respondents to base their answers on their most recently completed fiscal year. The most important things is for grantee responses to represent a full year of grant activity.