At the University of Wisconsin – Milwaukee (UWM) Learning Technology Center (LTC), the development and facilitation of all innovations moves through a pilot cycle or lifecycle developed at the UWMLTC. Innovations can include technologies, processes, and/or behaviors that have the capacity to positively influence pedagogical practices. The cycle includes identifying or assessing a pedagogical need or problem and then finding innovative solutions for improving instructional practices to overcome pedagogical challenges. This identification then translates into hypotheses or research questions that guide our exploration and study. Once the need and potential innovation have been identified, UWMLTC develops a justification for the initiative and research study. The main objectives of the UWMLTC initiatives are to identify, implement, evaluate, disseminate, diffuse, and improve the quality of solutions to increase the effectiveness of teaching and learning. These initiatives focus on developing rigorous research agendas pertaining to related phenomena in order to further explore the field as well as expand UWM’s reputation of excellence. Idea exploration and generation of new initiatives are tightly-linked to the need and justification for the pilot and research. It can be a lengthy and time-consuming endeavor during the initial planning stage.
The UWMLTC examines the landscape on campus as well as regional, national, and global trends to determine whether the initiative should move forward to guide educational goals. The goal of each initiative is to gather evidence to support the proposal and align the initiative with unit and institutional strategy. First, the UWMLTC conducts an environmental scan of higher educational institutions, educational organizations (e.g., New Media Consortium, EDUCAUSE Learning Initiatives, Sloan Consortium, PEW), and media outlets (e.g., Chronicle, Inside Higher Ed, eCampus News) that have explored or have documented practices and/or uses of the innovation or have highlighted further need for study. Many times these environmental scans are done on a continuing basis through the consumption of digital materials (e.g., articles), attendance at conference sessions, and networking with key organizations and leaders in the field.
In addition to gathering evidence of national, international and societal trends, the UWMLTC continually monitors the needs and demands of current faculty and students. The UWMLTC pays particular attention to the needs and desires of its faculty, staff, and students before implementing any research project. For example, the UWMLTC faculty support survey is one way in which we better understand the needs and wants of academic teaching staff and faculty at the university (See Appendix). National surveys of student learning needs and wants are also taken into account when assessing research needs (e.g., PEW, EDUCAUSE). Further, we keep a digital archive of instructor requests made to our email list so we can assess requests and interests by instructors. Thus, the commencement and completion of the project will be based upon careful examination and identification of the needs of the faculty and students. Initiatives move forward if there is an indication that instructors have interest in piloting an innovation (including attending faculty development, implementing the solution, and participating in the corresponding research and dissemination) and student readiness or benefits are documented. This information also helps the UWMLTC prioritize their resources and efforts.
Along with considering the perspectives of faculty and students, the UWMLTC looks for data, evidence, and theoretical propositions to indicate or support success. Staff at the center collect, analyze and synthesize existing literature and information in the field through the compilation of annotated bibliographies that take into account literature related to the problems and potential solutions identified. The current literature, research, and practices in education are just one of the considerations in planning an initiative. Another main component of planning is to align the initiative with the UWM mission and goals and LTC strategic goals. The UWM Chancellor’s new mission focused on research and teaching as well as the integration of the Provost’s Digital Future initiative have specified several strategic goals and tasks to be undertaken to meet those goals. The UWMLTC identifies strategic goals on an annual basis, and the LTC staff and resources are taken into account when deciding on whether to pilot a new initiative.
Once the UWMLTC develops justification for new pilot initiatives of emerging innovations through an examination of the needs of the campus community, university goals, national and international trends, and UWM resources, the UWMLTC manages the project throughout the lifecycle, from idea generation to diffusion to quality improvement. The UWMLTC determines the scope, timeline, launch date, deliverables, communication plan, project management, staff roles, and related responsibilities. Then, the UWMLTC considers readiness when determining initial resource investment (financial, instructor time and efforts, instructional support, student support, infrastructure, licensing), sustainability and scalability, and often a business model to launch and sustain the initiative. Specifically, the UWMLTC examines technology needs (e.g., delivery platform, identity access management, registration, enrollment, storage, new technology for instruction and/or support) and policy needs (e.g., access, financial aid). A lack of readiness in any of these areas may prolong implementation; however, once the initiatives are executed, there is a high rate of engagement and success.
The implementation phase is a very complex part of the pilot process. First, the UWMLTC examines which stakeholders will be involved in the implementation (faculty, students, instructional support, classroom support, help desk, tutoring, libraries, desktop, unit support, and other organizations). Second, if needed, the UWMLTC will initiate a vendor selection process, including evaluation of technologies and vendors for new necessary technologies. This includes a scan of the market, campus demonstrations with stakeholders, feedback gathering from stakeholders, implementation of technology evaluation criteria, and selection of a vendor (see Appendix). Purchasing policies and the need for a vendor RFP or statement of work may be needed at this point, which can vastly prolong an initiative. Third, the UWMLTC develops pedagogical models informed by the literature and survey data that suggest the greatest impact on student outcomes and then shares these pedagogical models in faculty development programs, which provide pedagogical and technically effective practices for the innovation being piloted. Fourth, the UWMLTC develops a faculty selection process that adheres to a competitive request for proposals (RFP) to participate in the initiative. A rubric is used to determine whether the instructor and/or course are an appropriate fit for the project. Instructors who are selected are required to attend the faculty development program, implement identified pedagogical models, attend monthly pilot focus groups, participate in data collection efforts, and assist in dissemination of findings. Faculty incentives are provided (usually a stipend) to supplement the efforts of the selected faculty.
After instructors have been selected and have completed a faculty development program highlighting effective practices and pedagogical models for use, the evaluation phase begins. The UWMLTC may utilize both quantitative and qualitative methodologies in evaluation of the project outcomes. Qualitatively, the faculty and staff involved develop a set of criteria to assess the usefulness of the innovation. Various debriefing sessions and focus groups are held during and after the initiative in order to gather faculty reports on the innovation for analysis. Also, a thematic analysis of narratives is conducted using individual comments and stories gathered during one-on-one training appointments and via email correspondence from faculty queries, comments, issues and problems during the project. Further, the faculty members’ perceived effectiveness of the application and digital content across disciplines are examined.
Once the project has ended, faculty are required to complete an instructor-based survey and provide an incentive to students to complete a survey about their experiences. The UWMLTC usually gathers anonymous student evaluations by survey, which contains a series of questions that ask about students’ attitudes towards the use of the innovation in the classroom as well as their evaluation of the course itself. In addition, surveys to examine students’ satisfaction, perceived learning, perceived performance, and perceived interactivity ( including engagement) are administered. Also, retention and performance data can be mined from a data warehouse. Various descriptive and multivariate analyses are completed to produce quantifiable results of the effectiveness of the innovative technology related to student satisfaction and perceived success. Once the research project is complete, the UWMLTC disseminates the results through presentations and publications on campus and beyond. Most importantly, conclusions are drawn on the efficacy of the innovation and the support to diffuse the innovation across campus.
The UWMLTC prides itself on staying abreast of technological and pedagogical developments in teaching and learning and is founded upon principles of rigorous research practices and dissemination to appropriate audiences. This practice focuses on the described piloting of emerging and innovative technologies and pedagogical solutions, including conducting research on the phenomenon under study and evaluating the impact of the innovation. With a broad research program and extensive experience with pilot projects, the UWMLTC staff is equipped and experienced in constructing, administering, and sharing valid and reliable research instruments and tools for analysis. The UWMLTC endeavors to study emerging technologies and innovative practices in pilot phases, including evaluating and researching the impact of those technologies on teaching and learning to inform our decisions and best practices at UWM. The research conducted within the UWMLTC has led to numerous accepted and invited presentations and publications to state, regional, national and international audiences. These efforts support institutional learning at UWM and have resulted in a national and international reputation for the UWMLTC, which has been well-documented by popular news and media outlets (e.g., The Chronicle of Higher Education).
The UWMLTC looks forward to continuing our research efforts through support for our Digital Future Evidence of Education Effectiveness proposal by sharing our research findings and effective practices to improve teaching at UWM through the Virtual Teaching Commons (also supported by a grant from the Digital Futures). Further, the UWMLTC looks forward to partnering with the UWM Libraries in the development of a new Faculty Commons in the East Wing to support faculty in their efforts to teach and research with technology.