Return to site

Assessment

Recap of Webinar 3 by Michal Gilon-Yanai

· Metrics,Assessment,Self Efficacy

Our first speaker was Prof. Steven Gedeon from Ryerson University in Canada. Ryerson has a large, AACSB accredited business school, which offers and a 4-year undergraduate entrepreneurship program. The school’s faculty has put much thought into developing a robust assessment program to measure the program’s impact.

Steve stressed the importance of developing assessment tools with specific and explicit goals and success criteria in mind. While different stakeholders at the university have different goals which drive them to view success differently, and consensus regarding the actual definition of entrepreneurship is hard to reach, it is crucial to clearly define the goals of the program, and accordingly, the goals of assessment. The primary goal of Ryerson’s program was defined as “student transformation”, which was found to be a common goal to most of the stakeholders.

The process followed by Ryerson in designing both the program and the assessment tools was to 1) define the specific learning goals under “student transformation, 2) embed these goals into the curriculum, as well as into graded course assignments, peer reviews, and self-assessment questionnaires, and 3) analyze and learn from the results and other feedback to drive continuous improvement.

To identify the specific goals, student transformation was broken down into three parts: 1) knowledge, 2) skills and competencies, and 3) attitudes, values and beliefs. At Ryerson, both curriculum and assessment are designed to comply with AACSB accreditation. According to AACSB, in designing the assessment program, an “assurances of learning” approach is required for skills and competencies. AACSB defines a specific list of desired skills, which include communication skills, critical thinking and teamwork. Interestingly, ‘Knowledge’ does not draw any special attention, since it is believed that universities typically over-focus on tests and quizzes to measure knowledge anyway. The “heart attributes” (attitudes, values and beliefs) are not addressed by AACSB at all, but Ryerson does measure them. These include core self-evaluation (self-esteem), entrepreneurial self-efficacy, and entrepreneurial intent. In measuring attitudes, the main principles that Steve highlights are:

  • Using a model (such as the “Theory of reasoned action” (Fishbein) and the “Theory of planned behavior” (Ajzen)
  • Using a validated assessment scale (e.g. ESE, ASTEE)
  • Measuring longitudinally (measuring students as they progress through the curriculum)
  • Applying a multi-method approach (faculty input, self- perception and peer-review) to overcome the problem of student over confidence
  • Whenever possible, using control groups

Steve provided many useful examples for the models that influenced the design of the Ryerson program, some of which he developed himself as part of his research, as well as the specific skills measured, and how (which scales, which questions are used etc.).

Our next speaker was Kare Moberg, from the Copenhagen Business School (CBS) and the Danish Foundation for Entrepreneurship. Kare introduced us to ASTEE, a framework developed in Europe over the past few years for measuring the impact of entrepreneurship education, from primary school to university.

The research project that led to the development of ASTEE used the pre-existing Entrepreneurship Self Efficacy (ESE) scales as a starting point. ESE is an established set of scales developed in the literature over the past three decades to address the challenge in measuring non-cognitive skills that are considered essential to entrepreneurship. However, Kare pointed out, these scales typically use business and entrepreneurship jargon that is not understood by all students, especially those majoring in other disciplines which often serve as control groups in program assessments. To mitigate this problem, Kare led a research project in which he developed a similar scale with more neutral language that would allow measuring the same concepts, without the specific ‘professional’ vocabulary. The new scale was validated in a large number of programs, and with control groups. It proved to be reliable, and the concepts measured by it where shown to be understood in the same way by different groups over time.

However, there was still one major limitation - the advanced statistical analysis required to assess the effects of different curricular designs made the scale complicated for educators to use. This was one of the reasons that motivated the development of ASTEE. The ASTEE project is led by a consortium of seven European partners. The group developed an assessment tool which covers knowledge, skills, mindsets and attitudes, intentions and behavior, and can be used by educators teaching at all levels. The ASTEE team also built and continues to develop a student database, to allow educators to compare their students to others in similar age groups and programs. Finally, to facilitate assessment even further, ASTEE is currently building an app that will automate the evaluation process.

Alex DeNoble from San Diego State University, president of USASBE was our next speaker. He is among the developers of ESE, and in his research he has demonstrated the importance of using specific scales for specific types of entrepreneurship. He has developed several specific scales, such as a scale for assessing Corporate Entrepreneurial & Innovation Leadership qualities. In his talk, Alex discussed the importance of assessment, despite common complaints raised by faculty – that it is too difficult to measure, too time consuming, etc. He stressed the importance of measuring ‘the right thing’, focusing on the competencies that students should acquire, and pointed to some of the common but often mistaken approaches to tracking the learning process, such as the number of students participating and winning in business plan competitions and/or launching new ventures upon graduation. He also stressed the importance of ‘closing the loop’ in assessment – informing faculty and taking corrective action where needed.

Dane Stangler of the Kauffman Foundation shared a few closing comments, referring to the earlier presentations. He stressed the importance of using control groups in assessment, and the focus on learning goals as the desired outcome. He too pointed to the damaging effects of ‘vanity metrics’, and proposed some additional outcomes that would be interesting to measure. For example, whether entrepreneurship programs serve as a “gateway drug” that introduces students to this career path and draws them to it, or whether reverse selection occurs, where student entrepreneurship goes down once people see what it involves.

Dane pointed out the difficulty of determining causality in studying the entrepreneurial activity of students graduating from entrepreneurship programs, and the need for longitudinal datasets that will follow people as they move from the university, through the ecosystem and different jobs, and sometimes only then into entrepreneurship careers.

Finally, Dane raised the question, whether entrepreneurship has become the general education of our time, arguably preparing students for any future job. If this is the case, then the ROI measured by many academic institutes, using economic development metrics such as the number of startups founded by graduates, may be a reason to worry about the future of these programsh text here.

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OKSubscriptions powered by Strikingly