- There has been significant recent activity in Australian entrepreneur support activity.
- Central leadership and a responsible organisation for entrepreneur ecosystem metrics matters, contributing to what people believe about entrepreneur ecosystem data availability, relevance, and accuracy.
- While having a central coordination of data is important, metrics also need to be tailored and available at the local and sector level.
- A common standard of metrics and a common language that is accessible for a broad audience are important.
- The challenge is complex and we need to understand the needs of a diverse number of communities, but small and immediate steps can and should be taken.
These are key messages that came from a survey and panel session focused on entrepreneur ecosystem measurement.
On 5 February 2020, a panel session on entrepreneur ecosystem measurement was held at the 17th annual Australian Centre for Entrepreneurship Research Exchange in Adelaide, South Australia. Contributors to the session who helped make it possible include the sponsorship of the QUT Business School, University of South Australia Business School with particular thanks to Allan O’Connor, and South Australia’s Future Industries Exchange for Entrepreneurship. Particular thanks to
This post outlines the results of the session and related survey for those who were unable to attend and continue the conversation and momentum.
WHO: Panelists and Participants
The panelists provided perspectives from global, national and local views; industry, academic, and investment domains; and fields of data science and practical impact. Each person had a depth of related experience and an interest and passion for a shared outcome.
- Sasan Bakhtiari: Senior Economist with the federal Department of Industry. Sasan brought a national view and experience in reporting on entrepreneurial data from the Australian Bureau of Statistics integration platform (2017 and 2019).
- Sameeksha Desai: Director of Knowledge Creation and Research at the Ewing Marion Kauffman Foundation and Associate Director, Institute for Development Strategies. Samee participated through video and brought an international perspective, along with an emphasis on local impact of entrepreneurial firms reflected in her TedX talk here.
- David B. Audretsch: Distinguished Professor, Ameritech Chair of Economic Development, Director, Institute for Development Strategies, Indiana University. David is an economist and leader in entrepreneur research, having written and contributed to countless journal articles and books on the topics related to policy, entrepreneurship, and economic growth.
- Char-lee Moyle: Innovation Metrics Mid-Career Research Fellow at QUT (Queensland University of Technology). Char-lee is a leader in integration projects between public and private entrepreneur data in Australia and has a background in research and application in tourism and regional resilience.
- Geoff Thomas: Principal, Axant; Investment Committee Member, Paspalis; Non-executive director with private equity experience. Geoff provides an investment perspective and helping to develop emerging businesses.
- Judy Halliday: Director, Science, Technology and Commercialisation at Department for Innovation and Skills – South Australia. Judy is instrumental in commercialisation strategy and policy in the South Australian government, building on experience that includes capital raising, research in the medical industry, and commercial engagement at Uniquest.
The panel was joined by a room full of participants, many of them delegates for the Australian Centre for Entrepreneurship Research Exchange (ACERE). Participants included researchers in the field of entrepreneurship, policy makers for economic development, university program managers, and entrepreneur support organisations such as innovation hubs accelerators, and coworking spaces.
The collective wisdom in the room contributed to a rich conversation about entrepreneur ecosystem metrics in Australia.
CONEXT: Setting the scene – Background and Context
The workshop opened with a historical overview for context, relating to innovation ecosystem measurement in Australia.
Some of the initiatives focused specifically on measuring entrepreneur and innovation impact include:
- The National Innovation System Report: A ten-year project which included an annual PDF report from 2010 to 2017, evolving into the digital Australian Innovation System Monitor
- Data Integration Partnership for Australia (DIPA): DIPA is a cross-agency initiative started in 2017 and involves over 20 government agencies in the three-year, $130.8 million program to “maximise the use and value of the Government’s data assets.”
- Metrics review: Two concurrent metrics reviews are underway, one being the Improving Innovation Indicators review and the second the Startup Metrics Working Group.
- Surveys: Regular surveys related to entrepreneur and innovation in Australia include: the ABS Business Characteristics Survey (2014 to present), the Global Entrepreneurship Monitor (1999 globally, from 2006 in AU), Startup Muster (2014-2018), and Startup Genome (2012 globally, from 2017 in Melbourne).
- Platforms: Several digital platforms provide startup data specific to Australia, including Techboard, Decode Ecosystems, Startup Status, TechSydney powered by RamenLife, and LaunchVic’s Laneways.
These various initiatives are converging towards an integrated approach, bringing together public and private data., including projects at the QUT Business School and University of Adelaide.
SURVEY INPUT: Pre-workshop survey results
In the weeks leading up to the workshop, a survey was sent out to understand needs and perspectives of those who use and benefit from entrepreneur ecosystem metrics.
Demographics
The survey received 35 responses from regions around the world. Respondents focused on national, state, and local boundaries when considering their ecosystem.
Respondents held multiple roles in the ecosystem, with diverse roles across the ecosystem.
Consider the measurement of the impact and outcomes of the entrepreneur ecosystem in your region. How effective do you feel this is in your region?
The survey asked whether the respondent felt the the measurement of impact and outcomes of entrepreneur support in their region was effective. Half of respondents felt measurement was very or moderately effective. The response from this group to the question of “What factors contributed to your response above?” highlighted both contributing and inhibiting factors:
- What doesn’t get evaluated is the opportunity cost of inaction, the time spent on the documentation etc vs the time that could be spent on getting on with the job and making an impact
- National Accrediting body, Federal and State gov do collect impact, but it is not well defined, reported impact is a portion of population served and impact is often misreported or unreported.
- Support from local townspeople
- Great amount of engagements
- Lack of synergy between all resources.
- We just agreed two days ago about how to measure a few things … as a first test
- Focus on vanity metrics from the ecosystem rather than true traction
- Need for greater interconnectedness
The other half of respondents felt measurement was slightly effective or not effective at all. The feedback from this group included:
- Fragmented innovation data and metrics
- Providers have no interest in measuring what’s working and what’s not.
- No consistent or all-encompassing measurement of the entrepreneurship ecosystem within regions
- Embryonic in terms of mapping and related leveraging of network knowledge
- Silos, there is no collaborated efforts
- Most metrics are at the individual level but each actor uses their own metrics.
How is the entrepreneur ecosystem and its impact in your region measured at the moment?
Respondents to the online survey were asked what they knew about how the impact of the entrepreneur ecosystem was measured in their region. Open responses were coded into the method of measurement, or “How”, the metrics, or “What”, and barriers.
Over half the responses used jobs and investment as metrics. Other metrics included the number of firms or founders, activity such as number of programs, specific activities such as pitch night presentations, and generic outputs.
Many respondents relied on individual reporting from programs or entrepreneur support organisations which related to concerns over silos and uncoordinated approaches. Several indicated that no measurement was done or they did not know. Other sources included data from the Australian Bureau of Statistics, custom surveys, and publicly available databases.
Barriers raised included:
- Connecting multiple data sources takes time and specialist skills.
- Data protection rules in Europe
- Double counting
- Inconsistent metrics, frequency
- No overarching systematic/effective approach
- Not measuring effectiveness of programs
- Not measuring impact
- Old, not since 2014
- Results lag policy and program development
- Silos
- Uncoordinated
Responsible organisation; Access; Relevance and value; Accuracy
Respondents to the online survey were then asked whether there was a responsible organisation in their region, and whether they felt they had access to metrics, whether those metrics were relevant and provided value, and whether the metrics were accurate.
Half the respondents agreed or somewhat agreed that there was one or more organisations in their region responsible for entrepreneur ecosystem metrics. Half of the respondents also felt that they had access to those metrics. Forty-four percent felt metrics were relevant and provided value and twenty-two percent felt the metrics were accurate.
What was interesting was the difference in the results based on whether the respondent believed there was an organisation in their region responsible for metrics.
Of respondents who believed there was a responsible organisation, 89% were likely to feel they had access, 56% felt those metrics were relevant and provided value, and a third believed they were accuracy.
Of the respondents who felt there was no organisation responsible for metrics in their region, only 11% felt they had access, a third believed the metrics were relevant and provided value, and 11% felt they were accurate.
Think ahead to two to three years from now. What would measurement of your entrepreneur ecosystem and its impact in your region look like if it was the best that it could be?
Respondents were asked about what would change in regards to measurement in their region. What is the best that it could look like? Open responses were coded into main themes. Many respondents referenced specific metrics that would be measured. Others wanted public access or some visible interface what was also related to a means for promotion, advocacy, and case studies. A desire for a central management of data and accountability was also prominent.
Other characteristics of a positive future included: regional data, standardised definitions, trend analysis, alignment with regional strategy, corporate investment, cross-border reporting and benchmarking, and using the metrics to improve program quality.
What are three things you believe need to happen to improve the measurement of your entrepreneur ecosystem and its impact in your region?
Respondents were asked what needed to happen to realise the future of ecosystem measurement. The open responses were coded into four broad categories: Capability, Funding, Method, and Structure.
Capability feedback involved improving education relating to measurement, acknowledging the specialist skill set required.
- Measurement education (1)
Funding related to how measurement is funding, with a need identified for dedicated support for non-metro areas with less resources as well as dedicated funding for measurement overall.
- Dedicated regional support for measurement (1)
- Increased government funding into measurement (3)
Method responses related to how measurement is done. Examples include integrating with corporate reporting, conducting a national survey, and ensuring there is a low impact on businesses for reporting.
- Change corporate reporting (1)
- National survey (1)
- Low impact on businesses from data collection (1)
Metrics responses related to what was being measured, including measuring longitudinally over time, incorporating social outcomes and impact measurement, developing KPIs for the ecosystem overall, integrating activity and outcome metrics, defining measures in general, and standardising measures for improved benchmarking over time and between regions.
- Longitudinal analysis (1)
- Impact measurement (1)
- Ecosystem KPIs (1)
- Measure activity (2)
- Define measures (2)
- Standardised definitions / clarity for benchmarking (5)
Structure themed responses were prominent in the feedback and involved the management approach, coordination, and institutions involved in measurement. Responses included a need for a collective view, global connections, reduced government and increased ecosystem involvement, local ownership of data,increasing the number of actors such as investors, a central management approach, using the metrics as an input into strategy, clarifying the boundaries of the ecosystem, and coordinating efforts.
- Collective view (1)
- Global connections (1)
- Reduced government / increased ecosystem leadership (1)
- Local ownership of data (1)
- Increase size of ecosystem (eg., Number of investors) (1)
- Central management (2)
- Input into strategy / action from metrics (2)
- Clarity of geographic boundaries / measure at the ecosystem level (2)
- Coordinated efforts (3)
PANEL: Reflections
Image credit: Siobhan Curran
The panel conversation confirmed much of the sentiment from the survey. The panel conversation was structured similar to the survey in terms of current state, future state, and next steps. Highlights from the panel conversation are considered in terms of leadership, and metrics and measurement.
Leadership
A need for leadership was identified, with comments about a lack of consensus on approach between states. Comments were made about learning from some of the Smart Specialisation strategies in other regions around the world and a need to work with others. The role of government was highlighted in terms of providing direction, giving confidence, and addressing underrepresented markets such as indigenous entrepreneurs. A need for a national approach to serve all of Australia was highlighted.
Method and Management
In regards to the method and approach to obtaining metrics, there was an acknowledgement that long cycle dynamics were not working and that people needed real time results. There has been some good progress at the national level, but there remains a gap at localised levels. Char-lee highlighted work that teams were doing with integration into government data, including the development of Entrepreneur Quality Index Modelling and program analysis.
Samee mentioned the need to keep different perspectives in mind and that we need to understand the needs of different stakeholders, from entrepreneurs to governments. But it was also raised that we need to balance the custom approach with standardisation and centralisation for efficient outcomes. There are benefits in a national survey and mapping deal flow across states, but also a need to adapt the data for localised requirements. This requires an approach that is collated and collaborative, accommodating the different needs of the diverse ecosystems.
The need was raised for metrics by industry and using data to explore how many firms may stop operating in different sectors. This involves examining different levels including the individual, the firm, and the portfolio. We need to analyse the way firms grow and where founders are formed. This can involve identifying “tent-pole” firms that are successful and working backwards to map their trajectory.
The panel acknowledged that it can be tough to know what to measure including determining direct and indirect factors. There can be opportunities in exploring alternative data sets including cross referencing social media with other data sets.
The conversation highlighted that the challenge of measuring the impact of entrepreneurial ecosystems is complex and global. It is also necessary to understand the impact of investment and decisions by policymakers, for support organisations to adapt services and programs, and for entrepreneurs to better navigate a dynamic commercial landscape.
This scope of what is needed requires a risk appetite – for those measuring impact of new firms to think of themselves as a startup. The question was raised:
“What can we measure now that will connect with needs of those who use the information?”
Panel questions
The survey and panel session highlighted the need and opportunity for collaboration and coordination of entrepreneur ecosystem measurement at a national level that is locally accessible and relevant. The follow-up questions from the audience reinforced the challenges.
One question to the panel that focused on the definition of entrepreneur reinforced the need for some form of standards or at least a selection of definitions that can be used for a common language. There is currently a project in Australia developing a standardised approach.
Another question from a manager of an innovation hub about application of the conversation for entrepreneurs highlighted the need to adapt measures for multiple audiences. David’s response raised a key point, however. Measuring entrepreneur activity is a lag measure to factors in the overall system. If the focus is only on entrepreneurs or even a form of organisation that supports entrepreneurs, we can overlook other factors that contribute to overall success in a region. We can also miss seeing negative impacts in other aspects of the system if we only focus on a limited view of entrepreneurial performance.
The questions and discussion could have continued for some time given the calibre of those in the room and the scope of the challenge. The session did achieve the goal of raising the profile of the challenge. The outcomes also contributed to the national progress towards a collective approach to measuring entrepreneur ecosystems.
Next steps
Entrepreneurial ecosystems are complex. Entrepreneurs navigate a constantly changing environment with multiple diverse stakeholders each with different priorities. Measuring the impact of these ecosystems is a challenging but worthwhile effort.
The survey and panel session highlighted a need for a central approach that is locally relevant, to understand the complexity while taking immediate action, for a common language that can be adapted to and understood by local communities, and collaboration across Australia that aligns with global efforts.
Fortunately, there is progress in Australia along this path. The national metrics review efforts are producing practical outcomes that are expected to be delivered later this year. Presentations at the ACERE conference shared about coordination between states and universities to develop common approaches to metrics. There is increased global awareness of aligned efforts, including work by Startup Commons in Europe and the Kauffman Foundation in the US.
Measurement of entrepreneurial ecosystems is a focused and growing discipline. The aim of this report is to share outcomes and attract like-minded people working on or interested in this area. For more information and to connect if you are involved in measuring outcomes and impact of entrepreneurs in your region or are interested in the efforts in this area, please connect as well as head over to the Australian Centre for Entrepreneurship Research.