Ako prebieha hodnotenie SME Instrument? Pozrite si podklady pre hodnotiteľov, kritériá a bodovanie

Návrhy projektov programu Horizont 2020 posudzujú nezávislí experti (hodnotitelia), ktorí hodnotia jednotlivé aspekty návrhu. Vďaka otvorenosti schémy sú všetky informácie verejné a na tejto stránke môžete nahliadnuť do zákulisia hodnotenia.

zo-zakulisia-hodnotenia-0

Samotné hodnotenie prebieha podľa štandardizovaných procesov, ktoré nájdete popísané tu:

Európska komisia zverejnila zoznamy hodnotiteľov pre projekty SME Instrument, ktorí boli zapojení do hodnotenia v rokoch 2015 a 2016. Pre každý rok je to vyše tisíc expertov. Pozrite sa kto hodnotil projekty SME Instrument za roky 2015 a 2016? V zozname je aj 23 Slovákov.

Od roku 2018 boli pre návrhy fázy 2 SME Instrument zavedené pohovory pre úspešných uchádzačov. Pozrite si ako prebiehajú pohovory a kto sú experti, ktorí vedú interviá.

Tiež bola zavedená nová povinnosť pre hodnotiteľov, ktorá súvisí s overovaním skutočností uvedených v návrhu (fact checking). Ide napríklad o avizovanie podozrivých skutočností (tzv. red flag), ako sú napr. e-maily, ktoré nie sú v doméne spoločnosti, bankový účet v inom meste/krajine ako je ústredie spoločnosti, chýbajúce základné informácie na webových stránkach či nesprávna/neexistujúca adresa. Inštrukcie k fact checking (PDF).

Pre hodnotiteľov sú pripravené podporné informácie vo forme FAQ či prezentácií, ktoré prinášajú podrobnosti z hodnotiaceho procesu. Pozrite si často kladené otázky ohľadne:

  • hodnotenia (správa projektov, bodovanie, subdodávky, konflikt záujmov)
  • zazmluvnenia a platieb (zmluvy s expertmi, platby)

Stiahnite si dokument: The Frequently asked Questions for the experts of the SME Instrument Programme (PDF)

Pozrite si prezentácie, ktoré pripravila Európska komisia pre hodnotiteľov, ktoré sumarizujú (po kliknutí na odkaz otvoríte príslušné PDF):

Pre riadne a korektné fungovanie najväčšieho európskeho programu pre výskum a vývoj – Horizont 2020 stále potrebuje Európska komisia nezávislých expertov, ktorí by hodnotili projekty, posudzovali ich návrhy, monitorovali implementáciu programov a dodržiavanie pravidiel.

Ďalšie a podrobnejšie informácie nájdete v časti FAQ Participant Portal-u (Support):  vyberte kategóriu “Expert evaluators, reviewers, monitors”. Tam tiež nájdete ďalšie rady a odpovede ako sa prihlásiť za experta a čo to v detailoch obnáša.

Pozrite si tiež:

POPIS HODNOTENIA (prevzaté z Participant portal):

Evaluation:

Award Criteria

Proposals are evaluated by experts on the basis of three award criteria: ‘impact’, ‘excellence’, and ‘quality and efficiency of implementation’. The aspects examined under each criterion are described in the table below.

IMPACT
50% WEIGHTING
Convincing description of substantial demand (including willingness to pay) for the innovation; demand generated by new ideas, with the potential to create new markets, is particularly sought after.

Total market size envisaged.
Convincing description of targeted users or customers of the innovation, how their needs have been addressed, why the users or customers identified will want to use or buy the product, service or business model, including compared to what is currently available if anything at all.
Phase 1 (only): Good understanding of need for a realistic and relevant analysis of market conditions, total potential market size and growth-rate, competitors and competitive offerings, key stakeholders, clear identification of opportunities for market introduction; potential for market creation is particularly sought after.

Phase 2 (only): Realistic and relevant analysis of market conditions and growth-rate, competitors and competitive offerings, key stakeholders, clear identification of opportunities for market introduction, market creation or disruption (e.g. via new value-chains).
Realistic and relevant description of how the innovation has the potential to scale-up the applicant company (or companies). This should be underpinned by a convincing business plan with a clear timeline, and complemented, where possible, by a track-record that includes financial data.
Alignment of proposal with overall strategy of applicant SME (or SMEs) and commitment of the team behind them. Demonstration of need for commercial and management experience, including understanding of the financial and organisational requirements for commercial exploitation and scaling up (and – Phase 2 only) as well as key third parties needed.
Phase1 (only): Outline of initial commercialisation plan and how this will be developed further (in-house development, licensing strategy, etc.).

Phase 2 (only): Realistic and relevant strategic plan for commercialisation, including approximate time-to-market or deployment. Activities to be undertaken after the project.

The ‘commercial strategy’ aspect is particularly examined in Step 2 of the evaluation of Phase 2 proposals.
European/global dimension of innovation with respect to both commercialisation and assessment of competitors and competitive offerings.
Phase 1 (only): Realistic and relevant description of knowledge protection status and strategy, need for ‘freedom to operate’ (i.e., possibility of commercial exploitation), and current IPR situation. Where relevant, description of potential regulatory requirements.

Phase 2 (only): Evidence of or realistic measures to ensure ‘freedom to operate’ (i.e., possibility of commercial exploitation), convincing knowledge-protection strategy, including current IPR filing status, IPR ownership and licensing issues. Regulatory and/or standards requirements addressed.
Taken as whole, to what extent the above elements are coherent and plausible.
EXCELLENCE
25% WEIGHTING
High-risk/high-potential innovation idea that has something that nobody else has. It should bebetter and/or significantly different to any alternative. Game-changing ideas or breakthrough innovations are particularly sought after.

Its high degree of novelty comes with a high chance of either success or failure.
Realistic description of current stage of development (Phase 2 only: TRL 6, or something analogous for non-technological innovations), and clear outline of steps planned to take this innovation to market.
Highly innovative solution that goes beyond the state of the art in comparison with existing or competing solutions, including on the basis of costs, ease of use and other relevant features as well as issues related to climate change or the environment, the gender dimension, any other benefits for society, or (Phase 1 only) includes plans for obtaining this information.
Very good understanding of both risks and opportunities related to successful market introduction of the innovation from both technical and commercial points of view or (Phase 1 only) includes convincing plans for obtaining this information.

Phase 2 only: Documentation on the technological, practical and economic feasibility of the innovation.

The ‘feasibility’ aspect is particularly examined in Step 2 of the evaluation of Phase 2 proposals.
Phase 1 (only): Objectives for the feasibility study and the approach and activities to be developed are consistent with the expected impact of the innovation.

Phase 2 (only): Objectives for the innovation proposal as well as the approach and activities to be developed are consistent with the expected impact (i.e. commercialisation or deployment resulting in company growth). Appropriate definition provided of specifications for outcome of project and criteria for success.
Taken as whole, to what extent the above elements are coherent and plausible.
QUALITY AND EFFICIENCY OF IMPLEMENTATION
25% WEIGHTING
Technical/business experience of the team, including management capacity to lead a growing team

Only Phase 1: If relevant, the proposal includes a plan to acquire missing competences.

Only Phase 2: If relevant, the proposal includes a plan to acquire missing competences, namely through partnerships and/or subcontracting*, and explains why and how they are selected (subcontractors must be selected using ‘best value-for-money’ principles).

The ‘team’ aspect is particularly examined in Step 2 of the evaluation of Phase 2 proposals.
Availability of resources required (personnel, facilities, networks, etc.) to develop project activities in the most suitable conditions.

Where relevant, complementarity of partners in a consortium.

Only Phase 2:

Where relevant, realistic description of how key stakeholders / partners / subcontractors could be involved* (subcontractors must be selected using ‘best value-for-money’ principles).

Where relevant, the estimated budget and the procedure planned for selecting the subcontractors are appropriate*.
Realistic timeframe and comprehensive description of implementation (work-packages, major deliverables and milestones, risk management) taking the company’s or applicant’s innovation ambitions and objectives into account.
*Subcontracting is acceptable to the extent required for the implementation of the proposed activities. Subcontracting may be an essential part of the implementation of the project, but should not be a disproportionate part of the total estimated eligible costs. Subcontractors must be selected using ‘best value-for-money’ principles.
Taken as whole, to what extent the above elements are coherent and plausible.

After each Phase 1 cut-off

  • Proposals are evaluated in one step.
  • A proposal is evaluated remotely by a number of evaluators with a mixture of technology, industry sector, business and finance expertise.
  • Each evaluator scores each of the three award criteria from 0 to 5. Scores with a resolution of one decimal place may be given.
  • The quality threshold of each criterion is 4 out of 5. The overall quality threshold, applying to the weighted sum of the three individual scores, is 13 out of 15.
  • The consensus score at the level of the three evaluation criteria is the median of the scores given by each evaluator. The overall consensus score is the weighted sum of these separate scores. Proposals that have passed all thresholds are ranked in the order of their final score.

If necessary, a panel review is organised remotely.

After each Phase 2 cut-off

Applications are evaluated in two steps.

Step 1: remote evaluation

  • A proposal is evaluated remotely by a number of evaluators with a mixture of technology, industry sector, business and finance expertise.
  • Each evaluator scores each of the three award criteria from 0 to 5. Scores with a resolution of one decimal place may be given.
  • The quality threshold of each criterion is 4 out of 5. The overall quality threshold, applying to the weighted sum of the three individual scores, is 13 out of 15.
  • The consensus score of a proposal at the level of the three evaluation criteria is the median of the scores given by each evaluator. The overall consensus score is the weighted sum of these scores.
  • Proposals that pass all quality thresholds will be considered for step 2.

Step 2: face-to-face interview

  • Starting with the highest-scoring proposal and in descending, sequential order, proposals are passed to Step 2 until, as a batch, either the total amount of EU funding requested is as close as possible to twice the budget available, or all proposals eligible for funding have been accounted for. The actual threshold to pass to Step 2 will therefore be dynamic and depend on the volume of proposals received that pass all quality thresholds.
  • Each applicant whose proposal has passed to Step 2 is invited to a face-to-face interview in Brussels.
  • Only staff of applicants can represent them. Representation by third parties is forbidden.
  • The interview is conducted by evaluators with a mixture of technology, industry sector, business and finance expertise.
  • During the interview, the applicant is posed questions designed to clarify aspects of the proposal evaluated in Step 1, in particular those indicated above under ‘award criteria’.
  • In Step 2, proposals will receive, in addition to the score in Step 1, an ‘A’ mark or a ‘B’ mark from the final panel review.
  • Only proposals that have passed all quality thresholds and receive an ‘A’ mark are proposed for funding.

For both Phase 1 and Phase 2

  • During the electronic proposal submission process, you can provide up to three names of persons that should not act as an evaluator of your proposal, for commercial or other reasons.
  • To set a priority order for proposals given the same consensus score in Phase 1, the following method is used:
  • Proposals are first prioritised according to scores given for the award criterion ‘impact’.
  • Where those scores are equal, priority is then determined using scores for the award criterion ‘excellence’.

If necessary, a further prioritisation is based on the degree of gender balance among the personnel named in the proposal as primarily responsible for carrying out the project.

Communication to applicants after evaluation procedure

Phase 1

For each proposal, applicants receive an evaluation summary report with the scores obtained and a qualitative assessment with respect to each of the aspects considered under each of the three award criteria.

Phase 2

Each applicant invited to an interview in Step 2 receives an invitation at the end of Step 1.

For each proposal, applicants receive an evaluation summary report with the scores obtained and a qualitative assessment with respect to each of the aspects considered under each of the three award criteria (Step 1 of the evaluation). For proposals that have passed to Step 2, the report will contain an A or B mark and an additional qualitative assessment.

Phase 1 and Phase 2 applicants meeting all quality thresholds but not receiving funding will receive a Seal of Excellence.

Indicative timetable for evaluation and grant agreement signature

  • Information about the outcome of the evaluation: maximum 2 months after the corresponding cut-off date set out above for Phase 1, and maximum 4 months after the corresponding cut-off date set out above for Phase 2.
  • Indicative date for the signing of grant agreements: maximum 3 months from the final date for submission for Phase 1 and maximum 6 months from the final date for submission for Phase 2.