Elaborating and Empirical Validation of the STI Monitoring and Evaluation Framework for Iran

Document Type : Research paper

Author

Assistant Professor of Science & Technology Policy, National Research Institute for Science Policy, Tehran, Iran

Abstract

This research was conducted to empirically validate the third-generation framework for monitoring and evaluating science, technology, and innovation (STI) by the Supreme Council of Science, Research, and Technology (SCSRT) in Iran. The framework is based on an integrated evaluation model (efficiency, effectiveness, and utility quadrant) and comprises 40 selected indicators. The primary objective was to assess the degree of Compliance, implementability, and added value of this framework before its formal deployment within Iran's science and technology governance system.
The study employed a mixed-methods approach (quantitative time-series trend analysis and qualitative expert-based analysis). Real-world data for 40 indicators were collected from national and international reports during the period of the Sixth Development Plan (2017-2022) and analyzed. The validity and reliability of the instruments were confirmed.
Key findings indicate that the average Compliance of the indicators with international standards is 85%. This alignment was 97% for the efficiency quadrant, 67% for effectiveness, and 92% for utility. In addition, the overall performance of the framework was rated as "weak," with a weighted average Compound Annual Growth Rate (CAGR) of 1.6%. The efficiency, effectiveness, and utility quadrants all performed at a "weak" level. Out of 34 indicators with available data, 21indicators were rated at "weak" or "very weak" levels. Furthermore, the most significant implementation challenges identified include weaknesses in "Data Accessibility, Analyzability, Being up-to-date" (DAB) (especially in the utility domain), fragmentation of information systems, and lack of integration in data collection processes. However, the added value of the third-generation evaluation framework compared to previous generations is evident in the integration of the value chain, reduction of indicators from 112 to 40, and the incorporation of "Beta Testing" for policy learning before full-scale implementation.
The overall conclusion suggests that the third-generation framework is structurally and theoretically ready for deployment, but requires modifications to impact indicators and a strengthening of data infrastructure. A pilot implementation of this framework, with continuous monitoring and benchmarking informed by successful national and international experiences, is recommended in the short term. This framework has the potential to serve as an effective model for a national intelligent STI monitoring system in Iran and other countries in the region.

Keywords

Main Subjects

References

Abbasi, M., & Ashrafi, M. (2011). Development of a framework for evaluating the performance of research credit in the country. Rahyaft, 21(48), 5-17. (Persian)
Acciai, C. (2023). Research and innovation policy design in France. In C. Acciai (Ed.), Policy design for research and innovation: Politics, institutions and interest intermediation practices (pp. 105-152). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-36628-4
Acharya, P., & Rahman, A. (2021). Compound Annual Growth Rate (CAGR) of select financial variables in Indian four-wheeler automobile companies. Wesleyan Journal of Research, 14(15).
Aghaei, P. (2021). Amendment of the executive regulations of the science, technology and innovation monitoring and evaluation system. Tehran: Research Project Report, National Research Institution for Science Policy. (Persian)
Bagheri Moghaddam, N., & Mosleh, E. (2016). Fostering sustainable technologies: A framework for analyzing the governance of innovation systems. The Journal of Science and Technology Policy Letters, 6(2), 117-139. (Persian)
Cho, H., Ahn, H., & Park, E. (2024). Data-driven analysis on the performance evaluation of national R&D projects in Korea. Evaluation and Program Planning, 102, 102383. https://doi.org/10.1016/j.evalprogplan.2023.102383
Contandriopoulos, D., & Brousselle, A. (2012). Evaluation models and evaluation use. Evaluation, 18(1), 61-77. https://doi.org/10.1177/1356389011430371
Corporate Finance Institute (CFI). (2023). Compound Annual Growth Rate (CAGR). Retrieved from: https://corporatefinanceinstitute.com/resources/wealth-management/compound-growth-rate/
Çubuk, M. (2023). R&D and innovation map of Turkey: Hybrid model approach. Turkish Journal of Science and Technology, 18(2), 487-502. https://doi.org/10.55525/tjst.1340408
Eurostat. (2018). The measurement of scientific, technological and innovation activities Oslo manual 2018 guidelines for collecting, reporting and using data on innovation: Guidelines for collecting, reporting and using data on innovation. Paris: OECD publishing.
Falkenheim, J. C., & Alexander, J. M. (2023). Academic research and development. Science & engineering indicators 2024. NSB-2023-26. National Science Foundation. Retrieved from: https://ncses.nsf.gov/pubs/nsb202326/assets/nsb202326.pdf
Farazkish, M. (2016). Designing an evaluation model for science, technology and innovation process of Iranian governmental organizations [PhD dissertation]. Tarbiat Modares University, Tehran. (Persian)
Gemici, E., & Gemici, Z. (2021). A comparative study on Turkey’s Science and Technology (S&T) indicators. Economics and Business Quarterly Reviews, 4(3), 126-143. https://doi.org/10.31014/aior.1992.04.03.376
Ghazinoory, R., Divsalar, A., & Ghazinoory, s. (2012). Evaluation of national R&D projects; Structures and methods. Tehran: Institute for International Energy Studies Press. (Persian)
Ghazinoory, S. & Farazkish, M. (2018). A modal for STI national evaluation-based efficiency, effectiveness and Utility index. Strategic Studies of Public Policy, 8(27), 205-229. (Persian)
Ghazinoory, S., Farazkish, M., Montazer, G. A., & Soltani, B. (2017). Designing a national science and technology evaluation system based on a new typology of international practices. Technological Forecasting and Social Change, 122, 119-127. https://doi.org/10.1016/j.techfore.2017.04.012
Ghazinoory, S., Farazkish, M., Nasri, S., & Mardani, A. (2023). Designing a Science, Technology, and Innovation (STI) evaluation dashboard: A comprehensive and multidimensional approach. Technology Analysis & Strategic Management, 35(8), 1005-1023. https://doi.org/10.1080/09537325.2021.1990877
Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2), 337-355. https://doi.org/10.25300/MISQ/2013/37.2.01
Harris, M., & Albury, D. (2009). Why radical innovation is needed to reinvent public services for the recession and beyond: The innovation imperative. London: The Lab Discussion Paper, NESTA.
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75-105. https://doi.org/10.2307/25148625
Khayyatian, M. S., Fartash, K. & Pourasgari, P. (2020). Development of a framework for monitoring and evaluation of Iran’s national system of science, technology and innovation. Strategy for Culture, 13(49), 119-154. (Persian) https://doi.org/10.22034/jsfc.2020.109868
Kosari, S., & Alizadeh, P. (2021). A comparative study of the governance of science, technology and innovation in Iran and selected countries. Rahyaft, 31(2), 1-22. (Persian) https://doi.org/10.22034/rahyaft.2021.10737.1223
Markiewicz, A., & Patrick, I. (2015). Developing monitoring and evaluation frameworks. Thousand Oaks: Sage Publications.
Montazer, G. A., Sharanji, M., Moradipor, H., & Farazkish, M. (2019). Sanandaj manual: The national evaluation model for research institutions. Tehran: University Publication Center. (Persian)
OECD. (2015). Frascati manual 2015: Guidelines for collecting and reporting data on research and experimental development. Paris: OECD Publishing.
OECD. (2021). OECD science, technology and innovation outlook 2021: Times of crisis and opportunity. Paris: OECD Publishing. https://doi.org/10.1787/75f79015-en
Orozco, L. A., Ordóñez-Matamoros, G., García-Estévez, J., Sierra-González, J. H., & Bortagaray, I. (2021). Science, technology, and innovation governance for social inclusion and sustainable development in Latin America. In L. A. Orozco, G. Ordóñez-Matamoros, J. H. Sierra-González, J. García-Estévez, I. Bortagaray (Eds.), Science, technology, and higher education: Governance approaches on social inclusion and sustainability in Latin America (pp. 1-18). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-80720-7_1
Partelow, S. (2023). What is a framework? Understanding their purpose, value, development and use. Journal of Environmental Studies and Sciences, 13(3), 510-519. https://doi.org/10.1007/s13412-023-00833-w
Patton, M. (2017). Developmental evaluation. In J. Pokorski, Z. Popis, T. Wyszyńska, & K. Hermann-Pawłowska (Eds.), Theory-based evaluation in complex environments (pp. 7-19). Warsaw: Polish Agency for Enterprise Development.
Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45-77. https://doi.org/10.2753/MIS0742-1222240302
Pinar, M., & Horne, T. J. (2022). Assessing research excellence: evaluating the research excellence framework. Research Evaluation, 31(2), 173-187. https://doi.org/10.1093/reseval/rvab042
Rodriguez, V., & Soeparwata, A. (2015). The governance of science, technology and innovation in ASEAN and its member states. Journal of the Knowledge Economy, 6(2), 228-249. https://doi.org/10.1007/s13132-012-0111-x
Stam, C. D. (2007). Making sense of knowledge productivity: Beta testing the KP‐enhancer. Journal of Intellectual Capital, 8(4), 628-640. https://doi.org/10.1108/14691930710830792
Supreme Council for Science, Research and Technology. (2021). Executive regulations for the national science, technology, and innovation monitoring and evaluation system. Retrieved from: https://www.atf.gov.ir/Content/media/digitallibrary/2021/10/book200/200.pdf (Persian)
Van Aken, J. E. (2004). Management research based on the paradigm of the design sciences: the quest for field‐tested and grounded technological rules. Journal of Management Studies, 41(2), 219-246. https://doi.org/10.1111/j.1467-6486.2004.00430.x
Vetterli, M., D’Amours, S., Bronstein, M., Daston, L., Diáz, S., Faist, J., ... & Stubbs, C. (2023). Assessment report of the CNRS (Centre National de la Recherche Scientifique) [Report]. High Council for Evaluation of Research and Higher Education. Retrieved from: https://art.torvergata.it/retrieve/3cdc5810-0bab-4dec-8235-3da93b138592/report-cnrs-2023.pdf
 

Files

Share

How to cite

Statistics