Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Consecuencias de la Política de Aseguramiento de Calidad en la Producción Científica de las Universidades Manabitas: un  
análisis cienciométrico  
DOI: https://doi.org/10.56124/tj.v6i11.0080  
CONSECUENCIAS DE LA POLÍTICA DE ASEGURAMIENTO DE CALIDAD EN LA PRODUCCIÓN  
CIENTÍFICA DE LAS UNIVERSIDADES MANABITAS: UN ANÁLISIS CIENCIOMÉTRICO  
CONSEQUENCES OF QUALITY ASSURANCE POLICY IN THE ACADEMIC PRODUCTION OF  
MANABÍ UNIVERSITIES: A SCIENTOMETRIC ANALYSIS  
Sornoza-Parrales Diego 1  
1
Western University (UWO). Ontario, Canadá. Correo: dsornoza@uwo.ca.  
ORCID ID: https://orcid.org/0000-0001-9319-9298.  
RESUMEN:  
El propósito de este estudio es explorar la influencia de la Política de Aseguramiento de la Calidad en la producción  
científica de las universidades ecuatorianas localizadas en la provincia de Manabí. Se utilizó una muestra de  
documentos publicados en SCOPUS para obtener una caracterización de su producción científica usando indicadores  
cienciométricos (número de artículos, distribución de citaciones e índice-h). Los resultados muestran que la  
producción científica se ha incrementado en número desde la implementación de la política, pero el 47% de los  
artículos producidos en las universidades de Manabí han recibido cero citas durante el periodo seleccionado. Para  
efectos de comparación, se ha incluído en el estudio una de las universidades mejor ranqueadas en el país que  
cuenta solo con el 15% de sus artículos sin citas. El gobierno depende de los resultados de las evaluaciones para  
definir el presupuesto de las universidades y esta vinculación de la evaluación con el presupuesto contribuye a  
incrementar las inequidades entre las Instituciones de Educación Superior en el Ecuador.  
Palabras Clave: calidad de la educación, evaluación de la educación, investigación.  
ABSTRACT:  
This study is aimed to explore the influence of the Research Quality Assurance Policy in the academic production of  
the Ecuadorian Universities located in the province of Manabí. A sample of documents published on journals indexed  
in SCOPUS was used to characterize their research production using scientometric indicators (number of items,  
distribution of citations and h-index). Results show that scientific production has increased since the policy  
implementation, but 47% of the papers produced in the universities of Manabí have received zero citations during  
the selected period. One of the top-ranked universities in the country was included in the study to establish a  
comparison, and only 15% of their papers have not been cited. The government relies on the evaluation outcomes  
to determine university budget allocation, increasing inequalities among Ecuador's Higher Education institutions.  
Keywords: educational quality, educational evaluation, research.  
Recibido: 05 de octubre de 2022; Aceptado: 19 de diciembre de 2022; Publicado: 09 de enero de 2023.  
141  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Sornoza-Parrales. (2023)  
1. INTRODUCTION  
This study explores the influence of the  
accreditation and quality assurance of  
Ecuadorian Higher Education Institutions  
(Colcha & Quinde, 2014). To achieve its  
supervisory goal, CACES carries out periodic  
processes of evaluation and accreditation  
based on the "Evaluation Model for  
Universities and Polytechnic Schools," an  
assessment tool created for accreditation  
purposes (CACES, 2015).  
Research Quality Assurance Policy in the  
academic production of the Ecuadorian  
Universities located in the province of  
Manabí. It provides a characterization of  
Manabí Universities’ research production  
using scientometric indicators (Fernández-  
Cano & Bueno, 1999).  
Ecuador has 55 Universities that integrate  
the Ecuadorian Higher Education System. To  
become a member of this system, all  
universities, including public and private,)  
must participate in the process of evaluation  
and accreditation, as defined by the  
country's higher education governing  
institutions: CES (Consejo de Educación  
Superior – Higher Education Council) and  
CACES (Consejo de Aseguramiento de la  
Calidad de la Educación Superior - Higher  
Education Quality Assurance Council).  
The Model of Evaluation sets the Quality  
standards for higher education institutions.  
It is developed considering different criteria  
(CACES, 2015):  
•
•
•
Institutional organization. This  
-
criterion considers institutional,  
organizational processes that enable  
higher education institutions to  
establish, control and evaluate the  
achievement of institutional objectives.  
Faculty Members. – This criterion  
evaluates the qualification of the faculty  
members and the existence of working  
conditions that promote an adequate  
development of their teaching, research  
and community outreach activities.  
CES has the objective of planning, regulating,  
and coordinating the Higher Education  
System in Ecuador and overseeing the  
relationships between higher education  
institutions, the government, and the  
Ecuadorian society (Colcha & Quinde, 2014).  
CACES, on the other hand, oversees policy  
Research. The research criterion  
evaluates institutional  
the  
development  
for  
the  
evaluation,  
dissemination of research objectives  
142  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Consecuencias de la Política de Aseguramiento de Calidad en la Producción Científica de las Universidades Manabitas: un  
análisis cienciométrico  
and plans and the results achieved by  
the researchers in higher education  
institutions. Research is considered a  
central element of universities and  
polytechnic schools and should be  
planned and executed, considering the  
availability of human talent and  
resources.  
The research criterion has two indicators  
related to the planning and management of  
research activities and funding and three  
indicators related to research outcomes  
(scientific papers and book publication). In  
terms of scoring for the universities  
accreditation, the most important of these  
indicators is the one related to the number  
of publications in journals indexed by  
SCOPUS or ISI Web of Knowledge(CACES,  
2015 p.26). With a weight of nine percent,  
this is the indicator with the most significant  
score in the entire model (CACES, n.d.).  
•
Community outreach. – This criterion  
measures the capacity of higher  
education institutions to address the  
necessities of their environment and the  
contribution of the universities and  
polytechnic schools to solve community  
Perspectives  
problems  
and  
generate  
social  
The Evaluation Model for Universities and  
Polytechnic Schools in Ecuador considers  
quality the "permanent search for  
excellence, pertinence, optimal production,  
knowledge mobilization and thinking  
development using auto-criticism, external  
criticism and continuous improvement"  
development.  
•
•
Infrastructure. – In this criterion, the  
characteristics of physical and digital  
infrastructure are examined, measuring  
its capacity to provide an adequate  
framework for the development of the  
activities of the academic community.  
Students. – This criterion considers the  
policy and initiatives of the higher  
education institutions to facilitate  
acceptable conditions for the students  
to achieve success in their academic  
formation.  
(CACES, 2015). The research production  
indicators are now fully integrated into the  
Ecuadorian universities' activities. Because  
policy reshapes the environments in which  
they have been introduced (Shore & Wright,  
2011), these indicators have implications for  
instructors and researchers' training,  
143  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Sornoza-Parrales. (2023)  
internal policy formulation, implementation  
and funding. Around 60% of the yearly  
budget allocation for each university is  
defined by the outcomes of the CACES  
periodic evaluations (CES, 2013).  
(Hernández et al., 2010). In this case, it can  
be helpful to show the policy's influence on  
research practices.  
SCOPUS was selected because it is one of the  
two only explicitly declared databases (the  
other is ISI Web of Knowledge) considered  
valid by the Evaluation Model (CACES, 2015).  
This work was conducted on the research  
papers produced by the universities in  
Manabí, using scientometric analysis.  
Fernández and Cano (Fernández-Cano &  
Bueno, 1999), scientometric research is  
based on different indicators to represent  
the structure of research production. Even  
though they describe several indicators  
(related to diachronic production, author's  
productivity, institutional productivity,  
citation analysis, analysis of content, among  
others), not all of them are used in this  
research.  
Quality is the main objective of the research  
indicators, but at this point, the indicators  
are calculated to prioritize quantity over  
relevance and impact. As long as the papers  
are published, no importance is given to  
what they are used for or their contribution  
to other researchers. This situation is  
evident in the four public universities  
located in the province of Manabí. This  
motivation toward increasing "productivity"  
(measured by the number of published  
papers) is not entirely aligned with the  
quality definition in the Ecuadorian Higher  
Education System.  
2
. METHODS AND DATA SOURCES  
An ad hoc selection was performed because  
this work focuses more on the institutional  
level. The chosen indicators are the number  
of items per year and institution, distribution  
of citation frequencies and h-index  
This research has a descriptive scope. The  
purpose of considering this scope is to  
describe the specific characteristics of the  
research production of the universities in the  
province of Manabí. Descriptive studies are  
helpful to show the dimensions and  
perspectives of different phenomena,  
situations, communities or contexts  
(indicators description can be found in Table  
1). The h- index for four universities was  
calculated to relate paper production in  
numbers with their impact. The h-index is  
144  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Consecuencias de la Política de Aseguramiento de Calidad en la Producción Científica de las Universidades Manabitas: un  
análisis cienciométrico  
commonly used for authors in different  
databases (SCOPUS, ISI WOK and Google  
Scholar), but its application is not restricted  
to groups of documents or institutions.  
Microsoft Excel 2016 was used to calculate  
the indicators and generate figures and  
tables.  
Table 1. Description of the scientometric indicators used in this study.  
Indicator  
A. Number of items per year  
Description  
Each year the published items (mainly articles) are counted.  
Although it is a common practice to group years, in this case,  
the only created group is the one formed by papers published  
before 2010. This will show the evolution of the generated  
documents before and after the research quality policy  
enactment.  
and institution  
B. Distribution of citation  
frequencies by papers  
I am using the following intervals: studies with no citations,  
with 1 to 10 citations, 11 to 20 citations, or more than 20  
citations.  
C. H-index  
"An entity has an h-index value of y if the entity has y  
publications that have all been cited at least y times" (Hodge  
&
Lacasse, 2011, p. 583)  
There are four public universities in Manabí:  
UTM (Universidad Técnica de Manabí),  
ULEAM (Universidad Laica Eloy Alfaro de  
Manabí), UNESUM (Universidad Estatal del  
Sur de Manabí), and ESPAM-MFL (Escuela  
Superior Politécnica Agropecuaria de  
Manabí – Manuel Félix López). The affiliation  
search performed using SCOPUS search tools  
showed that only UTM and ULEAM have  
institutional profiles on that database. An  
author search allowed to get the affiliation  
for UNESUM and ESPAM-MFL. Some of the  
results include an additional university:  
USFQ (Universidad San Francisco de Quito),  
the best-ranked university in the country at  
the date of this study (according to QS  
ranking ). USFQ will serve as a reference to  
better understand the differences among  
the universities in Ecuador.  
The sample selection process considered all  
types of documents (articles, conferences,  
book chapters, reviews, notes, editorials,  
letters) in SCOPUS, as long as they fulfill the  
following conditions:  
•
•
They were published in 2018 or before.  
They must be already published ("in  
press"  
documents  
were  
not  
considered).  
145  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Sornoza-Parrales. (2023)  
•
Authors must have provided an explicit  
affiliation with their respective  
Universities.  
papers while USFQ has produced 1.847,  
which is 16% of USFQ's production.  
Considering the 2013 -2018 period, the  
difference is still very high, with Manabí's  
Universities generating 289 items and USFQ  
3
. RESULTS  
1
168, with a proportion of 25%. Table 2  
Indicator A: Items per year and institution.  
Universities in Manabí have generated a  
combined total of 296 SCOPUS indexed  
details the production per year and  
institution.  
Table 2. Document production per year and institution. The yellow line represents the introduction of  
the Quality Assurance Policy.  
UTM  
UNESUM  
ESPAM-MFL  
ULEAM  
USFQ  
Before 2010  
5
0
0
0
2
0
7
12  
38  
51  
0
0
0
0
6
1
1
2
7
7
0
0
0
0
2
1
0
3
1
2
2
0
0
0
436  
55  
81  
107  
88  
92  
115  
179  
314  
380  
2
2
2
2
2
2
2
2
2
010  
011  
012  
013  
014  
015  
016  
017  
018  
3
11  
26  
25  
38  
43  
Indicator B: Distribution of citation  
frequencies. This indicator focuses on the  
number of citations each university's  
publication achieves. Only 157 (53%) of the  
items have been cited at least once, and 6 of  
those items have reached more than 20  
citations. The most common interval is 1 to  
10 citations per item with 135 documents in  
total (see table 3 for individual institutions'  
details).  
Table 3. Distribution of citations frequency  
UNESUM ESPAM-MFL ULEAM  
UTM  
#
USFQ  
#
f
#
f
#
f
#
f
F
Zero citations  
64  
56%  
43%  
16  
6
67%  
25%  
2
6
22%  
67%  
57  
74  
39%  
50%  
272  
772  
15%  
42%  
1
to 10 citations 49  
146  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Consecuencias de la Política de Aseguramiento de Calidad en la Producción Científica de las Universidades Manabitas: un  
análisis cienciométrico  
1
1 to 20  
citations  
1 to 30  
2
0
0
2%  
0%  
0%  
2
8%  
1
0
0
9
11%  
0%  
11  
4
7%  
292  
187  
324  
1847  
16%  
10%  
18%  
100%  
2
0
0%  
3%  
citations  
More than 30  
citations  
Total  
0
0%  
0%  
2
1%  
115 100%  
24  
100%  
100%  
148  
100%  
Indicator C: h-index. The highest h-index  
exceed a value of 7 (UTM:7, UNESUM: 5 and  
ESPAM-MFL:5).  
value in Manabí is 13, belonging to ULEAM  
(Table 4). The remaining universities do not  
Table 4. Citation percentages and h-index  
UTM  
UNESUM  
ESPAM-MFL  
ULEAM  
USFQ  
Number of  
115  
24  
9
148  
1847  
documents  
Cited documents  
Percentage of cited  
documents  
51  
44%  
8
33%  
7
78%  
91  
61%  
1575  
85%  
h-index  
7
5
5
13  
75  
Discussion  
access, subscription) over the number of  
citations. This could also show how the  
universities have adopted these practices in  
the province of Manabí.  
The paper production of universities in  
Manabí shows sustained yearly growth. It  
may not be the only cause, but it could be  
hypothesized that the model's formulation  
Comparing this indicator with the number of  
produced items raises questions about the  
purpose of research production in Manabí's  
universities. More in-depth research is  
needed to analyze how resources have been  
used and how universities in Manabí support  
the researchers' activities. Further studies  
could use the acknowledgment index to  
provide insights about budget allocation  
(in 2013) and implementation motivated  
researchers to publish more (Figure 1).  
Unfortunately, this progress is far from the  
best-ranked university in the country.  
It was beyond the scope of this paper, but it  
could be of interest to do more research  
about the influence of the access type (Open  
147  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Sornoza-Parrales. (2023)  
from the government to higher education  
institutions.  
study's scope only describes the impact of  
scientific production; still, future research  
can focus on the interaction between  
researchers, administrators, and the  
research offices' role as interpreters of the  
policy. The purpose of providing this  
characterization of the academic production  
of the universities in the province of Manabí  
is to generate discussion about the quality  
discourses that the government is  
promoting inside the higher education  
system in Ecuador and how those discourses  
are shaping the research practices in higher  
education institutions.  
4. CONCLUSIONS  
Comparing the values of Manabí Universities  
with USFQ's h- index (75) shows that even  
though the government policy aims to  
improve quality in all universities of the  
system, linking these results with the budget  
allocation seeks to reward prolific paper  
production as a sign of good resources  
usage. The actual problem is that in the way  
it is formulated now, instead of generating  
new opportunities for improving less  
developed universities with training and  
consulting, the quality model contributes to  
increasing inequalities between the higher  
Acknowledgements  
A preliminary poster version of this paper  
was presented at the 11th Robert Macmillan  
Symposium in Education (RMSE) in London,  
Ontario.  
education  
institutions  
in  
Ecuador.  
Universities with more experience and  
infrastructure will get more money, while  
the model punishes others with several  
restrictions, and they eventually will receive  
fewer resources.  
This work was supported by Scholarship No.  
CZ04-000651-2018,  
funded  
by  
the  
Ecuadorian Secretariat of Higher Education,  
Science, Technology and Innovation  
To conclude, it is essential to note that this  
work is just one piece in the analysis of the  
policy enactment; policy enactment is not a  
linear process and is heavily dependent on  
actors and context (Ball et al., 2012). This  
(SENESCYT)  
148  
Revista Científica y Arbitrada de Ciencias Sociales y Trabajo Social ‘‘Tejedora’’: Vol. 6 (Núm. 11) (ene-jun 2023). ISSN: 2697-3626  
Consecuencias de la Política de Aseguramiento de Calidad en la Producción Científica de las Universidades Manabitas: un  
análisis cienciométrico  
REFERENCES  
Fernández-Cano, A., & Bueno, A. (1999).  
Synthesizing scientometric patterns  
in Spanish educational research. In  
Scientometrics (Vol. 46, Issue 2, pp.  
Ball, S., Maguire, M., & Braun, A. (2012).  
Doing enactment research. In How  
Schools Do Policy: Policy Enactments  
in Secondary Schools (pp. 1–9). NY  
Routledge.  
349–367).  
Kluwer  
Academic  
Publishers.  
https://doi.org/10.1007/BF0246478  
3
CACES. (n.d.). Evaluación Institucional de  
Universidades y Escuelas Politécnicas  
Hernández, R., Fernández, C., & Baptista, M.  
-
Instrumentos de Cuantificación del  
(
2010). la  
Metodología  
de  
Desempeño Institucional.  
investigación. McGraw Hill Mexico.  
CACES. (2015). Modelo de evaluacion  
institucional de Universidades y  
Escuelas Politecnicas.  
Hodge, D. R., & Lacasse, J. R. (2011). Ranking  
disciplinary journals with the Google  
scholar h-index: a new tool for  
constructing cases for tenure,  
promotion, and other professional  
decisions. Journal of Social Work  
CES. (2013). Reglamento De Aplicación de la  
Fórmula de Asignación de Recursos  
destinados anualmente por parte del  
Estado a favor de las Instituciones de  
Educación Superior (Issue 021).  
Education, 579–596.  
https://doi.org/10.5175/JSWE.2011.  
01000024  
47(3),  
2
Colcha, G.,  
Implementación de un modelo y  
herramienta de inteligencia  
balanced scorecard y cuadro de  
&
Quinde, D. (2014).  
Shore, C.,  
&
Wright, S. (2011).  
Conceptualising policy: Technologies  
of governance and the politics of  
(
mando) para el mejoramiento de la  
gestión universitaria en el sistema  
administrativo, de la Escuela  
Superior Politécnica De Chimborazo,  
para el período 2014-2018. Escuela  
Politécnica de Chimborazo.  
visibility.  
In  
Policy  
Worlds:  
Anthropology and the analysis of  
contemporary power (pp. 1–26). NY:  
Berghahn.  
149