Optimizing development and operations from the project success perspective using the analytic hierarchy process

Computer Science and Information Technologies
Vol. 5, No. 3, November 2024, pp. 272~282
ISSN: 2722-3221, DOI: 10.11591/csit.v5i3.pp272-282  272
Journal homepage: http://iaesprime.com/index.php/csit
Optimizing development and operations from the project
success perspective using the analytic hierarchy process
Sani Novi Nugraheni, Teguh Raharjo, Ni Wayan Trisnawaty
Department of Information Technology, Faculty of Computer Science, University of Indonesia, Jakarta, Indonesia
Article Info ABSTRACT
Article history:
Received Aug 21, 2024
Revised Sept 27, 2024
Accepted Oct 7, 2024
By merging development and operation disciplines, the approach known as
development and operations (DevOps) can significantly improve the
efficiency and effectiveness of software development. Despite its potential
benefits, successfully implementing DevOps within traditional project
management frameworks presents significant challenges. This study
explores the critical factors influencing the implementation of DevOps
practices from the project management perspective, specifically focusing on
software development projects in the Ministry of Finance. This study utilizes
the analytic hierarchy process (AHP) to prioritize the critical elements of
project success criteria and DevOps factors necessary for effective
implementation. The findings indicate that stakeholder satisfaction, quality,
and value creation are the primary criteria for project success. Moreover,
knowledge and skills, collaboration and communication, and robust
infrastructure are pivotal factors for facilitating DevOps within project
management. The study provides actionable insights for organizations
aiming to improve their project outcomes by incorporating DevOps and
offers a systematic approach to decision-making using AHP. This study
recognizes limitations due to its focus on specific contexts and emphasizes
the need for future research in diverse organizational environments to
validate and expand these findings.
Keywords:
Agile methodologies
Analytic hierarchy process
Development and operations
implementation
Project management
Public sector Information
technology projects
This is an open access article under the CC BY-SA license.
Corresponding Author:
Sani Novi Nugraheni
Department of Information Technology, Faculty of Computer Science, University of Indonesia
Salemba Raya No.4, 10430, Jakarta, Indonesia
Email: sani.novi@ui.ac.id
1. INTRODUCTION
In the age of digitalization, public sector organizations are increasingly leveraging software and
information systems to enhance transparency and service delivery to the public [1]. This digital
transformation requires swiftly adapting to changing environments and evolving citizen expectations [2].
Public sector entities, such as the Ministry of Finance of Indonesia, must adopt technological solutions that
improve efficiency and ensure the delivery of high-quality public services [3], [4]. The shift towards digital
platforms has led these organizations to rethink their project management approaches, making agility a
crucial factor in the successful implementation and sustainability of these technologies.
The evolution of project management methodologies in system development has shifted from
waterfall models to Agile methodologies [5], [6]. Agile methods provide the flexibility and faster response
times that drive this transition [6]. Integrating development and operation (DevOps) with Agile practices has
emerged as a powerful strategy in software development projects, enhancing the continuous delivery of
software updates and new features [7], [8].

Comput Sci Inf Technol ISSN: 2722-3221 
Optimizing development and operations from the project success perspective … (Sani Novi Nugraheni)
273
DevOps has become a transformative force, revolutionizing the execution of software development
projects. DevOps combines the aspects of 'Development' and 'Operations,' which represents a cultural and
technical shift that promotes a collaborative environment, enabling the building, testing, and releasing of
software to occur more rapidly, frequently, and reliably [9]–[12]. The significant advantage of incorporating
DevOps with project management lies in its ability to enhance the agility of development processes, thereby
reducing time to market, minimizing the development lifecycle, and elevating product quality [9], [13].
Organizations increasingly adopt DevOps due to its significant benefits. Allied Market Research
projects the agile software development market to expand at a compound annual growth rate of 19.50% by
the end of 2026 [14], [15]. This approach has led to notable improvements in software quality and team
productivity, with 42% of enterprises reporting enhanced software quality and 47% noting increased
productivity. Further highlighting its efficacy, a Google Report in 2023 indicates that organizations with
capabilities in continuous integration (CI) and continuous deployment (CD) can improve organizational
performance by up to 2.4 times [16]. DevOps is utilized by approximately 61% of respondents in varying
fields. The benefits of implementing DevOps are substantial, as evidenced by a survey highlighted in the
Harvard Business Review. Respondents noted considerable enhancements in several key areas: speed to
market (70%), productivity (67%), customer relevance (67%), innovation (66%), and quality of products or
services (64%) [17]. Thus, it leads to better project outcomes.
However, the agility and responsiveness of DevOps do not come without challenges, especially in
the public sector, where projects must adhere strictly to predefined success criteria, including time, scope,
and budget constraints [18], [19]. Integrating DevOps into public sector projects requires careful
consideration of these success factors to ensure that the quick development and deployment processes do not
undermine the quality and reliability of the software [12]. Therefore, assessing how DevOps can meet these
stringent requirements is crucial for its adoption in high-stakes environments like government ministry. To
successfully implement DevOps, particularly in projects with specific requirements, organizations need to
pinpoint the necessary factors that are challenging [8]. Yusuf et al. examined how to improve local e-
government using DevOps [20]. They identified several critical success factors in implementing DevOps,
including collaboration and communication, automation, continuous feedback, and the integration of tools
and technology, as well as cultural shifts and skill development. A case study conducted by the Indonesian
Public Health Organization revealed that the development team’s mindset presents the most significant
challenge that must be addressed when implementing DevOps [21].
This study proposes a structured analysis using the analytic hierarchy process (AHP) to prioritize the
factors that affect the successful adoption of DevOps in software development projects, incorporating project
management elements to address the challenges. The AHP offers a methodical approach to decision-making
by decomposing the problem into a hierarchy of more comprehensible sub-problems for individual analysis
[22]–[24]. Therefore, this study's research question is: "What factors influence the DevOps implementation
from the project success perspective, and how can they be effectively prioritized using the AHP approach?".
By answering this question, the study aims to provide actionable insights organizations can use to optimize
their software development project by effectively integrating DevOps and project management practices.
Existing research often talks about the difficulties, benefits, or success factors of implementing
DevOps in the private sector [13], [25]–[27]. However, there aren't many studies that look specifically at the
responses to the critical success factors or their outcomes from a project management point of view when
implementing DevOps in the public sector. This gap highlights the need for detailed case-specific analyses
that investigate what project success criteria and DevOps key factors are managed. The success criteria and
factors identified from the previous studies will be ranked using the AHP. This study aims to bridge the gap
between DevOps's theoretical benefits and practical application and enhance project outcomes through
strategic implementations [28].
2. METHOD
This study explains research chronologically, including research design, research procedure,
theoretical framework, and data acquisition. This study adopts a quantitative method using the Analytic
Hierarchy Process to answer the research question.
2.1. Development and operations (DevOps)
Development and operations usually consist of two different departments. In the software
development process, there is a conflict between the need for change from the developer team and the fear of
change from the operation team [9]. The DevOps initiative focuses on enhancing collaboration between
development and operations personnel to address critical challenges like resistance to change and risky
deployments [9], [29]–[31].

 ISSN: 2722-3221
Comput Sci Inf Technol, Vol. 5, No. 3, November 2024: 272-282
274
DevOps, a contraction of "development" and "operations," refers to the technique that combines
software development and IT operations intending to cut the systems development life cycle while frequently
delivering features, fixes, and updates that closely align with business goals [12], [30], [31]. The essence of
DevOps is to promote collaboration and integration between the development and operations teams, which
historically functioned in siloes, to improve the agility and responsiveness of the software development
process [9], [28], [31]. The integration of continuous integration (CI) and CD into the DevOps methodology
ensures that software can be built, tested, and released to production rapidly and with high reliability [29].
2.2. Analytic hierarchy process
The AHP is a methodical approach for arranging and evaluating complicated decisions based on
mathematical and psychological principles. Saaty developed AHP in the 1970s, which assists decision-makers
in establishing priorities and making optimal choices by simplifying complex decisions into pairwise
comparisons and then integrating these findings [22]-[24]. AHP has been widely used across multiple
disciplines, such as project management, policy, finance, healthcare, and strategic decisions, demonstrating its
versatility and effectiveness in decision-making scenarios where multiple criteria are involved [24], [32], [33].
The AHP is highly effective at simplifying complex decision-making processes into a clear
hierarchical structure, as demonstrated in numerous studies. It organizes decision-making criteria into a
multi-tiered framework, ranging from overarching goals to specific alternatives, thus rendering the decision
process more methodical [34]. This structured methodology thoroughly assesses diverse and often conflicting
criteria, enhancing decision-making efficacy [35]. Furthermore, as a robust multi-criteria decision making
(MCDM) tool, AHP addresses complex decision challenges by providing a coherent framework that allows
for the comparative analysis of alternatives based on various factors [36]. The methodological rigor of AHP
ensures that decision-makers can navigate through complex decision landscapes with clarity and efficiency,
ultimately leading to optimal choices in scenarios with diverse and conflicting criteria.
2.3. Project success criteria
Project success criteria are also derived from previous studies, encompassing the triple constraint
and other criteria applicable today [37]–[39]. The project success criteria are depicted in the following sub-
section. From the comparison and gap analysis of previous studies, this study aims to conduct an empirical
study applying AHP to prioritize the DevOps success factor from the project management perspective.
Within project management, success criteria are the standards used to measure project success [37],
[38]. Traditionally, project success was measured against the "iron triangle" of scope, time, and cost [39],
[40]. However, contemporary views on project success encompass broader aspects such as stakeholder
satisfaction, project quality, and the overall business or public value derived from the project [10], [41]–[43].
Some literature has been identified to obtain project success criteria. The success criteria for a
software development project vary for each project. Table 1 summarizes project success criteria from
previous studies.
Table 1. Project success criteria
Project Success Criteia References
Time [10], [18], [39]-[43]
Cost [10], [18], [41], [42], [44], [45]
Scope [10], [18], [41], [44], [45]
Quality [10], [41], [42], [44], [45]
Risk [10], [41], [44], [45]
Stakeholder satisfaction [10], [39]-[41], [43]
Value creation [41], [43]
Table 1 represents various sources cited for different success criteria in project management. Critical
software development project measures are time, cost, scope, quality, risk, stakeholder satisfaction, and value
creation. The Project Management Institute (PMI) [44], Kerzner [45], and Marchewka [18] are frequently
referenced across multiple criteria, highlighting their comprehensive contributions to project management
literature. Adywiratama et al. [41] and Akbar et al. [10], along with others like Raharjo [42] and Takagi et al.
[43], are cited specifically for more focused aspects such as stakeholder satisfaction and value creation,
indicating specialized study in these areas within recent years.
2.4. DevOps success factors
Various studies have been conducted on the implementation of DevOps using various approaches.
The study by Khan and Shameem [46] identified that the bureaucracy factor in the deployment process is the

275
main challenge in DevOps activities. Akbar et al. [11] also conducted a similar study using the fuzzy-AHP
approach and revealed that the security pipeline factor, usage of orchestration systems, and organizational
matrix transparency are the main factors influencing the success of DevOps implementation. The study by
Akbar et al. [10] incorporates DevOps with the knowledge area of project management and prioritizes the
success factor of each knowledge area. Several studies have identified success factors using literature
reviews, such as those by Karunarathne et al. [47] Mishra and Otaiwi [12], and Azad and Hyrynsalmi [48].
Critical factors in implementing DevOps are obtained and incorporated from those previous studies with
project success criteria.
The successful implementation of DevOps hinges on several key factors. Lwakaterte et al. [49]
conclude that DevOps implementation success factors in cultural change, automation, lean management,
measurement and monitoring, and sharing and knowledge management [49]. Based on previous studies, six
factors can be identified, as shown in Table 2.
Table 2. DevOps success factors
DevOps Success Factors References
Top management support [27], [47]
Collaboration and communication [27], [46], [47]
Automation [12], [26], [27], [46], [48], [50]
Knowledge and skill [27], [46], [47]
Infrastructure, tools, and technology [25], [26], [47], [48], [51]
Transparency [25], [26], [46], [47]
Top management highlights the crucial role of senior management in endorsing and supporting
DevOps initiatives within an organization [27], [47]. Furthermore, effective collaboration and
communication are essential in a DevOps environment, where development, operations, and other
departments must work closely to achieve seamless software delivery [27], [46], [47]. Automation is one of
the DevOps principles that help reduce manual work, minimize errors, and speed up the release cycles,
thereby enhancing operational efficiency and enabling more frequent updates and improvements [12], [26],
[27], [46], [48], [50].
The success of DevOps also depends heavily on the knowledge and skills of the team. The condition
encompasses expertise in coding, system administration, integration, and tools optimization [27], [46], [47].
The role of infrastructure, tools, and technology in supporting project operations is explored by Azad and
Hyrynsalmi [48] and Chen [51] among others. Finally, transparency in DevOps involves clear visibility into
all processes and workflows for all stakeholders involved [25], [26], [46], [47].
2.5. Research framework
The research framework for integrating DevOps into project management involves understanding
how DevOps practices can align with project management success criteria to enhance project outcomes. By
applying the AHP methodology within this framework, organizations can systematically prioritize the
success factors in DevOps, ensuring that the most critical aspects are addressed to optimize project success
and operational efficiency. Based on the theories explained before, Figure 1 shows the research framework
for this study. At the upper level of the hierarchical structure, seven elements are considered project success
criteria. On the second level, six critical factors from previous studies significantly affect the implementation
of DevOps.
Figure 1. Research framework

 ISSN: 2722-3221
276
2.6. Research methodology
The study employs a hybrid approach, beginning with a qualitative study followed by quantitative
approaches [52]. Figure 2 illustrates the flow of this study. A literature review was conducted to derive
project success criteria and DevOps success factors. Three experts validated the hierarchical structure. These
experts were project managers in government and senior developers.
Four distinct stages are undertaken to rank key factors effectively using the AHP approach. The
initial stage involves defining the problem and the desired solutions and constructing a hierarchical structure,
as depicted in Figure 1. This stage breaks down complex issues into smaller, manageable components to
establish a hierarchy.
The second stage in the AHP involves determining priority elements. At this stage, priorities are
established for all elements within the hierarchy using pairwise comparisons, as illustrated in Table 3. This
process begins at the criteria level and continues with the factor elements. The scale used for pairwise
comparisons in the AHP is outlined in Table 4.
Figure 2. Research flow diagram
Table 3. Pairwise comparison
C1 C2 … Cn
C1 C11 C12 … C1n
C2 C21 C22 … C2n
… … … …
Cn Cn1 Cn2 … Cnn
Table 4. Scale for pairwise comparison
Importance Scale Explanation
1 Both factors hold the same level of importance.
3 One factor is slightly more important than the other.
5 One factor is more important than the other.
7 One factor is substantially more important than the other.
9 One factor is more important than others.
2, 4, 6, 8 The value is between two consecutive evaluations.
Questionnaires were distributed to eight respondents to obtain data. The questionnaires were
directed towards employees within the central information and communication technology (ICT) unit
responsible for developing applications utilizing DevOps methodologies. The characteristics of respondents
are shown in Table 5.

277
Table 5. Respondent characteristics
Respondent Job Position Experience (years)
E01 Team Leader, Senior Developer, System Analyst 22
E02 Team Leader, Senior Developer, System Analyst 13
E03 Senior Developer 13
E04 Senior Developer and IT Operation Specialist 20
E05 IT Operation Specialist 11
E08 IT Service Specialist 13
The third stage involves determining the eigenvector and eigenvalues. The eigenvector is used to
ascertain the relative impact of each criterion and factor on the research objective [24]. The final step in AHP
is calculating the consistency ratio (CR). In AHP, the CR is determined by comparing the consistency index
(CI) and the reference index (RI). Saaty and Vargas [24] have established that the consistency ratio should be
10% or below. If the assessment ratio is above 10%, the assessment needs to be revised by re-evaluating until
an appropriate assessment ratio is achieved. In (1) is the method to calculate the CR.
𝐶𝑅 =
𝐶𝐼
𝑅𝐼
≤ 10% (1)
The CI value is determined using the maximum eigenvector value (λmax) obtained from the
pairwise comparison matrix, as seen in (2). The RI value is a predetermined value, as depicted in Table 6.
𝐶𝐼 =
λ𝑚𝑎𝑥 − 𝑛
𝑛−1
(2)
Table 6. Reference index
Matrix 1 2 3 4 5 6 7 8 9 10
RI 0,00 0,00 0,58 0,90 1,12 1,24 1,32 1,41 1,45 1,49
The stages of the AHP methodology adopted for this study provide a robust framework for
examining the essential factors impacting the successful implementation of DevOps practices within project
management. This approach ensures reliable and actionable findings. This approach also offering a clear path
for organizations looking to integrate DevOps effectively.
3. RESULTS AND DISCUSSION
The questionnaire data from eight respondents was transformed into a pairwise matrix and managed
through a form available at http://www.bpmsg.com. The website facilitates the calculation of element
priorities within the pairwise comparison by applying the AHP method. The calculation output yielded
priority percentages for each criterion and factor and included the value of the CR. The CR value for each
individual and the combined respondents is below 10%, thus making the results of this questionnaire
acceptable.
3.1. Priority of project success criteria in DevOps
Prioritizing project success criteria in DevOps is critical to managing software development
projects. Eight respondents evaluated seven criteria using the AHP. Table 7 represents the priority of project
success criteria.
Table 7. Priority of project success criteria
Criteria Weight Gap Rank
Stakeholder satisfaction 0.271 - 1
Quality 0.219 0.052 2
Value creation 0.171 0.048 3
Risk 0.114 0.057 4
Scope 0.104 0.010 5
Time 0.083 0.021 6
Cost 0.038 0.045 7

 ISSN: 2722-3221
278
According to the calculation results, stakeholder satisfaction emerged as the most significant
criterion for project success, holding a weight value of 0.271 or 27.1%. This result differs from those found
by Castro et al. in their study conducted in Brazil [53]. They found that while stakeholder satisfaction is
considered a criterion for project success, it is not the primary factor. Project stakeholders include individuals
and groups participating in or affected by a project whose interests may positively or negatively influence the
project's outcomes [44]. The stakeholders are best positioned to assess the project's success [54], [55].
Therefore, it is crucial to define and manage stakeholder satisfaction as a project objective [44]. Effective
stakeholder engagement hinges on maintaining ongoing communication with all involved parties to grasp
their needs and expectations, swiftly resolve issues, balance conflicting interests, and ensure stakeholders are
actively involved in project decisions and activities [44].
Quality criteria are ranked second with a weight value of 0.219 or 21.9%, indicating that quality is a
significant criterion of project success. This finding aligns with the result of the study conducted by Lamprou
and Vagiona, which considered quality to be one of the most important success criteria for projects in Greece
[56]. The quality of the project can be regarded as the extent to which the expectations of project participants
are met [57]. In software development using DevOps, the plan for quality control and quality management
activities must be detailed and integrated [10], [44]. Senapathi and Buchan [8] explained that quickly
delivering quality software is one of the critical drivers for DevOps adoption.
Thirdly, value creation has a weight value of 0.171 or 17.1%, emphasizing its importance in project
outcomes. This finding aligns with a case study conducted by Adywiratama et al. in the Presidential
Advisory Council in Indonesia [41]. This criterion reflects the project's ability to deliver substantial benefits,
including innovative solutions that significantly enhance the impact of the result of the project [43]. The
prioritization of value creation highlights its role in ensuring the sustainability and broader acceptance of
project results, affirming that successful projects must align with business objectives and positive community
impact, thus fostering long-term value beyond immediate project outputs.
On the other hand, the triple constraint (cost, time, and scope) does not emerge as the primary
priority. These findings differ from the result of the study conducted by Adywiratama et al., which found that
triple constraint remains the most important success criterion in the Presidential Advisory Council in
Indonesia [41]. From a project management perspective within the public sector, this indicates a shift in focus
from traditional success metrics. Public sector projects have rigorously adhered to these constraints to ensure
accountability and effective use of public resources. However, this shift suggests that factors such as
stakeholder satisfaction and perhaps other qualitative measures are gaining prominence over the conventional
cost, time, and scope considerations. Additionally, the information system development project using
DevOps is carried out in-house, so it is not strictly bound by budget and time constraints.
3.2. Priority of key success factor of DevOps implementation
The weight values of DevOps success factors for each criterion were analyzed, and a spreadsheet
was subsequently used to calculate the matrix between criteria and success factors. The criteria and factors
are listed in the Tables 2 and 3, respectively. These are combined to form a 6×7 matrix, as depicted in
Table 8, elaborating on the weight values of each of the success factors.
Table 8. Factor and criteria matrix 6×7
Time Cost Scope Quality Risk Stakeholder
Satisfaction
Value
Creation
Final
Weight
Rank
0.083 0.038 0.104 0.219 0.114 0.271 0.171
Top management support 0.087 0.202 0.210 0.093 0.185 0.101 0.229 0.144724 4
Collaboration and
communication
0.294 0.111 0.182 0.210 0.155 0.230 0.163 0.201411 2
Automation 0.144 0.171 0.133 0.100 0.149 0.125 0.132 0.127615 5
Knowledge and skill 0.266 0.218 0.199 0.356 0.254 0.248 0.242 0.266568 1
Infrastructure 0.150 0.207 0.194 0.182 0.172 0.161 0.152 0.169581 3
Transparency 0.059 0.092 0.082 0.060 0.086 0.136 0.082 0.090743 6
According to Table 8, knowledge and skill are at the top of the hierarchy, weighted at 26.66%,
indicating that technical expertise and proficiency are the most critical elements in achieving DevOps
objectives. Next, collaboration and communication follow with a 20.14% weight value, emphasizing the need
for effective interaction and cooperation among team members and across departments. The infrastructure
factor, which includes the tools and technology that support DevOps implementation, holds a significant
Criteria
Factor

279
position, with a 16.96% weight, highlighting the need for a robust and adaptable technological foundation to
support continuous deployment and integration processes.
Knowledge and skill emerged as the primary factors contributing to DevOps's success. This
condition became the most crucial factor for creating value in cost, quality, risk, stakeholder satisfaction, and
project success criteria. This result aligns with the findings of the study conducted by Khan and Shameem
[46]. They stated that team members' lack of knowledge and data sharing became a significant obstacle to
DevOps implementation. It is not only essential knowledge about the project or software development;
members should also have a thorough understanding of DevOps tools [27], [46]. Therefore, DevOps team
members can optimize the utilization of these tools.
DevOps demands a combination of development and operation knowledge and skill. For
organizations, it is essential to have staff with the correct skills and expertise, as a shortage of adequately
skilled staff can hinder the progress of DevOps adoption [8], [47]. Improving knowledge and skills in
DevOps within an organization can be approached through various strategies. One effective method is
implementing structured training and certification programs focusing on both theoretical and practical aspects
of DevOps practices. This method includes courses on specific tools, software development practices, and
operational procedures integral to DevOps. Another approach is to encourage collaboration and knowledge
sharing among teams through regular meetings and discussion forums where team members can collectively
exchange ideas and solve problems. Above all, the organization needs top management support to foster a
culture of knowledge sharing.
Collaboration and communication became the second most crucial factor in DevOps implementation
for software development projects, with a weight value of 20.14%. This result aligns with the findings of the
study conducted by Jayakody and Wijayanayake, which identify a collaborative culture as one of the most
critical success factors in DevOps implementation [58]. According to Table 8, collaboration and
communication become the most critical factors from a time criteria perspective. As depicted in previous
studies, collaboration and communication is a significant factor in implementing DevOps [27], [46], [47],
[59], [60]. Collaboration and communication are the underlying culture of DevOps [47], [61]. Development
and operations teams must share responsibilities, leading to more effective team communication [60]. Team
dynamics, with knowledge and idea sharing, can improve collaboration within the team [60].
According to Table 8, infrastructure ranks third with a weight value of 16.96%. This condition
aligns with previous studies that indicate implementing DevOps requires infrastructure support [48], [51],
highlighting the necessity of robust infrastructure support for successful DevOps implementation. This result
also aligns with Akbar et al., which found that tools are less critical than cultural factors [11]. Organizations
looking to adopt DevOps should prioritize investments in their infrastructure, tools, and technology to ensure
they can fully leverage the benefits of DevOps methodologies. Investing in scalable and flexible
infrastructure solutions, such as cloud services and automated deployment tools, can facilitate more efficient
workflows, enhance collaboration across teams, and ultimately lead to faster and more reliable product
deliveries.
Following project success criteria, top management support plays a moderate role in DevOps
success. However, top management support is the most critical factor influencing project success regarding
scope criteria. Leadership plays a role in setting clear project objectives and defining project scope
boundaries. With a clear understanding of the project, top management and the project team can prevent
scope creep [62]. Regarding DevOps, Karunarathne et al. [47] stated that a lack of top management
commitment slows down DevOps implementation. Top management should direct and enable the change
process.
The lowest priorities are automation and transparency. This result differs from the findings of a
study conducted by Rehman et al. which identified transparency as one of the most critical success factors in
DevOps implementation [25]. Despite lower priorities, automation and transparency are also crucial in
DevOps implementation. Automation streamlines processes, reducing manual intervention and enhancing
efficiency, which is crucial for continuous integration and deployment practices that define DevOps. Thus, it
can reduce the time needed to implement the project. On the other hand, transparency ensures that all team
members have access to the same information, fostering an environment of trust and collaborative problem-
solving. Though not at the top of the priority list, both elements play vital roles in maintaining the integrity
and effectiveness of DevOps practices, ultimately contributing to the project's overall success.
4. CONCLUSION
This study aims to provide practical insights organizations can use to enhance their software
development projects by effectively integrating DevOps and project management practices. The findings
from this study reveal that the primary criterion for project success is stakeholder satisfaction, followed
closely by quality and value creation. These results underscore the importance of aligning project outputs

 ISSN: 2722-3221
280
with stakeholder expectations and emphasizing delivering high-quality products that generate substantial value.
Knowledge and skills, collaboration and communication, and infrastructure emerge as crucial elements in
DevOps software development projects. The proficiency and expertise of the development team in DevOps
practices are vital, as is the ability to effectively communicate and collaborate across various departments.
Additionally, robust infrastructure supports rapid iterations and deployments of DevOps methodologies.
From a practical standpoint, these findings suggest that organizations should focus on enhancing
stakeholder engagement processes to obtain their expectations, prioritizing continuous improvement in
product quality, and fostering an environment where innovation drives value creation. In theory, this study
adds to what is already known by showing how the success criteria for project management and the key
factors for DevOps implementation change over time. It gives a full picture of how these factors affect
project outcomes.
This study's limitations include its focus on specific contexts that may not universally apply,
potentially limiting the generalizability of the findings. Future work could investigate these relationships in
various organizational settings or industry sectors to confirm the results' applicability. Additionally,
examining the long-term impacts of these success factors and criteria in a broader range of DevOps
environments could provide deeper insights and further validate the current study's conclusions. Future works
can utilize another multi-criteria decision-making approach, such as PROMETHEE, fuzzy AHP, or TOPSIS.
ACKNOWLEDGEMENTS
We express deep gratitude to the Ministry of Communication and Information Technology
(KOMINFO) for their generous support and funding of this study, facilitated through the educational
framework of the University of Indonesia. This acknowledgment highlights the vital role that KOMINFO has
played not only in financially backing the study but also in enabling a collaborative academic environment
that fosters significant research advancements.
REFERENCES
[1] T. Mountasser and M. Abdellatif, “Digital transformation in public administration: a systematic literature review,” International
Journal of Professional Business Review, vol. 8, no. 10, p. e02372, 2023, doi: 10.26668/businessreview/2023.v8i10.2372.
[2] G. Li, M. Zhou, Z. Feng, M. Li, and H. Jiang, “Research on key influencing factors of e-government cloud service satisfaction,”
Wireless Personal Communications, vol. 127, no. 2, pp. 1117–1135, 2022, doi: 10.1007/s11277-021-08567-0.
[3] J. Xiao, L. Han, and H. Zhang, “Exploring driving factors of digital transformation among local governments: foundations for
smart city construction in China,” Sustainability (Switzerland), vol. 14, no. 22, 2022, doi: 10.3390/su142214980.
[4] A. David et al., “Understanding local government digital technology adoption strategies: a PRISMA review,” Sustainability
(Switzerland), vol. 15, no. 12, 2023, doi: 10.3390/su15129645.
[5] T. Thesing, C. Feldmann, and M. Burchardt, “Agile versus waterfall project management: decision model for selecting the
appropriate approach to a project,” Procedia Computer Science, vol. 181, pp. 746–756, 2021, doi: 10.1016/j.procs.2021.01.227.
[6] J. Leong, K. May Yee, O. Baitsegi, L. Palanisamy, and R. K. Ramasamy, “Hybrid project management between traditional
software development lifecycle and agile based product development for future sustainability,” Sustainability (Switzerland), vol.
15, no. 2, 2023, doi: 10.3390/su15021121.
[7] A. Hemon, B. Lyonnet, F. Rowe, and B. Fitzgerald, “From agile to DevOps: smart skills and collaborations,” Information Systems
Frontiers, vol. 22, no. 4, pp. 927–945, 2020, doi: 10.1007/s10796-019-09905-1.
[8] M. Senapathi, J. Buchan, and H. Osman, “DevOps capabilities, practices, and challenges: Insights from a case study,” ACM
International Conference Proceeding Series, vol. Part F1377, 2018, doi: 10.1145/3210459.3210465.
[9] M. Hüttermann, Beginning DevOps for developers. New York: Apress, 2012.
[10] M. A. Akbar, A. A. Khan, N. Islam, and S. Mahmood, “DevOps project management success factors: a decision-making
framework,” Software - Practice and Experience, vol. 54, no. 2, pp. 257–280, 2024, doi: 10.1002/spe.3269.
[11] M. A. Akbar, S. Mahmood, M. Shafiq, A. Alsanad, A. A. A. Alsanad, and A. Gumaei, “Identification and prioritization of
DevOps success factors using fuzzy-AHP approach,” Soft Computing, vol. 27, no. 4, pp. 1907–1931, 2023, doi: 10.1007/s00500-
020-05150-w.
[12] A. Mishra and Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, 2020, doi:
10.1016/j.cosrev.2020.100308.
[13] L. Riungu-Kalliosaari, S. Mäkinen, L. E. Lwakatare, J. Tiihonen, and T. Männistö, “DevOps adoption benefits and challenges in
practice: A case study,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and
Lecture Notes in Bioinformatics), vol. 10027 LNCS, pp. 590–597, 2016, doi: 10.1007/978-3-319-49094-6_44.
[14] Allied Market Research, Enterprise agile transformation services market size, share, competitive landscape and trend analysis
report, by organization size, by industry vertical, by service type, by methodology, by consumer type: global opportunity analysis
and industry forecast. 2023.
[15] J. Katariya, “DevOps Project Management: An Agile Approach to Managing Projects,” Https://Www.Moontechnolabs.Com,
2024. https://www.moontechnolabs.com/blog/devops-project-management/#:~:text=DevOps project management uses
automation,on delivering high-quality results.
[16] DevOps Research and Assessment and Google Cloud, Accelerate state of DevOps report 2023. 2023.
[17] H. B. Review, Competitive advantage through networking, vol. 36, no. 1. 2019.
[18] J. Marchewka, “Information technology project management providing measurable organizational value,” Wiley, vol. 5, no.
August, p. 128, 2016.
[19] K. Schwalbe, “Information technology project management,” Cengage Learning, 2015, doi: 10.7172/978-83-65402-07-
3.2015.wwz.4.

281
[20] M. Yusuf, M. K. Sophan, A. K. Darmawan, B. D. Satoto, A. Muntasa, and R. A. Nugroho, “E-government service management
system (e-govservice) to improve local e-government using DevOps approach,” 2023 6th International Conference on
Information and Communications Technology, ICOIACT 2023, pp. 309–314, 2023, doi: 10.1109/ICOIACT59844.2023.10455931.
[21] M. Yazid, A. Qahar, and T. Raharjo, “DevOps implementation challenges in the Indonesian public health organization,”
International Journal of Advanced Computer Science and Applications, vol. 14, no. 9, pp. 80–93, 2023, doi:
10.14569/IJACSA.2023.0140910.
[22] T. L. Saaty, “Decision-making with the AHP: Why is the principal eigenvector necessary,” European Journal of Operational
Research, vol. 145, no. 1, pp. 85–91, 2003, doi: 10.1016/S0377-2217(02)00227-8.
[23] R. W. Saaty, “The analytic hierarchy process-what it is and how it is used,” Mathematical Modelling, vol. 9, no. 3–5, pp. 161–
176, 1987, doi: 10.1016/0270-0255(87)90473-8.
[24] T. L. Saaty and L. G. Vargas, Models, methods, concepts & applications of the analytic hierarchy process. Springer, 2012.
[25] U. U. Rehman, T. Mahmood, M. Albaity, K. Hayat, and Z. Ali, “Identification and prioritization of DevOps success factors using
bipolar complex fuzzy setting with frank aggregation operators and analytical hierarchy process,” IEEE Access, vol. 10, pp.
74702–74721, 2022, doi: 10.1109/ACCESS.2022.3190611.
[26] M. Z. Toh, S. Sahibuddin, and R. A. Bakar, “A review on DevOps adoption in continuous delivery process,” Proceedings - 2021
International Conference on Software Engineering and Computer Systems and 4th International Conference on Computational
Science and Information Management, ICSECS-ICOCSIM 2021, pp. 98–103, 2021, doi: 10.1109/ICSECS52883.2021.00025.
[27] N. M. Noorani, A. T. Zamani, M. Alenezi, M. Shameem, and P. Singh, “Factor prioritization for effectively implementing
DevOps in software development organizations: a SWOT-AHP approach,” Axioms, vol. 11, no. 10, 2022, doi:
10.3390/axioms11100498.
[28] R. Jabbari, N. Bin Ali, K. Petersen, and B. Tanveer, “What is DevOps? A systematic mapping study on definitions and practices,”
ACM International Conference Proceeding Series, vol. 24-May-201, 2016, doi: 10.1145/2962695.2962707.
[29] A. M. Mowad, H. Fawareh, and M. A. Hassan, “Effect of using continuous integration (CI) and continuous delivery (CD)
deployment in DevOps to reduce the gap between developer and operation,” Proceedings - 2022 23rd International Arab
Conference on Information Technology, ACIT 2022, 2022, doi: 10.1109/ACIT57182.2022.9994139.
[30] M. Gall and F. Pigni, “Taking DevOps mainstream: a critical review and conceptual framework,” European Journal of
Information Systems, vol. 31, no. 5, pp. 548–567, 2022, doi: 10.1080/0960085X.2021.1997100.
[31] M. S. Khan, A. W. Khan, F. Khan, M. A. Khan, and T. K. Whangbo, “Critical challenges to adopt DevOps culture in software
organizations: a systematic review,” IEEE Access, vol. 10, pp. 14339–14349, 2022, doi: 10.1109/ACCESS.2022.3145970.
[32] Z. K. Szabo, Z. Szádoczki, S. Bozóki, G. C. Stanciulescu, and D. Szabo, “An analytic hierarchy process approach for
prioritisation of strategic objectives of sustainable development,” Sustainability (Switzerland), vol. 13, no. 4, pp. 1–26, 2021, doi:
10.3390/su13042254.
[33] C. Y. Chen and J. J. Huang, “Integrating dynamic bayesian networks and analytic hierarchy process for time-dependent multi-
criteria decision-making,” Mathematics, vol. 11, no. 10, 2023, doi: 10.3390/math11102362.
[34] S. Gupta, S. C. Misra, A. Singh, V. Kumar, and U. Kumar, “Identification of challenges and their ranking in the implementation
of cloud ERP: A comparative study for SMEs and large organizations,” International Journal of Quality and Reliability
Management, vol. 34, no. 7, pp. 1056–1072, 2017, doi: 10.1108/IJQRM-09-2015-0133.
[35] S. K. Sahoo and S. S. Goswami, “A comprehensive review of multiple criteria decision-making (MCDM) methods:
advancements, applications, and future directions,” Decision Making Advances, vol. 1, no. 1, pp. 25–48, 2023, doi:
10.31181/dma1120237.
[36] W. Wongvilaisakul, P. Netinant, and M. Rukhiran, “Dynamic multi-criteria decision making of graduate admission recommender
system: AHP and fuzzy AHP approaches,” Sustainability (Switzerland), vol. 15, no. 12, 2023, doi: 10.3390/su15129758.
[37] M. Albert, P. Balve, and K. Spang, “Evaluation of project success: a structured literature review,” International Journal of
Managing Projects in Business, vol. 10, no. 4, pp. 796–821, 2017, doi: 10.1108/IJMPB-01-2017-0004.
[38] C. Santos, V. Santos, A. Tavares, and J. Varajão, “Project management in public health: a systematic literature review on success
criteria and factors,” Portuguese Journal of Public Health, vol. 38, no. 1, pp. 37–48, 2020, doi: 10.1159/000509531.
[39] R. Atkinson, “Project management: Cost, time and quality, two best guesses and a phenomenon, its time to accept other success
criteria,” International Journal of Project Management, vol. 17, no. 6, pp. 337–342, 1999, doi: 10.1016/S0263-7863(98)00069-6.
[40] M. Bronte-stewart, “Beyond the iron triangle: evaluating aspects of success and failure using a project status model,” Computing
& Information Systems, vol. 19, no. 2, pp. 21–37, 2015, [Online]. Available:
https://eds.b.ebscohost.com/eds/pdfviewer/pdfviewer?vid=5&sid=d15f9a66-b33c-497c-8e07-
448c2e0d6334@sessionmgr113&hid=112.
[41] A. D. Adywiratama, C. Ko, T. Raharjo, and A. Wahbi, “Critical success factors for ICT project: A case study in project colocation
government data center,” Procedia Computer Science, vol. 197, pp. 385–392, 2021, doi: 10.1016/j.procs.2021.12.154.
[42] T. Raharjo, “Model of critical success factors for agile information technology project in Indonesia using analytic hierarchy
process (AHP),” ADI Journal on Recent Innovation (AJRI), vol. 5, no. 1Sp, pp. 68–77, 2023, doi: 10.34306/ajri.v5i1sp.968.
[43] N. Takagi, J. Varajão, T. Ventura, D. Ubialli, and T. Silva, “Managing success criteria and success factors in a BPM project: an
approach using PRINCE2 and success management on the public sector,” Cogent Business and Management, vol. 11, no. 1, 2024,
doi: 10.1080/23311975.2024.2336273.
[44] P. M. Institute, A Guide to the project management body of knowledge, vol. 34, no. 03. Pennsylvania: Project Management
Institute, Inc, 1996.
[45] H. Kerzner, “Key Performance Indicators,” in Project Management Metrics, KPIs, and Dashboards, Wiley, 2017, pp. 121–171.
[46] A. A. Khan and M. Shameem, “Multicriteria decision-making taxonomy for DevOps challenging factors using analytical
hierarchy process,” Journal of Software: Evolution and Process, vol. 32, no. 10, 2020, doi: 10.1002/smr.2263.
[47] M. A. W. Karunarathne, W. M. J. I. Wijayanayake, and A. P. K. J. Prasadika, “DevOps adoption in software development
organizations: a systematic literature review,” ICARC 2024 - 4th International Conference on Advanced Research in Computing:
Smart and Innovative Trends in Next Generation Computing Technologies, pp. 282–287, 2024, doi:
10.1109/ICARC61713.2024.10499789.
[48] N. Azad and S. Hyrynsalmi, “DevOps critical success factors-A systematic literature review,” Information and Software
Technology, vol. 157, 2023, doi: 10.1016/j.infsof.2023.107150.
[49] L. E. Lwakatare, P. Kuvaja, and M. Oivo, “Dimensions of devOps,” Lecture Notes in Business Information Processing, vol. 212,
pp. 212–217, 2015, doi: 10.1007/978-3-319-18612-2_19.
[50] P. Perera, R. Silva, and I. Perera, “Improve software quality through practicing DevOps,” 17th International Conference on
Advances in ICT for Emerging Regions, ICTer 2017 - Proceedings, vol. 2018-Janua, pp. 13–18, 2017, doi:
10.1109/ICTER.2017.8257807.

 ISSN: 2722-3221
282
[51] L. Chen, “Continuous delivery: Overcoming adoption challenges,” Journal of Systems and Software, vol. 128, pp. 72–86, 2017,
doi: 10.1016/j.jss.2017.02.013.
[52] T. C. John W. Edition, Qualitative, quantitative, and mixed methods approach. Research design qualitative quantitative and
mixed methods approach, vol. 4, no. June. Thousand Oaks, CA: Sage, 2009.
[53] M. S. Castro, B. Bahli, A. Barcaui, and R. Figueiredo, “Does one project success measure fit all? An empirical investigation of
Brazilian projects,” International Journal of Managing Projects in Business, vol. 14, no. 3, pp. 788–805, 2021, doi:
10.1108/IJMPB-01-2020-0028.
[54] S. Shayan, K. Pyung Kim, and V. W. Y. Tam, “Critical success factor analysis for effective risk management at the execution
stage of a construction project,” International Journal of Construction Management, vol. 22, no. 3, pp. 379–386, 2022, doi:
10.1080/15623599.2019.1624678.
[55] S. Subramonian, S. Alblooshi, and S. A. Husseini, “The relationship between project quality and stakeholder’s satisfaction
through project management office (PMO) in UAE construction industry,” Journal of Positive School Psychology, vol. 2022, no.
3, pp. 4787–4797, 2021, [Online]. Available: http://journalppw.com.
[56] A. Lamprou and D. G. Vagiona, “Identification and evaluation of success criteria and critical success factors in project success,”
Global Journal of Flexible Systems Management, vol. 23, no. 2, pp. 237–253, 2022, doi: 10.1007/s40171-022-00302-3.
[57] J. Chauhan, S. Subedi, A. Thakulla, and S. R. Khanal, “Project quality management,” 2023, doi: 10.1007/978-3-662-56328-1_9.
[58] J. A. V. M. K. Jayakody and W. M. J. I. Wijayanayake, “Critical success factors for DevOps adoption in information systems
development,” International Journal of Information Systems and Project Management, vol. 11, no. 3, pp. 60–82, 2023, doi:
10.12821/ijispm110304.
[59] J. Wettinger, U. Breitenbücher, and F. Leymann, “DevOpSlang - Bridging the gap between development and operations,” Lecture
Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol.
8745 LNCS, pp. 108–122, 2014, doi: 10.1007/978-3-662-44879-3_8.
[60] N. Azad, “Understanding DevOps critical success factors and organizational practices,” Proceedings - 5th International
Workshop on Software-Intensive Business: Towards Sustainable Software Business, IWSiB 2022, pp. 83–90, 2022, doi:
10.1145/3524614.3528627.
[61] M. Sánchez-Gordón and R. Colomo-Palacios, “Security as culture: a systematic literature review of DevSecOps,” Proceedings -
2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020, pp. 266–269, 2020, doi:
10.1145/3387940.3392233.
[62] M. Ajmal, M. Khan, and H. Al-Yafei, “Exploring factors behind project scope creep – stakeholders’ perspective,” International
Journal of Managing Projects in Business, vol. 13, no. 3, pp. 483–504, 2020, doi: 10.1108/IJMPB-10-2018-0228.
BIOGRAPHIES OF AUTHORS
Sani Novi Nugraheni is a master's candidate in the Department of Information
Technology at the Faculty of Computer Science, University of Indonesia. She specializes in
information technology, concentrating on IT governance, operations, security, and project
management. Additionally, she works professionally as an IT Operations Specialist at the
Ministry of Finance of Indonesia. She can be contacted at email: sani.novi@ui.ac.id or
sani.nugraheni@gmail.com.
Teguh Raharjo is a lecturer at the Faculty of Computer Science, University of
Indonesia. He does research in supply chain management, organizational studies, and business
administration. His research interests include software engineering and IT project
management. He is also a professional senior project manager at IBM Indonesia. He has
professional certifications in PMP, PMI-ACP, Prince2, Prince2 Agile, Certified Scrum Master
(CSM), and SAFe Agilist (SA). He can be contacted at email: teguhr2000@gmail.com.
Ni Wayan Trisnawaty is an information technology professional specializing in
project management and technology development. She has experience in several consultant
information technologies (IT) for various client sectors and industries. She works for an IT
consultant company focusing on technology development for the financial sector. She received
her Magister in Information Technology from the University of Indonesia. Aside from being a
professional in her field of work, she is also an assistant lecturer at the University of Indonesia
for bachelor's and master's degrees in various subject studies. She can be contacted at email:
ni.wayan05@ui.ac.id or niwayan.trisnawaty@gmail.com.

Optimizing development and operations from the project success perspective using the analytic hierarchy process

More Related Content

Similar to Optimizing development and operations from the project success perspective using the analytic hierarchy process (20)

More from CSITiaesprime (20)

Recently uploaded (20)

Optimizing development and operations from the project success perspective using the analytic hierarchy process