Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
📝 Abstract
Context: Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in and literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. Objective: This research aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. Method: We used systematic literature review (SLR) method for reviewing the peer-reviewed papers on continuous practices published between 2004 and 1st June 2016. We applied thematic analysis method for analysing the data extracted from reviewing 69 papers selected using predefined criteria. Results: We have identified thirty approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) “reducing build and test time in continuous integration (CI)”; (2) “increasing visibility and awareness on build and test results in CI”; (3) “supporting (semi-) automated continuous testing”; (4) “detecting violations, flaws and faults in CI”; (5) “addressing security and scalability issues in deployment pipeline”, and (6) “improving dependability and reliability of deployment process”. We have also determined a list of critical factors such as “testing (effort and time)”, “team awareness and transparency”, “good design principles”, “customer”, “highly skilled and motivated team”, “application domain”, and “appropriate infrastructure” that should be carefully considered when introducing continuous practices in a given organization.
💡 Analysis
Context: Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in and literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. Objective: This research aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. Method: We used systematic literature review (SLR) method for reviewing the peer-reviewed papers on continuous practices published between 2004 and 1st June 2016. We applied thematic analysis method for analysing the data extracted from reviewing 69 papers selected using predefined criteria. Results: We have identified thirty approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) “reducing build and test time in continuous integration (CI)”; (2) “increasing visibility and awareness on build and test results in CI”; (3) “supporting (semi-) automated continuous testing”; (4) “detecting violations, flaws and faults in CI”; (5) “addressing security and scalability issues in deployment pipeline”, and (6) “improving dependability and reliability of deployment process”. We have also determined a list of critical factors such as “testing (effort and time)”, “team awareness and transparency”, “good design principles”, “customer”, “highly skilled and motivated team”, “application domain”, and “appropriate infrastructure” that should be carefully considered when introducing continuous practices in a given organization.
📄 Content
1
Continuous Integration, Delivery and Deployment: A Systematic
Review on Approaches, Tools, Challenges and Practices
Mojtaba Shahina, Muhammad Ali Babara, Liming Zhub
a CREST – The Centre for Research on Engineering Software Technologies, The University of Adelaide, Australia
b Data61, Commonwealth Scientific and Industrial Research Organisation, Sydney, NSW 2015, Australia
mojtaba.shahin@adelaide.edu.au, ali.babar@adelaide.edu.au, liming.zhu@data61.csiro.au
Abstract—Context: Continuous practices, i.e., continuous
integration, delivery, and deployment, are the software
development industry practices that enable organizations to
frequently and reliably release new features and products.
With the increasing interest in and literature on continuous
practices, it is important to systematically review and
synthesize the approaches, tools, challenges, and practices
reported for adopting and implementing continuous practices.
Objective: This research aimed at systematically reviewing
the state of the art of continuous practices to classify
approaches and tools, identify challenges and practices in this
regard, and identify the gaps for future research.
Method: We used systematic literature review (SLR) method
for reviewing the peer-reviewed papers on continuous
practices published between 2004 and 1st June 2016. We
applied thematic analysis method for analysing the data
extracted from reviewing 69 papers selected using predefined
criteria.
Results: We have identified thirty approaches and associated
tools, which facilitate the implementation of continuous
practices in the following ways: (1) ―reducing build and test
time in continuous integration (CI)‖; (2) ―increasing visibility
and awareness on build and test results in CI‖; (3)
―supporting (semi-) automated continuous testing‖; (4)
―detecting violations, flaws and faults in CI‖; (5) ―addressing
security and scalability issues in deployment pipeline‖, and
(6) ―improving dependability and reliability of deployment
process‖. We have also determined a list of critical factors
such as ―testing (effort and time)‖, ―team awareness and
transparency‖, ―good design principles‖, ―customer‖, ―highly
skilled and motivated team‖, ―application domain‖, and
―appropriate
infrastructure‖
that
should
be
carefully
considered when introducing continuous practices in a given
organization. The majority of the reviewed papers were
validation (34.7%) and evaluation (36.2%) research types.
This review also reveals that continuous practices have been
successfully applied to both greenfield and maintenance
projects.
Conclusion: Continuous practices have become an important
area of software engineering research and practice. Whilst the
reported approaches, tools, and practices are addressing a
wide range of challenges, there are several challenges and
gaps which require future research work for: improving the
capturing and reporting of contextual information in the
studies reporting different aspects of continuous practices;
gaining a deep understanding of how software-intensive
systems should be (re-) architected to support continuous
practices; addressing the lack of knowledge and tools for
engineering processes of designing and running secure
deployment pipelines.
Index Terms— continuous integration, continuous delivery,
continuous deployment, continuous software engineering,
systematic literature review, empirical software engineering
I. INTRODUCTION
With
increasing
competition
in
software
market,
organizations pay significant attention and allocate resources
to develop and deliver high-quality software at much
accelerated pace [1]. Continuous Integration (CI), Continuous
DElivery (CDE), and Continuous Deployment (CD), called
continuous practices for this study, are some of the practices
aimed
at
helping
organisations
to
accelerate
their
development and delivery of software features without
compromising quality [2]. Whilst CI advocates integrating
work-in-progress multiple times per day, CDE and CD are
about ability to quickly and reliably release values to
customers by bringing automation support as much as
possible [3, 4].
Continuous practices are expected to provide several
benefits such as: (1) getting more and quick feedback from
the software development process and customers; (2) having
frequent and reliable releases, which lead to improved
customer satisfaction and product quality; (3) through CD,
the connection between development and operations teams is
strengthened and manual tasks can be eliminated [5, 6]. A
growing number of industrial cases indicate that the
continuous practices are making inroad in software
development industrial practices across various domains and
sizes of organizations [5, 7, 8]. At the same time, adopting
continuous practices is not a trivial task since organizational
processes, practices, and tool may not be ready to support the
highly complex
This content is AI-processed based on ArXiv data.