Software Development Craftsmanship with Java: “The DevOps”
This is part 2 of my “Software Development Craftsmanship with Java” post series, and this time is called the “DevOps”. You can check my first post about the Software Development Craftsmanship with Java over here:
https://medium.com/@simeon.yan.iliev/software-development-craftsmanship-with-java-8dad654f2693
Intro
This post is written, based on my own experience, from the perspective of the development and delivery of Java based applications. But most likely the definitions and concepts mentioned in this post will be valid for other type of applications as well.
As a start lets try to give some basic definitions about the main topics that we will touch in the post. We will try to repeat them few times from different perspectives, because one of my goals is to help my readers to understand them well, but also to remember them. So let us start….
Continuous Integration (CI)— is the Automation to build and test applications, whenever new code commits are pushed into the code repository branch.
Or with other words, CI just basically means that the developers working code copies are synchronised with the shared from other developers mainline branch code copies, up to several times a day.
Continuous Delivery (CD)— is Continuous Integration + the Automation for the preparation of the release packages for production, and the automation for the integration tests, which means also passing the final quality assurance checks. As a result the packages for the release are delivered and ready for deployment to production. But the deployment to production itself is not done yet automatically, it is still done manually (Within CI/CD, the Releases to customers are done quite often, but are still made on demand and triggered or done fully manually).
Or with other words, CD is described as the logical evolution of Continuous Integration: Always be able to put a product/release into production! By this we just mean having a release package ready and assembled, configured, tested, prepared with the production configuration.
Continuous Deployment (CD - strange has same CD abbreviation again)— is Continuous Delivery + the Automation for the release itself to production. Means the release itself is done without any explicit human intervention (No manual preparation, No manual triggering or verification or approval steps. The Release to production is on-going automated process).
Or with other words, CD (Continuous Deployment) is described as the logical next step after Continuous Delivery: Automatically deploy the product/releases into production whenever it passes quality assurance. This includes stuff like, additional configuration of the release packages, configuration of the production environment, the implementation of the release strategy so no downtime or incidents caused etc.
Agile is focused on removing process barriers and enabling the key stakeholders: developers and customers, to collaborate more closely on accelerating delivery. Agile highlights the constancy of change and acknowledges that as software producers, we don’t often know everything we need to successfully conceive, develop, and deliver high-quality software in monolithic life cycles.
DevOps on the other hand focuses on limitations of culture and roles as agile development process. The intention of DevOps is to avoid the negative impact that overspecialisation and stove-piping roles in an organisation have on preventing rapid or even effective response to production issues. DevOps organisations break down the barriers between Operations and Engineering by cross-training each team in the other’s skills. This approach improves everyone’s ability to appreciate and participate in each other’s tasks and leads to more high-quality collaboration and more frequent communication.
You can check the final summary in the end of the post for even shorter definitions :). Or read as well the definitions I got from Amazon DevOps manual a little bit bellow in the post.
DevOps: The beginning
It become very popular, to say that the place where we all want to be is production. As well we are shifting left... Hm…, I’m quite some time in this industry, I mean in the software development, and I saw a lot of history (in context of the development and delivery of Java applications I mean)…., so why not to share a bit about it.
Long time ago, we were developing Java based software, developing the tests, and after that manually executing them. But wait, before even that we would manually find and download all the needed for our project libraries (finding manually latest or proper versions; stuff like Struts, Hibernate or any other framework). And we would add and reference them to our development projects, again manually. So running the tests, If all of them were passing OK, we would manually commit to single central code repository (not even distributed). From there someone would manually check out and release into Test/QA environment, if needed integrate with external libraries, execute manually the integration tests. And if all passed OK, somebody will manually build or trigger the building of the application (this will include the configuration and preparation of the release files, but also preparing the deployment to production, stuff like configure the Database, the Tomcat Application server, also configure the VMs, or the Physical machines, this would include the required IP, Network, Firewall etc). Once that is done, somebody will get those files, execute the configuration changes and manually apply/add/replace the packages to production environment and thus by doing this will release. Those were sweat times I remember, we were building those heavy monolithic applications…
Lets draw the main points again:
So, every arrow over there, is a lot of manual work for somebody, either for the Developers, either the Operations. Multiple times a day, or multiple times a week, or multiple times a month, somebody manually had to do that. Ok it was not that bad, for some of the tasks we had shell scripts :).
Continues Integration & Continues Delivery
At some point developers become too tired of every time configuring their dependencies and their packages and releases. So some smart people came up with the ideas of introducing of automated package management, also automated dependency management. So we suddenly all started to use tool like Maven, which provided automatic dependency management, and also providing central maven repository with all kind of open source packages, ready to be used straight away into your project. Wow it was so cool, was saving so much time in setup our projects and manage their packages and dependencies.
Once that was there somebody came with ideas of automatic the integration of code from the repositories and branches together from the different development teams, and different projects, resolve the dependencies automatically, prepare the packages automatically, but also building them, automate and run the integration tests, automatically release to the QA environment etc. Tools like Jenkins appeared. It was so cool, suddenly you could save so much work. So can you imagine: you push your code to your Git Branch, it passes the Unit tests and Code reviews, and after a while you automatically end up with a ready to release production WAR or JAR files. Almost no manual work in here.
So here how Continues Integration and Continues Delivery came into play.
Continues Deployment
Though we had Continues Integration and Continues Delivery, we were still missing something. The Production???; We still had heavy preparation for production, long planned releases, sometimes taking months, most of the times releases done manually. Means we had to prepare the Machine or the VM configurations, Database configurations/migrations, Search engine rebuild/recreation etc. It was pain, not to mention the release notes and announcements of the new release etc. You need to remember that at this point we were still building monoliths.
But something changed after that, suddenly micro-services become popular, with them the containers become popular. Suddenly we could build and screw up and build again the entire application setup (I mean the entire environment) for like less than 5 minutes. Cloud solutions become good enough and also popular…., So we suddenly saw the possibilities to automate the release to production as well. I mean, why not, you can build production release already with the Continues Delivery, now we can add the missing point. So now due to micro-services and containerisation, we could easily (by easily means without causing incidents) release small pieces of our applications, quite often. So at this point of time we have CI/CD/CD :).
DevOps
Having automated all the work for the integration, testing, delivery and deployment of our environments and applications, we didn’t need that much Operations guys anymore in terms of classical infrastructure provisioning. Classical operations work don’t exist anymore on the scale it was needed in the past. Having Cloud to provision infrastructure automatically, using containerisation to deliver the micro-service based applications to production easy and fast in automated manner. Operations guys should deal more with automation, means scripting and programming. Skills for tools like Ansible automation, Python scripting, or Terraform orchestration, had to be added to the Shell scripting knowledge they already had. The Operations guys, had to change, and had to be moved into the development teams, so now Developers and Operations guys have to share the same responsibility and work for the entire process. So here how we came to Developer(Dev) and Operations(Ops) type of teams, we are having in right now.
NoOps
Some people say that the future is in NoOps. With the introduction of the server-less programming and advanced PaaS solutions where the entire infra and ops work being handled by automation of the platform itself, the classical Ops work will be minimised to a point where it will be needed only for the PaaS and Servless platform providers. But let see what the future will bring, and how we will get there.
Some useful quotations about CI/CD/CD from Amazon DevOps documentation*:
“Continuous integration (CI) is a software development practice where developers regularly merge their code changes into a central repository, after which automated builds and tests are run. CI most often refers to the build or integration stage of the software release process and requires both an automation component (e.g., a CI or build service) and a cultural component (e.g., learning to integrate frequently). The key goals of CI are to find and address bugs more quickly, improve software quality, and reduce the time it takes to validate and release new software updates.”
“Continuous delivery (CD) is a software development practice where code changes are automatically built, tested, and prepared for production release. It expands on continuous integration by deploying all code changes to a testing environment, a production environment, or both after the build stage has been completed. Continuous delivery can be fully automated with a workflow process or partially automated with manual steps at critical points. When continuous delivery is properly implemented, developers always have a deployment-ready build artifact that has passed through a standardized test process.”
“With Continuous deployment, revisions are deployed to a production environment automatically without explicit approval from a developer, making the entire software release process automated. This, in turn, allows for a continuous customer feedback loop early in the product lifecycle. Continuous Delivery Is Not Continuous Deployment
One misconception about continuous delivery is that it means every change committed is applied to production immediately after passing automated tests. However, the point of continuous delivery is not to apply every change to production immediately, but to ensure that every change is ready to go to production. Before deploying a change to production, you can implement a decision process to ensure that the production deployment is authorized and audited. This decision can be made by a person and then executed by the tooling.”“DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity. Using DevOps principles, organizations can evolve and improve products at a faster pace than organizations that use traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.”
Final Summary
Here’s a quick and easy way to differentiate Agile, DevOps, and CI/CD/CD:
- CI/CD focuses on software-defined life cycles highlighting tools that emphasise automation.
- CD (Continues Deployment) focuses on the regular and automated delivery practices to production.
- Agile focuses on processes highlighting change while accelerating delivery.
- DevOps focuses on culture highlighting roles that emphasise responsiveness.
- NoOps is associated with PaaS clouds and Server-less in which case we do not need anymore classical operations work to deliver applications.
