By: Rajiv Kadayam
Technology processes are changing just as rapidly as technology itself. DevSecOps, even with its limited presence in the federal government, has dramatically accelerated the development and delivery of software at agencies like the U.S. Department of Homeland Security (DHS) and the General Services Administration. The question is no longer “why” but “how” and “how quickly” can DevSecOps be adopted and scaled across an organization to accelerate delivery of secure IT solutions to customers and citizens.
Federal agencies classify procurement of IT services under two major categories:[1] development, modernization and enhancement (DME) and operations and maintenance (O&M). Back in the day of the waterfall software development life cycle (SDLC), vendors segmented the work according to DME and O&M. This practice influenced the federal contracting process, and we started to see procurements and performance work statements labeled as such. Per the White House report on 2018 IT spending, of the $48.5 billion reported to the IT Dashboard, 78.5% ($38.1 billion) is O&M spending — a staggering portion of the budget.
Federal agencies have an opportunity to stem this disproportionate spending trend and upgrade their traditional DME and O&M models to a modern DevSecOps approach. O&M is focused on ‘keeping the lights on’ and status quo, where DevSecOps incentivizes teams to be innovative, therefore, extracting greater value from IT dollars spent. The benefits of a DevSecOps model are overwhelming and undeniable: gaining more value and capabilities from IT investments, a boost in mission performance, the delivery of new capabilities faster, the ability to deliver digital services at speed and with quality, the integration of security from inception in the SDLC, more time to innovate rather than fix and maintain — the list goes on.So, how should agencies begin their journey to DevSecOps? This white paper lays out the milestones to reach for new and early adopters of DevSecOps.
Learn, Assess and Set Goals
LEARN. DevSecOps breaks down the artificial silos between development, testing, security, operations and other teams and enables free and open collaboration to promote faster delivery of secure and reliable software. No one is better suited to determine the DevSecOps strategy for your organization than you and your team, who have an immersive understanding of the business, culture and competency. Before you start the strategy, do some research about DevSecOps. The following resources can help you get started:
- Build your understanding and awareness of DevSecOps concepts by reading books such as The DevSecOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations by Gene Kim, Jez Humble and Patrick Debois.
- Seek out DevSecOps communities as a source for learning, inspiration and ideas to help you get started on your DevSecOps journey. Find and attend a local DevSecOpsDays meetup. For example, at least three chapters operate in the Washington, D.C., metropolitan area: NOVA DevSecOps, DevSecOps DC and the Baltimore DevSecOps Meetup. DevSecOpsDays.org organizes two-day conferences across the country every year.
- Take a course in DevSecOps. MindMajix offers a highly rated course.
ASSESS. Evaluate the current IT delivery model your organization uses. Document the challenges and problems that adoption of DevSecOps could solve. Interview your team members and other stakeholders to gain a more objective, 360-degree view of the constraints in your existing IT delivery model; identify opportunities for improvement; and isolate criteria for success. This formal assessment will help you build organizational momentum to disrupt the status quo.
SET GOALS. Defining measurable goals and criteria for success is essential for motivating the team to achieve a targeted objective. Capture simple metrics that are important to your team, such as improving the frequency of releases to production and mean time to resolution. Once you have consensus on the goals and objectives of your DevSecOps transformation, it’s time to plan and implement changes.
Practice Being Agile
Before you can embrace DevSecOps, you must practice being lean and agile. Start with the basic Scrum methodology, which is ideal for development-focused projects because it instills a time-boxed software development process with regular and frequent delivery of new releases to promote faster feedback and learning. For O&M projects that focus more on sustainment, Kanban may be better because it provides visible work-in-progress queues, enabling the team to track work assignments, identify bottlenecks, measure throughput and optimize delivery. Beyond these prescriptive methods, it is more important to practice “being agile” rather than “doing agile,” where the emphasis is on delivering value rather than perfectly following an Agile method.
Establish a Culture of Collaboration
The importance of creating a culture that embodies collaboration, cooperation and a collective mindset cannot be overemphasized in a DevSecOps transformation effort. These fundamental and innate human qualities must be unlocked before any organization can succeed in its mission. Legacy IT delivery models created silo mindsets by organizing teams for every phase of the SDLC. Dev teams “threw over the fence” deployment instructions to operations teams. Great software does not get built by throwing great ideas over any fence to any team to build. Great software is possible only through continuous interaction, experimentation, feedback and learning, made possible by a DevSecOps mindset and culture. DevSecOps disrupts the siloed model by reorganizing and empowering teams to be responsible and accountable for the end-to-end SDLC for one or more systems. In doing so, DevSecOps challenges teams to work together more effectively to solve problems and more quickly deliver fixes, new features and capabilities.
Automate Your Pipeline
Yet another crucial ingredient for meeting the increasing expectations on speed, security and quality of software delivery is automation. Although an overly simplified term, “automation” embodies modern software development and agile engineering techniques such as:
- Test-driven development
- Continuous integration (CI)/continuous delivery (CD)
- Infrastructure as Code
- Security Compliance as Code
Start by automating the processes closest to production, such as deployment and operations. In this way, you force collaboration between development and operations teams. Teams can then progressively automate repetitive functions left in the pipeline, including unit, integration and user interface tests; software builds; and code inspection. These techniques help teams “shift left” and “fail early” to minimize the cost of remediation and the chances of recurrence. A holistic investment in culture, people and automation is a fundamental necessity for organizations aspiring to adopt and benefit from DevSecOps.
Pilot DevSecOps
DevSecOps is all about learning through practice, and early adopters can begin by conducting small, short-term pilots. Follow these steps to pilot DevSecOps in your organization:
- Identify two medium-sized, medium-risk projects as a prototype in which teams can experiment with DevSecOps practices over two to three months. The pilot will provide an opportunity to learn, gauge feasibility and identify areas for improvement.
- Form an integrated, cross-functional team, and implement basic automation, such as CI, automatic builds and automated testing processes, to progressively mature your CD, eventually automating deployment and operations. Ultimately, the objective is to deliver meaningful value to customers through functional, secure, high-quality software iteratively delivered at high velocity. A key mantra of DevSecOps is CI, which means having the ability to quantifiably measure processes, quality, security and operations. This measurement then becomes the baseline on which teams can improve.
- At the end of the pilot, document experiences, techniques and approaches that were successful, and find opportunities for improvement. The documentation helps inform the approach and next steps for further adoption of DevSecOps in more projects and across the organization.
Develop a DevSecOps Playbook, and Scale Adoption
The playbook provides an opportunity to standardize practices not only from a technical perspective but also from a procurement perspective. DevSecOps is not just a blend of development and operations; it is an infusion of every function of the organization. For federal agencies that contract out most of their IT needs, acquisitions and procurement play a key role. Procurement professionals must have a seat at the table to help bridge the legal chasm that often exists between procurement policies and the dynamics of a DevSecOps project.
How should we define “acceptable performance?” How do we determine and measure quality? These are key questions that DevSecOps can help answer to ensure consistent adoption across the organization. Socialize your DevSecOps Playbook with the rest of the organization, plan broadscale adoption, and iteratively refine and evolve the playbook by incorporating innovation and approaches discovered during the transformation journey.
Evolve From DevSecOps to DevSecOps
DevSecOps is a portmanteau of development and operations, but it encompasses every aspect of the development and delivery process, including requirements, design, all forms of testing and security. DevSecOps naturally evolved into DevSecOps to emphasize and treat security as a first-class citizen. DevSecOps mandates that security engineers be part of the cross-functional team, taking full responsibility for:
- Securing engineering of the applications, systems and related security testing up front.
- Ensuring compliance with required National Institute of Standards and Technology Special Publication 800-53 and similar security controls and securing authority to operate (ATO).
- Ongoing protection of systems and data through continuous security monitoring.
- Responding to emerging threats and reacting to security attacks and breaches.
DevSecOps does not guarantee fool-proof security for your applications and data —nothing does. Rather, it helps instill a security best practices discipline that is enabled through design and code. Throughout the development life cycle, security is embedded in the following ways:
- In the design phase, system and software architecture is engineered for security protection, adhering to mandated security controls, established industry standards and emerging practices.
- During the SDLC, code is regularly tested for potential vulnerabilities and iteratively remediated before it is deployed to production. Security and compliance testing, automated through code, helps catch security regressions and automates compliance verification, thereby speeding up the assessment and accreditation as well as ATO processes.
- In post deployment and production, environments are continuously monitored for vulnerabilities, suspicious behaviors and imminent attacks to enable early detection, faster response and restoration of security protection. DevSecOps empowers teams to operate at high velocity, which is essential for securing and protecting federal information systems.
Fuel the Journey and Keep Moving
It is important to understand that DevSecOps is a journey that never ends; therefore, it requires a regular supply of fuel to maintain momentum. Like snowflakes, no two DevSecOps organizations, at a point in time, should look the same because they are constantly exploring how to get better and faster at delivering value. To facilitate this evolution, a safe space for sharing ideas through demonstrations, brown-bag lunches and presentations will inspire, promote collaboration, and generate new and better approaches for delivering greater value to customers. Sharing at a philosophical level is a cognitive exercise that can facilitate organizational learning and trigger thought processes to generate innovative ideas. Through these discoveries, organizations can ensure that they are constantly moving and getting better at their DevSecOps craft.
Carpe Diem!
We are at a tipping point: Aging infrastructure, systems and processes pose an existential risk for federal agencies in executing their mission. The Technology Modernization Fund provides a real opportunity not only to modernize IT systems but also to modernize the processes by which IT is built and delivered. Using the steps and best practices discussed in this white paper, Pyramid Systems successfully transformed its projects at DHS and a financial regulatory agency to a DevSecOps model. These efforts resulted in savings of more than $1 million per year, achieved more than 90% sustainable code coverage, enabled daily deployments and improved overall productivity by an order of magnitude. We are continuing that journey to elevate mission capabilities and improve outcomes for our customers.
If you are interested in learning more about our services and how we can help you kickstart your DevSecOps journey, please email info@pyramidsystems.com.
DOWNLOAD OUR INFOGRAPHIC
About Pyramid
Pyramid Systems is an award-winning small business technology solutions company. We drive digital transformation at federal agencies through modernization, DevSecOps, cloud, analytics and biometrics services. Our talented agile engineers apply human-centered design and modern software development techniques to produce next-generation solutions. Pyramid’s proven methods and tools empower forward-thinking innovations; accelerate delivery of production-ready software; and create secure, high-quality solutions that last. For more information about Pyramid Systems, visit www.pyramidsystems.com.
2677 Prosperity Ave., Suite 700, Fairfax, VA 22031 | P: 703-553-0800 | www.pyramidsystems.com
[1] DME is typically a discrete contract that involves the development of new systems or the addition of new capabilities to an existing system through custom software development, commercial off-the-shelf software or low code platforms. The second category is the ubiquitous O&M contract, which is much broader in scope and involves sustainment of hardware, software, network, applications, security and most other IT assets. In the context of applications and systems, O&M typically involves maintenance of existing code and data, managing production and nonproduction environments and a help desk.