With the rapid growth of critical software infrastructure globally and the open source components therein, likewise there is rapid growth in vulnerabilities and malicious targeting of these systems through those vulnerabilities. Governing bodies around the world have taken notice, and we’ve seen an explosion in proposed regulation and legislation around these issues.
In 2021, following the aftermath of the Log4Shell vulnerability, the US Office of the National Cyber Director (OCND), in collaboration with other government offices, established the Open Source Software Security Initiative (OS3I) interagency working group with the goal of channeling government resources to foster greater open source software security. Over the past year, OS3I identified several focus areas, including reducing the proliferation of memory-unsafe programming languages and designing implementation requirements for secure and privacy-preserving security attestations.
As part of their work, the OS3I issued a Request for Information (RFI) in August, which sought input from the public on:
- How should the Federal Government contribute to driving down the most important systemic risks in open source software?
- How can the Federal Government help foster the long-term sustainability of open source software communities?
- How should open source software security solutions be implemented from a technical and resourcing perspective?
Potential areas of focus they sought input on to address these questions included:
- Secure open-source software foundations
- Sustaining open source software communities and governance
- Behavioral and economic incentives to secure the open source software ecosystem
As one of the most preeminent and active open source software foundations in the world, the Apache Software Foundation welcomed the opportunity to submit a response based on our decades-long experience in open source software development, communities, and foundations. Our full response is also shared below. Thanks are extended to Mark Cox (Vice-President, Security), Arnout Engelen, Ruth Suehle (Executive Vice-President), Dirk-Willem van Gulik (Vice-President, Public Affairs), and Henri Yandell for their work on this response.
Secure Open Source Software Foundations
Fostering the adoption of memory-safe programming languages
Memory safety is an important property of programming languages: without it, bugs in otherwise-mundane parts of the application commonly have far-reaching consequences. Still, it is just one aspect of “defense-in-depth.” Vulnerabilities can also occur in memory-safe codebases (as was the case for the Log4Shell vulnerability). This means other techniques, such as runtime process isolation, remain important as well.
In some cases there are circumstances under which memory safety is not possible or realistic, and even when using a memory-safe language, the nature of the program may require disabling these safety features. Memory safety must be a goal, not an absolute requirement.
Replacing critical components written in memory-unsafe languages with alternatives written in memory-safe languages can reduce the attack surface. A key requirement for the long-term success and sustainability of such components is making sure they are supported by an active community that has the capacity and capability to maintain the rewritten code base.
For larger projects, rather than attempting complete, feature-by-feature rewrites, the development of alternatives that can functionally replace existing (memory-unsafe) software has a higher chance of success: aiming to keep full feature and behavioral compatibility with the old codebase would lead to a large scope for the new project and hamper opportunities for improvements that might conflict with the exact behavior of the to-be-replaced project.
More importantly, a direct feature-by-feature rewrite would require buy-in from the project that is to be rewritten, as it would require the involvement of developers who are intimately involved in both the old and the new codebases. This might be hard to achieve culturally, and even when everyone is in agreement, the time involved would stretch the existing community exceedingly thin. By contrast, a community around a new functional replacement can form naturally: interested contributors from the to-be-replaced project can be attracted to provide their subject matter expertise, and it is easier to onboard new contributors who can provide a fresh perspective on the problem space.
Because of these considerations, we recommend encouraging the development of new, independent projects in memory-safe languages (and ensuring, in parallel, sufficient engineers entering the market with the practical skills needed for this), with a focus on verification and testing on the protocol-compliance level. This is more valuable compared to low-level code refactoring and verification testing.
Reducing entire classes of vulnerabilities at scale
Creating code analysis tooling that can detect potentially problematic code patterns is one way that is used to reduce entire classes of vulnerabilities at scale. However, our experience is that such tools produce a large number of false positives, and it takes considerable further effort to then perform necessary analysis, taking into account the context and security model that applies to the software to determine whether a flagged potential problem is, in fact, an actual problem for a given project.
Flooding projects with more such potential problems (that often turn out not to be) will exhaust them, providing limited value. For this reason, initiatives that develop and run code analysis tooling should allocate significant developer resources to analyze the tool results before sharing them with the respective projects. If the project is open to funding, sufficient funding for the projects to be able to parse and act on the results is another solution.
Strengthening the software supply chain
As we develop better ways to distribute information about potential vulnerabilities, “security alert fatigue” is increasingly becoming a problem: both maintainers of software with dependencies and operators deploying it are at risk of being overwhelmed by the number of potential problems as described above. Techniques that allow all stakeholders to participate in risk assessments and share the results of that analysis, such as VEX/VDR infrastructure, will become increasingly important.
Additionally, when introducing new guidelines for strengthening the software supply chain, consideration must be given to the unintended consequences: for example, “create a new release whenever a dependency issues an advisory” might seem reasonable in theory, but in practice leads to a significant increase in the number of releases, putting a strain on the entire supply chain. For that reason, more fine-grained analysis on whether the project is, in fact, affected by the problem described in the advisory may be the only practical solution.
Sustaining Open Source Software Communities and Governance
Open source software is now inextricably a critical component of nearly all software globally. This makes sustainable open source software development crucial to society. Companies have come to rely on this software as a resource free of cost without recognizing the long-term impact of ignoring the ecosystem’s needs. Yet by and large, this software is developed wholly by volunteers.
For example, at the Apache Software Foundation (one of the oldest and largest open source foundations), all developers, board members, and officers of the foundation are volunteers. Some of them may be allowed (or even encouraged or directed) by their employers to work on open source software, but the foundation itself does not pay or direct its volunteers.
Instead it largely acts as a professional engineering body that supports a large number of individual communities to the great benefit of industry, by providing a place for volunteers to communicate and innovate. The foundation provides a home, a neutral, third-party space, for the professionals of the software industry to collaboratively work. This work includes setting a range of baseline standards, such as good release engineering, security/bug triage and followup, CVE management, responsible disclosure and downstream communication about releases, and, when needed, security advisories.
The Apache Software Foundation is by no means the only open source software foundation; many other foundations exist, with varying community and governance models.
Outside of these organized foundation spaces providing structure and financial support, one of the largest segments of the open source ecosystem is made up of small, independent developer communities. With perhaps only one or two maintainers, these projects are extremely crucial, from both economic and innovation perspectives. Should those people decide they can no longer afford (in whatever way) to keep doing the work for free, their corporate users are sometimes—often!—surprised to discover that the project they depended on is no longer updated.
While that scenario of individual developers may seem like an outlier, it is not an insignificant portion of the ecosystem. The 2022 Census II of Free and Open Source Software found that 94% of projects had fewer than ten developers accounting for more than 90% of the lines of code added. In 49 of the top 50 non-npm projects reviewed, nearly a quarter of them had only one developer responsible for more than 80% of the lines of code added.
It is vital to the sustainability of software projects and communities that the companies that leverage and take advantage of the software also actively participate in the communities that created it. While financial support can be a component, direct participation by a company’s engineers within the community that maintains the code is vastly preferred, as this drives a self-sustaining win for both sides. The community grows in terms of resources and diversity and gets firsthand, detailed information from the companies downstream on how the software is used, its problems, and their needs as users and consumers of the software. Those downstream companies get a level of understanding and ability to provide input and potentially make features they value a reality by allowing volunteers in their employ to work on the projects.
In other words, it can be much more valuable to both the companies and the open source projects for the companies to give employees the time and freedom to participate in the projects than it is to contribute money to a foundation. This is not to say that financial support is not necessary or helpful; a combination of support in both aspects is ideal.
Behavioral and Economic Incentives to Secure the Open Source Software Ecosystem
While the general intentions of Europe’s Cyber Resilience Act (CRA) and Product Liability Directive (PLD) are welcomed in general–and their much-needed bolstering of security in particular–we are concerned by the proposed specifics. Proposed regulation imposes obligations too early in the supply chain, risking stifling innovation and jeopardizing the ability of open source organizations, such as the Apache Software Foundation, to make a positive contribution by coordinating the downstream industry with regard to security.
When restrictions are necessary, they should be implemented when the software is deployed for commercial gain, rather than when it is created. Restrictions applied too early in the software lifecycle are a serious threat to the sustainability of organizations like the Apache Software Foundation, and, subsequently, these organizations’ current, proven, role in securing the supply chain. Without the governance the foundations provide to the open source communities, the industry may have no choice but to go back to developers sharing code privately and informally–if at all.
That would mean losing the well-governed structure that today ensures that security processes are followed, CVEs are filed, and responsible disclosure is adhered to. Additionally, it breaks the current proven ability of open source to act as an innovation catalyst.
In designing compensation models intended to benefit security, it is important to earmark them specifically for building security improvements. Many compensation models today focus on the creation of new tools and features, which incentivizes a minimal approach to security by not extending to the needs for security work beyond tooling. The more frequently funded “pure” security-related activities, such as audits and tooling (e.g., fuzzers and dependency scanners), may identify potential problems but leave the actual work of security improvements out of scope. This currently leaves a gap in funding for development activities that are explicitly focused on improving security (ideally done by people also intimately involved in general feature development of the project).
When regulation is imposed when the software is deployed for commercial gain, such funding is often less of an issue. The focus can then instead be on encouraging (and removing friction from) the process that sees such downstream entities contribute fixes upstream, i.e., to align with the existing win-win collaboration between industry and contributors that has made open source the dominant innovation driver in software.