Lessons from Log4Shell: Building a CRA-Ready Log4j

By: Piotr P. Karwasz, VP Logging, Apache Software Foundation

The disclosure of Log4Shell (CVE-2021-44228) on December 9, 2021 did not just expose a vulnerability: it exposed a way of building software that was no longer fit for purpose, and it helped bring the European Cyber Resilience Act into being.

I recently hosted a session for the Open Regulatory Compliance community’s CRA Monday series to tell the story from the inside: what the Apache Logging team actually did in the years after Log4Shell to rebuild the project as something CRA-ready.

This blog recaps and expands upon that session; you can also watch the recording or view the slides.

A Wake-Up Call for the Software Ecosystem

Lessons from Log4j perspective

Since Log4j is mostly consumed as a dependency rather than built upon, the lessons the Apache Software Foundation’s Logging Services team drew from Log4Shell were different from those of the broader ecosystem. The problems were not about visibility into our own dependencies, but about the state of the project itself:

• Documentation was hard to navigate, with many features either undocumented or described only in terms a new contributor could not act on.
• The release process was antiquated, understood by only a handful of people, and run on personal hardware: a single point of failure that nobody had reason to address until a crisis made it unavoidable.
• Builds were slow and tests were flaky, meaning a failure late in a multi-hour process sent you back to the beginning.

None of these were unique to Log4j. Log4Shell made them impossible to ignore, and addressing them put us on a path that anticipates much of what the CRA now asks of software maintainers.

Documentation: from maintainer knowledge to public record

Logging is not always safe. There are real security concerns: CRLF injection from unstructured logging; sensitive information leaking into debug output; and injection of Log4j formatting patterns through user-supplied strings. Before Log4Shell, much of this knowledge lived in the heads of a few maintainers: not written down, not discoverable, and not actionable for the thousands of teams depending on the library.

We rewrote the documentation website from scratch. The goal was to turn that private knowledge base into a public record by:

• Covering security best practices and an explicit security model
• Providing reference documentation generated directly from code, so it stays in sync as the library evolves
• Making Log4j’s versioning policy and support status explicit and visible, both required for CRA attestationsMoving the issue tracker from JIRA closer to the code in GitHub Issues
• Mirroring some discussions on both GitHub Discussions and mailing lists 

The results were measurable: more documentation pull requests, more site visits, a useful proxy for coverage and clarity, and noticeably better answers from LLMs trained on our new content.

Release process: from manual to reproducible

In December 2021, Log4j’s tests ran on a Jenkins instance, binaries were built on maintainer machines, signing was manual, and builds were not reproducible. A full binary and site build literally took hours. This was not unusual for open source projects, but it created real risks around build integrity, and it was clearly not sustainable.

By September 2024 we had migrated to GitHub Actions, achieved reproducible builds signed by a CI GPG key only known to ASF admins, parallelized tests, and reduced the build-and-deploy cycle to around 30 minutes.

Currently:

• The CI pipeline now automatically stages releases up to the voting phase: the first project in The ASF to do this.
• We are working on integration with Apache Trusted Releases, which will bring automation to the voting and publishing steps as well.
• We are working on full-SLSA build and source attestations, which will make us one of the first ASF projects to achieve this. This includes SLSA source level 4, requiring a non-author review for every commit: a critical guarantee for a project at the center of the most significant supply-chain incident in recent memory.

Machine-readable metadata: SBOMs, VEX, and beyond

One of the most concrete CRA requirements is the expectation that software comes with machine-readable security information. We now publish CycloneDX SBOMs to Maven Central, which references a Vulnerability Disclosure Report, a machine-readable version of our CVE list, on our website. This gives downstream users a complete, well-curated source of vulnerability information, unaffected by the data loss that public vulnerability databases sometimes introduce when converting between formats. It is also open to improvements by contributors.

The next step is Vulnerability Exploitability eXchange (VEX) statements, generated automatically through an open source toolset we are developing with OpenRefactory. The system combines:

• An AI-backed Root Cause Service that identifies vulnerable methods
• A Call Graph Service that maps per-component call graphs
• A VEX Generation Service that determines the maximum reachable path and generates enriched VEX statements, which we call VEXplanations

We are currently testing this within Apache Solr and plan to extend it to Log4j and Commons. The goal is to give downstream users a machine-readable guarantee of no known exploitable vulnerabilities, assessed automatically rather than by hand.

We are also planning to support Common Lifecycle Enumeration (ECMA-428), the machine-readable equivalent of our supported versions list, generated ASF-wide through Apache Trusted Releases.

Vulnerability handling: from ad hoc to structured

Since Log4Shell, the Logging Services team has put in place several structural improvements to vulnerability handling:

• A dedicated reporting address (security@logging.apache.org) separate from the general ASF security team
• An explicit threat model that clearly separates the security responsibilities of Log4j from those of its consumers
• A bug bounty program hosted on YesWeHack, funded by the Sovereign Tech Resilience program

Since July 2024, the program has received 140 reports across Log4cxx, Log4j, and Log4net, yielding 10 CVEs.

At the architectural level, Log4j 3 addresses the attack surface problem directly. Log4j 2 was built for a pre-Maven world and shipped as a monolithic core with many optional dependencies bundled in. Log4j 3 modularises most of that core, so each module only pulls in what it actually needs. Smaller surface area means fewer things that can go wrong.

Community sustainability: the hardest and unsolved problem

All of the above is meaningless if the project burns out. And that risk is real. Log4j currently has two active maintainers doing most of the work. Log4cxx and Log4net are in a similar position. This is not a criticism of the community. It is the structural reality of most open source projects, and it is the problem that CRA compliance pressure will make worse if it is not addressed alongside the technical work.

We are exploring two directions:

• The Open Source Economy initiative: offering consulting and compliance attestations as a funded model, where fees support maintainers, infrastructure, and upstream dependencies.
• ECMA TC-54’s CONTRIBUTING.yaml specification: a machine-readable format for describing a project’s maintenance status, contribution needs, and support expectations, so that users and organisations can understand what they are depending on.

There is also a cultural dimension. Many open source communities grew up in an era when maintainers could commit directly to the repository and build software on their own machines. Shifting to a model where maintainers review all contributions and CI builds everything is the right move for security, but it is genuinely less fun, and communities resist it. Making that transition while keeping contributors engaged is one of the real challenges of building a CRA-ready project.

What CRA readiness actually looks like

The CRA introduces a set of “due diligence” requirements for manufacturers and voluntary attestations for OSS projects. What I hope this post makes clear is that meeting them is not a compliance exercise you bolt on at the end. It is the output of years of work on documentation, build integrity, machine-readable metadata, vulnerability processes, and community health.

The good news is that the direction was right before the regulation arrived. Log4Shell forced hard questions that led, eventually, to a project that is genuinely more secure, more transparent, and more sustainable than it was in 2021. The CRA gives that work a formal framework and, ideally, the organisational backing to fund it. For other open source projects facing the same pressures, that is perhaps the most useful takeaway: the path to CRA readiness and the path to a healthier, more sustainable project are the same path.

Get involved

The path to a CRA-ready ecosystem is not walked by one project. If any of this matters to you, here is where to start:

• Contribute to Log4j directly. Code, documentation, and review all count. Start at logging.apache.org, or report security issues to security@logging.apache.org.
• Help across the ASF. The Contributing to Apache Security guide is the way in.
• Shape OSS regulatory response. The Open Regulatory Compliance Working Group is where CRA implementation for open source is being worked out in public.
• Define what sustainable maintenance means. ECMA TC54 TG4 is building the machine-readable standards for project health and support.
• Secure the supply chain. The SLSA Community is advancing the build-integrity framework Log4j will soon use.