The Unbearable Weight of Massive CVE Numbers

We have seen the number of reported software vulnerabilities increase continuously over the last few years, leaving engineering and security teams under the constant pressure to react and deploy patches. If you run scanners to identify unpatched vulnerabilities in a large software environment, reports containing 5-or-more-digit findings are no rarity.

At the same time, you get a variety of approaches and recommendations to handle vulnerability vulnerability management if you talk to security professionals and read available standards and reports.

We will look into the various aspects of vulnerability scanning, triaging, and patching in the post below and add our own additional perspective – giving you one more to choose from.

Vulnerability Scanning

Vulnerability scanning is a broad term but has evolved to often refer to scanning for missing patches as opposed to e.g. SAST or DAST. A variety of products and open-source solutions are available to support different types of systems and environments. The technical aspects of scanning for vulnerabilities are covered in many publications, however, good reporting on identified vulnerabilities can be just as important as the actual scanning.

While most solutions use the term vulnerabilities for what they report, we like to refer to them as issues (you may also hear finding from other solutions). A missing patch means that a vulnerability exists in a piece of software that is present in your environment, however, it does not mean that there actually is a vulnerability in your environment.

The distinction comes down to whether your environment is actually exploitable via the vulnerability present in a piece of code, which depends heavily on the functions actually imported from a library, the used configuration flags, or type of input being passed to an application.

Various data points support this differentiation, for example 25,082 CVE entries are reported for 2022, however, the Known Exploited Vulnerabilities Catalog (KEV) by CISA only lists 557 entries for 2022. Niels Provos and Peiter Zatko gave a must-watch presentation on When data contradicts security best practices, indicating that timely updates of all software packages may put you at a bigger risk than updating less often! This claim can anecdotally be supported by many security practitioners who in sum have triaged tens of thousands of software vulnerabilities and ended up determining that only a very little percentage of those actually affect their environments.

There are also debatable CVE instances on a regular basis (see e.g. here and here) which illustrate the strain our vulnerability handling ecosystem is under, sometimes even leading to differing ratings for the same CVE (here vs here). Even when the scoring is consistent, the rating can be nuanced: The CVSS definition of Attack Vector indicates vulnerabilities should be rated as exploitable via network if any attack path can be constructed for remote exploitation, however, the attack path then often depends on a more niche use of the vulnerable software (good example and comment).

Finding numbers sometimes get even more inflated by scanners by reporting a certain vulnerability multiple times (e.g. because a CVE entry, a language-specific security advisory, and a Linux distribution security advisory exists and matches) or wrongly reporting certain vulnerabilities as fixable because a distribution vulnerability feed is broken.

All the factors above illustrate the challenges when scanning for vulnerabilities and, even more so, when wanting to act on the results. They are also meant to illustrate the complexity of our modern software landscape and not any individual shortcomings. Security practitioners sharing results, engineers maintaining open source, and security researchers discovering bugs put in amazing effort to make software more secure and we are excited to contribute to it.

Patching

You may have read the last paragraph and thought about reducing the effort you put into patching – and this is where we can place our plot twist! We still recommend developing a strong and timely patching culture. This is based on the simple fact that you will have to update your software at some point – there are very few exceptions to this. If you have not patched your software for a while and did not develop workflows for it, it will be an enormous effort to eventually do it and keep you from reacting quickly when it is actually needed – even if that might not happen too often. If you have strong patching and reaction capabilities, you might fall victim to short-lived backdoored packages, but you will also have the capability to react swiftly.

Regular patching will force you to develop workflows, automation, and testing for your software, essentially reducing the burden it may present. If handled in a mostly automated way or integrated into your typical engineering workflow, it may take less resources than triaging vulnerabilities individually or spending weeks on updating your software when a vulnerability is actually relevant – which often also happens at the worst of times.

Many environments also require certain metrics to be met for the number of results from vulnerability scanners. While those metrics are worth another lengthy blogpost, you often just have to deal with them and regular patching is a good way to do so.

Applying standard, short patching cycles to the majority of your software will also enable you to spend triaging efforts on the systems that you can’t patch regularly – they probably jump to your mind as you are reading this.

Triaging

When you have your patch management process and automation established, you will have some spare time on your hands (for sure there won’t be any other pressing projects?!?) to focus on triaging the remaining issues, i.e. to identify which issues you actually have to act on first.

CVE entries come with CVSS severity ratings that are the most common input for triaging processes and you are most likely familiar with those. We will describe a few additional data sources that may be helpful for your particular environment.

Already mentioned above, the Known Exploited Vulnerabilities Catalog (KEV) by CISA lists vulnerabilities for which known exploits exist, which is probably one of the strongest indicators that you need to take action. However, we also saw above that those numbers are much lower and only cover ‘known knowns’.

First’s EPSS score is attempting to fill the gap between vulnerability release and availability of a publicly known exploit: They estimate “the likelihood (probability) that a software vulnerability will be exploited in the wild”, giving an indication which vulnerabilities may soon require you to act.

This post on EPSS gives some great background information, but also recommends to pair it with “environment-specific techniques”. Those can comprise utilizing the Vulnerability Exploitability eXchange (VEX), to determine whether your specific software packages are affected by certain vulnerabilities or a custom prioritization mechanism, where you track your most exposed or business-relevant software and focus on findings in this software first.

It is often recommended to extend this custom prioritization by automated exposure analysis, where your environment is scanned from public networks to determine which software is exposed and should thus be most thoroughly triaged.

However, external exposure analysis only gives you insight into the outermost layer of your environment, and most modern application environments grow way more complicated than that.

This complexity can only be truly covered by analyzing the overall environment for exposure, blast radius, and attack chains through the complete environment: