CVE-2022-49394: Vulnerability in Linux Linux

Severity: highType: vulnerabilityCVE-2022-49394

In the Linux kernel, the following vulnerability has been resolved: blk-iolatency: Fix inflight count imbalances and IO hangs on offline iolatency needs to track the number of inflight IOs per cgroup. As this tracking can be expensive, it is disabled when no cgroup has iolatency configured for the device. To ensure that the inflight counters stay balanced, iolatency_set_limit() freezes the request_queue while manipulating the enabled counter, which ensures that no IO is in flight and thus all counters are zero. Unfortunately, iolatency_set_limit() isn't the only place where the enabled counter is manipulated. iolatency_pd_offline() can also dec the counter and trigger disabling. As this disabling happens without freezing the q, this can easily happen while some IOs are in flight and thus leak the counts. This can be easily demonstrated by turning on iolatency on an one empty cgroup while IOs are in flight in other cgroups and then removing the cgroup. Note that iolatency shouldn't have been enabled elsewhere in the system to ensure that removing the cgroup disables iolatency for the whole device. The following keeps flipping on and off iolatency on sda: echo +io > /sys/fs/cgroup/cgroup.subtree_control while true; do mkdir -p /sys/fs/cgroup/test echo '8:0 target=100000' > /sys/fs/cgroup/test/io.latency sleep 1 rmdir /sys/fs/cgroup/test sleep 1 done and there's concurrent fio generating direct rand reads: fio --name test --filename=/dev/sda --direct=1 --rw=randread \ --runtime=600 --time_based --iodepth=256 --numjobs=4 --bs=4k while monitoring with the following drgn script: while True: for css in css_for_each_descendant_pre(prog['blkcg_root'].css.address_of_()): for pos in hlist_for_each(container_of(css, 'struct blkcg', 'css').blkg_list): blkg = container_of(pos, 'struct blkcg_gq', 'blkcg_node') pd = blkg.pd[prog['blkcg_policy_iolatency'].plid] if pd.value_() == 0: continue iolat = container_of(pd, 'struct iolatency_grp', 'pd') inflight = iolat.rq_wait.inflight.counter.value_() if inflight: print(f'inflight={inflight} {disk_name(blkg.q.disk).decode("utf-8")} ' f'{cgroup_path(css.cgroup).decode("utf-8")}') time.sleep(1) The monitoring output looks like the following: inflight=1 sda /user.slice inflight=1 sda /user.slice ... inflight=14 sda /user.slice inflight=13 sda /user.slice inflight=17 sda /user.slice inflight=15 sda /user.slice inflight=18 sda /user.slice inflight=17 sda /user.slice inflight=20 sda /user.slice inflight=19 sda /user.slice <- fio stopped, inflight stuck at 19 inflight=19 sda /user.slice inflight=19 sda /user.slice If a cgroup with stuck inflight ends up getting throttled, the throttled IOs will never get issued as there's no completion event to wake it up leading to an indefinite hang. This patch fixes the bug by unifying enable handling into a work item which is automatically kicked off from iolatency_set_min_lat_nsec() which is called from both iolatency_set_limit() and iolatency_pd_offline() paths. Punting to a work item is necessary as iolatency_pd_offline() is called under spinlocks while freezing a request_queue requires a sleepable context. This also simplifies the code reducing LOC sans the comments and avoids the unnecessary freezes which were happening whenever a cgroup's latency target is newly set or cleared.

AI Analysis

Technical Summary

CVE-2022-49394 is a vulnerability in the Linux kernel's blk-iolatency subsystem, which manages IO latency control per cgroup. The vulnerability arises from an imbalance in tracking the number of inflight IO operations when iolatency is enabled or disabled dynamically. Specifically, the function iolatency_set_limit() correctly freezes the request queue to ensure all inflight IO counters are zero before changing the enabled state. However, iolatency_pd_offline() also decrements the enabled counter and can disable iolatency without freezing the queue, leading to inflight IO counts leaking and becoming inconsistent. This inconsistency can cause IO operations to hang indefinitely, especially when a cgroup with stuck inflight IOs is throttled, as the completion events needed to wake IOs never occur. The issue can be reproduced by rapidly enabling and disabling iolatency on a device (e.g., /dev/sda) while concurrent IO workloads run, causing inflight counters to become stuck and IOs to hang. The patch resolves this by unifying enable/disable handling into a work item triggered from iolatency_set_min_lat_nsec(), which is called from both affected code paths. This approach avoids freezing the request queue under spinlocks and simplifies the code, preventing inflight count imbalances and IO hangs. The vulnerability affects multiple Linux kernel versions as identified by commit hashes. No known exploits are reported in the wild, and no CVSS score is assigned yet.

Potential Impact

For European organizations relying on Linux servers, especially those using cgroup-based IO latency controls (common in containerized environments, cloud infrastructure, and multi-tenant systems), this vulnerability can cause IO operations to hang indefinitely. This can lead to degraded system performance, application stalls, and potential denial of service conditions impacting critical services. Systems performing high IO workloads with cgroup latency controls enabled are particularly at risk. The hang can affect availability and operational continuity, potentially disrupting business-critical applications and services. Since Linux is widely deployed across European enterprises, cloud providers, and public sector infrastructure, the impact could be significant if unpatched kernels are used in production environments. However, exploitation requires specific conditions (enabling/disabling iolatency on cgroups with inflight IO), so the risk is more pronounced in environments actively using these kernel features.

Mitigation Recommendations

1. Apply the official Linux kernel patch that fixes the iolatency inflight count imbalance as soon as it becomes available for your distribution. 2. Audit and monitor usage of blk-iolatency cgroup controls on your systems; avoid dynamically enabling/disabling iolatency on devices with active IO workloads. 3. Implement monitoring for stuck inflight IO counters or IO hangs related to cgroup latency controls using kernel tracing or custom scripts similar to the provided drgn example. 4. For containerized or virtualized environments, review cgroup configurations to minimize unnecessary toggling of IO latency controls. 5. Test kernel updates in staging environments to ensure stability of IO latency features before production rollout. 6. Engage with Linux distribution vendors for backported fixes if using long-term support kernels. 7. Consider fallback IO scheduling or throttling mechanisms if iolatency features are not critical, to reduce exposure.