Kyle Swenson | 8d8f654 | 2021-03-15 11:02:55 -0600 | [diff] [blame] | 1 | DMA Buffer Sharing API Guide |
| 2 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 3 | |
| 4 | Sumit Semwal |
| 5 | <sumit dot semwal at linaro dot org> |
| 6 | <sumit dot semwal at ti dot com> |
| 7 | |
| 8 | This document serves as a guide to device-driver writers on what is the dma-buf |
| 9 | buffer sharing API, how to use it for exporting and using shared buffers. |
| 10 | |
| 11 | Any device driver which wishes to be a part of DMA buffer sharing, can do so as |
| 12 | either the 'exporter' of buffers, or the 'user' of buffers. |
| 13 | |
| 14 | Say a driver A wants to use buffers created by driver B, then we call B as the |
| 15 | exporter, and A as buffer-user. |
| 16 | |
| 17 | The exporter |
| 18 | - implements and manages operations[1] for the buffer |
| 19 | - allows other users to share the buffer by using dma_buf sharing APIs, |
| 20 | - manages the details of buffer allocation, |
| 21 | - decides about the actual backing storage where this allocation happens, |
| 22 | - takes care of any migration of scatterlist - for all (shared) users of this |
| 23 | buffer, |
| 24 | |
| 25 | The buffer-user |
| 26 | - is one of (many) sharing users of the buffer. |
| 27 | - doesn't need to worry about how the buffer is allocated, or where. |
| 28 | - needs a mechanism to get access to the scatterlist that makes up this buffer |
| 29 | in memory, mapped into its own address space, so it can access the same area |
| 30 | of memory. |
| 31 | |
| 32 | dma-buf operations for device dma only |
| 33 | -------------------------------------- |
| 34 | |
| 35 | The dma_buf buffer sharing API usage contains the following steps: |
| 36 | |
| 37 | 1. Exporter announces that it wishes to export a buffer |
| 38 | 2. Userspace gets the file descriptor associated with the exported buffer, and |
| 39 | passes it around to potential buffer-users based on use case |
| 40 | 3. Each buffer-user 'connects' itself to the buffer |
| 41 | 4. When needed, buffer-user requests access to the buffer from exporter |
| 42 | 5. When finished with its use, the buffer-user notifies end-of-DMA to exporter |
| 43 | 6. when buffer-user is done using this buffer completely, it 'disconnects' |
| 44 | itself from the buffer. |
| 45 | |
| 46 | |
| 47 | 1. Exporter's announcement of buffer export |
| 48 | |
| 49 | The buffer exporter announces its wish to export a buffer. In this, it |
| 50 | connects its own private buffer data, provides implementation for operations |
| 51 | that can be performed on the exported dma_buf, and flags for the file |
| 52 | associated with this buffer. All these fields are filled in struct |
| 53 | dma_buf_export_info, defined via the DEFINE_DMA_BUF_EXPORT_INFO macro. |
| 54 | |
| 55 | Interface: |
| 56 | DEFINE_DMA_BUF_EXPORT_INFO(exp_info) |
| 57 | struct dma_buf *dma_buf_export(struct dma_buf_export_info *exp_info) |
| 58 | |
| 59 | If this succeeds, dma_buf_export allocates a dma_buf structure, and |
| 60 | returns a pointer to the same. It also associates an anonymous file with this |
| 61 | buffer, so it can be exported. On failure to allocate the dma_buf object, |
| 62 | it returns NULL. |
| 63 | |
| 64 | 'exp_name' in struct dma_buf_export_info is the name of exporter - to |
| 65 | facilitate information while debugging. It is set to KBUILD_MODNAME by |
| 66 | default, so exporters don't have to provide a specific name, if they don't |
| 67 | wish to. |
| 68 | |
| 69 | DEFINE_DMA_BUF_EXPORT_INFO macro defines the struct dma_buf_export_info, |
| 70 | zeroes it out and pre-populates exp_name in it. |
| 71 | |
| 72 | |
| 73 | 2. Userspace gets a handle to pass around to potential buffer-users |
| 74 | |
| 75 | Userspace entity requests for a file-descriptor (fd) which is a handle to the |
| 76 | anonymous file associated with the buffer. It can then share the fd with other |
| 77 | drivers and/or processes. |
| 78 | |
| 79 | Interface: |
| 80 | int dma_buf_fd(struct dma_buf *dmabuf, int flags) |
| 81 | |
| 82 | This API installs an fd for the anonymous file associated with this buffer; |
| 83 | returns either 'fd', or error. |
| 84 | |
| 85 | 3. Each buffer-user 'connects' itself to the buffer |
| 86 | |
| 87 | Each buffer-user now gets a reference to the buffer, using the fd passed to |
| 88 | it. |
| 89 | |
| 90 | Interface: |
| 91 | struct dma_buf *dma_buf_get(int fd) |
| 92 | |
| 93 | This API will return a reference to the dma_buf, and increment refcount for |
| 94 | it. |
| 95 | |
| 96 | After this, the buffer-user needs to attach its device with the buffer, which |
| 97 | helps the exporter to know of device buffer constraints. |
| 98 | |
| 99 | Interface: |
| 100 | struct dma_buf_attachment *dma_buf_attach(struct dma_buf *dmabuf, |
| 101 | struct device *dev) |
| 102 | |
| 103 | This API returns reference to an attachment structure, which is then used |
| 104 | for scatterlist operations. It will optionally call the 'attach' dma_buf |
| 105 | operation, if provided by the exporter. |
| 106 | |
| 107 | The dma-buf sharing framework does the bookkeeping bits related to managing |
| 108 | the list of all attachments to a buffer. |
| 109 | |
| 110 | Until this stage, the buffer-exporter has the option to choose not to actually |
| 111 | allocate the backing storage for this buffer, but wait for the first buffer-user |
| 112 | to request use of buffer for allocation. |
| 113 | |
| 114 | |
| 115 | 4. When needed, buffer-user requests access to the buffer |
| 116 | |
| 117 | Whenever a buffer-user wants to use the buffer for any DMA, it asks for |
| 118 | access to the buffer using dma_buf_map_attachment API. At least one attach to |
| 119 | the buffer must have happened before map_dma_buf can be called. |
| 120 | |
| 121 | Interface: |
| 122 | struct sg_table * dma_buf_map_attachment(struct dma_buf_attachment *, |
| 123 | enum dma_data_direction); |
| 124 | |
| 125 | This is a wrapper to dma_buf->ops->map_dma_buf operation, which hides the |
| 126 | "dma_buf->ops->" indirection from the users of this interface. |
| 127 | |
| 128 | In struct dma_buf_ops, map_dma_buf is defined as |
| 129 | struct sg_table * (*map_dma_buf)(struct dma_buf_attachment *, |
| 130 | enum dma_data_direction); |
| 131 | |
| 132 | It is one of the buffer operations that must be implemented by the exporter. |
| 133 | It should return the sg_table containing scatterlist for this buffer, mapped |
| 134 | into caller's address space. |
| 135 | |
| 136 | If this is being called for the first time, the exporter can now choose to |
| 137 | scan through the list of attachments for this buffer, collate the requirements |
| 138 | of the attached devices, and choose an appropriate backing storage for the |
| 139 | buffer. |
| 140 | |
| 141 | Based on enum dma_data_direction, it might be possible to have multiple users |
| 142 | accessing at the same time (for reading, maybe), or any other kind of sharing |
| 143 | that the exporter might wish to make available to buffer-users. |
| 144 | |
| 145 | map_dma_buf() operation can return -EINTR if it is interrupted by a signal. |
| 146 | |
| 147 | |
| 148 | 5. When finished, the buffer-user notifies end-of-DMA to exporter |
| 149 | |
| 150 | Once the DMA for the current buffer-user is over, it signals 'end-of-DMA' to |
| 151 | the exporter using the dma_buf_unmap_attachment API. |
| 152 | |
| 153 | Interface: |
| 154 | void dma_buf_unmap_attachment(struct dma_buf_attachment *, |
| 155 | struct sg_table *); |
| 156 | |
| 157 | This is a wrapper to dma_buf->ops->unmap_dma_buf() operation, which hides the |
| 158 | "dma_buf->ops->" indirection from the users of this interface. |
| 159 | |
| 160 | In struct dma_buf_ops, unmap_dma_buf is defined as |
| 161 | void (*unmap_dma_buf)(struct dma_buf_attachment *, |
| 162 | struct sg_table *, |
| 163 | enum dma_data_direction); |
| 164 | |
| 165 | unmap_dma_buf signifies the end-of-DMA for the attachment provided. Like |
| 166 | map_dma_buf, this API also must be implemented by the exporter. |
| 167 | |
| 168 | |
| 169 | 6. when buffer-user is done using this buffer, it 'disconnects' itself from the |
| 170 | buffer. |
| 171 | |
| 172 | After the buffer-user has no more interest in using this buffer, it should |
| 173 | disconnect itself from the buffer: |
| 174 | |
| 175 | - it first detaches itself from the buffer. |
| 176 | |
| 177 | Interface: |
| 178 | void dma_buf_detach(struct dma_buf *dmabuf, |
| 179 | struct dma_buf_attachment *dmabuf_attach); |
| 180 | |
| 181 | This API removes the attachment from the list in dmabuf, and optionally calls |
| 182 | dma_buf->ops->detach(), if provided by exporter, for any housekeeping bits. |
| 183 | |
| 184 | - Then, the buffer-user returns the buffer reference to exporter. |
| 185 | |
| 186 | Interface: |
| 187 | void dma_buf_put(struct dma_buf *dmabuf); |
| 188 | |
| 189 | This API then reduces the refcount for this buffer. |
| 190 | |
| 191 | If, as a result of this call, the refcount becomes 0, the 'release' file |
| 192 | operation related to this fd is called. It calls the dmabuf->ops->release() |
| 193 | operation in turn, and frees the memory allocated for dmabuf when exported. |
| 194 | |
| 195 | NOTES: |
| 196 | - Importance of attach-detach and {map,unmap}_dma_buf operation pairs |
| 197 | The attach-detach calls allow the exporter to figure out backing-storage |
| 198 | constraints for the currently-interested devices. This allows preferential |
| 199 | allocation, and/or migration of pages across different types of storage |
| 200 | available, if possible. |
| 201 | |
| 202 | Bracketing of DMA access with {map,unmap}_dma_buf operations is essential |
| 203 | to allow just-in-time backing of storage, and migration mid-way through a |
| 204 | use-case. |
| 205 | |
| 206 | - Migration of backing storage if needed |
| 207 | If after |
| 208 | - at least one map_dma_buf has happened, |
| 209 | - and the backing storage has been allocated for this buffer, |
| 210 | another new buffer-user intends to attach itself to this buffer, it might |
| 211 | be allowed, if possible for the exporter. |
| 212 | |
| 213 | In case it is allowed by the exporter: |
| 214 | if the new buffer-user has stricter 'backing-storage constraints', and the |
| 215 | exporter can handle these constraints, the exporter can just stall on the |
| 216 | map_dma_buf until all outstanding access is completed (as signalled by |
| 217 | unmap_dma_buf). |
| 218 | Once all users have finished accessing and have unmapped this buffer, the |
| 219 | exporter could potentially move the buffer to the stricter backing-storage, |
| 220 | and then allow further {map,unmap}_dma_buf operations from any buffer-user |
| 221 | from the migrated backing-storage. |
| 222 | |
| 223 | If the exporter cannot fulfill the backing-storage constraints of the new |
| 224 | buffer-user device as requested, dma_buf_attach() would return an error to |
| 225 | denote non-compatibility of the new buffer-sharing request with the current |
| 226 | buffer. |
| 227 | |
| 228 | If the exporter chooses not to allow an attach() operation once a |
| 229 | map_dma_buf() API has been called, it simply returns an error. |
| 230 | |
| 231 | Kernel cpu access to a dma-buf buffer object |
| 232 | -------------------------------------------- |
| 233 | |
| 234 | The motivation to allow cpu access from the kernel to a dma-buf object from the |
| 235 | importers side are: |
| 236 | - fallback operations, e.g. if the devices is connected to a usb bus and the |
| 237 | kernel needs to shuffle the data around first before sending it away. |
| 238 | - full transparency for existing users on the importer side, i.e. userspace |
| 239 | should not notice the difference between a normal object from that subsystem |
| 240 | and an imported one backed by a dma-buf. This is really important for drm |
| 241 | opengl drivers that expect to still use all the existing upload/download |
| 242 | paths. |
| 243 | |
| 244 | Access to a dma_buf from the kernel context involves three steps: |
| 245 | |
| 246 | 1. Prepare access, which invalidate any necessary caches and make the object |
| 247 | available for cpu access. |
| 248 | 2. Access the object page-by-page with the dma_buf map apis |
| 249 | 3. Finish access, which will flush any necessary cpu caches and free reserved |
| 250 | resources. |
| 251 | |
| 252 | 1. Prepare access |
| 253 | |
| 254 | Before an importer can access a dma_buf object with the cpu from the kernel |
| 255 | context, it needs to notify the exporter of the access that is about to |
| 256 | happen. |
| 257 | |
| 258 | Interface: |
| 259 | int dma_buf_begin_cpu_access(struct dma_buf *dmabuf, |
| 260 | size_t start, size_t len, |
| 261 | enum dma_data_direction direction) |
| 262 | |
| 263 | This allows the exporter to ensure that the memory is actually available for |
| 264 | cpu access - the exporter might need to allocate or swap-in and pin the |
| 265 | backing storage. The exporter also needs to ensure that cpu access is |
| 266 | coherent for the given range and access direction. The range and access |
| 267 | direction can be used by the exporter to optimize the cache flushing, i.e. |
| 268 | access outside of the range or with a different direction (read instead of |
| 269 | write) might return stale or even bogus data (e.g. when the exporter needs to |
| 270 | copy the data to temporary storage). |
| 271 | |
| 272 | This step might fail, e.g. in oom conditions. |
| 273 | |
| 274 | 2. Accessing the buffer |
| 275 | |
| 276 | To support dma_buf objects residing in highmem cpu access is page-based using |
| 277 | an api similar to kmap. Accessing a dma_buf is done in aligned chunks of |
| 278 | PAGE_SIZE size. Before accessing a chunk it needs to be mapped, which returns |
| 279 | a pointer in kernel virtual address space. Afterwards the chunk needs to be |
| 280 | unmapped again. There is no limit on how often a given chunk can be mapped |
| 281 | and unmapped, i.e. the importer does not need to call begin_cpu_access again |
| 282 | before mapping the same chunk again. |
| 283 | |
| 284 | Interfaces: |
| 285 | void *dma_buf_kmap(struct dma_buf *, unsigned long); |
| 286 | void dma_buf_kunmap(struct dma_buf *, unsigned long, void *); |
| 287 | |
| 288 | There are also atomic variants of these interfaces. Like for kmap they |
| 289 | facilitate non-blocking fast-paths. Neither the importer nor the exporter (in |
| 290 | the callback) is allowed to block when using these. |
| 291 | |
| 292 | Interfaces: |
| 293 | void *dma_buf_kmap_atomic(struct dma_buf *, unsigned long); |
| 294 | void dma_buf_kunmap_atomic(struct dma_buf *, unsigned long, void *); |
| 295 | |
| 296 | For importers all the restrictions of using kmap apply, like the limited |
| 297 | supply of kmap_atomic slots. Hence an importer shall only hold onto at most 2 |
| 298 | atomic dma_buf kmaps at the same time (in any given process context). |
| 299 | |
| 300 | dma_buf kmap calls outside of the range specified in begin_cpu_access are |
| 301 | undefined. If the range is not PAGE_SIZE aligned, kmap needs to succeed on |
| 302 | the partial chunks at the beginning and end but may return stale or bogus |
| 303 | data outside of the range (in these partial chunks). |
| 304 | |
| 305 | Note that these calls need to always succeed. The exporter needs to complete |
| 306 | any preparations that might fail in begin_cpu_access. |
| 307 | |
| 308 | For some cases the overhead of kmap can be too high, a vmap interface |
| 309 | is introduced. This interface should be used very carefully, as vmalloc |
| 310 | space is a limited resources on many architectures. |
| 311 | |
| 312 | Interfaces: |
| 313 | void *dma_buf_vmap(struct dma_buf *dmabuf) |
| 314 | void dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr) |
| 315 | |
| 316 | The vmap call can fail if there is no vmap support in the exporter, or if it |
| 317 | runs out of vmalloc space. Fallback to kmap should be implemented. Note that |
| 318 | the dma-buf layer keeps a reference count for all vmap access and calls down |
| 319 | into the exporter's vmap function only when no vmapping exists, and only |
| 320 | unmaps it once. Protection against concurrent vmap/vunmap calls is provided |
| 321 | by taking the dma_buf->lock mutex. |
| 322 | |
| 323 | 3. Finish access |
| 324 | |
| 325 | When the importer is done accessing the range specified in begin_cpu_access, |
| 326 | it needs to announce this to the exporter (to facilitate cache flushing and |
| 327 | unpinning of any pinned resources). The result of any dma_buf kmap calls |
| 328 | after end_cpu_access is undefined. |
| 329 | |
| 330 | Interface: |
| 331 | void dma_buf_end_cpu_access(struct dma_buf *dma_buf, |
| 332 | size_t start, size_t len, |
| 333 | enum dma_data_direction dir); |
| 334 | |
| 335 | |
| 336 | Direct Userspace Access/mmap Support |
| 337 | ------------------------------------ |
| 338 | |
| 339 | Being able to mmap an export dma-buf buffer object has 2 main use-cases: |
| 340 | - CPU fallback processing in a pipeline and |
| 341 | - supporting existing mmap interfaces in importers. |
| 342 | |
| 343 | 1. CPU fallback processing in a pipeline |
| 344 | |
| 345 | In many processing pipelines it is sometimes required that the cpu can access |
| 346 | the data in a dma-buf (e.g. for thumbnail creation, snapshots, ...). To avoid |
| 347 | the need to handle this specially in userspace frameworks for buffer sharing |
| 348 | it's ideal if the dma_buf fd itself can be used to access the backing storage |
| 349 | from userspace using mmap. |
| 350 | |
| 351 | Furthermore Android's ION framework already supports this (and is otherwise |
| 352 | rather similar to dma-buf from a userspace consumer side with using fds as |
| 353 | handles, too). So it's beneficial to support this in a similar fashion on |
| 354 | dma-buf to have a good transition path for existing Android userspace. |
| 355 | |
| 356 | No special interfaces, userspace simply calls mmap on the dma-buf fd. |
| 357 | |
| 358 | 2. Supporting existing mmap interfaces in importers |
| 359 | |
| 360 | Similar to the motivation for kernel cpu access it is again important that |
| 361 | the userspace code of a given importing subsystem can use the same interfaces |
| 362 | with a imported dma-buf buffer object as with a native buffer object. This is |
| 363 | especially important for drm where the userspace part of contemporary OpenGL, |
| 364 | X, and other drivers is huge, and reworking them to use a different way to |
| 365 | mmap a buffer rather invasive. |
| 366 | |
| 367 | The assumption in the current dma-buf interfaces is that redirecting the |
| 368 | initial mmap is all that's needed. A survey of some of the existing |
| 369 | subsystems shows that no driver seems to do any nefarious thing like syncing |
| 370 | up with outstanding asynchronous processing on the device or allocating |
| 371 | special resources at fault time. So hopefully this is good enough, since |
| 372 | adding interfaces to intercept pagefaults and allow pte shootdowns would |
| 373 | increase the complexity quite a bit. |
| 374 | |
| 375 | Interface: |
| 376 | int dma_buf_mmap(struct dma_buf *, struct vm_area_struct *, |
| 377 | unsigned long); |
| 378 | |
| 379 | If the importing subsystem simply provides a special-purpose mmap call to set |
| 380 | up a mapping in userspace, calling do_mmap with dma_buf->file will equally |
| 381 | achieve that for a dma-buf object. |
| 382 | |
| 383 | 3. Implementation notes for exporters |
| 384 | |
| 385 | Because dma-buf buffers have invariant size over their lifetime, the dma-buf |
| 386 | core checks whether a vma is too large and rejects such mappings. The |
| 387 | exporter hence does not need to duplicate this check. |
| 388 | |
| 389 | Because existing importing subsystems might presume coherent mappings for |
| 390 | userspace, the exporter needs to set up a coherent mapping. If that's not |
| 391 | possible, it needs to fake coherency by manually shooting down ptes when |
| 392 | leaving the cpu domain and flushing caches at fault time. Note that all the |
| 393 | dma_buf files share the same anon inode, hence the exporter needs to replace |
| 394 | the dma_buf file stored in vma->vm_file with it's own if pte shootdown is |
| 395 | required. This is because the kernel uses the underlying inode's address_space |
| 396 | for vma tracking (and hence pte tracking at shootdown time with |
| 397 | unmap_mapping_range). |
| 398 | |
| 399 | If the above shootdown dance turns out to be too expensive in certain |
| 400 | scenarios, we can extend dma-buf with a more explicit cache tracking scheme |
| 401 | for userspace mappings. But the current assumption is that using mmap is |
| 402 | always a slower path, so some inefficiencies should be acceptable. |
| 403 | |
| 404 | Exporters that shoot down mappings (for any reasons) shall not do any |
| 405 | synchronization at fault time with outstanding device operations. |
| 406 | Synchronization is an orthogonal issue to sharing the backing storage of a |
| 407 | buffer and hence should not be handled by dma-buf itself. This is explicitly |
| 408 | mentioned here because many people seem to want something like this, but if |
| 409 | different exporters handle this differently, buffer sharing can fail in |
| 410 | interesting ways depending upong the exporter (if userspace starts depending |
| 411 | upon this implicit synchronization). |
| 412 | |
| 413 | Other Interfaces Exposed to Userspace on the dma-buf FD |
| 414 | ------------------------------------------------------ |
| 415 | |
| 416 | - Since kernel 3.12 the dma-buf FD supports the llseek system call, but only |
| 417 | with offset=0 and whence=SEEK_END|SEEK_SET. SEEK_SET is supported to allow |
| 418 | the usual size discover pattern size = SEEK_END(0); SEEK_SET(0). Every other |
| 419 | llseek operation will report -EINVAL. |
| 420 | |
| 421 | If llseek on dma-buf FDs isn't support the kernel will report -ESPIPE for all |
| 422 | cases. Userspace can use this to detect support for discovering the dma-buf |
| 423 | size using llseek. |
| 424 | |
| 425 | Miscellaneous notes |
| 426 | ------------------- |
| 427 | |
| 428 | - Any exporters or users of the dma-buf buffer sharing framework must have |
| 429 | a 'select DMA_SHARED_BUFFER' in their respective Kconfigs. |
| 430 | |
| 431 | - In order to avoid fd leaks on exec, the FD_CLOEXEC flag must be set |
| 432 | on the file descriptor. This is not just a resource leak, but a |
| 433 | potential security hole. It could give the newly exec'd application |
| 434 | access to buffers, via the leaked fd, to which it should otherwise |
| 435 | not be permitted access. |
| 436 | |
| 437 | The problem with doing this via a separate fcntl() call, versus doing it |
| 438 | atomically when the fd is created, is that this is inherently racy in a |
| 439 | multi-threaded app[3]. The issue is made worse when it is library code |
| 440 | opening/creating the file descriptor, as the application may not even be |
| 441 | aware of the fd's. |
| 442 | |
| 443 | To avoid this problem, userspace must have a way to request O_CLOEXEC |
| 444 | flag be set when the dma-buf fd is created. So any API provided by |
| 445 | the exporting driver to create a dmabuf fd must provide a way to let |
| 446 | userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd(). |
| 447 | |
| 448 | - If an exporter needs to manually flush caches and hence needs to fake |
| 449 | coherency for mmap support, it needs to be able to zap all the ptes pointing |
| 450 | at the backing storage. Now linux mm needs a struct address_space associated |
| 451 | with the struct file stored in vma->vm_file to do that with the function |
| 452 | unmap_mapping_range. But the dma_buf framework only backs every dma_buf fd |
| 453 | with the anon_file struct file, i.e. all dma_bufs share the same file. |
| 454 | |
| 455 | Hence exporters need to setup their own file (and address_space) association |
| 456 | by setting vma->vm_file and adjusting vma->vm_pgoff in the dma_buf mmap |
| 457 | callback. In the specific case of a gem driver the exporter could use the |
| 458 | shmem file already provided by gem (and set vm_pgoff = 0). Exporters can then |
| 459 | zap ptes by unmapping the corresponding range of the struct address_space |
| 460 | associated with their own file. |
| 461 | |
| 462 | References: |
| 463 | [1] struct dma_buf_ops in include/linux/dma-buf.h |
| 464 | [2] All interfaces mentioned above defined in include/linux/dma-buf.h |
| 465 | [3] https://lwn.net/Articles/236486/ |