summaryrefslogtreecommitdiffstats
path: root/drivers/iommu/intel-iommu.c
Commit message (Collapse)AuthorAgeFilesLines
...
* | Merge git://git.infradead.org/intel-iommuLinus Torvalds2016-11-271-0/+13
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull IOMMU fixes from David Woodhouse: "Two minor fixes. The first fixes the assignment of SR-IOV virtual functions to the correct IOMMU unit, and the second fixes the excessively large (and physically contiguous) PASID tables used with SVM" * git://git.infradead.org/intel-iommu: iommu/vt-d: Fix PASID table allocation iommu/vt-d: Fix IOMMU lookup for SR-IOV Virtual Functions
| * | iommu/vt-d: Fix IOMMU lookup for SR-IOV Virtual FunctionsAshok Raj2016-10-301-0/+13
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The VT-d specification (§8.3.3) says: ‘Virtual Functions’ of a ‘Physical Function’ are under the scope of the same remapping unit as the ‘Physical Function’. The BIOS is not required to list all the possible VFs in the scope tables, and arguably *shouldn't* make any attempt to do so, since there could be a huge number of them. This has been broken basically for ever — the VF is never going to match against a specific unit's scope, so it ends up being assigned to the INCLUDE_ALL IOMMU. Which was always actually correct by coincidence, but now we're looking at Root-Complex integrated devices with SR-IOV support it's going to start being wrong. Fix it to simply use pci_physfn() before doing the lookup for PCI devices. Cc: stable@vger.kernel.org Signed-off-by: Sainath Grandhi <sainath.grandhi@intel.com> Signed-off-by: Ashok Raj <ashok.raj@intel.com> Signed-off-by: David Woodhouse <dwmw2@infradead.org>
* / iommu/vt-d: Fix dead-locks in disable_dmar_iommu() pathJoerg Roedel2016-11-081-2/+12
|/ | | | | | | | | | It turns out that the disable_dmar_iommu() code-path tried to get the device_domain_lock recursivly, which will dead-lock when this code runs on dmar removal. Fix both code-paths that could lead to the dead-lock. Fixes: 55d940430ab9 ('iommu/vt-d: Get rid of domain->iommu_lock') Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/vt-d: Make sure RMRRs are mapped before domain goes publicJoerg Roedel2016-09-051-7/+20
| | | | | | | | | | | | | When a domain is allocated through the get_valid_domain_for_dev path, it will be context-mapped before the RMRR regions are mapped in the page-table. This opens a short time window where device-accesses to these regions fail and causing DMAR faults. Fix this by mapping the RMRR regions before the domain is context-mapped. Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/vt-d: Split up get_domain_for_dev functionJoerg Roedel2016-09-051-21/+55
| | | | | | | | | | Split out the search for an already existing domain and the context mapping of the device to the new domain. This allows to map possible RMRR regions into the domain before it is context mapped. Signed-off-by: Joerg Roedel <jroedel@suse.de>
* dma-mapping: use unsigned long for dma_attrsKrzysztof Kozlowski2016-08-041-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The dma-mapping core and the implementations do not change the DMA attributes passed by pointer. Thus the pointer can point to const data. However the attributes do not have to be a bitfield. Instead unsigned long will do fine: 1. This is just simpler. Both in terms of reading the code and setting attributes. Instead of initializing local attributes on the stack and passing pointer to it to dma_set_attr(), just set the bits. 2. It brings safeness and checking for const correctness because the attributes are passed by value. Semantic patches for this change (at least most of them): virtual patch virtual context @r@ identifier f, attrs; @@ f(..., - struct dma_attrs *attrs + unsigned long attrs , ...) { ... } @@ identifier r.f; @@ f(..., - NULL + 0 ) and // Options: --all-includes virtual patch virtual context @r@ identifier f, attrs; type t; @@ t f(..., struct dma_attrs *attrs); @@ identifier r.f; @@ f(..., - NULL + 0 ) Link: http://lkml.kernel.org/r/1468399300-5399-2-git-send-email-k.kozlowski@samsung.com Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com> Acked-by: Vineet Gupta <vgupta@synopsys.com> Acked-by: Robin Murphy <robin.murphy@arm.com> Acked-by: Hans-Christian Noren Egtvedt <egtvedt@samfundet.no> Acked-by: Mark Salter <msalter@redhat.com> [c6x] Acked-by: Jesper Nilsson <jesper.nilsson@axis.com> [cris] Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch> [drm] Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com> Acked-by: Joerg Roedel <jroedel@suse.de> [iommu] Acked-by: Fabien Dessenne <fabien.dessenne@st.com> [bdisp] Reviewed-by: Marek Szyprowski <m.szyprowski@samsung.com> [vb2-core] Acked-by: David Vrabel <david.vrabel@citrix.com> [xen] Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> [xen swiotlb] Acked-by: Joerg Roedel <jroedel@suse.de> [iommu] Acked-by: Richard Kuo <rkuo@codeaurora.org> [hexagon] Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> [m68k] Acked-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> [s390] Acked-by: Bjorn Andersson <bjorn.andersson@linaro.org> Acked-by: Hans-Christian Noren Egtvedt <egtvedt@samfundet.no> [avr32] Acked-by: Vineet Gupta <vgupta@synopsys.com> [arc] Acked-by: Robin Murphy <robin.murphy@arm.com> [arm64 and dma-iommu] Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* Merge tag 'iommu-updates-v4.8' of ↵Linus Torvalds2016-08-011-3/+3
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu Pull IOMMU updates from Joerg Roedel: - big-endian support and preparation for defered probing for the Exynos IOMMU driver - simplifications in iommu-group id handling - support for Mediatek generation one IOMMU hardware - conversion of the AMD IOMMU driver to use the generic IOVA allocator. This driver now also benefits from the recent scalability improvements in the IOVA code. - preparations to use generic DMA mapping code in the Rockchip IOMMU driver - device tree adaption and conversion to use generic page-table code for the MSM IOMMU driver - an iova_to_phys optimization in the ARM-SMMU driver to greatly improve page-table teardown performance with VFIO - various other small fixes and conversions * tag 'iommu-updates-v4.8' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu: (59 commits) iommu/amd: Initialize dma-ops domains with 3-level page-table iommu/amd: Update Alias-DTE in update_device_table() iommu/vt-d: Return error code in domain_context_mapping_one() iommu/amd: Use container_of to get dma_ops_domain iommu/amd: Flush iova queue before releasing dma_ops_domain iommu/amd: Handle IOMMU_DOMAIN_DMA in ops->domain_free call-back iommu/amd: Use dev_data->domain in get_domain() iommu/amd: Optimize map_sg and unmap_sg iommu/amd: Introduce dir2prot() helper iommu/amd: Implement timeout to flush unmap queues iommu/amd: Implement flush queue iommu/amd: Allow NULL pointer parameter for domain_flush_complete() iommu/amd: Set up data structures for flush queue iommu/amd: Remove align-parameter from __map_single() iommu/amd: Remove other remains of old address allocator iommu/amd: Make use of the generic IOVA allocator iommu/amd: Remove special mapping code for dma_ops path iommu/amd: Pass gfp-flags to iommu_map_page() iommu/amd: Implement apply_dm_region call-back iommu/amd: Create a list of reserved iova addresses ...
| *-----. Merge branches 'x86/amd', 'x86/vt-d', 'arm/exynos', 'arm/mediatek', ↵Joerg Roedel2016-07-261-3/+3
| |\ \ \ \ | | | | | | | | | | | | | | | | | | 'arm/msm', 'arm/rockchip', 'arm/smmu' and 'core' into next
| | | * | | iommu/vt-d: Return error code in domain_context_mapping_one()Wei Yang2016-07-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In 'commit <55d940430ab9> ("iommu/vt-d: Get rid of domain->iommu_lock")', the error handling path is changed a little, which makes the function always return 0. This path fixes this. Signed-off-by: Wei Yang <richard.weiyang@gmail.com> Fixes: 55d940430ab9 ('iommu/vt-d: Get rid of domain->iommu_lock') Cc: stable@vger.kernel.org # v4.3+ Signed-off-by: Joerg Roedel <jroedel@suse.de>
| | | * | | iommu/vt-d: Reduce extra first level entry in iommu->domainsWei Yang2016-06-151-2/+2
| | |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In commit <8bf478163e69> ("iommu/vt-d: Split up iommu->domains array"), it it splits iommu->domains in two levels. Each first level contains 256 entries of second level. In case of the ndomains is exact a multiple of 256, it would have one more extra first level entry for current implementation. This patch refines this calculation to reduce the extra first level entry. Signed-off-by: Wei Yang <richard.weiyang@gmail.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* | / / / Add braces to avoid "ambiguous ‘else’" compiler warningsLinus Torvalds2016-07-281-1/+2
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some of our "for_each_xyz()" macro constructs make gcc unhappy about lack of braces around if-statements inside or outside the loop, because the loop construct itself has a "if-then-else" statement inside of it. The resulting warnings look something like this: drivers/gpu/drm/i915/i915_debugfs.c: In function ‘i915_dump_lrc’: drivers/gpu/drm/i915/i915_debugfs.c:2103:6: warning: suggest explicit braces to avoid ambiguous ‘else’ [-Wparentheses] if (ctx != dev_priv->kernel_context) ^ even if the code itself is fine. Since the warning is fairly easy to avoid by adding a braces around the if-statement near the for_each_xyz() construct, do so, rather than disabling the otherwise potentially useful warning. (The if-then-else statements used in the "for_each_xyz()" constructs are designed to be inherently safe even with no braces, but in this case it's quite understandable that gcc isn't really able to tell that). This finally leaves the standard "allmodconfig" build with just a handful of remaining warnings, so new and valid warnings hopefully will stand out. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* | | / iommu/vt-d: Fix infinite loop in free_all_cpu_cached_iovasAaron Campbell2016-07-041-2/+2
| |_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Per VT-d spec Section 10.4.2 ("Capability Register"), the maximum number of possible domains is 64K; indeed this is the maximum value that the cap_ndoms() macro will expand to. Since the value 65536 will not fix in a u16, the 'did' variable must be promoted to an int, otherwise the test for < 65536 will always be true and the loop will never end. The symptom, in my case, was a hung machine during suspend. Fixes: 3bd4f9112f87 ("iommu/vt-d: Fix overflow of iommu->domains array") Signed-off-by: Aaron Campbell <aaron@monkey.org> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* | | iommu/vt-d: Fix overflow of iommu->domains arrayJan Niehusmann2016-06-271-1/+1
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | The valid range of 'did' in get_iommu_domain(*iommu, did) is 0..cap_ndoms(iommu->cap), so don't exceed that range in free_all_cpu_cached_iovas(). The user-visible impact of the out-of-bounds access is the machine hanging on suspend-to-ram. It is, in fact, a kernel panic, but due to already suspended devices, that's often not visible to the user. Fixes: 22e2f9fa63b0 ("iommu/vt-d: Use per-cpu IOVA caching") Signed-off-by: Jan Niehusmann <jan@gondor.com> Tested-By: Marius Vlad <marius.c.vlad@intel.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* | iommu/vt-d: Enable QI on all IOMMUs before setting root entryJoerg Roedel2016-06-171-5/+12
|/ | | | | | | | | | | | This seems to be required on some X58 chipsets on systems with more than one IOMMU. QI does not work until it is enabled on all IOMMUs in the system. Reported-by: Dheeraj CVR <cvr.dheeraj@gmail.com> Tested-by: Dheeraj CVR <cvr.dheeraj@gmail.com> Fixes: 5f0a7f7614a9 ('iommu/vt-d: Make root entry visible for hardware right after allocation') Cc: stable@vger.kernel.org Signed-off-by: Joerg Roedel <jroedel@suse.de>
* Merge git://git.infradead.org/intel-iommuLinus Torvalds2016-05-271-94/+224
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull intel IOMMU updates from David Woodhouse: "This patchset improves the scalability of the Intel IOMMU code by resolving two spinlock bottlenecks and eliminating the linearity of the IOVA allocator, yielding up to ~5x performance improvement and approaching 'iommu=off' performance" * git://git.infradead.org/intel-iommu: iommu/vt-d: Use per-cpu IOVA caching iommu/iova: introduce per-cpu caching to iova allocation iommu/vt-d: change intel-iommu to use IOVA frame numbers iommu/vt-d: avoid dev iotlb logic for domains with no dev iotlbs iommu/vt-d: only unmap mapped entries iommu/vt-d: correct flush_unmaps pfn usage iommu/vt-d: per-cpu deferred invalidation queues iommu/vt-d: refactoring of deferred flush entries
| * iommu/vt-d: Use per-cpu IOVA cachingOmer Peleg2016-04-201-12/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | Commit 9257b4a2 ('iommu/iova: introduce per-cpu caching to iova allocation') introduced per-CPU IOVA caches to massively improve scalability. Use them. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased, cleaned up and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> [dwmw2: split out VT-d part into a separate patch] Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * iommu/vt-d: change intel-iommu to use IOVA frame numbersOmer Peleg2016-04-201-32/+29Star
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make intel-iommu map/unmap/invalidate work with IOVA pfns instead of pointers to "struct iova". This avoids using the iova struct from the IOVA red-black tree and the resulting explicit find_iova() on unmap. This patch will allow us to cache IOVAs in the next patch, in order to avoid rbtree operations for the majority of map/unmap operations. Note: In eliminating the find_iova() operation, we have also eliminated the sanity check previously done in the unmap flow. Arguably, this was overhead that is better avoided in production code, but it could be brought back as a debug option for driver development. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased, fixed to not break iova api, and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * iommu/vt-d: avoid dev iotlb logic for domains with no dev iotlbsOmer Peleg2016-04-201-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch avoids taking the device_domain_lock in iommu_flush_dev_iotlb() for domains with no dev iotlb devices. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [gvdl@google.com: fixed locking issues] Signed-off-by: Godfrey van der Linden <gvdl@google.com> [mad@cs.technion.ac.il: rebased and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * iommu/vt-d: only unmap mapped entriesOmer Peleg2016-04-201-11/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current unmap implementation unmaps the entire area covered by the IOVA range, which is a power-of-2 aligned region. The corresponding map, however, only maps those pages originally mapped by the user. This discrepancy can lead to unmapping of already unmapped entries, which is unneeded work. With this patch, only mapped pages are unmapped. This is also a baseline for a map/unmap implementation based on IOVAs and not iova structures, which will allow caching. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * iommu/vt-d: correct flush_unmaps pfn usageOmer Peleg2016-04-201-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Change flush_unmaps() to correctly pass iommu_flush_iotlb_psi() dma addresses. (x86_64 mm and dma have the same size for pages at the moment, but this usage improves consistency.) Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * iommu/vt-d: per-cpu deferred invalidation queuesOmer Peleg2016-04-201-41/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The IOMMU's IOTLB invalidation is a costly process. When iommu mode is not set to "strict", it is done asynchronously. Current code amortizes the cost of invalidating IOTLB entries by batching all the invalidations in the system and performing a single global invalidation instead. The code queues pending invalidations in a global queue that is accessed under the global "async_umap_flush_lock" spinlock, which can result is significant spinlock contention. This patch splits this deferred queue into multiple per-cpu deferred queues, and thus gets rid of the "async_umap_flush_lock" and its contention. To keep existing deferred invalidation behavior, it still invalidates the pending invalidations of all CPUs whenever a CPU reaches its watermark or a timeout occurs. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased, cleaned up and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * iommu/vt-d: refactoring of deferred flush entriesOmer Peleg2016-04-201-19/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, deferred flushes' info is striped between several lists in the flush tables. Instead, move all information about a specific flush to a single entry in this table. This patch does not introduce any functional change. Signed-off-by: Omer Peleg <omer@cs.technion.ac.il> [mad@cs.technion.ac.il: rebased and reworded the commit message] Signed-off-by: Adam Morrison <mad@cs.technion.ac.il> Reviewed-by: Shaohua Li <shli@fb.com> Reviewed-by: Ben Serebrin <serebrin@google.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| |
| \
*-. | Merge branches 'arm/io-pgtable', 'arm/rockchip', 'arm/omap', 'x86/vt-d', ↵Joerg Roedel2016-05-091-2/+2
|\ \| | | | | | | | | | 'ppc/pamu', 'core' and 'x86/amd' into next
| | * iommu/vt-d: Silence an uninitialized variable warningDan Carpenter2016-04-071-1/+1
| |/ |/| | | | | | | | | | | | | | | My static checker complains that "dma_alias" is uninitialized unless we are dealing with a pci device. This is true but harmless. Anyway, we can flip the condition around to silence the warning. Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
| * x86/vt-d: Fix comment for dma_pte_free_pagetable()Michael S. Tsirkin2016-04-051-1/+1
|/ | | | | | | | | | | dma_pte_free_pagetable no longer depends on last level ptes being clear, it clears them itself. Fix up the comment to match. Cc: Jiang Liu <jiang.liu@linux.intel.com> Suggested-by: Alex Williamson <alex.williamson@redhat.com> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/vt-d: Use BUS_NOTIFY_REMOVED_DEVICE in hotplug pathJoerg Roedel2016-02-291-2/+2
| | | | | | | | | | | | | | | In the PCI hotplug path of the Intel IOMMU driver, replace the usage of the BUS_NOTIFY_DEL_DEVICE notifier, which is executed before the driver is unbound from the device, with BUS_NOTIFY_REMOVED_DEVICE, which runs after that. This fixes a kernel BUG being triggered in the VT-d code when the device driver tries to unmap DMA buffers and the VT-d driver already destroyed all mappings. Reported-by: Stefani Seibold <stefani@seibold.net> Cc: stable@vger.kernel.org # v4.3+ Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/vt-d: Don't skip PCI devices when disabling IOTLBJeremy McNicoll2016-01-291-1/+1
| | | | | | | | | | Fix a simple typo when disabling IOTLB on PCI(e) devices. Fixes: b16d0cb9e2fc ("iommu/vt-d: Always enable PASID/PRI PCI capabilities before ATS") Cc: stable@vger.kernel.org # v4.4 Signed-off-by: Jeremy McNicoll <jmcnicol@redhat.com> Reviewed-by: Alex Williamson <alex.williamson@redhat.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* Revert "scatterlist: use sg_phys()"Dan Williams2015-12-151-2/+2
| | | | | | | | | | | | | | | | | | | | | | | commit db0fa0cb0157 "scatterlist: use sg_phys()" did replacements of the form: phys_addr_t phys = page_to_phys(sg_page(s)); phys_addr_t phys = sg_phys(s) & PAGE_MASK; However, this breaks platforms where sizeof(phys_addr_t) > sizeof(unsigned long). Revert for 4.3 and 4.4 to make room for a combined helper in 4.5. Cc: <stable@vger.kernel.org> Cc: Jens Axboe <axboe@fb.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Russell King <linux@arm.linux.org.uk> Cc: David Woodhouse <dwmw2@infradead.org> Cc: Andrew Morton <akpm@linux-foundation.org> Fixes: db0fa0cb0157 ("scatterlist: use sg_phys()") Suggested-by: Joerg Roedel <joro@8bytes.org> Reported-by: Vitaly Lavrov <vel21ripn@gmail.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
* mm, page_alloc: distinguish between being unable to sleep, unwilling to ↵Mel Gorman2015-11-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sleep and avoiding waking kswapd __GFP_WAIT has been used to identify atomic context in callers that hold spinlocks or are in interrupts. They are expected to be high priority and have access one of two watermarks lower than "min" which can be referred to as the "atomic reserve". __GFP_HIGH users get access to the first lower watermark and can be called the "high priority reserve". Over time, callers had a requirement to not block when fallback options were available. Some have abused __GFP_WAIT leading to a situation where an optimisitic allocation with a fallback option can access atomic reserves. This patch uses __GFP_ATOMIC to identify callers that are truely atomic, cannot sleep and have no alternative. High priority users continue to use __GFP_HIGH. __GFP_DIRECT_RECLAIM identifies callers that can sleep and are willing to enter direct reclaim. __GFP_KSWAPD_RECLAIM to identify callers that want to wake kswapd for background reclaim. __GFP_WAIT is redefined as a caller that is willing to enter direct reclaim and wake kswapd for background reclaim. This patch then converts a number of sites o __GFP_ATOMIC is used by callers that are high priority and have memory pools for those requests. GFP_ATOMIC uses this flag. o Callers that have a limited mempool to guarantee forward progress clear __GFP_DIRECT_RECLAIM but keep __GFP_KSWAPD_RECLAIM. bio allocations fall into this category where kswapd will still be woken but atomic reserves are not used as there is a one-entry mempool to guarantee progress. o Callers that are checking if they are non-blocking should use the helper gfpflags_allow_blocking() where possible. This is because checking for __GFP_WAIT as was done historically now can trigger false positives. Some exceptions like dm-crypt.c exist where the code intent is clearer if __GFP_DIRECT_RECLAIM is used instead of the helper due to flag manipulations. o Callers that built their own GFP flags instead of starting with GFP_KERNEL and friends now also need to specify __GFP_KSWAPD_RECLAIM. The first key hazard to watch out for is callers that removed __GFP_WAIT and was depending on access to atomic reserves for inconspicuous reasons. In some cases it may be appropriate for them to use __GFP_HIGH. The second key hazard is callers that assembled their own combination of GFP flags instead of starting with something like GFP_KERNEL. They may now wish to specify __GFP_KSWAPD_RECLAIM. It's almost certainly harmless if it's missed in most cases as other activity will wake kswapd. Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Christoph Lameter <cl@linux.com> Cc: David Rientjes <rientjes@google.com> Cc: Vitaly Wool <vitalywool@gmail.com> Cc: Rik van Riel <riel@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* Merge tag 'iommu-updates-v4.4' of ↵Linus Torvalds2015-11-061-29/+54
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu Pull iommu updates from Joerg Roedel: "This time including: - A new IOMMU driver for s390 pci devices - Common dma-ops support based on iommu-api for ARM64. The plan is to use this as a basis for ARM32 and hopefully other architectures as well in the future. - MSI support for ARM-SMMUv3 - Cleanups and dead code removal in the AMD IOMMU driver - Better RMRR handling for the Intel VT-d driver - Various other cleanups and small fixes" * tag 'iommu-updates-v4.4' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu: (41 commits) iommu/vt-d: Fix return value check of parse_ioapics_under_ir() iommu/vt-d: Propagate error-value from ir_parse_ioapic_hpet_scope() iommu/vt-d: Adjust the return value of the parse_ioapics_under_ir iommu: Move default domain allocation to iommu_group_get_for_dev() iommu: Remove is_pci_dev() fall-back from iommu_group_get_for_dev iommu/arm-smmu: Switch to device_group call-back iommu/fsl: Convert to device_group call-back iommu: Add device_group call-back to x86 iommu drivers iommu: Add generic_device_group() function iommu: Export and rename iommu_group_get_for_pci_dev() iommu: Revive device_group iommu-ops call-back iommu/amd: Remove find_last_devid_on_pci() iommu/amd: Remove first/last_device handling iommu/amd: Initialize amd_iommu_last_bdf for DEV_ALL iommu/amd: Cleanup buffer allocation iommu/amd: Remove cmd_buf_size and evt_buf_size from struct amd_iommu iommu/amd: Align DTE flag definitions iommu/amd: Remove old alias handling code iommu/amd: Set alias DTE in do_attach/do_detach iommu/amd: WARN when __[attach|detach]_device are called with irqs enabled ...
| *-----. Merge branches 'x86/vt-d', 'arm/omap', 'arm/smmu', 's390', 'core' and ↵Joerg Roedel2015-11-021-29/+54
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 'x86/amd' into next Conflicts: drivers/iommu/amd_iommu_types.h
| | | | * | iommu: Add device_group call-back to x86 iommu driversJoerg Roedel2015-10-221-0/+1
| | | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Set the device_group call-back to pci_device_group() for the Intel VT-d and the AMD IOMMU driver. Signed-off-by: Joerg Roedel <jroedel@suse.de>
| | * | | iommu/vt-d: Switch from ioremap_cache to memremapDan Williams2015-10-141-9/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In preparation for deprecating ioremap_cache() convert its usage in intel-iommu to memremap. This also eliminates the mishandling of the __iomem annotation in the implementation. Cc: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
| | * | | iommu/vt-d: Create RMRR mappings in newly allocated domainsJoerg Roedel2015-10-051-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently the RMRR entries are created only at boot time. This means they will vanish when the domain allocated at boot time is destroyed. This patch makes sure that also newly allocated domains will get RMRR mappings. Signed-off-by: Joerg Roedel <jroedel@suse.de>
| | * | | iommu/vt-d: Split iommu_prepare_identity_mapJoerg Roedel2015-10-051-20/+22
| | | |/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Split the part of the function that fetches the domain out and put the rest into into a domain_prepare_identity_map, so that the code can also be used with when the domain is already known. Signed-off-by: Joerg Roedel <jroedel@suse.de>
* | | | Merge git://git.infradead.org/intel-iommuLinus Torvalds2015-11-061-48/+255
|\ \ \ \ | |/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull intel iommu updates from David Woodhouse: "This adds "Shared Virtual Memory" (aka PASID support) for the Intel IOMMU. This allows devices to do DMA using process address space, translated through the normal CPU page tables for the relevant mm. With corresponding support added to the i915 driver, this has been tested with the graphics device on Skylake. We don't have the required TLP support in our PCIe root ports for supporting discrete devices yet, so it's only integrated devices that can do it so far" * git://git.infradead.org/intel-iommu: (23 commits) iommu/vt-d: Fix rwxp flags in SVM device fault callback iommu/vt-d: Expose struct svm_dev_ops without CONFIG_INTEL_IOMMU_SVM iommu/vt-d: Clean up pasid_enabled() and ecs_enabled() dependencies iommu/vt-d: Handle Caching Mode implementations of SVM iommu/vt-d: Fix SVM IOTLB flush handling iommu/vt-d: Use dev_err(..) in intel_svm_device_to_iommu(..) iommu/vt-d: fix a loop in prq_event_thread() iommu/vt-d: Fix IOTLB flushing for global pages iommu/vt-d: Fix address shifting in page request handler iommu/vt-d: shift wrapping bug in prq_event_thread() iommu/vt-d: Fix NULL pointer dereference in page request error case iommu/vt-d: Implement SVM_FLAG_SUPERVISOR_MODE for kernel access iommu/vt-d: Implement SVM_FLAG_PRIVATE_PASID to allocate unique PASIDs iommu/vt-d: Add callback to device driver on page faults iommu/vt-d: Implement page request handling iommu/vt-d: Generalise DMAR MSI setup to allow for page request events iommu/vt-d: Implement deferred invalidate for SVM iommu/vt-d: Add basic SVM PASID support iommu/vt-d: Always enable PASID/PRI PCI capabilities before ATS iommu/vt-d: Add initial support for PASID tables ...
| * | | iommu/vt-d: Clean up pasid_enabled() and ecs_enabled() dependenciesDavid Woodhouse2015-10-241-7/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When booted with intel_iommu=ecs_off we were still allocating the PASID tables even though we couldn't actually use them. We really want to make the pasid_enabled() macro depend on ecs_enabled(). Which is unfortunate, because currently they're the other way round to cope with the Broadwell/Skylake problems with ECS. Instead of having ecs_enabled() depend on pasid_enabled(), which was never something that made me happy anyway, make it depend in the normal case on the "broken PASID" bit 28 *not* being set. Then pasid_enabled() can depend on ecs_enabled() as it should. And we also don't need to mess with it if we ever see an implementation that has some features requiring ECS (like PRI) but which *doesn't* have PASID support. Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Use dev_err(..) in intel_svm_device_to_iommu(..)Sudeep Dutt2015-10-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This will give a little bit of assistance to those developing drivers using SVM. It might cause a slight annoyance to end-users whose kernel disables the IOMMU when drivers are trying to use it. But the fix there is to fix the kernel to enable the IOMMU. Signed-off-by: Sudeep Dutt <sudeep.dutt@intel.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Implement page request handlingDavid Woodhouse2015-10-151-1/+21
| | | | | | | | | | | | | | | | | | | | | | | | Largely based on the driver-mode implementation by Jesse Barnes. Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Implement deferred invalidate for SVMDavid Woodhouse2015-10-151-0/+2
| | | | | | | | | | | | | | | | Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Add basic SVM PASID supportDavid Woodhouse2015-10-151-0/+104
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This provides basic PASID support for endpoint devices, tested with a version of the i915 driver. Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Always enable PASID/PRI PCI capabilities before ATSDavid Woodhouse2015-10-151-35/+75
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The behaviour if you enable PASID support after ATS is undefined. So we have to enable it first, even if we don't know whether we'll need it. This is safe enough; unless we set up a context that permits it, the device can't actually *do* anything with it. Also shift the feature detction to dmar_insert_one_dev_info() as it only needs to happen once. Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Add initial support for PASID tablesDavid Woodhouse2015-10-151-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | Add CONFIG_INTEL_IOMMU_SVM, and allocate PASID tables on supported hardware. Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Introduce intel_iommu=pasid28, and pasid_enabled() macroDavid Woodhouse2015-10-151-6/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As long as we use an identity mapping to work around the worst of the hardware bugs which caused us to defeature it and change the definition of the capability bit, we *can* use PASID support on the devices which advertised it in bit 28 of the Extended Capability Register. Allow people to do so with 'intel_iommu=pasid28' on the command line. Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
| * | | iommu/vt-d: Fix ATSR handling for Root-Complex integrated endpointsDavid Woodhouse2015-10-151-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The VT-d specification says that "Software must enable ATS on endpoint devices behind a Root Port only if the Root Port is reported as supporting ATS transactions." We walk up the tree to find a Root Port, but for integrated devices we don't find one — we get to the host bridge. In that case we *should* allow ATS. Currently we don't, which means that we are incorrectly failing to use ATS for the integrated graphics. Fix that. We should never break out of this loop "naturally" with bus==NULL, since we'll always find bridge==NULL in that case (and now return 1). So remove the check for (!bridge) after the loop, since it can never happen. If it did, it would be worthy of a BUG_ON(!bridge). But since it'll oops anyway in that case, that'll do just as well. Cc: stable@vger.kernel.org Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
* | | | Merge tag 'for-linus-20151021' of git://git.infradead.org/intel-iommuLinus Torvalds2015-10-211-4/+8
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull intel-iommu bugfix from David Woodhouse: "This contains a single fix, for when the IOMMU API is used to overlay an existing mapping comprised of 4KiB pages, with a mapping that can use superpages. For the *first* superpage in the new mapping, we were correctly¹ freeing the old bottom-level page table page and clearing the link to it, before installing the superpage. For subsequent superpages, however, we weren't. This causes a memory leak, and a warning about setting a PTE which is already set. ¹ Well, not *entirely* correctly. We just free the page table pages right there and then, which is wrong. In fact they should only be freed *after* the IOTLB is flushed so we know the hardware will no longer be looking at them.... and in fact I note that the IOTLB flush is completely missing from the intel_iommu_map() code path, although it needs to be there if it's permitted to overwrite existing mappings. Fixing those is somewhat more intrusive though, and will probably need to wait for 4.4 at this point" * tag 'for-linus-20151021' of git://git.infradead.org/intel-iommu: iommu/vt-d: fix range computation when making room for large pages
| * | | iommu/vt-d: fix range computation when making room for large pagesChristian Zander2015-10-131-4/+8
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In preparation for the installation of a large page, any small page tables that may still exist in the target IOV address range are removed. However, if a scatter/gather list entry is large enough to fit more than one large page, the address space for any subsequent large pages is not cleared of conflicting small page tables. This can cause legitimate mapping requests to fail with errors of the form below, potentially followed by a series of IOMMU faults: ERROR: DMA PTE for vPFN 0xfde00 already set (to 7f83a4003 not 7e9e00083) In this example, a 4MiB scatter/gather list entry resulted in the successful installation of a large page @ vPFN 0xfdc00, followed by a failed attempt to install another large page @ vPFN 0xfde00, due to the presence of a pointer to a small page table @ 0x7f83a4000. To address this problem, compute the number of large pages that fit into a given scatter/gather list entry, and use it to derive the last vPFN covered by the large page(s). Cc: stable@vger.kernel.org Signed-off-by: Christian Zander <christian@nervanasys.com> Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
* | | Merge tag 'iommu-fixes-v4.3-rc5' of ↵Linus Torvalds2015-10-131-0/+1
|\ \ \ | |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu Pull IOMMU fixes from Joerg Roedel: "A few fixes piled up: - Fix for a suspend/resume issue where PCI probing code overwrote dev->irq for the MSI irq of the AMD IOMMU. - Fix for a kernel crash when a 32 bit PCI device was assigned to a KVM guest. - Fix for a possible memory leak in the VT-d driver - A couple of fixes for the ARM-SMMU driver" * tag 'iommu-fixes-v4.3-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu: iommu/amd: Fix NULL pointer deref on device detach iommu/amd: Prevent binding other PCI drivers to IOMMU PCI devices iommu/vt-d: Fix memory leak in dmar_insert_one_dev_info() iommu/arm-smmu: Use correct address mask for CMD_TLBI_S2_IPA iommu/arm-smmu: Ensure IAS is set correctly for AArch32-capable SMMUs iommu/io-pgtable-arm: Don't use dma_to_phys()
| * | iommu/vt-d: Fix memory leak in dmar_insert_one_dev_info()Sudip Mukherjee2015-09-291-0/+1
| |/ | | | | | | | | | | | | | | We are returning NULL if we are not able to attach the iommu to the domain but while returning we missed freeing info. Signed-off-by: Sudip Mukherjee <sudip@vectorindia.org> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* | Merge git://git.infradead.org/intel-iommuLinus Torvalds2015-10-021-3/+5
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull IOVA fixes from David Woodhouse: "The main fix here is the first one, fixing the over-allocation of size-aligned requests. The other patches simply make the existing IOVA code available to users other than the Intel VT-d driver, with no functional change. I concede the latter really *should* have been submitted during the merge window, but since it's basically risk-free and people are waiting to build on top of it and it's my fault I didn't get it in, I (and they) would be grateful if you'd take it" * git://git.infradead.org/intel-iommu: iommu: Make the iova library a module iommu: iova: Export symbols iommu: iova: Move iova cache management to the iova library iommu/iova: Avoid over-allocating when size-aligned
generated by cgit v1.2.3-55-g7522 (git 2.39.0) at 2024-07-10 22:45:14 +0200