summaryrefslogtreecommitdiffstats
path: root/arch/arm/mm/flush.c
Commit message (Collapse)AuthorAgeFilesLines
* ARM: 8237/1: fix flush_pfn_aliasJungseung Lee2014-12-031-1/+1
| | | | | | | | | | | | L1_CACHE_BYTES could be larger than real L1 cache line size. In that case, flush_pfn_alias() would omit to flush last bytes as much as L1_CACHE_BYTES - real cache line size. So fix end address to "to + PAGE_SIZE - 1". The bottom bits of the address is LINELEN. that is ignored by mcrr. Signed-off-by: Jungseung Lee <js07.lee@gmail.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* arm: mm: enable RCU fast_gupSteve Capper2014-10-101-0/+15
| | | | | | | | | | | | | | | | | | | | | | Activate the RCU fast_gup for ARM. We also need to force THP splits to broadcast an IPI s.t. we block in the fast_gup page walker. As THP splits are comparatively rare, this should not lead to a noticeable performance degradation. Some pre-requisite functions pud_write and pud_page are also added. Signed-off-by: Steve Capper <steve.capper@linaro.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Dann Frazier <dann.frazier@canonical.com> Cc: Hugh Dickins <hughd@google.com> Cc: Russell King <rmk@arm.linux.org.uk> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mel Gorman <mel@csn.ul.ie> Cc: Will Deacon <will.deacon@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* ARM: 8043/1: uprobes need icache flush after xol writeVictor Kamensky2014-05-261-5/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After instruction write into xol area, on ARM V7 architecture code need to flush dcache and icache to sync them up for given set of addresses. Having just 'flush_dcache_page(page)' call is not enough - it is possible to have stale instruction sitting in icache for given xol area slot address. Introduce arch_uprobe_ixol_copy weak function that by default calls uprobes copy_to_page function and than flush_dcache_page function and on ARM define new one that handles xol slot copy in ARM specific way flush_uprobe_xol_access function shares/reuses implementation with/of flush_ptrace_access function and takes care of writing instruction to user land address space on given variety of different cache types on ARM CPUs. Because flush_uprobe_xol_access does not have vma around flush_ptrace_access was split into two parts. First that retrieves set of condition from vma and common that receives those conditions as flags. Note ARM cache flush function need kernel address through which instruction write happened, so instead of using uprobes copy_to_page function changed code to explicitly map page and do memcpy. Note arch_uprobe_copy_ixol function, in similar way as copy_to_user_page function, has preempt_disable/preempt_enable. Signed-off-by: Victor Kamensky <victor.kamensky@linaro.org> Acked-by: Oleg Nesterov <oleg@redhat.com> Reviewed-by: David A. Long <dave.long@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 7923/1: mm: fix dcache flush logic for compound high pagesSteven Capper2013-12-291-3/+3
| | | | | | | | | | | | | | | | | When given a compound high page, __flush_dcache_page will only flush the first page of the compound page repeatedly rather than the entire set of constituent pages. This error was introduced by: 0b19f93 ARM: mm: Add support for flushing HugeTLB pages. This patch corrects the logic such that all constituent pages are now flushed. Cc: stable@vger.kernel.org # 3.10+ Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* Merge branch 'devel-stable' into for-nextRussell King2013-06-291-10/+15
|\ | | | | | | | | | | Conflicts: arch/arm/Makefile arch/arm/include/asm/glue-proc.h
| * ARM: mm: Add support for flushing HugeTLB pages.Steve Capper2013-06-041-10/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On ARM we use the __flush_dcache_page function to flush the dcache of pages when needed; usually when the PG_dcache_clean bit is unset and we are setting a PTE. A HugeTLB page is represented as a compound page consisting of an array of pages. Thus to flush the dcache of a HugeTLB page, one must flush more than a single page. This patch modifies __flush_dcache_page such that all constituent pages of a HugeTLB page are flushed. Signed-off-by: Steve Capper <steve.capper@linaro.org> Reviewed-by: Will Deacon <will.deacon@arm.com>
| |
| \
*-. \ Merge branches 'fixes', 'mcpm', 'misc' and 'mmci' into for-nextRussell King2013-06-291-1/+1
|\ \ \ | | |/ | |/|
| | * ARM: 7746/1: mm: lazy cache flushing on non-mapped pagesMing Lei2013-06-061-1/+1
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently flush_dcache_page() thinks pages as non-mapped if mapping_mapped(mapping) return false. This approach is very coase: - mmap on part of file may cause all pages backed on the file being thought as mmaped - file-backed pages aren't mapped into user space actually if the memory mmaped on the file isn't accessed This patch uses page_mapped() to decide if the page has been mapped. From the attached test code, I find there is much performance improvement(>25%) when accessing page caches via read under this situations, so memcpy benefits a lot from not flushing cache under this situation. No. read time without the patch No. read time with the patch ================================================================ No. 0, time 22615636 us No. 0, time 22014717 us No. 1, time 4387851 us No. 1, time 3113184 us No. 2, time 4276535 us No. 2, time 3005244 us No. 3, time 4259821 us No. 3, time 3001565 us No. 4, time 4263811 us No. 4, time 3002748 us No. 5, time 4258486 us No. 5, time 3004104 us No. 6, time 4253009 us No. 6, time 3002188 us No. 7, time 4262809 us No. 7, time 2998196 us No. 8, time 4264525 us No. 8, time 3007255 us No. 9, time 4267795 us No. 9, time 3005094 us 1), No.0. is to read the file from storage device, and others are to read the file from page caches basically. 2), file size is 512M, and is on ext4 over usb mass storage. 3), the test is done on Pandaboard. unsigned int sum = 0; unsigned long sum_val = 0; static unsigned long tv_diff(struct timeval *tv1, struct timeval *tv2) { return (tv2->tv_sec - tv1->tv_sec) * 1000000 + (tv2->tv_usec - tv1->tv_usec); } int main(int argc, char *argv[]) { char *mbuf, fbuf; int fd; int i; unsigned long page_size, size; struct stat stat; struct timeval t1, t2; unsigned char *rbuf = malloc(32 * page_size); if (!rbuf) { printf(" %sn", "malloc failed"); exit(-1); } page_size = getpagesize(); fd = open(argv[1], O_RDWR); assert(fd >= 0); fstat(fd, &stat); size = stat.st_size; printf("%s: file %s, size %lu, page size %lun", argv[0], argv[1], size, page_size); gettimeofday(&t1, NULL); mbuf = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); if (!mbuf) { printf(" %sn", "mmap failed"); exit(-1); } for (i = 0 ; i < size ; i += (page_size * 32)) { int rcnt; lseek(fd, i, SEEK_SET); rcnt = read(fd, rbuf, page_size * 32); if (rcnt != page_size * 32) { printf("%s: read faildn", __func__); exit(-1); } } free(rbuf); munmap(mbuf, size); gettimeofday(&t2, NULL); printf("tread mmaped time: %luusn", tv_diff(&t1, &t2)); close(fd); } Cc: Michel Lespinasse <walken@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Nicolas Pitre <nicolas.pitre@linaro.org> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Ming Lei <ming.lei@canonical.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* / ARM: 7755/1: handle user space mapped pages in flush_kernel_dcache_pageSimon Baatz2013-06-171-0/+33
|/ | | | | | | | | | | | | | | Commit f8b63c1 made flush_kernel_dcache_page a no-op assuming that the pages it needs to handle are kernel mapped only. However, for example when doing direct I/O, pages with user space mappings may occur. Thus, continue to do lazy flushing if there are no user space mappings. Otherwise, flush the kernel cache lines directly. Signed-off-by: Simon Baatz <gmbnomis@gmail.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: <stable@vger.kernel.org> # 3.2+ Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 7693/1: mm: clean-up in order to reduce to call kmap_high_get()Joonsoo Kim2013-04-171-6/+9
| | | | | | | | | | | | | | | In kmap_atomic(), kmap_high_get() is invoked for checking already mapped area. In __flush_dcache_page() and dma_cache_maint_page(), we explicitly call kmap_high_get() before kmap_atomic() when cache_is_vipt(), so kmap_high_get() can be invoked twice. This is useless operation, so remove one. v2: change cache_is_vipt() to cache_is_vipt_nonaliasing() in order to be self-documented Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* mm: replace vma prio_tree with an interval treeMichel Lespinasse2012-10-091-2/+1Star
| | | | | | | | | | | | | | | | | | | | | | Implement an interval tree as a replacement for the VMA prio_tree. The algorithms are similar to lib/interval_tree.c; however that code can't be directly reused as the interval endpoints are not explicitly stored in the VMA. So instead, the common algorithm is moved into a template and the details (node type, how to get interval endpoints from the node, etc) are filled in using the C preprocessor. Once the interval tree functions are available, using them as a replacement to the VMA prio tree is a relatively simple, mechanical job. Signed-off-by: Michel Lespinasse <walken@google.com> Cc: Rik van Riel <riel@redhat.com> Cc: Hillf Danton <dhillf@gmail.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Woodhouse <dwmw2@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* ARM: 7487/1: mm: avoid setting nG bit for user mappings that aren't presentWill Deacon2012-08-111-2/+0Star
| | | | | | | | | | | | | | | | | | | | | Swap entries are encoding in ptes such that !pte_present(pte) and pte_file(pte). The remaining bits of the descriptor are used to identify the swapfile and offset within it to the swap entry. When writing such a pte for a user virtual address, set_pte_at unconditionally sets the nG bit, which (in the case of LPAE) will corrupt the swapfile offset and lead to a BUG: [ 140.494067] swap_free: Unused swap offset entry 000763b4 [ 140.509989] BUG: Bad page map in process rs:main Q:Reg pte:0ec76800 pmd:8f92e003 This patch fixes the problem by only setting the nG bit for user mappings that are actually present. Cc: <stable@vger.kernel.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* Merge branch 'for-linus' of git://git.linaro.org/people/rmk/linux-armLinus Torvalds2012-03-301-9/+5Star
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull more ARM updates from Russell King. This got a fair number of conflicts with the <asm/system.h> split, but also with some other sparse-irq and header file include cleanups. They all looked pretty trivial, though. * 'for-linus' of git://git.linaro.org/people/rmk/linux-arm: (59 commits) ARM: fix Kconfig warning for HAVE_BPF_JIT ARM: 7361/1: provide XIP_VIRT_ADDR for no-MMU builds ARM: 7349/1: integrator: convert to sparse irqs ARM: 7259/3: net: JIT compiler for packet filters ARM: 7334/1: add jump label support ARM: 7333/2: jump label: detect %c support for ARM ARM: 7338/1: add support for early console output via semihosting ARM: use set_current_blocked() and block_sigmask() ARM: exec: remove redundant set_fs(USER_DS) ARM: 7332/1: extract out code patch function from kprobes ARM: 7331/1: extract out insn generation code from ftrace ARM: 7330/1: ftrace: use canonical Thumb-2 wide instruction format ARM: 7351/1: ftrace: remove useless memory checks ARM: 7316/1: kexec: EOI active and mask all interrupts in kexec crash path ARM: Versatile Express: add NO_IOPORT ARM: get rid of asm/irq.h in asm/prom.h ARM: 7319/1: Print debug info for SIGBUS in user faults ARM: 7318/1: gic: refactor irq_start assignment ARM: 7317/1: irq: avoid NULL check in for_each_irq_desc loop ARM: 7315/1: perf: add support for the Cortex-A7 PMU ...
| * ARM: pgtable: consolidate set_pte_ext(TOP_PTE,...) + tlb flushRussell King2012-01-261-6/+4Star
| | | | | | | | | | | | | | | | | | | | | | | | | | A number of places establish a PTE in our top page table and immediately flush the TLB. Rather than having this at every callsite, provide an inline function for this purpose. This changes some global tlb flushes to be local; each time we setup one of these mappings, we always do it with preemption disabled which would prevent us migrating to another CPU. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
| * ARM: pgtable: move TOP_PTE address definitions to arch/arm/mm/mm.hRussell King2012-01-261-5/+3Star
| | | | | | | | | | | | | | | | | | Move the TOP_PTE address definitions to one central place so that it's easy to discover what they're being used for. This helps to ensure that there are no overlaps. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* | Disintegrate asm/system.h for ARMDavid Howells2012-03-281-1/+0Star
|/ | | | | | | | Disintegrate asm/system.h for ARM. Signed-off-by: David Howells <dhowells@redhat.com> cc: Russell King <linux@arm.linux.org.uk> cc: linux-arm-kernel@lists.infradead.org
*-. Merge branches 'consolidate', 'ep93xx', 'fixes', 'misc', 'mmci', 'remove' ↵Russell King2011-05-231-3/+4
|\ \ | | | | | | | | | and 'spear' into for-linus
| | * ARM: 6901/1: remove unneeded check of the cache_is_vipt_nonaliasing()saeed bishara2011-05-161-2/+2
| |/ |/| | | | | | | | | | | | | | | when cache_is_vipt_nonaliasing(), we always have pte_exec() true at the end of this function, so no need for the additional check. Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Saeed Bishara <saeed@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
| * ARM: 6899/1: fix the note about dcache lazy flushing for SMP systemssaeed bishara2011-05-201-1/+2
|/ | | | | | Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Saeed Bishara <saeed@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* Merge branch 'for-next' of ↵Linus Torvalds2011-01-131-1/+0Star
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial * 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial: (43 commits) Documentation/trace/events.txt: Remove obsolete sched_signal_send. writeback: fix global_dirty_limits comment runtime -> real-time ppc: fix comment typo singal -> signal drivers: fix comment typo diable -> disable. m68k: fix comment typo diable -> disable. wireless: comment typo fix diable -> disable. media: comment typo fix diable -> disable. remove doc for obsolete dynamic-printk kernel-parameter remove extraneous 'is' from Documentation/iostats.txt Fix spelling milisec -> ms in snd_ps3 module parameter description Fix spelling mistakes in comments Revert conflicting V4L changes i7core_edac: fix typos in comments mm/rmap.c: fix comment sound, ca0106: Fix assignment to 'channel'. hrtimer: fix a typo in comment init/Kconfig: fix typo anon_inodes: fix wrong function name in comment fix comment typos concerning "consistent" poll: fix a typo in comment ... Fix up trivial conflicts in: - drivers/net/wireless/iwlwifi/iwl-core.c (moved to iwl-legacy.c) - fs/ext4/ext4.h Also fix missed 'diabled' typo in drivers/net/bnx2x/bnx2x.h while at it.
| * ARM, mm: Don't include smp_plat.h twice in flush.cJesper Juhl2010-11-151-1/+0Star
| | | | | | | | | | | | | | It's enough to include the asm/smp_plat.h once in arch/arm/mm/flush.c Signed-off-by: Jesper Juhl <jj@chaosbits.net> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
* | ARM: get rid of kmap_high_l1_vipt()Nicolas Pitre2010-12-191-3/+4
|/ | | | | | | | | | | | | Since commit 3e4d3af501 "mm: stack based kmap_atomic()", it is no longer necessary to carry an ad hoc version of kmap_atomic() added in commit 7e5a69e83b "ARM: 6007/1: fix highmem with VIPT cache and DMA" to cope with reentrancy. In fact, it is now actively wrong to rely on fixed kmap type indices (namely KM_L1_CACHE) as kmap_atomic() totally ignores them now and a concurrent instance of it may reuse any slot for any purpose. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
* ARM: 6386/1: flush_ptrace_access: invalidate correct I-cache aliasWill Deacon2010-10-041-3/+20
| | | | | | | | | | | | | | | | | copy_to_user_page can be used by access_process_vm to write to an executable page of a process using a mapping acquired by kmap. For systems with I-cache aliasing, flushing the I-cache using the Kernel mapping may leave stale data in the I-cache if the user mapping is of a different colour. This patch introduces a flush_icache_alias function to flush.c, which calls flush_icache_range with a mapping of the specified colour. flush_ptrace_access is then modified to call this new function instead of coherent_kern_range in the case of an aliasing I-cache and a non-aliasing D-cache. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 6381/1: Use lazy cache flushing on ARMv7 SMP systemsCatalin Marinas2010-09-191-9/+4Star
| | | | | | | | | | | | | | | | | ARMv7 processors like Cortex-A9 broadcast the cache maintenance operations in hardware. This patch allows the flush_dcache_page/update_mmu_cache pair to work in lazy flushing mode similar to the UP case. Note that cache flushing on SMP systems now takes place via the set_pte_at() call (__sync_icache_dcache) and there is no race with other CPUs executing code from the new PTE before the cache flushing took place. Tested-by: Rabin Vincent <rabin.vincent@stericsson.com> Cc: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 6380/1: Introduce __sync_icache_dcache() for VIPT cachesCatalin Marinas2010-09-191-0/+30
| | | | | | | | | | | | | | | | | | On SMP systems, there is a small chance of a PTE becoming visible to a different CPU before the current cache maintenance operations in update_mmu_cache(). To avoid this, cache maintenance must be handled in set_pte_at() (similar to IA-64 and PowerPC). This patch provides a unified VIPT cache handling mechanism and implements the __sync_icache_dcache() function for ARMv6 onwards architectures. It is called from set_pte_at() and replaces the update_mmu_cache(). The latter is still used on VIVT hardware where a vm_area_struct is required. Tested-by: Rabin Vincent <rabin.vincent@stericsson.com> Cc: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 6379/1: Assume new page cache pages have dirty D-cacheCatalin Marinas2010-09-191-1/+2
| | | | | | | | | | | | | | | | There are places in Linux where writes to newly allocated page cache pages happen without a subsequent call to flush_dcache_page() (several PIO drivers including USB HCD). This patch changes the meaning of PG_arch_1 to be PG_dcache_clean and always flush the D-cache for a newly mapped page in update_mmu_cache(). The patch also sets the PG_arch_1 bit in the DMA cache maintenance function to avoid additional cache flushing in update_mmu_cache(). Tested-by: Rabin Vincent <rabin.vincent@stericsson.com> Cc: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 6378/1: Allow lazy cache flushing via PG_arch_1 for highmem pagesCatalin Marinas2010-09-191-1/+1
| | | | | | | | | | | Commit d73cd42 forced non-lazy cache flushing of highmem pages in flush_dcache_page(). This isn't needed since __flush_dcache_page() (called lazily from update_mmu_cache) can handle highmem pages (fixed by commit 7e5a69e). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: 6007/1: fix highmem with VIPT cache and DMANicolas Pitre2010-04-141-10/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The VIVT cache of a highmem page is always flushed before the page is unmapped. This cache flush is explicit through flush_cache_kmaps() in flush_all_zero_pkmaps(), or through __cpuc_flush_dcache_area() in kunmap_atomic(). There is also an implicit flush of those highmem pages that were part of a process that just terminated making those pages free as the whole VIVT cache has to be flushed on every task switch. Hence unmapped highmem pages need no cache maintenance in that case. However unmapped pages may still be cached with a VIPT cache because the cache is tagged with physical addresses. There is no need for a whole cache flush during task switching for that reason, and despite the explicit cache flushes in flush_all_zero_pkmaps() and kunmap_atomic(), some highmem pages that were mapped in user space end up still cached even when they become unmapped. So, we do have to perform cache maintenance on those unmapped highmem pages in the context of DMA when using a VIPT cache. Unfortunately, it is not possible to perform that cache maintenance using physical addresses as all the L1 cache maintenance coprocessor functions accept virtual addresses only. Therefore we have no choice but to set up a temporary virtual mapping for that purpose. And of course the explicit cache flushing when unmapping a highmem page on a system with a VIPT cache now can go, which should increase performance. While at it, because the code in __flush_dcache_page() has to be modified anyway, let's also make sure the mapped highmem pages are pinned with kmap_high_get() for the duration of the cache maintenance operation. Because kunmap() does unmap highmem pages lazily, it was reported by Gary King <GKing@nvidia.com> that those pages ended up being unmapped during cache maintenance on SMP causing segmentation faults. Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: Fix ptrace accessesRussell King2009-12-141-8/+43
| | | | Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: add size argument to __cpuc_flush_dcache_pageRussell King2009-12-141-2/+2
| | | | | | | ... and rename the function since it no longer operates on just pages. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: I-cache: flush executable mappings in flush_cache_range()Russell King2009-12-041-2/+1Star
| | | | | | | | | | | | | | | | | | | | | Dirk Behme reported instability on ARM11 SMP (VIPT non-aliasing cache) caused by the dynamic linker changing protection on text pages to write GOT entries. The problem is due to an interaction between the write faulting code providing new anonymous pages which are incoherent with the I-cache due to write buffering, and the I-cache not having been invalidated. a4db94d plugs the hole with the data cache coherency. This patch provides the other half of the fix by flushing the I-cache in flush_cache_range() for VM_EXEC VMAs (which is what we have when the region is being made executable again.) This ensures that the I-cache will be up to date with the newly COW'd pages. Note: if users are writing instructions, then they still need to use the ARM sys_cacheflush API to ensure that the caches are correctly synchronized. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: I-cache: avoid flushing in flush_cache_mm()Russell King2009-12-041-1/+0Star
| | | | | | | | | | | | | | | | | | flush_cache_mm() is called in two cases: 1. when a process exits, just before the page tables are torn down. We can allow the stale lines to evict themselves over time without causing any harm. 2. when a process forks, and we've allocated a new ASID. The instruction cache issues are dealt with as pages are brought into the new process address space. Flushing the I-cache here is therefore unnecessary. However, we must keep the VIPT aliasing D-cache flush to ensure that any dirty cache lines are not written back after the pages have been reallocated for some other use - which would result in corruption. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: I-cache: Add invalidation for VIVT ASID tagged cachesRussell King2009-12-041-0/+6
| | | | Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: Remove __flush_icache_all() from __flush_dcache_page()Russell King2009-12-041-3/+1Star
| | | | | | | | Both call sites for __flush_dcache_page() end up calling __flush_icache_all() themselves, so having __flush_dcache_page() do this as well is wasteful. Remove the duplicated icache flushing. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: Move __flush_icache_all() out of flush_pfn_alias()Russell King2009-12-041-3/+8
| | | | Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: ZERO_PAGE: Avoid flush_dcache_page() for zero pageRussell King2009-12-011-1/+10
| | | | | | | The zero page is read-only, and has its cache state cleared during boot. No further maintanence for this page is required. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: Avoid evaluating page_address() multiple timesRussell King2009-12-011-2/+4
| | | | | | | | | | | | page_address() is a function call rather than a macro, and so: if (page_address(page)) do_something(page_address(page)); results in two calls to this function. This is unnecessary; remove the duplication. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: Avoid duplicated implementation for VIVT cache flushingRussell King2009-12-011-13/+4Star
| | | | | | | | We had two copies of the wrapper code for VIVT cache flushing - one in asm/cacheflush.h and one in arch/arm/mm/flush.c. Reduce this down to one common copy. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* ARM: Fix errata 411920 workaroundsRussell King2009-10-291-25/+6Star
| | | | | | | | | | Errata 411920 indicates that any "invalidate entire instruction cache" operation can fail if the right conditions are present. This is not limited just to those operations in flush.c, but elsewhere. Place the workaround in the already existing __flush_icache_all() function instead. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* cpumask: use mm_cpumask() wrapper: armRusty Russell2009-09-241-5/+5
| | | | | | | | | Makes code futureproof against the impending change to mm->cpu_vm_mask. It's also a chance to use the new cpumask_ ops which take a pointer (the older ones are deprecated, but there's no hurry for arch code). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ARM: 5687/1: fix an oops with highmemNicolas Pitre2009-09-021-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | In xdr_partial_copy_from_skb() there is that sequence: kaddr = kmap_atomic(*ppage, KM_SKB_SUNRPC_DATA); [...] flush_dcache_page(*ppage); kunmap_atomic(kaddr, KM_SKB_SUNRPC_DATA); Mixing flush_dcache_page() and kmap_atomic() is a bit odd, especially since kunmap_atomic() must deal with cache issues already. OTOH the non-highmem case must use flush_dcache_page() as kunmap_atomic() becomes a no op with no cache maintenance. Problem is that with highmem the implementation of kmap_atomic() doesn't set page->virtual, and page_address(page) returns 0 in that case. Here flush_dcache_page() calls __flush_dcache_page() which calls __cpuc_flush_dcache_page(page_address(page)) resulting in a kernel oops. None of the kmap_atomic() implementations uses set_page_address(). Hence we can assume page_address() is always expected to return 0 in that case. Let's conditionally call __cpuc_flush_dcache_page() only when the page address is non zero, and perform that test only when highmem is configured. Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] 5488/1: ARM errata: Invalidation of the Instruction Cache operation ↵Catalin Marinas2009-04-301-2/+21
| | | | | | | | | | can fail This patch implements the recommended workaround for erratum 411920 (ARM1136, ARM1156, ARM1176). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] kmap supportNicolas Pitre2009-03-161-1/+1
| | | | | | | | | | | | | | | | | | | | The kmap virtual area borrows a 2MB range at the top of the 16MB area below PAGE_OFFSET currently reserved for kernel modules and/or the XIP kernel. This 2MB corresponds to the range covered by 2 consecutive second-level page tables, or a single pmd entry as seen by the Linux page table abstraction. Because XIP kernels are unlikely to be seen on systems needing highmem support, there shouldn't be any shortage of VM space for modules (14 MB for modules is still way more than twice the typical usage). Because the virtual mapping of highmem pages can go away at any moment after kunmap() is called on them, we need to bypass the delayed cache flushing provided by flush_dcache_page() in that case. The atomic kmap versions are based on fixmaps, and __cpuc_flush_dcache_page() is used directly in that case. Signed-off-by: Nicolas Pitre <nico@marvell.com>
* [ARM] cachetype: move definitions to separate headerRussell King2008-09-011-0/+1
| | | | | | | | Rather than pollute asm/cacheflush.h with the cache type definitions, move them to asm/cachetype.h, and include this new header where necessary. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] 5092/1: Fix the I-cache invalidation on ARMv6 and later CPUsCatalin Marinas2008-07-031-0/+2
| | | | | | | | | This patch adds the I-cache invalidation in update_mmu_cache if the corresponding vma is marked as executable. It also invalidates the I-cache if a thread migrates to a CPU it never ran on. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] Resolve fuse and direct-IO failures due to missing cache flushesRussell King2007-01-081-0/+39
| | | | | | | | | | | | | | | fuse does not work on ARM due to cache incoherency issues - fuse wants to use get_user_pages() to copy data from the current process into kernel space. However, since this accesses userspace via the kernel mapping, the kernel mapping can be out of date wrt data written to userspace. This can lead to unpredictable behaviour (in the case of fuse) or data corruption for direct-IO. This resolves debian bug #402876 Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] Unuse another Linux PTE bitRussell King2006-12-131-1/+1
| | | | | | | | | | L_PTE_ASID is not really required to be stored in every PTE, since we can identify it via the address passed to set_pte_at(). So, create set_pte_ext() which takes the address of the PTE to set, the Linux PTE value, and the additional CPU PTE bits which aren't encoded in the Linux PTE value. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] 3853/1: Fix flush_ptrace_access() thinko for nonaliasing VIPT cache caseGeorge G. Davis2006-09-251-1/+1
| | | | | | | | | | Fix thinko in the flush_ptrace_access() "if (expr)" for the ARM VIPT non-aliasing cache case. We only need to flush cache when VM_EXEC is set in vma->vm_flags but "if (expr) always evaluates to true on UP systems for the ARM VIPT non-aliasing cache case. Signed-off-by: George G. Davis <gdavis@mvista.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] Cleanup arch/arm/mm a littleRussell King2006-09-201-2/+2
| | | | | | | | | | Move top_pmd into arch/arm/mm/mm.h - nothing outside arch/arm/mm references it. Move the repeated definition of TOP_PTE into mm/mm.h, as well as a few function prototypes. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
* [ARM] 3762/1: Fix ptrace cache coherency bug for ARM1136 VIPT nonaliasing ↵George G. Davis2006-09-021-0/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Harvard caches Patch from George G. Davis Resolve ARM1136 VIPT non-aliasing cache coherency issues observed when using ptrace to set breakpoints and cleanup copy_{to,from}_user_page() while we're here as requested by Russell King because "it's also far too heavy on non-v6 CPUs". NOTES: 1. Only access_process_vm() calls copy_{to,from}_user_page(). 2. access_process_vm() calls get_user_pages() to pin down the "page". 3. get_user_pages() calls flush_dcache_page(page) which ensures cache coherency between kernel and userspace mappings of "page". However flush_dcache_page(page) may not invalidate I-Cache over this range for all cases, specifically, I-Cache is not invalidated for the VIPT non-aliasing case. So memory is consistent between kernel and user space mappings of "page" but I-Cache may still be hot over this range. IOW, we don't have to worry about flush_cache_page() before memcpy(). 4. Now, for the copy_to_user_page() case, after memcpy(), we must flush the caches so memory is consistent with kernel cache entries and invalidate the I-Cache if this mm region is executable. We don't need to do anything after memcpy() for the copy_from_user_page() case since kernel cache entries will be invalidated via the same process above if we access "page" again. The flush_ptrace_access() function (borrowed from SPARC64 implementation) is added to handle cache flushing after memcpy() for the copy_to_user_page() case. Signed-off-by: George G. Davis <gdavis@mvista.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
generated by cgit v1.2.3-55-g7522 (git 2.39.0) at 2024-11-12 21:52:23 +0100