From de30ec47302c101c7badc8fe687641fd75e596e7 Mon Sep 17 00:00:00 2001 From: Nicholas Mc Guire Date: Sat, 17 Jan 2015 05:05:34 +0100 Subject: sched/completion: Remove unnecessary ->wait.lock serialization when reading completion state Signed-off-by: Nicholas Mc Guire Signed-off-by: Peter Zijlstra (Intel) Cc: Linus Torvalds Link: http://lkml.kernel.org/r/1421467534-22834-1-git-send-email-der.herr@hofr.at Signed-off-by: Ingo Molnar --- kernel/sched/completion.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) (limited to 'kernel/sched') diff --git a/kernel/sched/completion.c b/kernel/sched/completion.c index 607f852b4d04..9d1fe32da232 100644 --- a/kernel/sched/completion.c +++ b/kernel/sched/completion.c @@ -288,13 +288,6 @@ EXPORT_SYMBOL(try_wait_for_completion); */ bool completion_done(struct completion *x) { - unsigned long flags; - int ret = 1; - - spin_lock_irqsave(&x->wait.lock, flags); - if (!x->done) - ret = 0; - spin_unlock_irqrestore(&x->wait.lock, flags); - return ret; + return !!ACCESS_ONCE(x->done); } EXPORT_SYMBOL(completion_done); -- cgit v1.2.3-55-g7522 From 7c34e3180a01c800a40bc8535654d5735802fc1b Mon Sep 17 00:00:00 2001 From: Nicholas Mc Guire Date: Fri, 23 Jan 2015 12:41:47 +0100 Subject: sched/completion: Add lock-free checking of the blocking case The "thread would block" case can be checked without grabbing ->wait.lock. [ If the check does not return early then grab the lock and recheck. A memory barrier is not needed as complete() and complete_all() imply a barrier. The ACCESS_ONCE() is needed for calls in a loop that, if inlined, could optimize out the re-fetching of x->done. ] Signed-off-by: Nicholas Mc Guire Signed-off-by: Peter Zijlstra (Intel) Cc: Linus Torvalds Link: http://lkml.kernel.org/r/1422013307-13200-1-git-send-email-der.herr@hofr.at Signed-off-by: Ingo Molnar --- kernel/sched/completion.c | 9 +++++++++ 1 file changed, 9 insertions(+) (limited to 'kernel/sched') diff --git a/kernel/sched/completion.c b/kernel/sched/completion.c index 9d1fe32da232..7052d3fd4e7b 100644 --- a/kernel/sched/completion.c +++ b/kernel/sched/completion.c @@ -268,6 +268,15 @@ bool try_wait_for_completion(struct completion *x) unsigned long flags; int ret = 1; + /* + * Since x->done will need to be locked only + * in the non-blocking case, we check x->done + * first without taking the lock so we can + * return early in the blocking case. + */ + if (!ACCESS_ONCE(x->done)) + return 0; + spin_lock_irqsave(&x->wait.lock, flags); if (!x->done) ret = 0; -- cgit v1.2.3-55-g7522