Pull SMP/hotplug updates from Thomas Gleixner:
"A small set of updates for SMP and CPU hotplug:
- Abort disabling secondary CPUs in the freezer when a wakeup is
pending instead of evaluating it only after all CPUs have been
offlined.
- Remove the shared annotation for the strict per CPU cfd_data in the
smp function call core code.
- Remove the return values of smp_call_function() and on_each_cpu()
as they are unconditionally 0. Fixup the few callers which actually
bothered to check the return value"
* 'smp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
smp: Remove smp_call_function() and on_each_cpu() return values
smp: Do not mark call_function_data as shared
cpu/hotplug: Abort disabling secondary CPUs if wakeup is pending
cpu/hotplug: Fix notify_cpu_starting() reference in bringup_wait_for_ap()
The return value is fixed. Remove it and amend the callers.
[ tglx: Fixup arm/bL_switcher and powerpc/rtas ]
Signed-off-by: Nadav Amit <namit@vmware.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Link: https://lkml.kernel.org/r/20190613064813.8102-2-namit@vmware.com
Based on 2 normalized pattern(s):
this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation
this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation #
extracted by the scancode license scanner the SPDX license identifier
GPL-2.0-only
has been chosen to replace the boilerplate/reference in 4122 file(s).
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Enrico Weigelt <info@metux.net>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Fix up affected files that include this signal functionality via sched.h.
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
We are going to move scheduler ABI details to <uapi/linux/sched/types.h>,
which will be used from a number of .c files.
Create empty placeholder header that maps to <linux/types.h>.
Include the new header in the files that are going to need it.
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Use the new tick_suspend/resume_local() and get rid of the
homebrewn implementation of these in the ARM bL switcher. The
check for the cpumask is completely pointless. There is no harm
to suspend a per cpu tick device unconditionally. If that's a
real issue then we fix it proper at the core level and not with
some completely undocumented hacks in some random core code.
Move the tick internals to the core code, now that this nuisance
is gone.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
[ rjw: Rebase, changelog ]
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Russell King <rmk+kernel@arm.linux.org.uk>
Link: http://lkml.kernel.org/r/1655112.Ws17YsMfN7@vostro.rjw.lan
Signed-off-by: Ingo Molnar <mingo@kernel.org>
'enum clock_event_mode' is used for two purposes today:
- to pass mode to the driver of clockevent device::set_mode().
- for managing state of the device for clockevents core.
For supporting new modes/states we have moved away from the
legacy set_mode() callback to new per-mode/state callbacks. New
modes/states shouldn't be exposed to the legacy (now OBSOLOTE)
callbacks and so we shouldn't add new states to 'enum
clock_event_mode'.
Lets have separate enums for the two use cases mentioned above.
Keep using the earlier enum for legacy set_mode() callback and
mark it OBSOLETE. And add another enum to clearly specify the
possible states of a clockevent device.
This also renames the newly added per-mode callbacks to reflect
state changes.
We haven't got rid of 'mode' member of 'struct
clock_event_device' as it is used by some of the clockevent
drivers and it would automatically die down once we migrate
those drivers to the new interface. It ('mode') is only updated
now for the drivers using the legacy interface.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Suggested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Preeti U Murthy <preeti@linux.vnet.ibm.com>
Cc: linaro-kernel@lists.linaro.org
Cc: linaro-networking@linaro.org
Cc: linux-arm-kernel@lists.infradead.org
Link: http://lkml.kernel.org/r/b6b0143a8a57bd58352ad35e08c25424c879c0cb.1425037853.git.viresh.kumar@linaro.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Use the nanoseconds based interface instead of converting from a
timespec.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: linux-arm-kernel@lists.infradead.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
The content of /sys/devices/system/cpu/cpu*/online is still 1 for those
CPUs that the switcher has removed even though the global state in
/sys/devices/system/cpu/online is updated correctly.
It turns out that commit 0902a9044f ("Driver core: Use generic
offline/online for CPU offline/online") has changed the way those files
retrieve their content by relying on on the generic attribute handling
code. The switcher, by calling cpu_down() directly, bypasses this
handling and the attribute value doesn't get updated.
Fix this by calling device_offline()/device_online() instead.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
The switcher should not depend on MAX_CLUSTER to determine ifit should
be activated or not. In a multiplatform kernel binary it is possible to
have dual-cluster and quad-cluster platforms configured in. In that case
MAX_CLUSTER which is a build time limit should be 4 and that shouldn't
prevent the switcher from working if the kernel is booted on a b.L
dual-cluster system.
In bL_switcher_halve_cpus() we already have a runtime validation check
to make sure we're dealing with only two clusters, so booting on a quad
cluster system will be caught and switcher activation aborted.
However, the b.L switcher must ensure the MCPM layer is initialized on
the booted hardware before doing anything. The mcpm_is_available()
function is added to that effect.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Tested-by: Abhilash Kesavan <kesavan.abhilash@gmail.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Commit 6dedcca610 ("hotplug, powerpc, x86: Remove
cpu_hotplug_driver_lock())" removes the the definition of
cpu_hotplug_driver_{lock,unlock} APIs, thereby causing a build error.
Replace these calls with {lock,unlock}_device_hotplug().
Signed-off-by: Tushar Behera <tushar.behera@linaro.org>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
When the switcher is active, there is no straightforward way to
figure out which logical CPU a given physical CPU maps to.
This patch provides a function
bL_switcher_get_logical_index(mpidr), which is analogous to
get_logical_index().
This function returns the logical CPU on which the specified
physical CPU is grouped (or -EINVAL if unknown).
If the switcher is inactive or not present, -EUNATCH is returned instead.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
This patch exports a bL_switcher_trace_trigger() function to
provide a means for drivers using the trace events to get the
current status when starting a trace session.
Calling this function is equivalent to pinging the trace_trigger
file in sysfs.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
When tracing switching, an external tracer needs a way to bootstrap
its knowledge of the logical<->physical CPU mapping.
This patch adds a sysfs attribute trace_trigger. A write to this
attribute will generate a power:cpu_migrate_current event for each
online CPU, indicating the current physical CPU for each logical
CPU.
Activating or deactivating the switcher also generates these
events, so that the tracer knows about the resulting remapping of
affected CPUs.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
This patch adds simple trace events to the b.L switcher code
to allow tracing of CPU migration events.
To make use of the trace events, you will need:
CONFIG_FTRACE=y
CONFIG_ENABLE_DEFAULT_TRACERS=y
The following events are added:
* power:cpu_migrate_begin
* power:cpu_migrate_finish
each with the following data:
u64 timestamp;
u32 cpu_hwid;
power:cpu_migrate_begin occurs immediately before the
switcher-specific migration operations start.
power:cpu_migrate_finish occurs immediately when migration is
completed.
The cpu_hwid field contains the ID fields of the MPIDR.
* For power:cpu_migrate_begin, cpu_hwid is the ID of the outbound
physical CPU (equivalent to (from_phys_cpu,from_phys_cluster)).
* For power:cpu_migrate_finish, cpu_hwid is the ID of the inbound
physical CPU (equivalent to (to_phys_cpu,to_phys_cluster)).
By design, the cpu_hwid field is masked in the same way as the
device tree cpu node reg property, allowing direct correlation to
the DT description of the hardware.
The timestamp is added in order to minimise timing noise. An
accurate system-wide clock should be used for generating this
(hopefully getnstimeofday is appropriate, but it could be changed).
It could be any monotonic shared clock, since the aim is to allow
accurate deltas to be computed. We don't necessarily care about
accurate synchronisation with wall clock time.
In practice, each switch takes place on a single logical CPU,
and the trace infrastructure should guarantee that events are
well-ordered with respect to a single logical CPU.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
In some cases, a significant delay may be observed between the moment
a request for a CPU to come up is made and the moment it is ready to
start executing kernel code. This is especially true when a whole
cluster has to be powered up which may take in the order of miliseconds.
It is therefore a good idea to let the outbound CPU continue to execute
code in the mean time, and be notified when the inbound is ready before
performing the actual switch.
This is achieved by registering a completion block with the appropriate
IPI callback, and programming the sending of an IPI by the early assembly
code prior to entering the main kernel code. Once the IPI is delivered
to the outbound CPU, the completion block is "completed" and the switcher
thread is resumed.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Let's wait for the inbound CPU to come up and snoop some of the outbound
CPU cache before bringing the outbound CPU down. That should be more
efficient than going down right away.
Possible improvements might involve some monitoring of the CCI event
counters.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
There is no explicit way to know when a switch started via
bL_switch_request() is complete. This can lead to unpredictable
behaviour when the switcher is controlled by a subsystem which
makes dynamic decisions (such as cpufreq).
The CPU PM notifier is not really suitable for signalling
completion, because the CPU could get suspended and resumed for
other, independent reasons while a switch request is in flight.
Adding a whole new notifier for this seems excessive, and may tempt
people to put heavyweight code on this path.
This patch implements a new bL_switch_request_cb() function that
allows for a per-request lightweight callback, private between the
switcher and the caller of bL_switch_request_cb().
Overlapping switches on a single CPU are considered incorrect if
they are requested via bL_switch_request_cb() with a callback (they
will lead to an unpredictable final state without explicit external
synchronisation to force the requests into a particular order).
Queuing requests robustly would be overkill because only one
subsystem should be attempting to control the switcher at any time.
Overlapping requests of this kind will be failed with -EBUSY to
indicate that the second request won't take effect and the
completer will never be called for it.
bL_switch_request() is retained as a wrapper round the new function,
with the old, fire-and-forget semantics. In this case the last request
will always win. The request may still be denied if a previous request
with a completer is still pending.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Some subsystems will need to respond synchronously to runtime
enabling and disabling of the switcher.
This patch adds a dedicated notifier interface to support such
subsystems. Pre- and post- enable/disable notifications are sent
to registered callbacks, allowing safe transition of non-b.L-
transparent subsystems across these control transitions.
Notifier callbacks may veto switcher (de)activation on pre notifications
only. Post notifications won't revert the action.
If enabling or disabling of the switcher fails after the pre-change
notification has been sent, subsystems which have registered
notifiers can be left in an inappropriate state.
This patch sends a suitable post-change notification on failure,
indicating that the old state has been reestablished.
For example, a failed initialisation will result in the following
sequence:
BL_NOTIFY_PRE_ENABLE
/* switcher initialisation fails */
BL_NOTIFY_POST_DISABLE
It is the responsibility of notified subsystems to respond in an
appropriate way.
Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Some subsystems will need to know for sure whether the switcher is
enabled or disabled during certain critical regions.
This patch provides a simple mutex-based mechanism to discover
whether the switcher is enabled and temporarily lock out further
enable/disable:
* bL_switcher_get_enabled() returns true iff the switcher is
enabled and temporarily inhibits enable/disable.
* bL_switcher_put_enabled() permits enable/disable of the switcher
again after a previous call to bL_switcher_get_enabled().
Signed-off-by: Dave Martin <dave.martin@linaro.org>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Trying to support both the switcher and CPU hotplug at the same time
is tricky due to ambiguous semantics. So let's at least prevent users
from messing around with those logical CPUs the switcher has removed
and those which were not active when the switcher was activated.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Up to now, the logical CPU was somehow tied to the physical CPU number
within a cluster. This causes problems when forcing the boot CPU to be
different from the first enumerated CPU in the device tree creating a
discrepancy between logical and physical CPU numbers.
Let's make the pairing completely independent from physical CPU numbers.
Let's keep only those logical CPUs with same initial CPU cluster to create
a uniform scheduler profile without having to modify any of the probed
topology and compute capacity data. This has the potential to create
a non contiguous CPU numbering space when the switcher is active with
potential impact on buggy user space tools. It is however better to fix
those tools rather than making the switcher code more intrusive.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Reviewed-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
By adding no_bL_switcher to the kernel cmdline string, the switcher
won't be activated automatically at boot time. It is still possible
to activate it later with:
echo 1 > /sys/kernel/bL_switcher/active
Signed-off-by: Nicolas Pitre <nico@linaro.org>
The /sys/kernel/bL_switcher/enable file allows to enable or disable
the switcher by writing 1 or 0 to it respectively. It is still enabled
by default on boot.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Currently, GIC IDs are hardcoded making the code dependent on the 4+4 b.L
configuration. Let's allow for GIC IDs to be discovered upon switcher
initialization to support other b.L configurations such as the 1+1 one,
or 2+3 as on the VExpress TC2.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
In a regular kernel configuration, all the CPUs are initially available.
But the switcher execution model uses half of them at any time. Instead
of hacking the DTB to remove half of the CPUs, let's remove them at
run time and make sure we still have a working switcher configuration.
This way, the same DTB can be used whether or not the switcher is used.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
The workqueues are problematic as they may be contended.
They can't be scheduled with top priority either. Also the optimization
in bL_switch_request() to skip the workqueue entirely when the target CPU
and the calling CPU were the same didn't allow for bL_switch_request() to
be called from atomic context, as might be the case for some cpufreq
drivers.
Let's move to dedicated kthreads instead.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Per-CPU timers that are shutdown when a CPU is switched over must be disabled
upon switching and reprogrammed on the inbound CPU by relying on the
clock events management API. save/restore sequence is executed with irqs
disabled as mandated by the clock events API.
The next_event is an absolute time, hence, when the inbound CPU resumes,
if the timer has expired the min delta is forced into the tick device to
fire after few cycles.
This patch adds switching support for clock events that are per-CPU and
have to be migrated when a switch takes place; the cpumask of the clock
event device is checked against the cpumask of the current cpu, and if
they match, the clockevent device mode is saved and it is put in
shutdown mode. Resume code reprogrammes the tick device accordingly.
Tested on A15/A7 fast models and architected timers.
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Nicolas Pitre <nico@linaro.org>
This is the core code implementing big.LITTLE switcher functionality.
Rationale for this code is available here:
http://lwn.net/Articles/481055/
The main entry point for a switch request is:
void bL_switch_request(unsigned int cpu, unsigned int new_cluster_id)
If the calling CPU is not the wanted one, this wrapper takes care of
sending the request to the appropriate CPU with schedule_work_on().
At the moment the core switch operation is handled by bL_switch_to()
which must be called on the CPU for which a switch is requested.
What this code does:
* Return early if the current cluster is the wanted one.
* Close the gate in the kernel entry vector for both the inbound
and outbound CPUs.
* Wake up the inbound CPU so it can perform its reset sequence in
parallel up to the kernel entry vector gate.
* Migrate all interrupts in the GIC targeting the outbound CPU
interface to the inbound CPU interface, including SGIs. This is
performed by gic_migrate_target() in drivers/irqchip/irq-gic.c.
* Call cpu_pm_enter() which takes care of flushing the VFP state to
RAM and save the CPU interface config from the GIC to RAM.
* Modify the cpu_logical_map to refer to the inbound physical CPU.
* Call cpu_suspend() which saves the CPU state (general purpose
registers, page table address) onto the stack and store the
resulting stack pointer in an array indexed by the updated
cpu_logical_map, then call the provided shutdown function.
This happens in arch/arm/kernel/sleep.S.
At this point, the provided shutdown function executed by the outbound
CPU ungates the inbound CPU. Therefore the inbound CPU:
* Picks up the saved stack pointer in the array indexed by its MPIDR
in arch/arm/kernel/sleep.S.
* The MMU and caches are re-enabled using the saved state on the
provided stack, just like if this was a resume operation from a
suspended state.
* Then cpu_suspend() returns, although this is on the inbound CPU
rather than the outbound CPU which called it initially.
* The function cpu_pm_exit() is called which effect is to restore the
CPU interface state in the GIC using the state previously saved by
the outbound CPU.
* Exit of bL_switch_to() to resume normal kernel execution on the
new CPU.
However, the outbound CPU is potentially still running in parallel while
the inbound CPU is resuming normal kernel execution, hence we need
per CPU stack isolation to execute bL_do_switch(). After the outbound
CPU has ungated the inbound CPU, it calls mcpm_cpu_power_down() to:
* Clean its L1 cache.
* If it is the last CPU still alive in its cluster (last man standing),
it also cleans its L2 cache and disables cache snooping from the other
cluster.
* Power down the CPU (or whole cluster).
Code called from bL_do_switch() might end up referencing 'current' for
some reasons. However, 'current' is derived from the stack pointer.
With any arbitrary stack, the returned value for 'current' and any
dereferenced values through it are just random garbage which may lead to
segmentation faults.
The active page table during the execution of bL_do_switch() is also a
problem. There is no guarantee that the inbound CPU won't destroy the
corresponding task which would free the attached page table while the
outbound CPU is still running and relying on it.
To solve both issues, we borrow some of the task space belonging to
the init/idle task which, by its nature, is lightly used and therefore
is unlikely to clash with our usage. The init task is also never going
away.
Right now the logical CPU number is assumed to be equivalent to the
physical CPU number within each cluster. The kernel should also be
booted with only one cluster active. These limitations will be lifted
eventually.
Signed-off-by: Nicolas Pitre <nico@linaro.org>
Linus Torvalds
Nadav Amit
Thomas Gleixner
Ingo Molnar
Sebastian Andrzej Siewior
Viresh Kumar
Nicolas Pitre
Tushar Behera
Dave Martin
Lorenzo Pieralisi