2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* Architecture-specific setup.
|
|
|
|
*
|
|
|
|
* Copyright (C) 1998-2003 Hewlett-Packard Co
|
|
|
|
* David Mosberger-Tang <davidm@hpl.hp.com>
|
2005-04-23 01:44:40 +04:00
|
|
|
* 04/11/17 Ashok Raj <ashok.raj@intel.com> Added CPU Hotplug Support
|
[IA64] Extend notify_die() hooks for IA64
notify_die() added for MCA_{MONARCH,SLAVE,RENDEZVOUS}_{ENTER,PROCESS,LEAVE} and
INIT_{MONARCH,SLAVE}_{ENTER,PROCESS,LEAVE}. We need multiple
notification points for these events because they can take many seconds
to run which has nasty effects on the behaviour of the rest of the
system.
DIE_SS replaced by a generic DIE_FAULT which checks the vector number,
to allow interception of faults other than SS.
DIE_MACHINE_{HALT,RESTART} added to allow last minute close down
processing, especially when the halt/restart routines are called from
error handlers.
DIE_OOPS added.
The check for kprobe's break numbers has been moved from traps.c to
kprobes.c, allowing DIE_BREAK to be used for any additional break
numbers, i.e. it is no longer kprobes specific.
Hooks for kernel debuggers and kernel dumpers added, ENTER and LEAVE.
Both of these disable the system for long periods which impact on
watchdogs and heartbeat systems in general. More patches to come that
use these events to reset watchdogs and heartbeats.
unregister_die_notifier() added and both routines exported. Requested
by Dean Nelson.
Lock removed from {un,}register_die_notifier. notifier_chain_register()
already takes a lock. Also the generic notifier chain locking is being
reworked to distinguish between callbacks that can block and those that
cannot, the lock in {un,}register_die_notifier would interfere with
that change. http://marc.theaimsgroup.com/?l=linux-kernel&m=113018709002036&w=2
Leading white space removed from arch/ia64/kernel/kprobes.c.
Typo in mca.c in original version of this patch found & fixed by Dean
Nelson.
Signed-off-by: Keith Owens <kaos@sgi.com>
Acked-by: Dean Nelson <dcn@sgi.com>
Acked-by: Anil Keshavamurthy <anil.s.keshavamurthy@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
2005-11-07 22:27:13 +03:00
|
|
|
*
|
|
|
|
* 2005-10-07 Keith Owens <kaos@sgi.com>
|
|
|
|
* Add notify_die() hooks.
|
2005-04-17 02:20:36 +04:00
|
|
|
*/
|
|
|
|
#include <linux/cpu.h>
|
|
|
|
#include <linux/pm.h>
|
|
|
|
#include <linux/elf.h>
|
|
|
|
#include <linux/errno.h>
|
|
|
|
#include <linux/kallsyms.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/mm.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 11:04:11 +03:00
|
|
|
#include <linux/slab.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/notifier.h>
|
|
|
|
#include <linux/personality.h>
|
|
|
|
#include <linux/sched.h>
|
|
|
|
#include <linux/stddef.h>
|
|
|
|
#include <linux/thread_info.h>
|
|
|
|
#include <linux/unistd.h>
|
|
|
|
#include <linux/efi.h>
|
|
|
|
#include <linux/interrupt.h>
|
|
|
|
#include <linux/delay.h>
|
2007-05-08 11:27:03 +04:00
|
|
|
#include <linux/kdebug.h>
|
2007-12-18 22:46:38 +03:00
|
|
|
#include <linux/utsname.h>
|
2008-10-06 21:43:06 +04:00
|
|
|
#include <linux/tracehook.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#include <asm/cpu.h>
|
|
|
|
#include <asm/delay.h>
|
|
|
|
#include <asm/elf.h>
|
|
|
|
#include <asm/irq.h>
|
2007-02-06 00:49:10 +03:00
|
|
|
#include <asm/kexec.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <asm/pgalloc.h>
|
|
|
|
#include <asm/processor.h>
|
|
|
|
#include <asm/sal.h>
|
|
|
|
#include <asm/tlbflush.h>
|
|
|
|
#include <asm/uaccess.h>
|
|
|
|
#include <asm/unwind.h>
|
|
|
|
#include <asm/user.h>
|
|
|
|
|
|
|
|
#include "entry.h"
|
|
|
|
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
# include <asm/perfmon.h>
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#include "sigframe.h"
|
|
|
|
|
|
|
|
void (*ia64_mark_idle)(int);
|
|
|
|
|
2010-11-03 19:06:14 +03:00
|
|
|
unsigned long boot_option_idle_override = IDLE_NO_OVERRIDE;
|
2005-04-17 02:20:36 +04:00
|
|
|
EXPORT_SYMBOL(boot_option_idle_override);
|
2010-02-26 00:09:52 +03:00
|
|
|
void (*pm_idle) (void);
|
|
|
|
EXPORT_SYMBOL(pm_idle);
|
|
|
|
void (*pm_power_off) (void);
|
|
|
|
EXPORT_SYMBOL(pm_power_off);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
void
|
|
|
|
ia64_do_show_stack (struct unw_frame_info *info, void *arg)
|
|
|
|
{
|
|
|
|
unsigned long ip, sp, bsp;
|
|
|
|
char buf[128]; /* don't make it so big that it overflows the stack! */
|
|
|
|
|
|
|
|
printk("\nCall Trace:\n");
|
|
|
|
do {
|
|
|
|
unw_get_ip(info, &ip);
|
|
|
|
if (ip == 0)
|
|
|
|
break;
|
|
|
|
|
|
|
|
unw_get_sp(info, &sp);
|
|
|
|
unw_get_bsp(info, &bsp);
|
|
|
|
snprintf(buf, sizeof(buf),
|
|
|
|
" [<%016lx>] %%s\n"
|
|
|
|
" sp=%016lx bsp=%016lx\n",
|
|
|
|
ip, sp, bsp);
|
|
|
|
print_symbol(buf, ip);
|
|
|
|
} while (unw_unwind(info) >= 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
show_stack (struct task_struct *task, unsigned long *sp)
|
|
|
|
{
|
|
|
|
if (!task)
|
|
|
|
unw_init_running(ia64_do_show_stack, NULL);
|
|
|
|
else {
|
|
|
|
struct unw_frame_info info;
|
|
|
|
|
|
|
|
unw_init_from_blocked_task(&info, task);
|
|
|
|
ia64_do_show_stack(&info, NULL);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
dump_stack (void)
|
|
|
|
{
|
|
|
|
show_stack(NULL, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
EXPORT_SYMBOL(dump_stack);
|
|
|
|
|
|
|
|
void
|
|
|
|
show_regs (struct pt_regs *regs)
|
|
|
|
{
|
|
|
|
unsigned long ip = regs->cr_iip + ia64_psr(regs)->ri;
|
|
|
|
|
|
|
|
print_modules();
|
2007-10-19 10:40:41 +04:00
|
|
|
printk("\nPid: %d, CPU %d, comm: %20s\n", task_pid_nr(current),
|
|
|
|
smp_processor_id(), current->comm);
|
2007-12-18 22:46:38 +03:00
|
|
|
printk("psr : %016lx ifs : %016lx ip : [<%016lx>] %s (%s)\n",
|
|
|
|
regs->cr_ipsr, regs->cr_ifs, ip, print_tainted(),
|
|
|
|
init_utsname()->release);
|
2005-04-17 02:20:36 +04:00
|
|
|
print_symbol("ip is at %s\n", ip);
|
|
|
|
printk("unat: %016lx pfs : %016lx rsc : %016lx\n",
|
|
|
|
regs->ar_unat, regs->ar_pfs, regs->ar_rsc);
|
|
|
|
printk("rnat: %016lx bsps: %016lx pr : %016lx\n",
|
|
|
|
regs->ar_rnat, regs->ar_bspstore, regs->pr);
|
|
|
|
printk("ldrs: %016lx ccv : %016lx fpsr: %016lx\n",
|
|
|
|
regs->loadrs, regs->ar_ccv, regs->ar_fpsr);
|
|
|
|
printk("csd : %016lx ssd : %016lx\n", regs->ar_csd, regs->ar_ssd);
|
|
|
|
printk("b0 : %016lx b6 : %016lx b7 : %016lx\n", regs->b0, regs->b6, regs->b7);
|
|
|
|
printk("f6 : %05lx%016lx f7 : %05lx%016lx\n",
|
|
|
|
regs->f6.u.bits[1], regs->f6.u.bits[0],
|
|
|
|
regs->f7.u.bits[1], regs->f7.u.bits[0]);
|
|
|
|
printk("f8 : %05lx%016lx f9 : %05lx%016lx\n",
|
|
|
|
regs->f8.u.bits[1], regs->f8.u.bits[0],
|
|
|
|
regs->f9.u.bits[1], regs->f9.u.bits[0]);
|
|
|
|
printk("f10 : %05lx%016lx f11 : %05lx%016lx\n",
|
|
|
|
regs->f10.u.bits[1], regs->f10.u.bits[0],
|
|
|
|
regs->f11.u.bits[1], regs->f11.u.bits[0]);
|
|
|
|
|
|
|
|
printk("r1 : %016lx r2 : %016lx r3 : %016lx\n", regs->r1, regs->r2, regs->r3);
|
|
|
|
printk("r8 : %016lx r9 : %016lx r10 : %016lx\n", regs->r8, regs->r9, regs->r10);
|
|
|
|
printk("r11 : %016lx r12 : %016lx r13 : %016lx\n", regs->r11, regs->r12, regs->r13);
|
|
|
|
printk("r14 : %016lx r15 : %016lx r16 : %016lx\n", regs->r14, regs->r15, regs->r16);
|
|
|
|
printk("r17 : %016lx r18 : %016lx r19 : %016lx\n", regs->r17, regs->r18, regs->r19);
|
|
|
|
printk("r20 : %016lx r21 : %016lx r22 : %016lx\n", regs->r20, regs->r21, regs->r22);
|
|
|
|
printk("r23 : %016lx r24 : %016lx r25 : %016lx\n", regs->r23, regs->r24, regs->r25);
|
|
|
|
printk("r26 : %016lx r27 : %016lx r28 : %016lx\n", regs->r26, regs->r27, regs->r28);
|
|
|
|
printk("r29 : %016lx r30 : %016lx r31 : %016lx\n", regs->r29, regs->r30, regs->r31);
|
|
|
|
|
|
|
|
if (user_mode(regs)) {
|
|
|
|
/* print the stacked registers */
|
|
|
|
unsigned long val, *bsp, ndirty;
|
|
|
|
int i, sof, is_nat = 0;
|
|
|
|
|
|
|
|
sof = regs->cr_ifs & 0x7f; /* size of frame */
|
|
|
|
ndirty = (regs->loadrs >> 19);
|
|
|
|
bsp = ia64_rse_skip_regs((unsigned long *) regs->ar_bspstore, ndirty);
|
|
|
|
for (i = 0; i < sof; ++i) {
|
|
|
|
get_user(val, (unsigned long __user *) ia64_rse_skip_regs(bsp, i));
|
|
|
|
printk("r%-3u:%c%016lx%s", 32 + i, is_nat ? '*' : ' ', val,
|
|
|
|
((i == sof - 1) || (i % 3) == 2) ? "\n" : " ");
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
show_stack(NULL, NULL);
|
|
|
|
}
|
|
|
|
|
2009-09-14 22:13:37 +04:00
|
|
|
/* local support for deprecated console_print */
|
|
|
|
void
|
|
|
|
console_print(const char *s)
|
|
|
|
{
|
|
|
|
printk(KERN_EMERG "%s", s);
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
void
|
2008-05-09 10:26:35 +04:00
|
|
|
do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
if (fsys_mode(current, &scr->pt)) {
|
2008-05-09 10:26:35 +04:00
|
|
|
/*
|
|
|
|
* defer signal-handling etc. until we return to
|
|
|
|
* privilege-level 0.
|
|
|
|
*/
|
2005-04-17 02:20:36 +04:00
|
|
|
if (!ia64_psr(&scr->pt)->lp)
|
|
|
|
ia64_psr(&scr->pt)->lp = 1;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
if (current->thread.pfm_needs_checking)
|
2008-05-09 10:26:35 +04:00
|
|
|
/*
|
|
|
|
* Note: pfm_handle_work() allow us to call it with interrupts
|
|
|
|
* disabled, and may enable interrupts within the function.
|
|
|
|
*/
|
2005-04-17 02:20:36 +04:00
|
|
|
pfm_handle_work();
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/* deal with pending signal delivery */
|
2008-05-09 10:26:35 +04:00
|
|
|
if (test_thread_flag(TIF_SIGPENDING)) {
|
|
|
|
local_irq_enable(); /* force interrupt enable */
|
2007-01-23 19:03:17 +03:00
|
|
|
ia64_do_signal(scr, in_syscall);
|
2008-05-09 10:26:35 +04:00
|
|
|
}
|
2007-12-12 17:23:34 +03:00
|
|
|
|
2008-10-06 21:43:06 +04:00
|
|
|
if (test_thread_flag(TIF_NOTIFY_RESUME)) {
|
|
|
|
clear_thread_flag(TIF_NOTIFY_RESUME);
|
|
|
|
tracehook_notify_resume(&scr->pt);
|
KEYS: Add a keyctl to install a process's session keyring on its parent [try #6]
Add a keyctl to install a process's session keyring onto its parent. This
replaces the parent's session keyring. Because the COW credential code does
not permit one process to change another process's credentials directly, the
change is deferred until userspace next starts executing again. Normally this
will be after a wait*() syscall.
To support this, three new security hooks have been provided:
cred_alloc_blank() to allocate unset security creds, cred_transfer() to fill in
the blank security creds and key_session_to_parent() - which asks the LSM if
the process may replace its parent's session keyring.
The replacement may only happen if the process has the same ownership details
as its parent, and the process has LINK permission on the session keyring, and
the session keyring is owned by the process, and the LSM permits it.
Note that this requires alteration to each architecture's notify_resume path.
This has been done for all arches barring blackfin, m68k* and xtensa, all of
which need assembly alteration to support TIF_NOTIFY_RESUME. This allows the
replacement to be performed at the point the parent process resumes userspace
execution.
This allows the userspace AFS pioctl emulation to fully emulate newpag() and
the VIOCSETTOK and VIOCSETTOK2 pioctls, all of which require the ability to
alter the parent process's PAG membership. However, since kAFS doesn't use
PAGs per se, but rather dumps the keys into the session keyring, the session
keyring of the parent must be replaced if, for example, VIOCSETTOK is passed
the newpag flag.
This can be tested with the following program:
#include <stdio.h>
#include <stdlib.h>
#include <keyutils.h>
#define KEYCTL_SESSION_TO_PARENT 18
#define OSERROR(X, S) do { if ((long)(X) == -1) { perror(S); exit(1); } } while(0)
int main(int argc, char **argv)
{
key_serial_t keyring, key;
long ret;
keyring = keyctl_join_session_keyring(argv[1]);
OSERROR(keyring, "keyctl_join_session_keyring");
key = add_key("user", "a", "b", 1, keyring);
OSERROR(key, "add_key");
ret = keyctl(KEYCTL_SESSION_TO_PARENT);
OSERROR(ret, "KEYCTL_SESSION_TO_PARENT");
return 0;
}
Compiled and linked with -lkeyutils, you should see something like:
[dhowells@andromeda ~]$ keyctl show
Session Keyring
-3 --alswrv 4043 4043 keyring: _ses
355907932 --alswrv 4043 -1 \_ keyring: _uid.4043
[dhowells@andromeda ~]$ /tmp/newpag
[dhowells@andromeda ~]$ keyctl show
Session Keyring
-3 --alswrv 4043 4043 keyring: _ses
1055658746 --alswrv 4043 4043 \_ user: a
[dhowells@andromeda ~]$ /tmp/newpag hello
[dhowells@andromeda ~]$ keyctl show
Session Keyring
-3 --alswrv 4043 4043 keyring: hello
340417692 --alswrv 4043 4043 \_ user: a
Where the test program creates a new session keyring, sticks a user key named
'a' into it and then installs it on its parent.
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: James Morris <jmorris@namei.org>
2009-09-02 12:14:21 +04:00
|
|
|
if (current->replacement_session_keyring)
|
|
|
|
key_replace_session_keyring();
|
2008-10-06 21:43:06 +04:00
|
|
|
}
|
|
|
|
|
2007-12-12 17:23:34 +03:00
|
|
|
/* copy user rbs to kernel rbs */
|
2008-05-09 10:26:35 +04:00
|
|
|
if (unlikely(test_thread_flag(TIF_RESTORE_RSE))) {
|
|
|
|
local_irq_enable(); /* force interrupt enable */
|
2007-12-12 17:23:34 +03:00
|
|
|
ia64_sync_krbs();
|
2008-05-09 10:26:35 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
local_irq_disable(); /* force interrupt disable */
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2005-04-12 00:45:00 +04:00
|
|
|
static int pal_halt = 1;
|
|
|
|
static int can_do_pal_halt = 1;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
static int __init nohalt_setup(char * str)
|
|
|
|
{
|
2005-08-09 02:25:00 +04:00
|
|
|
pal_halt = can_do_pal_halt = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
__setup("nohalt", nohalt_setup);
|
|
|
|
|
2005-05-04 03:21:45 +04:00
|
|
|
void
|
2005-04-12 00:45:00 +04:00
|
|
|
update_pal_halt_status(int status)
|
|
|
|
{
|
|
|
|
can_do_pal_halt = pal_halt && status;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* We use this if we don't have any better idle routine..
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
default_idle (void)
|
|
|
|
{
|
2005-04-19 07:06:47 +04:00
|
|
|
local_irq_enable();
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
while (!need_resched()) {
|
2007-08-07 17:49:32 +04:00
|
|
|
if (can_do_pal_halt) {
|
|
|
|
local_irq_disable();
|
|
|
|
if (!need_resched()) {
|
|
|
|
safe_halt();
|
|
|
|
}
|
|
|
|
local_irq_enable();
|
|
|
|
} else
|
2005-04-17 02:20:36 +04:00
|
|
|
cpu_relax();
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_HOTPLUG_CPU
|
|
|
|
/* We don't actually take CPU down, just spin without interrupts. */
|
|
|
|
static inline void play_dead(void)
|
|
|
|
{
|
2005-04-23 01:44:40 +04:00
|
|
|
unsigned int this_cpu = smp_processor_id();
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Ack it */
|
|
|
|
__get_cpu_var(cpu_state) = CPU_DEAD;
|
|
|
|
|
|
|
|
max_xtp();
|
|
|
|
local_irq_disable();
|
2005-04-23 01:44:40 +04:00
|
|
|
idle_task_exit();
|
|
|
|
ia64_jump_to_sal(&sal_boot_rendez_state[this_cpu]);
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
2005-04-23 01:44:40 +04:00
|
|
|
* The above is a point of no-return, the processor is
|
|
|
|
* expected to be in SAL loop now.
|
2005-04-17 02:20:36 +04:00
|
|
|
*/
|
2005-04-23 01:44:40 +04:00
|
|
|
BUG();
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
#else
|
|
|
|
static inline void play_dead(void)
|
|
|
|
{
|
|
|
|
BUG();
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_HOTPLUG_CPU */
|
|
|
|
|
2008-02-08 22:53:09 +03:00
|
|
|
static void do_nothing(void *unused)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2008-02-08 22:53:09 +03:00
|
|
|
}
|
2005-03-31 08:40:00 +04:00
|
|
|
|
2008-02-08 22:53:09 +03:00
|
|
|
/*
|
|
|
|
* cpu_idle_wait - Used to ensure that all the CPUs discard old value of
|
|
|
|
* pm_idle and update to new pm_idle value. Required while changing pm_idle
|
|
|
|
* handler on SMP systems.
|
|
|
|
*
|
|
|
|
* Caller must have changed pm_idle to the new value before the call. Old
|
|
|
|
* pm_idle value will not be used by any CPU after the return of this function.
|
|
|
|
*/
|
|
|
|
void cpu_idle_wait(void)
|
|
|
|
{
|
|
|
|
smp_mb();
|
|
|
|
/* kick all the CPUs so that they exit out of pm_idle */
|
2008-06-06 13:18:06 +04:00
|
|
|
smp_call_function(do_nothing, NULL, 1);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(cpu_idle_wait);
|
|
|
|
|
|
|
|
void __attribute__((noreturn))
|
|
|
|
cpu_idle (void)
|
|
|
|
{
|
|
|
|
void (*mark_idle)(int) = ia64_mark_idle;
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
int cpu = smp_processor_id();
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* endless idle loop with no priority at all */
|
|
|
|
while (1) {
|
[PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code
Fernando Lopez-Lezcano reported frequent scheduling latencies and audio
xruns starting at the 2.6.18-rt kernel, and those problems persisted all
until current -rt kernels. The latencies were serious and unjustified by
system load, often in the milliseconds range.
After a patient and heroic multi-month effort of Fernando, where he
tested dozens of kernels, tried various configs, boot options,
test-patches of mine and provided latency traces of those incidents, the
following 'smoking gun' trace was captured by him:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup <<...>-5856> (37 0)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (c01262ba 0 0)
IRQ_19-1479 1D..1 0us : resched_task (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __spin_unlock_irqrestore (try_to_wake_up)
...
<idle>-0 1...1 11us!: default_idle (cpu_idle)
...
<idle>-0 0Dn.1 602us : smp_apic_timer_interrupt (c0103baf 1 0)
...
<...>-5856 0D..2 618us : __switch_to (__schedule)
<...>-5856 0D..2 618us : __schedule <<idle>-0> (20 162)
<...>-5856 0D..2 619us : __spin_unlock_irq (__schedule)
<...>-5856 0...1 619us : trace_stop_sched_switched (__schedule)
<...>-5856 0D..1 619us : trace_stop_sched_switched <<...>-5856> (37 0)
what is visible in this trace is that CPU#1 ran try_to_wake_up() for
PID:5856, it placed PID:5856 on CPU#0's runqueue and ran resched_task()
for CPU#0. But it decided to not send an IPI that no CPU - due to
TS_POLLING. But CPU#0 never woke up after its NEED_RESCHED bit was set,
and only rescheduled to PID:5856 upon the next lapic timer IRQ. The
result was a 600+ usecs latency and a missed wakeup!
the bug turned out to be an idle-wakeup bug introduced into the mainline
kernel this summer via an optimization in the x86_64 tree:
commit 495ab9c045e1b0e5c82951b762257fe1c9d81564
Author: Andi Kleen <ak@suse.de>
Date: Mon Jun 26 13:59:11 2006 +0200
[PATCH] i386/x86-64/ia64: Move polling flag into thread_info_status
During some profiling I noticed that default_idle causes a lot of
memory traffic. I think that is caused by the atomic operations
to clear/set the polling flag in thread_info. There is actually
no reason to make this atomic - only the idle thread does it
to itself, other CPUs only read it. So I moved it into ti->status.
the problem is this type of change:
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ current_thread_info()->status &= ~TS_POLLING;
smp_mb__after_clear_bit();
while (!need_resched()) {
local_irq_disable();
this changes clear_thread_flag() to an explicit clearing of TS_POLLING.
clear_thread_flag() is defined as:
clear_bit(flag, &ti->flags);
and clear_bit() is a LOCK-ed atomic instruction on all x86 platforms:
static inline void clear_bit(int nr, volatile unsigned long * addr)
{
__asm__ __volatile__( LOCK_PREFIX
"btrl %1,%0"
hence smp_mb__after_clear_bit() is defined as a simple compile barrier:
#define smp_mb__after_clear_bit() barrier()
but the explicit TS_POLLING clearing introduced by the patch:
+ current_thread_info()->status &= ~TS_POLLING;
is not an atomic op! So the clearing of the TS_POLLING bit is freely
reorderable with the reading of the NEED_RESCHED bit - and both now
reside in different memory addresses.
CPU idle wakeup very much depends on ordered memory ops, the clearing of
the TS_POLLING flag must always be done before we test need_resched()
and hit the idle instruction(s). [Symmetrically, the wakeup code needs
to set NEED_RESCHED before it tests the TS_POLLING flag, so memory
ordering is paramount.]
Fernando's dual-core Athlon64 system has a sufficiently advanced memory
ordering model so that it triggered this scenario very often.
( And it also turned out that the reason why these latencies never
triggered on my testsystems is that i routinely use idle=poll, which
was the only idle variant not affected by this bug. )
The fix is to change the smp_mb__after_clear_bit() to an smp_mb(), to
act as an absolute barrier between the TS_POLLING write and the
NEED_RESCHED read. This affects almost all idling methods (default,
ACPI, APM), on all 3 x86 architectures: i386, x86_64, ia64.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Tested-by: Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-22 12:11:56 +03:00
|
|
|
if (can_do_pal_halt) {
|
2006-06-26 15:59:11 +04:00
|
|
|
current_thread_info()->status &= ~TS_POLLING;
|
[PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code
Fernando Lopez-Lezcano reported frequent scheduling latencies and audio
xruns starting at the 2.6.18-rt kernel, and those problems persisted all
until current -rt kernels. The latencies were serious and unjustified by
system load, often in the milliseconds range.
After a patient and heroic multi-month effort of Fernando, where he
tested dozens of kernels, tried various configs, boot options,
test-patches of mine and provided latency traces of those incidents, the
following 'smoking gun' trace was captured by him:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup <<...>-5856> (37 0)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (c01262ba 0 0)
IRQ_19-1479 1D..1 0us : resched_task (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __spin_unlock_irqrestore (try_to_wake_up)
...
<idle>-0 1...1 11us!: default_idle (cpu_idle)
...
<idle>-0 0Dn.1 602us : smp_apic_timer_interrupt (c0103baf 1 0)
...
<...>-5856 0D..2 618us : __switch_to (__schedule)
<...>-5856 0D..2 618us : __schedule <<idle>-0> (20 162)
<...>-5856 0D..2 619us : __spin_unlock_irq (__schedule)
<...>-5856 0...1 619us : trace_stop_sched_switched (__schedule)
<...>-5856 0D..1 619us : trace_stop_sched_switched <<...>-5856> (37 0)
what is visible in this trace is that CPU#1 ran try_to_wake_up() for
PID:5856, it placed PID:5856 on CPU#0's runqueue and ran resched_task()
for CPU#0. But it decided to not send an IPI that no CPU - due to
TS_POLLING. But CPU#0 never woke up after its NEED_RESCHED bit was set,
and only rescheduled to PID:5856 upon the next lapic timer IRQ. The
result was a 600+ usecs latency and a missed wakeup!
the bug turned out to be an idle-wakeup bug introduced into the mainline
kernel this summer via an optimization in the x86_64 tree:
commit 495ab9c045e1b0e5c82951b762257fe1c9d81564
Author: Andi Kleen <ak@suse.de>
Date: Mon Jun 26 13:59:11 2006 +0200
[PATCH] i386/x86-64/ia64: Move polling flag into thread_info_status
During some profiling I noticed that default_idle causes a lot of
memory traffic. I think that is caused by the atomic operations
to clear/set the polling flag in thread_info. There is actually
no reason to make this atomic - only the idle thread does it
to itself, other CPUs only read it. So I moved it into ti->status.
the problem is this type of change:
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ current_thread_info()->status &= ~TS_POLLING;
smp_mb__after_clear_bit();
while (!need_resched()) {
local_irq_disable();
this changes clear_thread_flag() to an explicit clearing of TS_POLLING.
clear_thread_flag() is defined as:
clear_bit(flag, &ti->flags);
and clear_bit() is a LOCK-ed atomic instruction on all x86 platforms:
static inline void clear_bit(int nr, volatile unsigned long * addr)
{
__asm__ __volatile__( LOCK_PREFIX
"btrl %1,%0"
hence smp_mb__after_clear_bit() is defined as a simple compile barrier:
#define smp_mb__after_clear_bit() barrier()
but the explicit TS_POLLING clearing introduced by the patch:
+ current_thread_info()->status &= ~TS_POLLING;
is not an atomic op! So the clearing of the TS_POLLING bit is freely
reorderable with the reading of the NEED_RESCHED bit - and both now
reside in different memory addresses.
CPU idle wakeup very much depends on ordered memory ops, the clearing of
the TS_POLLING flag must always be done before we test need_resched()
and hit the idle instruction(s). [Symmetrically, the wakeup code needs
to set NEED_RESCHED before it tests the TS_POLLING flag, so memory
ordering is paramount.]
Fernando's dual-core Athlon64 system has a sufficiently advanced memory
ordering model so that it triggered this scenario very often.
( And it also turned out that the reason why these latencies never
triggered on my testsystems is that i routinely use idle=poll, which
was the only idle variant not affected by this bug. )
The fix is to change the smp_mb__after_clear_bit() to an smp_mb(), to
act as an absolute barrier between the TS_POLLING write and the
NEED_RESCHED read. This affects almost all idling methods (default,
ACPI, APM), on all 3 x86 architectures: i386, x86_64, ia64.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Tested-by: Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-22 12:11:56 +03:00
|
|
|
/*
|
|
|
|
* TS_POLLING-cleared state must be visible before we
|
|
|
|
* test NEED_RESCHED:
|
|
|
|
*/
|
|
|
|
smp_mb();
|
|
|
|
} else {
|
2006-06-26 15:59:11 +04:00
|
|
|
current_thread_info()->status |= TS_POLLING;
|
[PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code
Fernando Lopez-Lezcano reported frequent scheduling latencies and audio
xruns starting at the 2.6.18-rt kernel, and those problems persisted all
until current -rt kernels. The latencies were serious and unjustified by
system load, often in the milliseconds range.
After a patient and heroic multi-month effort of Fernando, where he
tested dozens of kernels, tried various configs, boot options,
test-patches of mine and provided latency traces of those incidents, the
following 'smoking gun' trace was captured by him:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup <<...>-5856> (37 0)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (c01262ba 0 0)
IRQ_19-1479 1D..1 0us : resched_task (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __spin_unlock_irqrestore (try_to_wake_up)
...
<idle>-0 1...1 11us!: default_idle (cpu_idle)
...
<idle>-0 0Dn.1 602us : smp_apic_timer_interrupt (c0103baf 1 0)
...
<...>-5856 0D..2 618us : __switch_to (__schedule)
<...>-5856 0D..2 618us : __schedule <<idle>-0> (20 162)
<...>-5856 0D..2 619us : __spin_unlock_irq (__schedule)
<...>-5856 0...1 619us : trace_stop_sched_switched (__schedule)
<...>-5856 0D..1 619us : trace_stop_sched_switched <<...>-5856> (37 0)
what is visible in this trace is that CPU#1 ran try_to_wake_up() for
PID:5856, it placed PID:5856 on CPU#0's runqueue and ran resched_task()
for CPU#0. But it decided to not send an IPI that no CPU - due to
TS_POLLING. But CPU#0 never woke up after its NEED_RESCHED bit was set,
and only rescheduled to PID:5856 upon the next lapic timer IRQ. The
result was a 600+ usecs latency and a missed wakeup!
the bug turned out to be an idle-wakeup bug introduced into the mainline
kernel this summer via an optimization in the x86_64 tree:
commit 495ab9c045e1b0e5c82951b762257fe1c9d81564
Author: Andi Kleen <ak@suse.de>
Date: Mon Jun 26 13:59:11 2006 +0200
[PATCH] i386/x86-64/ia64: Move polling flag into thread_info_status
During some profiling I noticed that default_idle causes a lot of
memory traffic. I think that is caused by the atomic operations
to clear/set the polling flag in thread_info. There is actually
no reason to make this atomic - only the idle thread does it
to itself, other CPUs only read it. So I moved it into ti->status.
the problem is this type of change:
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ current_thread_info()->status &= ~TS_POLLING;
smp_mb__after_clear_bit();
while (!need_resched()) {
local_irq_disable();
this changes clear_thread_flag() to an explicit clearing of TS_POLLING.
clear_thread_flag() is defined as:
clear_bit(flag, &ti->flags);
and clear_bit() is a LOCK-ed atomic instruction on all x86 platforms:
static inline void clear_bit(int nr, volatile unsigned long * addr)
{
__asm__ __volatile__( LOCK_PREFIX
"btrl %1,%0"
hence smp_mb__after_clear_bit() is defined as a simple compile barrier:
#define smp_mb__after_clear_bit() barrier()
but the explicit TS_POLLING clearing introduced by the patch:
+ current_thread_info()->status &= ~TS_POLLING;
is not an atomic op! So the clearing of the TS_POLLING bit is freely
reorderable with the reading of the NEED_RESCHED bit - and both now
reside in different memory addresses.
CPU idle wakeup very much depends on ordered memory ops, the clearing of
the TS_POLLING flag must always be done before we test need_resched()
and hit the idle instruction(s). [Symmetrically, the wakeup code needs
to set NEED_RESCHED before it tests the TS_POLLING flag, so memory
ordering is paramount.]
Fernando's dual-core Athlon64 system has a sufficiently advanced memory
ordering model so that it triggered this scenario very often.
( And it also turned out that the reason why these latencies never
triggered on my testsystems is that i routinely use idle=poll, which
was the only idle variant not affected by this bug. )
The fix is to change the smp_mb__after_clear_bit() to an smp_mb(), to
act as an absolute barrier between the TS_POLLING write and the
NEED_RESCHED read. This affects almost all idling methods (default,
ACPI, APM), on all 3 x86 architectures: i386, x86_64, ia64.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Tested-by: Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-22 12:11:56 +03:00
|
|
|
}
|
2005-11-16 01:37:05 +03:00
|
|
|
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
if (!need_resched()) {
|
|
|
|
void (*idle)(void);
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
min_xtp();
|
|
|
|
#endif
|
2005-03-31 08:40:00 +04:00
|
|
|
rmb();
|
2005-04-17 02:20:36 +04:00
|
|
|
if (mark_idle)
|
|
|
|
(*mark_idle)(1);
|
|
|
|
|
|
|
|
idle = pm_idle;
|
|
|
|
if (!idle)
|
|
|
|
idle = default_idle;
|
|
|
|
(*idle)();
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
if (mark_idle)
|
|
|
|
(*mark_idle)(0);
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_SMP
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
normal_xtp();
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
}
|
2011-03-21 14:33:18 +03:00
|
|
|
schedule_preempt_disabled();
|
2005-04-17 02:20:36 +04:00
|
|
|
check_pgt_cache();
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-09 08:39:04 +03:00
|
|
|
if (cpu_is_offline(cpu))
|
2005-04-17 02:20:36 +04:00
|
|
|
play_dead();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
ia64_save_extra (struct task_struct *task)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
unsigned long info;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
if ((task->thread.flags & IA64_THREAD_DBG_VALID) != 0)
|
|
|
|
ia64_save_debug_regs(&task->thread.dbr[0]);
|
|
|
|
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
|
|
|
|
pfm_save_regs(task);
|
|
|
|
|
|
|
|
info = __get_cpu_var(pfm_syst_info);
|
|
|
|
if (info & PFM_CPUINFO_SYST_WIDE)
|
|
|
|
pfm_syst_wide_update_task(task, info, 0);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
ia64_load_extra (struct task_struct *task)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
unsigned long info;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
if ((task->thread.flags & IA64_THREAD_DBG_VALID) != 0)
|
|
|
|
ia64_load_debug_regs(&task->thread.dbr[0]);
|
|
|
|
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
if ((task->thread.flags & IA64_THREAD_PM_VALID) != 0)
|
|
|
|
pfm_load_regs(task);
|
|
|
|
|
|
|
|
info = __get_cpu_var(pfm_syst_info);
|
|
|
|
if (info & PFM_CPUINFO_SYST_WIDE)
|
|
|
|
pfm_syst_wide_update_task(task, info, 1);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Copy the state of an ia-64 thread.
|
|
|
|
*
|
|
|
|
* We get here through the following call chain:
|
|
|
|
*
|
|
|
|
* from user-level: from kernel:
|
|
|
|
*
|
|
|
|
* <clone syscall> <some kernel call frames>
|
|
|
|
* sys_clone :
|
|
|
|
* do_fork do_fork
|
|
|
|
* copy_thread copy_thread
|
|
|
|
*
|
|
|
|
* This means that the stack layout is as follows:
|
|
|
|
*
|
|
|
|
* +---------------------+ (highest addr)
|
|
|
|
* | struct pt_regs |
|
|
|
|
* +---------------------+
|
|
|
|
* | struct switch_stack |
|
|
|
|
* +---------------------+
|
|
|
|
* | |
|
|
|
|
* | memory stack |
|
|
|
|
* | | <-- sp (lowest addr)
|
|
|
|
* +---------------------+
|
|
|
|
*
|
|
|
|
* Observe that we copy the unat values that are in pt_regs and switch_stack. Spilling an
|
|
|
|
* integer to address X causes bit N in ar.unat to be set to the NaT bit of the register,
|
|
|
|
* with N=(X & 0x1ff)/8. Thus, copying the unat value preserves the NaT bits ONLY if the
|
|
|
|
* pt_regs structure in the parent is congruent to that of the child, modulo 512. Since
|
|
|
|
* the stack is page aligned and the page size is at least 4KB, this is always the case,
|
|
|
|
* so there is nothing to worry about.
|
|
|
|
*/
|
|
|
|
int
|
2009-04-03 03:56:59 +04:00
|
|
|
copy_thread(unsigned long clone_flags,
|
2005-04-17 02:20:36 +04:00
|
|
|
unsigned long user_stack_base, unsigned long user_stack_size,
|
|
|
|
struct task_struct *p, struct pt_regs *regs)
|
|
|
|
{
|
2010-02-08 21:42:17 +03:00
|
|
|
extern char ia64_ret_from_clone;
|
2005-04-17 02:20:36 +04:00
|
|
|
struct switch_stack *child_stack, *stack;
|
|
|
|
unsigned long rbs, child_rbs, rbs_size;
|
|
|
|
struct pt_regs *child_ptregs;
|
|
|
|
int retval = 0;
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
/*
|
|
|
|
* For SMP idle threads, fork_by_hand() calls do_fork with
|
|
|
|
* NULL regs.
|
|
|
|
*/
|
|
|
|
if (!regs)
|
|
|
|
return 0;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
stack = ((struct switch_stack *) regs) - 1;
|
|
|
|
|
|
|
|
child_ptregs = (struct pt_regs *) ((unsigned long) p + IA64_STK_OFFSET) - 1;
|
|
|
|
child_stack = (struct switch_stack *) child_ptregs - 1;
|
|
|
|
|
|
|
|
/* copy parent's switch_stack & pt_regs to child: */
|
|
|
|
memcpy(child_stack, stack, sizeof(*child_ptregs) + sizeof(*child_stack));
|
|
|
|
|
|
|
|
rbs = (unsigned long) current + IA64_RBS_OFFSET;
|
|
|
|
child_rbs = (unsigned long) p + IA64_RBS_OFFSET;
|
|
|
|
rbs_size = stack->ar_bspstore - rbs;
|
|
|
|
|
|
|
|
/* copy the parent's register backing store to the child: */
|
|
|
|
memcpy((void *) child_rbs, (void *) rbs, rbs_size);
|
|
|
|
|
|
|
|
if (likely(user_mode(child_ptregs))) {
|
2010-02-08 21:42:17 +03:00
|
|
|
if (clone_flags & CLONE_SETTLS)
|
2005-04-17 02:20:36 +04:00
|
|
|
child_ptregs->r13 = regs->r16; /* see sys_clone2() in entry.S */
|
|
|
|
if (user_stack_base) {
|
|
|
|
child_ptregs->r12 = user_stack_base + user_stack_size - 16;
|
|
|
|
child_ptregs->ar_bspstore = user_stack_base;
|
|
|
|
child_ptregs->ar_rnat = 0;
|
|
|
|
child_ptregs->loadrs = 0;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Note: we simply preserve the relative position of
|
|
|
|
* the stack pointer here. There is no need to
|
|
|
|
* allocate a scratch area here, since that will have
|
|
|
|
* been taken care of by the caller of sys_clone()
|
|
|
|
* already.
|
|
|
|
*/
|
|
|
|
child_ptregs->r12 = (unsigned long) child_ptregs - 16; /* kernel sp */
|
|
|
|
child_ptregs->r13 = (unsigned long) p; /* set `current' pointer */
|
|
|
|
}
|
|
|
|
child_stack->ar_bspstore = child_rbs + rbs_size;
|
2010-02-08 21:42:17 +03:00
|
|
|
child_stack->b0 = (unsigned long) &ia64_ret_from_clone;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* copy parts of thread_struct: */
|
|
|
|
p->thread.ksp = (unsigned long) child_stack - 16;
|
|
|
|
|
|
|
|
/* stop some PSR bits from being inherited.
|
|
|
|
* the psr.up/psr.pp bits must be cleared on fork but inherited on execve()
|
|
|
|
* therefore we must specify them explicitly here and not include them in
|
|
|
|
* IA64_PSR_BITS_TO_CLEAR.
|
|
|
|
*/
|
|
|
|
child_ptregs->cr_ipsr = ((child_ptregs->cr_ipsr | IA64_PSR_BITS_TO_SET)
|
|
|
|
& ~(IA64_PSR_BITS_TO_CLEAR | IA64_PSR_PP | IA64_PSR_UP));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* NOTE: The calling convention considers all floating point
|
|
|
|
* registers in the high partition (fph) to be scratch. Since
|
|
|
|
* the only way to get to this point is through a system call,
|
|
|
|
* we know that the values in fph are all dead. Hence, there
|
|
|
|
* is no need to inherit the fph state from the parent to the
|
|
|
|
* child and all we have to do is to make sure that
|
|
|
|
* IA64_THREAD_FPH_VALID is cleared in the child.
|
|
|
|
*
|
|
|
|
* XXX We could push this optimization a bit further by
|
|
|
|
* clearing IA64_THREAD_FPH_VALID on ANY system call.
|
|
|
|
* However, it's not clear this is worth doing. Also, it
|
|
|
|
* would be a slight deviation from the normal Linux system
|
|
|
|
* call behavior where scratch registers are preserved across
|
|
|
|
* system calls (unless used by the system call itself).
|
|
|
|
*/
|
|
|
|
# define THREAD_FLAGS_TO_CLEAR (IA64_THREAD_FPH_VALID | IA64_THREAD_DBG_VALID \
|
|
|
|
| IA64_THREAD_PM_VALID)
|
|
|
|
# define THREAD_FLAGS_TO_SET 0
|
|
|
|
p->thread.flags = ((current->thread.flags & ~THREAD_FLAGS_TO_CLEAR)
|
|
|
|
| THREAD_FLAGS_TO_SET);
|
|
|
|
ia64_drop_fpu(p); /* don't pick up stale state from a CPU's fph */
|
|
|
|
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
if (current->thread.pfm_context)
|
|
|
|
pfm_inherit(p, child_ptregs);
|
|
|
|
#endif
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
do_copy_task_regs (struct task_struct *task, struct unw_frame_info *info, void *arg)
|
|
|
|
{
|
2007-07-11 19:26:30 +04:00
|
|
|
unsigned long mask, sp, nat_bits = 0, ar_rnat, urbs_end, cfm;
|
|
|
|
unsigned long uninitialized_var(ip); /* GCC be quiet */
|
2005-04-17 02:20:36 +04:00
|
|
|
elf_greg_t *dst = arg;
|
|
|
|
struct pt_regs *pt;
|
|
|
|
char nat;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
memset(dst, 0, sizeof(elf_gregset_t)); /* don't leak any kernel bits to user-level */
|
|
|
|
|
|
|
|
if (unw_unwind_to_user(info) < 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
unw_get_sp(info, &sp);
|
|
|
|
pt = (struct pt_regs *) (sp + 16);
|
|
|
|
|
|
|
|
urbs_end = ia64_get_user_rbs_end(task, pt, &cfm);
|
|
|
|
|
|
|
|
if (ia64_sync_user_rbs(task, info->sw, pt->ar_bspstore, urbs_end) < 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
ia64_peek(task, info->sw, urbs_end, (long) ia64_rse_rnat_addr((long *) urbs_end),
|
|
|
|
&ar_rnat);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* coredump format:
|
|
|
|
* r0-r31
|
|
|
|
* NaT bits (for r0-r31; bit N == 1 iff rN is a NaT)
|
|
|
|
* predicate registers (p0-p63)
|
|
|
|
* b0-b7
|
|
|
|
* ip cfm user-mask
|
|
|
|
* ar.rsc ar.bsp ar.bspstore ar.rnat
|
|
|
|
* ar.ccv ar.unat ar.fpsr ar.pfs ar.lc ar.ec
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* r0 is zero */
|
|
|
|
for (i = 1, mask = (1UL << i); i < 32; ++i) {
|
|
|
|
unw_get_gr(info, i, &dst[i], &nat);
|
|
|
|
if (nat)
|
|
|
|
nat_bits |= mask;
|
|
|
|
mask <<= 1;
|
|
|
|
}
|
|
|
|
dst[32] = nat_bits;
|
|
|
|
unw_get_pr(info, &dst[33]);
|
|
|
|
|
|
|
|
for (i = 0; i < 8; ++i)
|
|
|
|
unw_get_br(info, i, &dst[34 + i]);
|
|
|
|
|
|
|
|
unw_get_rp(info, &ip);
|
|
|
|
dst[42] = ip + ia64_psr(pt)->ri;
|
|
|
|
dst[43] = cfm;
|
|
|
|
dst[44] = pt->cr_ipsr & IA64_PSR_UM;
|
|
|
|
|
|
|
|
unw_get_ar(info, UNW_AR_RSC, &dst[45]);
|
|
|
|
/*
|
|
|
|
* For bsp and bspstore, unw_get_ar() would return the kernel
|
|
|
|
* addresses, but we need the user-level addresses instead:
|
|
|
|
*/
|
|
|
|
dst[46] = urbs_end; /* note: by convention PT_AR_BSP points to the end of the urbs! */
|
|
|
|
dst[47] = pt->ar_bspstore;
|
|
|
|
dst[48] = ar_rnat;
|
|
|
|
unw_get_ar(info, UNW_AR_CCV, &dst[49]);
|
|
|
|
unw_get_ar(info, UNW_AR_UNAT, &dst[50]);
|
|
|
|
unw_get_ar(info, UNW_AR_FPSR, &dst[51]);
|
|
|
|
dst[52] = pt->ar_pfs; /* UNW_AR_PFS is == to pt->cr_ifs for interrupt frames */
|
|
|
|
unw_get_ar(info, UNW_AR_LC, &dst[53]);
|
|
|
|
unw_get_ar(info, UNW_AR_EC, &dst[54]);
|
|
|
|
unw_get_ar(info, UNW_AR_CSD, &dst[55]);
|
|
|
|
unw_get_ar(info, UNW_AR_SSD, &dst[56]);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
do_dump_task_fpu (struct task_struct *task, struct unw_frame_info *info, void *arg)
|
|
|
|
{
|
|
|
|
elf_fpreg_t *dst = arg;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
memset(dst, 0, sizeof(elf_fpregset_t)); /* don't leak any "random" bits */
|
|
|
|
|
|
|
|
if (unw_unwind_to_user(info) < 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* f0 is 0.0, f1 is 1.0 */
|
|
|
|
|
|
|
|
for (i = 2; i < 32; ++i)
|
|
|
|
unw_get_fr(info, i, dst + i);
|
|
|
|
|
|
|
|
ia64_flush_fph(task);
|
|
|
|
if ((task->thread.flags & IA64_THREAD_FPH_VALID) != 0)
|
|
|
|
memcpy(dst + 32, task->thread.fph, 96*16);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
do_copy_regs (struct unw_frame_info *info, void *arg)
|
|
|
|
{
|
|
|
|
do_copy_task_regs(current, info, arg);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
do_dump_fpu (struct unw_frame_info *info, void *arg)
|
|
|
|
{
|
|
|
|
do_dump_task_fpu(current, info, arg);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
ia64_elf_core_copy_regs (struct pt_regs *pt, elf_gregset_t dst)
|
|
|
|
{
|
|
|
|
unw_init_running(do_copy_regs, dst);
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
dump_fpu (struct pt_regs *pt, elf_fpregset_t dst)
|
|
|
|
{
|
|
|
|
unw_init_running(do_dump_fpu, dst);
|
|
|
|
return 1; /* f0-f31 are always valid so we always return 1 */
|
|
|
|
}
|
|
|
|
|
|
|
|
long
|
2010-08-18 02:52:56 +04:00
|
|
|
sys_execve (const char __user *filename,
|
|
|
|
const char __user *const __user *argv,
|
|
|
|
const char __user *const __user *envp,
|
2005-04-17 02:20:36 +04:00
|
|
|
struct pt_regs *regs)
|
|
|
|
{
|
|
|
|
char *fname;
|
|
|
|
int error;
|
|
|
|
|
|
|
|
fname = getname(filename);
|
|
|
|
error = PTR_ERR(fname);
|
|
|
|
if (IS_ERR(fname))
|
|
|
|
goto out;
|
|
|
|
error = do_execve(fname, argv, envp, regs);
|
|
|
|
putname(fname);
|
|
|
|
out:
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
pid_t
|
|
|
|
kernel_thread (int (*fn)(void *), void *arg, unsigned long flags)
|
|
|
|
{
|
|
|
|
extern void start_kernel_thread (void);
|
|
|
|
unsigned long *helper_fptr = (unsigned long *) &start_kernel_thread;
|
|
|
|
struct {
|
|
|
|
struct switch_stack sw;
|
|
|
|
struct pt_regs pt;
|
|
|
|
} regs;
|
|
|
|
|
|
|
|
memset(®s, 0, sizeof(regs));
|
|
|
|
regs.pt.cr_iip = helper_fptr[0]; /* set entry point (IP) */
|
|
|
|
regs.pt.r1 = helper_fptr[1]; /* set GP */
|
|
|
|
regs.pt.r9 = (unsigned long) fn; /* 1st argument */
|
|
|
|
regs.pt.r11 = (unsigned long) arg; /* 2nd argument */
|
|
|
|
/* Preserve PSR bits, except for bits 32-34 and 37-45, which we can't read. */
|
|
|
|
regs.pt.cr_ipsr = ia64_getreg(_IA64_REG_PSR) | IA64_PSR_BN;
|
|
|
|
regs.pt.cr_ifs = 1UL << 63; /* mark as valid, empty frame */
|
|
|
|
regs.sw.ar_fpsr = regs.pt.ar_fpsr = ia64_getreg(_IA64_REG_AR_FPSR);
|
|
|
|
regs.sw.ar_bspstore = (unsigned long) current + IA64_RBS_OFFSET;
|
|
|
|
regs.sw.pr = (1 << PRED_KERNEL_STACK);
|
|
|
|
return do_fork(flags | CLONE_VM | CLONE_UNTRACED, 0, ®s.pt, 0, NULL, NULL);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(kernel_thread);
|
|
|
|
|
|
|
|
/* This gets called from kernel_thread() via ia64_invoke_thread_helper(). */
|
|
|
|
int
|
|
|
|
kernel_thread_helper (int (*fn)(void *), void *arg)
|
|
|
|
{
|
|
|
|
return (*fn)(arg);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Flush thread state. This is called when a thread does an execve().
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
flush_thread (void)
|
|
|
|
{
|
|
|
|
/* drop floating-point and debug-register state if it exists: */
|
|
|
|
current->thread.flags &= ~(IA64_THREAD_FPH_VALID | IA64_THREAD_DBG_VALID);
|
|
|
|
ia64_drop_fpu(current);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Clean up state associated with current thread. This is called when
|
|
|
|
* the thread calls exit().
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
exit_thread (void)
|
|
|
|
{
|
2005-06-28 02:17:12 +04:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
ia64_drop_fpu(current);
|
|
|
|
#ifdef CONFIG_PERFMON
|
|
|
|
/* if needed, stop monitoring and flush state to perfmon context */
|
|
|
|
if (current->thread.pfm_context)
|
|
|
|
pfm_exit_thread(current);
|
|
|
|
|
|
|
|
/* free debug register resources */
|
|
|
|
if (current->thread.flags & IA64_THREAD_DBG_VALID)
|
|
|
|
pfm_release_debug_registers(current);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
unsigned long
|
|
|
|
get_wchan (struct task_struct *p)
|
|
|
|
{
|
|
|
|
struct unw_frame_info info;
|
|
|
|
unsigned long ip;
|
|
|
|
int count = 0;
|
|
|
|
|
2007-05-18 22:46:52 +04:00
|
|
|
if (!p || p == current || p->state == TASK_RUNNING)
|
|
|
|
return 0;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* Note: p may not be a blocked task (it could be current or
|
|
|
|
* another process running on some other CPU. Rather than
|
|
|
|
* trying to determine if p is really blocked, we just assume
|
|
|
|
* it's blocked and rely on the unwind routines to fail
|
|
|
|
* gracefully if the process wasn't really blocked after all.
|
|
|
|
* --davidm 99/12/15
|
|
|
|
*/
|
|
|
|
unw_init_from_blocked_task(&info, p);
|
|
|
|
do {
|
2007-05-18 22:46:52 +04:00
|
|
|
if (p->state == TASK_RUNNING)
|
|
|
|
return 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
if (unw_unwind(&info) < 0)
|
|
|
|
return 0;
|
|
|
|
unw_get_ip(&info, &ip);
|
|
|
|
if (!in_sched_functions(ip))
|
|
|
|
return ip;
|
|
|
|
} while (count++ < 16);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
cpu_halt (void)
|
|
|
|
{
|
|
|
|
pal_power_mgmt_info_u_t power_info[8];
|
|
|
|
unsigned long min_power;
|
|
|
|
int i, min_power_state;
|
|
|
|
|
|
|
|
if (ia64_pal_halt_info(power_info) != 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
min_power_state = 0;
|
|
|
|
min_power = power_info[0].pal_power_mgmt_info_s.power_consumption;
|
|
|
|
for (i = 1; i < 8; ++i)
|
|
|
|
if (power_info[i].pal_power_mgmt_info_s.im
|
|
|
|
&& power_info[i].pal_power_mgmt_info_s.power_consumption < min_power) {
|
|
|
|
min_power = power_info[i].pal_power_mgmt_info_s.power_consumption;
|
|
|
|
min_power_state = i;
|
|
|
|
}
|
|
|
|
|
|
|
|
while (1)
|
|
|
|
ia64_pal_halt(min_power_state);
|
|
|
|
}
|
|
|
|
|
2007-02-06 00:49:10 +03:00
|
|
|
void machine_shutdown(void)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_HOTPLUG_CPU
|
|
|
|
int cpu;
|
|
|
|
|
|
|
|
for_each_online_cpu(cpu) {
|
|
|
|
if (cpu != smp_processor_id())
|
|
|
|
cpu_down(cpu);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_KEXEC
|
|
|
|
kexec_disable_iosapic();
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
void
|
|
|
|
machine_restart (char *restart_cmd)
|
|
|
|
{
|
[IA64] Extend notify_die() hooks for IA64
notify_die() added for MCA_{MONARCH,SLAVE,RENDEZVOUS}_{ENTER,PROCESS,LEAVE} and
INIT_{MONARCH,SLAVE}_{ENTER,PROCESS,LEAVE}. We need multiple
notification points for these events because they can take many seconds
to run which has nasty effects on the behaviour of the rest of the
system.
DIE_SS replaced by a generic DIE_FAULT which checks the vector number,
to allow interception of faults other than SS.
DIE_MACHINE_{HALT,RESTART} added to allow last minute close down
processing, especially when the halt/restart routines are called from
error handlers.
DIE_OOPS added.
The check for kprobe's break numbers has been moved from traps.c to
kprobes.c, allowing DIE_BREAK to be used for any additional break
numbers, i.e. it is no longer kprobes specific.
Hooks for kernel debuggers and kernel dumpers added, ENTER and LEAVE.
Both of these disable the system for long periods which impact on
watchdogs and heartbeat systems in general. More patches to come that
use these events to reset watchdogs and heartbeats.
unregister_die_notifier() added and both routines exported. Requested
by Dean Nelson.
Lock removed from {un,}register_die_notifier. notifier_chain_register()
already takes a lock. Also the generic notifier chain locking is being
reworked to distinguish between callbacks that can block and those that
cannot, the lock in {un,}register_die_notifier would interfere with
that change. http://marc.theaimsgroup.com/?l=linux-kernel&m=113018709002036&w=2
Leading white space removed from arch/ia64/kernel/kprobes.c.
Typo in mca.c in original version of this patch found & fixed by Dean
Nelson.
Signed-off-by: Keith Owens <kaos@sgi.com>
Acked-by: Dean Nelson <dcn@sgi.com>
Acked-by: Anil Keshavamurthy <anil.s.keshavamurthy@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
2005-11-07 22:27:13 +03:00
|
|
|
(void) notify_die(DIE_MACHINE_RESTART, restart_cmd, NULL, 0, 0, 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
(*efi.reset_system)(EFI_RESET_WARM, 0, 0, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
machine_halt (void)
|
|
|
|
{
|
[IA64] Extend notify_die() hooks for IA64
notify_die() added for MCA_{MONARCH,SLAVE,RENDEZVOUS}_{ENTER,PROCESS,LEAVE} and
INIT_{MONARCH,SLAVE}_{ENTER,PROCESS,LEAVE}. We need multiple
notification points for these events because they can take many seconds
to run which has nasty effects on the behaviour of the rest of the
system.
DIE_SS replaced by a generic DIE_FAULT which checks the vector number,
to allow interception of faults other than SS.
DIE_MACHINE_{HALT,RESTART} added to allow last minute close down
processing, especially when the halt/restart routines are called from
error handlers.
DIE_OOPS added.
The check for kprobe's break numbers has been moved from traps.c to
kprobes.c, allowing DIE_BREAK to be used for any additional break
numbers, i.e. it is no longer kprobes specific.
Hooks for kernel debuggers and kernel dumpers added, ENTER and LEAVE.
Both of these disable the system for long periods which impact on
watchdogs and heartbeat systems in general. More patches to come that
use these events to reset watchdogs and heartbeats.
unregister_die_notifier() added and both routines exported. Requested
by Dean Nelson.
Lock removed from {un,}register_die_notifier. notifier_chain_register()
already takes a lock. Also the generic notifier chain locking is being
reworked to distinguish between callbacks that can block and those that
cannot, the lock in {un,}register_die_notifier would interfere with
that change. http://marc.theaimsgroup.com/?l=linux-kernel&m=113018709002036&w=2
Leading white space removed from arch/ia64/kernel/kprobes.c.
Typo in mca.c in original version of this patch found & fixed by Dean
Nelson.
Signed-off-by: Keith Owens <kaos@sgi.com>
Acked-by: Dean Nelson <dcn@sgi.com>
Acked-by: Anil Keshavamurthy <anil.s.keshavamurthy@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
2005-11-07 22:27:13 +03:00
|
|
|
(void) notify_die(DIE_MACHINE_HALT, "", NULL, 0, 0, 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
cpu_halt();
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
machine_power_off (void)
|
|
|
|
{
|
|
|
|
if (pm_power_off)
|
|
|
|
pm_power_off();
|
|
|
|
machine_halt();
|
|
|
|
}
|
|
|
|
|