microsoft
/
WSL2-Linux-Kernel
зеркало из https://github.com/microsoft/WSL2-Linux-Kernel.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
WSL2-Linux-Kernel/arch/s390/kernel/entry.S

1299 строки
33 KiB
ArmAsm
Исходник Обычный вид История

License cleanup: add SPDX GPL-2.0 license identifier to files with no license Many source files in the tree are missing licensing information, which makes it harder for compliance tools to determine the correct license. By default all files without license information are under the default license of the kernel, which is GPL version 2. Update the files which contain no license information with the 'GPL-2.0' SPDX license identifier. The SPDX identifier is a legally binding shorthand, which can be used instead of the full boiler plate text. This patch is based on work done by Thomas Gleixner and Kate Stewart and Philippe Ombredanne. How this work was done: Patches were generated and checked against linux-4.14-rc6 for a subset of the use cases: - file had no licensing information it it. - file was a */uapi/* one with no licensing information in it, - file was a */uapi/* one with existing licensing information, Further patches will be generated in subsequent months to fix up cases where non-standard license headers were used, and references to license had to be inferred by heuristics based on keywords. The analysis to determine which SPDX License Identifier to be applied to a file was done in a spreadsheet of side by side results from of the output of two independent scanners (ScanCode & Windriver) producing SPDX tag:value files created by Philippe Ombredanne. Philippe prepared the base worksheet, and did an initial spot review of a few 1000 files. The 4.13 kernel was the starting point of the analysis with 60,537 files assessed. Kate Stewart did a file by file comparison of the scanner results in the spreadsheet to determine which SPDX license identifier(s) to be applied to the file. She confirmed any determination that was not immediately clear with lawyers working with the Linux Foundation. Criteria used to select files for SPDX license identifier tagging was: - Files considered eligible had to be source code files. - Make and config files were included as candidates if they contained >5 lines of source - File already had some variant of a license header in it (even if <5 lines). All documentation files were explicitly excluded. The following heuristics were used to determine which SPDX license identifiers to apply. - when both scanners couldn't find any license traces, file was considered to have no license information in it, and the top level COPYING file license applied. For non */uapi/* files that summary was: SPDX license identifier # files ---------------------------------------------------|------- GPL-2.0 11139 and resulted in the first patch in this series. If that file was a */uapi/* path one, it was "GPL-2.0 WITH Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was: SPDX license identifier # files ---------------------------------------------------|------- GPL-2.0 WITH Linux-syscall-note 930 and resulted in the second patch in this series. - if a file had some form of licensing information in it, and was one of the */uapi/* ones, it was denoted with the Linux-syscall-note if any GPL family license was found in the file or had no licensing in it (per prior point). Results summary: SPDX license identifier # files ---------------------------------------------------|------ GPL-2.0 WITH Linux-syscall-note 270 GPL-2.0+ WITH Linux-syscall-note 169 ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21 ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17 LGPL-2.1+ WITH Linux-syscall-note 15 GPL-1.0+ WITH Linux-syscall-note 14 ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5 LGPL-2.0+ WITH Linux-syscall-note 4 LGPL-2.1 WITH Linux-syscall-note 3 ((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3 ((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1 and that resulted in the third patch in this series. - when the two scanners agreed on the detected license(s), that became the concluded license(s). - when there was disagreement between the two scanners (one detected a license but the other didn't, or they both detected different licenses) a manual inspection of the file occurred. - In most cases a manual inspection of the information in the file resulted in a clear resolution of the license that should apply (and which scanner probably needed to revisit its heuristics). - When it was not immediately clear, the license identifier was confirmed with lawyers working with the Linux Foundation. - If there was any question as to the appropriate license identifier, the file was flagged for further research and to be revisited later in time. In total, over 70 hours of logged manual review was done on the spreadsheet to determine the SPDX license identifiers to apply to the source files by Kate, Philippe, Thomas and, in some cases, confirmation by lawyers working with the Linux Foundation. Kate also obtained a third independent scan of the 4.13 code base from FOSSology, and compared selected files where the other two scanners disagreed against that SPDX file, to see if there was new insights. The Windriver scanner is based on an older version of FOSSology in part, so they are related. Thomas did random spot checks in about 500 files from the spreadsheets for the uapi headers and agreed with SPDX license identifier in the files he inspected. For the non-uapi files Thomas did random spot checks in about 15000 files. In initial set of patches against 4.14-rc6, 3 files were found to have copy/paste license identifier errors, and have been fixed to reflect the correct identifier. Additionally Philippe spent 10 hours this week doing a detailed manual inspection and review of the 12,461 patched files from the initial patch version early this week with: - a full scancode scan run, collecting the matched texts, detected license ids and scores - reviewing anything where there was a license detected (about 500+ files) to ensure that the applied SPDX license was correct - reviewing anything where there was no detection but the patch license was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied SPDX license was correct This produced a worksheet with 20 files needing minor correction. This worksheet was then exported into 3 different .csv files for the different types of files to be modified. These .csv files were then reviewed by Greg. Thomas wrote a script to parse the csv files and add the proper SPDX tag to the file, in the format that the file expected. This script was further refined by Greg based on the output to detect more types of files automatically and to distinguish between header and source .c files (which need different comment types.) Finally Greg ran the script using the .csv files to generate the patches. Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com> Reviewed-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 17:07:57 +03:00
/* SPDX-License-Identifier: GPL-2.0 */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* S390 low-level entry points.
*
s390/comments: unify copyright messages and remove file names Remove the file name from the comment at top of many files. In most cases the file name was wrong anyway, so it's rather pointless. Also unify the IBM copyright statement. We did have a lot of sightly different statements and wanted to change them one after another whenever a file gets touched. However that never happened. Instead people start to take the old/"wrong" statements to use as a template for new files. So unify all of them in one go. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2012-07-20 13:15:04 +04:00
* Copyright IBM Corp. 1999, 2012
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
* Author(s): Martin Schwidefsky (schwidefsky@de.ibm.com),
[S390] Whitespace cleanup. Huge s390 assembly files whitespace cleanup. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-28 18:56:37 +04:00
* Hartmut Penner (hp@de.ibm.com),
* Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com),
[PATCH] s390: improved machine check handling Improved machine check handling. Kernel is now able to receive machine checks while in kernel mode (system call, interrupt and program check handling). Also register validation is now performed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-26 01:55:30 +04:00
* Heiko Carstens <heiko.carstens@de.ibm.com>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
*/
[S390] Fix couple of section mismatches. Fix couple of section mismatches. And since we touch the code anyway change the IPL code to use C99 initializers. Cc: Michael Holzheu <holzheu@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-02-05 18:50:40 +03:00
#include <linux/init.h>
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
#include <linux/linkage.h>
s390/entry.S: use assembler alternatives Replace the open coded alternatives for the BPOFF, BPON, BPENTER, and BPEXIT macros with the new magic from asm/alternatives-asm.h to make the code easier to read. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:27:36 +03:00
#include <asm/alternative-asm.h>
s390/exceptions: switch to relative exception table entries This is the s390 port of 70627654 "x86, extable: Switch to relative exception table entries". Reduces the size of our exception tables by 50% on 64 bit builds. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-09-05 15:26:11 +04:00
#include <asm/processor.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#include <asm/cache.h>
s390/nmi: do register validation as early as possible The validation of the CPU registers in the machine check handler is currently split into two parts. The first part is done at the start of the low level mcck_int_handler function, this includes the CPU timer register and the general purpose registers. The second part is done a bit later in s390_do_machine_check for all the other registers, including the control registers, floating pointer control, vector or floating pointer registers, the access registers, the guarded storage registers, the TOD programmable registers and the clock comparator. This is working fine to far but in theory a future extensions could cause the C code to use registers that are not validated yet. A better approach is to validate all CPU registers in "safe" assembler code before any C function is called. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-12 14:24:48 +03:00
#include <asm/ctl_reg.h>
s390/clean-up: use CFI_* macros in entry.S Commit f19fbd5ed642 ("s390: introduce execute-trampolines for branches") introduces .cfi_* assembler directives. Instead of using the directives directly, use the macros from asm/dwarf.h. This also ensures that the dwarf debug information are created in the .debug_frame section. Fixes: f19fbd5ed642 ("s390: introduce execute-trampolines for branches") Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-02-19 13:27:09 +03:00
#include <asm/dwarf.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#include <asm/errno.h>
#include <asm/ptrace.h>
#include <asm/thread_info.h>
kbuild: m68k,parisc,ppc,ppc64,s390,xtensa use generic asm-offsets.h support Delete obsoleted parts form arch makefiles and rename to asm-offsets.h Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
2005-09-09 22:57:26 +04:00
#include <asm/asm-offsets.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#include <asm/unistd.h>
#include <asm/page.h>
s390/sigp: use sigp order code defines in assembly code Use sigp order code defines in assembly code as well. With this change all places that use sigp constants should have been converted to use self describing defines instead of directly using constants. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-06-04 17:05:43 +04:00
#include <asm/sigp.h>
s390: convert interrupt handling to use generic hardirq With the introduction of PCI it became apparent that s390 should convert to generic hardirqs as too many drivers do not have the correct dependency for GENERIC_HARDIRQS. On the architecture level s390 does not have irq lines. It has external interrupts, I/O interrupts and adapter interrupts. This patch hard-codes all external interrupts as irq #1, all I/O interrupts as irq #2 and all adapter interrupts as irq #3. The additional information from the lowcore associated with the interrupt is stored in the pt_regs of the interrupt frame, where the interrupt handler can pick it up. For PCI/MSI interrupts the adapter interrupt handler scans the relevant bit fields and calls generic_handle_irq with the virtual irq number for the MSI interrupt. Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-27 11:01:09 +04:00
#include <asm/irq.h>
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
#include <asm/vx-insn.h>
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
#include <asm/setup.h>
#include <asm/nmi.h>
s390: move exports to definitions Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2016-01-12 21:30:03 +03:00
#include <asm/export.h>
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
#include <asm/nospec-insn.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
__PT_R0 = __PT_GPRS
__PT_R1 = __PT_GPRS + 8
__PT_R2 = __PT_GPRS + 16
__PT_R3 = __PT_GPRS + 24
__PT_R4 = __PT_GPRS + 32
__PT_R5 = __PT_GPRS + 40
__PT_R6 = __PT_GPRS + 48
__PT_R7 = __PT_GPRS + 56
__PT_R8 = __PT_GPRS + 64
__PT_R9 = __PT_GPRS + 72
__PT_R10 = __PT_GPRS + 80
__PT_R11 = __PT_GPRS + 88
__PT_R12 = __PT_GPRS + 96
__PT_R13 = __PT_GPRS + 104
__PT_R14 = __PT_GPRS + 112
__PT_R15 = __PT_GPRS + 120
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/thread_info: get rid of THREAD_ORDER define We have the s390 specific THREAD_ORDER define and the THREAD_SIZE_ORDER define which is also used in common code. Both have exactly the same semantics. Therefore get rid of THREAD_ORDER and always use THREAD_SIZE_ORDER instead. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-14 16:39:16 +03:00
STACK_SHIFT = PAGE_SHIFT + THREAD_SIZE_ORDER
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
STACK_SIZE = 1 << STACK_SHIFT
s390: lowcore stack pointer offsets Store the stack pointers in the lowcore for the kernel stack, the async stack and the panic stack with the offset required for the first user. This avoids an unnecessary add instruction on the system call path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-04-24 12:20:43 +04:00
STACK_INIT = STACK_SIZE - STACK_FRAME_OVERHEAD - __PT_SIZE
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/uprobes: architecture backend for uprobes Signed-off-by: Jan Willeke <willeke@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-09-22 18:39:06 +04:00
_TIF_WORK = (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_NEED_RESCHED | \
s390: add support for TIF_NOTIFY_SIGNAL Wire up TIF_NOTIFY_SIGNAL handling for s390. Cc: linux-s390@vger.kernel.org Acked-by: Heiko Carstens <hca@linux.ibm.com> Acked-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-10-10 00:34:12 +03:00
_TIF_UPROBE | _TIF_GUARDED_STORAGE | _TIF_PATCH_PENDING | \
_TIF_NOTIFY_SIGNAL)
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
_TIF_TRACE = (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SECCOMP | \
_TIF_SYSCALL_TRACEPOINT)
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
_CIF_WORK = (_CIF_FPU)
s390/kvm: avoid global config of vm.alloc_pgste=1 The system control vm.alloc_pgste is used to control the size of the page tables, either 2K or 4K. The idea is that a KVM host sets the vm.alloc_pgste control to 1 which causes *all* new processes to run with 4K page tables. For a non-kvm system the control should stay off to save on memory used for page tables. Trouble is that distributions choose to set the control globally to be able to run KVM guests. This wastes memory on non-KVM systems. Introduce the PT_S390_PGSTE ELF segment type to "mark" the qemu executable with it. All executables with this (empty) segment in its ELF phdr array will be started with 4K page tables. Any executable without PT_S390_PGSTE will run with the default 2K page tables. This removes the need to set vm.alloc_pgste=1 for a KVM host and minimizes the waste of memory for page tables. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-06-07 15:10:24 +03:00
_PIF_WORK = (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/lpp: use assembler alternatives for the LPP instruction With the new macros for CPU alternatives the MACHINE_FLAG_LPP check around the LPP instruction can be optimized. After this is done the flag can be removed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:23:33 +03:00
_LPP_OFFSET = __LC_LPP
[PATCH] lockdep: irqtrace subsystem, s390 support irqtrace support for s390. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Arjan van de Ven <arjan@infradead.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-07-03 11:24:46 +04:00
.macro TRACE_IRQS_ON
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
#ifdef CONFIG_TRACE_IRQFLAGS
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
basr %r2,%r0
brasl %r14,trace_hardirqs_on_caller
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
#endif
[PATCH] lockdep: irqtrace subsystem, s390 support irqtrace support for s390. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Arjan van de Ven <arjan@infradead.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-07-03 11:24:46 +04:00
.endm
.macro TRACE_IRQS_OFF
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
#ifdef CONFIG_TRACE_IRQFLAGS
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
basr %r2,%r0
brasl %r14,trace_hardirqs_off_caller
[S390] Fix irq tracing and lockdep_sys_exit calls. Current support for TRACE_IRQFLAGS and lockdep_sys_exit is broken. IRQ flag tracing is broken for program checks. Even worse is that the newly introduced calls to lockdep_sys_exit are in the critical section code which is not supposed to call any C functions. In addition the checks if locks are still held are also done when returning to kernel code which is broken as well. Fix all this by disabling interrupts and machine checks at the exit paths and then do the appropriate checks and calls. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2007-11-20 13:13:32 +03:00
#endif
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
.endm
[S390] Fix irq tracing and lockdep_sys_exit calls. Current support for TRACE_IRQFLAGS and lockdep_sys_exit is broken. IRQ flag tracing is broken for program checks. Even worse is that the newly introduced calls to lockdep_sys_exit are in the critical section code which is not supposed to call any C functions. In addition the checks if locks are still held are also done when returning to kernel code which is broken as well. Fix all this by disabling interrupts and machine checks at the exit paths and then do the appropriate checks and calls. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2007-11-20 13:13:32 +03:00
.macro LOCKDEP_SYS_EXIT
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
#ifdef CONFIG_LOCKDEP
tm __PT_PSW+1(%r11),0x01 # returning to user ?
jz .+10
[S390] Fix irq tracing and lockdep_sys_exit calls. Current support for TRACE_IRQFLAGS and lockdep_sys_exit is broken. IRQ flag tracing is broken for program checks. Even worse is that the newly introduced calls to lockdep_sys_exit are in the critical section code which is not supposed to call any C functions. In addition the checks if locks are still held are also done when returning to kernel code which is broken as well. Fix all this by disabling interrupts and machine checks at the exit paths and then do the appropriate checks and calls. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2007-11-20 13:13:32 +03:00
brasl %r14,lockdep_sys_exit
[PATCH] lockdep: irqtrace subsystem, s390 support irqtrace support for s390. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Arjan van de Ven <arjan@infradead.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-07-03 11:24:46 +04:00
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
.endm
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
.macro CHECK_STACK savearea
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
#ifdef CONFIG_CHECK_STACK
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
tml %r15,STACK_SIZE - CONFIG_STACK_GUARD
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lghi %r14,\savearea
jz stack_overflow
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
#endif
.endm
s390/mm: add debug user asce support Verify on exit to user space that always - the primary ASCE (cr1) is set to kernel ASCE - the secondary ASCE (cr7) is set to user ASCE If this is not the case: panic since something went terribly wrong. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:41 +03:00
.macro DEBUG_USER_ASCE
#ifdef CONFIG_DEBUG_USER_ASCE
brasl %r14,debug_user_asce
#endif
.endm
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
.macro CHECK_VMAP_STACK savearea,oklabel
#ifdef CONFIG_VMAP_STACK
lgr %r14,%r15
nill %r14,0x10000 - STACK_SIZE
oill %r14,STACK_INIT
clg %r14,__LC_KERNEL_STACK
je \oklabel
clg %r14,__LC_ASYNC_STACK
je \oklabel
clg %r14,__LC_NODAT_STACK
je \oklabel
clg %r14,__LC_RESTART_STACK
je \oklabel
lghi %r14,\savearea
j stack_overflow
#else
j \oklabel
#endif
.endm
s390/idle: fix accounting with machine checks When a machine check interrupt is triggered during idle, the code is using the async timer/clock for idle time calculation. It should use the machine check enter timer/clock which is passed to the macro. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-06 12:47:47 +03:00
.macro SWITCH_ASYNC savearea,timer,clock
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
tmhh %r8,0x0001 # interrupting from user ?
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
jnz 4f
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
#if IS_ENABLED(CONFIG_KVM)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r14,%r9
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
larl %r13,.Lsie_gmap
slgr %r14,%r13
lghi %r13,.Lsie_done - .Lsie_gmap
clgr %r14,%r13
jhe 0f
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lghi %r11,\savearea # inside critical section, do cleanup
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
brasl %r14,.Lcleanup_sie
#endif
0: larl %r13,.Lpsw_idle_exit
cgr %r13,%r9
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
jne 3f
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
larl %r1,smp_cpu_mtid
llgf %r1,0(%r1)
ltgr %r1,%r1
jz 2f # no SMT, skip mt_cycles calculation
.insn rsy,0xeb0000000017,%r1,5,__SF_EMPTY+80(%r15)
larl %r3,mt_cycles
ag %r3,__LC_PERCPU_OFFSET
la %r4,__SF_EMPTY+16(%r15)
1: lg %r0,0(%r3)
slg %r0,0(%r4)
alg %r0,64(%r4)
stg %r0,0(%r3)
la %r3,8(%r3)
la %r4,8(%r4)
brct %r1,1b
s390/idle: fix accounting with machine checks When a machine check interrupt is triggered during idle, the code is using the async timer/clock for idle time calculation. It should use the machine check enter timer/clock which is passed to the macro. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-06 12:47:47 +03:00
2: mvc __CLOCK_IDLE_EXIT(8,%r2), \clock
mvc __TIMER_IDLE_EXIT(8,%r2), \timer
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
# account system time going idle
ni __LC_CPU_FLAGS+7,255-_CIF_ENABLED_WAIT
lg %r13,__LC_STEAL_TIMER
alg %r13,__CLOCK_IDLE_ENTER(%r2)
slg %r13,__LC_LAST_UPDATE_CLOCK
stg %r13,__LC_STEAL_TIMER
mvc __LC_LAST_UPDATE_CLOCK(8),__CLOCK_IDLE_EXIT(%r2)
lg %r13,__LC_SYSTEM_TIMER
alg %r13,__LC_LAST_UPDATE_TIMER
slg %r13,__TIMER_IDLE_ENTER(%r2)
stg %r13,__LC_SYSTEM_TIMER
mvc __LC_LAST_UPDATE_TIMER(8),__TIMER_IDLE_EXIT(%r2)
nihh %r8,0xfcfd # clear wait state and irq bits
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
3: lg %r14,__LC_ASYNC_STACK # are we already on the target stack?
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
slgr %r14,%r15
s390/nmi: use the normal asynchronous stack for machine checks If a machine checks is received while the CPU is in the kernel, only the s390_do_machine_check function will be called. The call to s390_handle_mcck is postponed until the CPU returns to user space. Because of this it is safe to use the asynchronous stack for machine checks even if the CPU is already handling an interrupt. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:28:14 +03:00
srag %r14,%r14,STACK_SHIFT
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
jnz 5f
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
CHECK_STACK \savearea
s390: lowcore stack pointer offsets Store the stack pointers in the lowcore for the kernel stack, the async stack and the panic stack with the offset required for the first user. This avoids an unnecessary add instruction on the system call path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-04-24 12:20:43 +04:00
aghi %r15,-(STACK_FRAME_OVERHEAD + __PT_SIZE)
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
j 6f
4: UPDATE_VTIME %r14,%r15,\timer
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
s390/idle: add missing mt_cycles calculation During removal of the critical section cleanup the calculation of mt_cycles during idle was removed. This causes invalid accounting on systems with SMT enabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 00:03:32 +03:00
5: lg %r15,__LC_ASYNC_STACK # load async stack
6: la %r11,STACK_FRAME_OVERHEAD(%r15)
[S390] Whitespace cleanup. Huge s390 assembly files whitespace cleanup. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-28 18:56:37 +04:00
.endm
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/kernel: squeeze a few more cycles out of the system call handler Reorder the instructions of UPDATE_VTIME to improve superscalar execution, remove duplicate checks for problem-state from the asynchronous interrupt handlers, and move the check for problem-state from the synchronous exit path to the program check path as it is only needed for program checks inside the kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:27:48 +03:00
.macro UPDATE_VTIME w1,w2,enter_timer
lg \w1,__LC_EXIT_TIMER
lg \w2,__LC_LAST_UPDATE_TIMER
slg \w1,\enter_timer
slg \w2,__LC_EXIT_TIMER
alg \w1,__LC_USER_TIMER
alg \w2,__LC_SYSTEM_TIMER
stg \w1,__LC_USER_TIMER
stg \w2,__LC_SYSTEM_TIMER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __LC_LAST_UPDATE_TIMER(8),\enter_timer
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
.endm
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
.macro RESTORE_SM_CLEAR_PER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r8,__LC_RETURN_PSW
ni __LC_RETURN_PSW,0xbf
ssm __LC_RETURN_PSW
[S390] cleanup lowcore access from program checks Read all required fields for program checks from the lowcore in the first level interrupt handler in entry[64].S. If the context that caused the fault was enabled for interrupts we can now re-enable the irqs in entry[64].S. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-10-25 18:10:37 +04:00
.endm
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
.macro ENABLE_INTS
stosm __SF_EMPTY(%r15),3
.endm
.macro ENABLE_INTS_TRACE
TRACE_IRQS_ON
ENABLE_INTS
.endm
.macro DISABLE_INTS
stnsm __SF_EMPTY(%r15),0xfc
.endm
.macro DISABLE_INTS_TRACE
DISABLE_INTS
TRACE_IRQS_OFF
.endm
s390/time: always use stckf instead of stck if available The store clock fast instruction saves a couple of instructions compared to the store clock instruction. Always use stckf instead of stck if it is available. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Reviewed-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-09 18:27:39 +04:00
.macro STCK savearea
s390/time: simply Kconfig dependency Use HAVE_MARCH_Z9_109_FEATURES to figure out if stckf is available at compile time. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-14 12:35:22 +04:00
#ifdef CONFIG_HAVE_MARCH_Z9_109_FEATURES
s390/time: always use stckf instead of stck if available The store clock fast instruction saves a couple of instructions compared to the store clock instruction. Always use stckf instead of stck if it is available. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Reviewed-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-09 18:27:39 +04:00
.insn s,0xb27c0000,\savearea # store clock fast
#else
.insn s,0xb2050000,\savearea # store clock
#endif
.endm
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
/*
* The TSTMSK macro generates a test-under-mask instruction by
* calculating the memory offset for the specified mask value.
* Mask value can be any constant. The macro shifts the mask
* value to calculate the memory offset for the test-under-mask
* instruction.
*/
.macro TSTMSK addr, mask, size=8, bytepos=0
.if (\bytepos < \size) && (\mask >> 8)
.if (\mask & 0xff)
.error "Mask exceeds byte boundary"
.endif
TSTMSK \addr, "(\mask >> 8)", \size, "(\bytepos + 1)"
.exitm
.endif
.ifeq \mask
.error "Mask must not be zero"
.endif
off = \size - \bytepos - 1
tm off+\addr, \mask
.endm
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
.macro BPOFF
s390/entry.S: use assembler alternatives Replace the open coded alternatives for the BPOFF, BPON, BPENTER, and BPEXIT macros with the new magic from asm/alternatives-asm.h to make the code easier to read. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:27:36 +03:00
ALTERNATIVE "", ".long 0xb2e8c000", 82
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
.endm
.macro BPON
s390/entry.S: use assembler alternatives Replace the open coded alternatives for the BPOFF, BPON, BPENTER, and BPEXIT macros with the new magic from asm/alternatives-asm.h to make the code easier to read. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:27:36 +03:00
ALTERNATIVE "", ".long 0xb2e8d000", 82
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
.endm
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
.macro BPENTER tif_ptr,tif_mask
s390/entry.S: use assembler alternatives Replace the open coded alternatives for the BPOFF, BPON, BPENTER, and BPEXIT macros with the new magic from asm/alternatives-asm.h to make the code easier to read. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:27:36 +03:00
ALTERNATIVE "TSTMSK \tif_ptr,\tif_mask; jz .+8; .long 0xb2e8d000", \
"", 82
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
.endm
.macro BPEXIT tif_ptr,tif_mask
TSTMSK \tif_ptr,\tif_mask
s390/entry.S: use assembler alternatives Replace the open coded alternatives for the BPOFF, BPON, BPENTER, and BPEXIT macros with the new magic from asm/alternatives-asm.h to make the code easier to read. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:27:36 +03:00
ALTERNATIVE "jz .+8; .long 0xb2e8c000", \
"jnz .+8; .long 0xb2e8d000", 82
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
.endm
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
GEN_BR_THUNK %r9
GEN_BR_THUNK %r14
GEN_BR_THUNK %r14,%r11
s390: introduce execute-trampolines for branches Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and -mfunction_return= compiler options to create a kernel fortified against the specte v2 attack. With CONFIG_EXPOLINE=y all indirect branches will be issued with an execute type instruction. For z10 or newer the EXRL instruction will be used, for older machines the EX instruction. The typical indirect call basr %r14,%r1 is replaced with a PC relative call to a new thunk brasl %r14,__s390x_indirect_jump_r1 The thunk contains the EXRL/EX instruction to the indirect branch __s390x_indirect_jump_r1: exrl 0,0f j . 0: br %r1 The detour via the execute type instruction has a performance impact. To get rid of the detour the new kernel parameter "nospectre_v2" and "spectre_v2=[on,off,auto]" can be used. If the parameter is specified the kernel and module code will be patched at runtime. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-26 14:46:47 +03:00
[S390] add kprobes annotations Add kprobes annotations to get the massive 'probe kernel.function("*") {}' stress test working. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-01-05 14:47:25 +03:00
.section .kprobes.text, "ax"
s390: have unique symbol for __switch_to address After linking there are several symbols for the same address that the __switch_to symbol points to. E.g.: 000000000089b9c0 T __kprobes_text_start 000000000089b9c0 T __lock_text_end 000000000089b9c0 T __lock_text_start 000000000089b9c0 T __sched_text_end 000000000089b9c0 T __switch_to When disassembling with "objdump -d" this results in a missing __switch_to function. It would be named __kprobes_text_start instead. To unconfuse objdump add a nop in front of the kprobes text section. That way __switch_to appears again. Obviously this solution is sort of a hack, since it also depends on link order if this works or not. However it is the best I can come up with for now. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-06-30 13:40:25 +03:00
.Ldummy:
/*
* This nop exists only in order to avoid that __switch_to starts at
* the beginning of the kprobes text section. In that case we would
* have several symbols at the same address. E.g. objdump would take
* an arbitrary symbol name when disassembling this code.
* With the added nop in between the __switch_to symbol is unique
* again.
*/
nop 0
[S390] add kprobes annotations Add kprobes annotations to get the massive 'probe kernel.function("*") {}' stress test working. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-01-05 14:47:25 +03:00
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
ENTRY(__bpon)
.globl __bpon
BPON
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BR_EX %r14
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(__bpon)
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* Scheduler resume function, called by switch_to
* gpr2 = (task_struct *) prev
* gpr3 = (task_struct *) next
* Returns:
* gpr2 = prev
*/
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(__switch_to)
s390: fix race on TIF_MCCK_PENDING There is a small race window in the __switch_to code in regard to the transfer of the TIF_MCCK_PENDING bit from the previous to the next task. The bit is transferred before the task struct pointer and the thread-info pointer for the next task has been stored to lowcore. If a machine check sets the TIF_MCCK_PENDING bit between the transfer code and the store of current/thread_info the bit is still set for the previous task. And if the previous task has terminated it can get lost. The effect is that a pending CRW is not retrieved until the next machine checks sets TIF_MCCK_PENDING. To fix this reorder __switch_to to first store the task struct and thread-info pointer and then do the transfer of the bit. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-15 11:20:06 +04:00
stmg %r6,%r15,__SF_GPRS(%r15) # store gprs of prev task
s390: rework __switch_to() to allow larger task_struct offsets If GCC_PLUGIN_RANDSTRUCT is enabled the members of task_struct will be shuffled around. The offsets of the "pid" and "stack" members within task_struct may not necessarily fit into 12 bits anymore, which causes compile errors within __switch_to, since instructions are used, which only have a 12 bit displacement field. Therefore rework __switch_to, to allow for larger offsets. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-11-16 16:54:04 +03:00
lghi %r4,__TASK_stack
lghi %r1,__TASK_thread
s390/kasan: increase instrumented stack size to 64k Increase kasan instrumented kernel stack size from 32k to 64k. Other architectures seems to get away with just doubling kernel stack size under kasan, but on s390 this appears to be not enough due to bigger frame size. The particular pain point is kasan inlined checks (CONFIG_KASAN_INLINE vs CONFIG_KASAN_OUTLINE). With inlined checks one particular case hitting stack overflow is fs sync on xfs filesystem: #0 [9a0681e8] 704 bytes check_usage at 34b1fc #1 [9a0684a8] 432 bytes check_usage at 34c710 #2 [9a068658] 1048 bytes validate_chain at 35044a #3 [9a068a70] 312 bytes __lock_acquire at 3559fe #4 [9a068ba8] 440 bytes lock_acquire at 3576ee #5 [9a068d60] 104 bytes _raw_spin_lock at 21b44e0 #6 [9a068dc8] 1992 bytes enqueue_entity at 2dbf72 #7 [9a069590] 1496 bytes enqueue_task_fair at 2df5f0 #8 [9a069b68] 64 bytes ttwu_do_activate at 28f438 #9 [9a069ba8] 552 bytes try_to_wake_up at 298c4c #10 [9a069dd0] 168 bytes wake_up_worker at 23f97c #11 [9a069e78] 200 bytes insert_work at 23fc2e #12 [9a069f40] 648 bytes __queue_work at 2487c0 #13 [9a06a1c8] 200 bytes __queue_delayed_work at 24db28 #14 [9a06a290] 248 bytes mod_delayed_work_on at 24de84 #15 [9a06a388] 24 bytes kblockd_mod_delayed_work_on at 153e2a0 #16 [9a06a3a0] 288 bytes __blk_mq_delay_run_hw_queue at 158168c #17 [9a06a4c0] 192 bytes blk_mq_run_hw_queue at 1581a3c #18 [9a06a580] 184 bytes blk_mq_sched_insert_requests at 15a2192 #19 [9a06a638] 1024 bytes blk_mq_flush_plug_list at 1590f3a #20 [9a06aa38] 704 bytes blk_flush_plug_list at 1555028 #21 [9a06acf8] 320 bytes schedule at 219e476 #22 [9a06ae38] 760 bytes schedule_timeout at 21b0aac #23 [9a06b130] 408 bytes wait_for_common at 21a1706 #24 [9a06b2c8] 360 bytes xfs_buf_iowait at fa1540 #25 [9a06b430] 256 bytes __xfs_buf_submit at fadae6 #26 [9a06b530] 264 bytes xfs_buf_read_map at fae3f6 #27 [9a06b638] 656 bytes xfs_trans_read_buf_map at 10ac9a8 #28 [9a06b8c8] 304 bytes xfs_btree_kill_root at e72426 #29 [9a06b9f8] 288 bytes xfs_btree_lookup_get_block at e7bc5e #30 [9a06bb18] 624 bytes xfs_btree_lookup at e7e1a6 #31 [9a06bd88] 2664 bytes xfs_alloc_ag_vextent_near at dfa070 #32 [9a06c7f0] 144 bytes xfs_alloc_ag_vextent at dff3ca #33 [9a06c880] 1128 bytes xfs_alloc_vextent at e05fce #34 [9a06cce8] 584 bytes xfs_bmap_btalloc at e58342 #35 [9a06cf30] 1336 bytes xfs_bmapi_write at e618de #36 [9a06d468] 776 bytes xfs_iomap_write_allocate at ff678e #37 [9a06d770] 720 bytes xfs_map_blocks at f82af8 #38 [9a06da40] 928 bytes xfs_writepage_map at f83cd6 #39 [9a06dde0] 320 bytes xfs_do_writepage at f85872 #40 [9a06df20] 1320 bytes write_cache_pages at 73dfe8 #41 [9a06e448] 208 bytes xfs_vm_writepages at f7f892 #42 [9a06e518] 88 bytes do_writepages at 73fe6a #43 [9a06e570] 872 bytes __writeback_single_inode at a20cb6 #44 [9a06e8d8] 664 bytes writeback_sb_inodes at a23be2 #45 [9a06eb70] 296 bytes __writeback_inodes_wb at a242e0 #46 [9a06ec98] 928 bytes wb_writeback at a2500e #47 [9a06f038] 848 bytes wb_do_writeback at a260ae #48 [9a06f388] 536 bytes wb_workfn at a28228 #49 [9a06f5a0] 1088 bytes process_one_work at 24a234 #50 [9a06f9e0] 1120 bytes worker_thread at 24ba26 #51 [9a06fe40] 104 bytes kthread at 26545a #52 [9a06fea8] kernel_thread_starter at 21b6b62 To be able to increase the stack size to 64k reuse LLILL instruction in __switch_to function to load 64k - STACK_FRAME_OVERHEAD - __PT_SIZE (65192) value as unsigned. Reported-by: Benjamin Block <bblock@linux.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-10-26 16:29:59 +03:00
llill %r5,STACK_INIT
s390: rework __switch_to() to allow larger task_struct offsets If GCC_PLUGIN_RANDSTRUCT is enabled the members of task_struct will be shuffled around. The offsets of the "pid" and "stack" members within task_struct may not necessarily fit into 12 bits anymore, which causes compile errors within __switch_to, since instructions are used, which only have a 12 bit displacement field. Therefore rework __switch_to, to allow for larger offsets. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-11-16 16:54:04 +03:00
stg %r15,__THREAD_ksp(%r1,%r2) # store kernel stack of prev
s390/kasan: increase instrumented stack size to 64k Increase kasan instrumented kernel stack size from 32k to 64k. Other architectures seems to get away with just doubling kernel stack size under kasan, but on s390 this appears to be not enough due to bigger frame size. The particular pain point is kasan inlined checks (CONFIG_KASAN_INLINE vs CONFIG_KASAN_OUTLINE). With inlined checks one particular case hitting stack overflow is fs sync on xfs filesystem: #0 [9a0681e8] 704 bytes check_usage at 34b1fc #1 [9a0684a8] 432 bytes check_usage at 34c710 #2 [9a068658] 1048 bytes validate_chain at 35044a #3 [9a068a70] 312 bytes __lock_acquire at 3559fe #4 [9a068ba8] 440 bytes lock_acquire at 3576ee #5 [9a068d60] 104 bytes _raw_spin_lock at 21b44e0 #6 [9a068dc8] 1992 bytes enqueue_entity at 2dbf72 #7 [9a069590] 1496 bytes enqueue_task_fair at 2df5f0 #8 [9a069b68] 64 bytes ttwu_do_activate at 28f438 #9 [9a069ba8] 552 bytes try_to_wake_up at 298c4c #10 [9a069dd0] 168 bytes wake_up_worker at 23f97c #11 [9a069e78] 200 bytes insert_work at 23fc2e #12 [9a069f40] 648 bytes __queue_work at 2487c0 #13 [9a06a1c8] 200 bytes __queue_delayed_work at 24db28 #14 [9a06a290] 248 bytes mod_delayed_work_on at 24de84 #15 [9a06a388] 24 bytes kblockd_mod_delayed_work_on at 153e2a0 #16 [9a06a3a0] 288 bytes __blk_mq_delay_run_hw_queue at 158168c #17 [9a06a4c0] 192 bytes blk_mq_run_hw_queue at 1581a3c #18 [9a06a580] 184 bytes blk_mq_sched_insert_requests at 15a2192 #19 [9a06a638] 1024 bytes blk_mq_flush_plug_list at 1590f3a #20 [9a06aa38] 704 bytes blk_flush_plug_list at 1555028 #21 [9a06acf8] 320 bytes schedule at 219e476 #22 [9a06ae38] 760 bytes schedule_timeout at 21b0aac #23 [9a06b130] 408 bytes wait_for_common at 21a1706 #24 [9a06b2c8] 360 bytes xfs_buf_iowait at fa1540 #25 [9a06b430] 256 bytes __xfs_buf_submit at fadae6 #26 [9a06b530] 264 bytes xfs_buf_read_map at fae3f6 #27 [9a06b638] 656 bytes xfs_trans_read_buf_map at 10ac9a8 #28 [9a06b8c8] 304 bytes xfs_btree_kill_root at e72426 #29 [9a06b9f8] 288 bytes xfs_btree_lookup_get_block at e7bc5e #30 [9a06bb18] 624 bytes xfs_btree_lookup at e7e1a6 #31 [9a06bd88] 2664 bytes xfs_alloc_ag_vextent_near at dfa070 #32 [9a06c7f0] 144 bytes xfs_alloc_ag_vextent at dff3ca #33 [9a06c880] 1128 bytes xfs_alloc_vextent at e05fce #34 [9a06cce8] 584 bytes xfs_bmap_btalloc at e58342 #35 [9a06cf30] 1336 bytes xfs_bmapi_write at e618de #36 [9a06d468] 776 bytes xfs_iomap_write_allocate at ff678e #37 [9a06d770] 720 bytes xfs_map_blocks at f82af8 #38 [9a06da40] 928 bytes xfs_writepage_map at f83cd6 #39 [9a06dde0] 320 bytes xfs_do_writepage at f85872 #40 [9a06df20] 1320 bytes write_cache_pages at 73dfe8 #41 [9a06e448] 208 bytes xfs_vm_writepages at f7f892 #42 [9a06e518] 88 bytes do_writepages at 73fe6a #43 [9a06e570] 872 bytes __writeback_single_inode at a20cb6 #44 [9a06e8d8] 664 bytes writeback_sb_inodes at a23be2 #45 [9a06eb70] 296 bytes __writeback_inodes_wb at a242e0 #46 [9a06ec98] 928 bytes wb_writeback at a2500e #47 [9a06f038] 848 bytes wb_do_writeback at a260ae #48 [9a06f388] 536 bytes wb_workfn at a28228 #49 [9a06f5a0] 1088 bytes process_one_work at 24a234 #50 [9a06f9e0] 1120 bytes worker_thread at 24ba26 #51 [9a06fe40] 104 bytes kthread at 26545a #52 [9a06fea8] kernel_thread_starter at 21b6b62 To be able to increase the stack size to 64k reuse LLILL instruction in __switch_to function to load 64k - STACK_FRAME_OVERHEAD - __PT_SIZE (65192) value as unsigned. Reported-by: Benjamin Block <bblock@linux.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-10-26 16:29:59 +03:00
lg %r15,0(%r4,%r3) # start of kernel stack of next
agr %r15,%r5 # end of kernel stack of next
s390: fix race on TIF_MCCK_PENDING There is a small race window in the __switch_to code in regard to the transfer of the TIF_MCCK_PENDING bit from the previous to the next task. The bit is transferred before the task struct pointer and the thread-info pointer for the next task has been stored to lowcore. If a machine check sets the TIF_MCCK_PENDING bit between the transfer code and the store of current/thread_info the bit is still set for the previous task. And if the previous task has terminated it can get lost. The effect is that a pending CRW is not retrieved until the next machine checks sets TIF_MCCK_PENDING. To fix this reorder __switch_to to first store the task struct and thread-info pointer and then do the transfer of the bit. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-15 11:20:06 +04:00
stg %r3,__LC_CURRENT # store task struct of next
stg %r15,__LC_KERNEL_STACK # store end of kernel stack
s390: rework __switch_to() to allow larger task_struct offsets If GCC_PLUGIN_RANDSTRUCT is enabled the members of task_struct will be shuffled around. The offsets of the "pid" and "stack" members within task_struct may not necessarily fit into 12 bits anymore, which causes compile errors within __switch_to, since instructions are used, which only have a 12 bit displacement field. Therefore rework __switch_to, to allow for larger offsets. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-11-16 16:54:04 +03:00
lg %r15,__THREAD_ksp(%r1,%r3) # load kernel stack of next
aghi %r3,__TASK_pid
mvc __LC_CURRENT_PID(4,%r0),0(%r3) # store pid of next
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
lmg %r6,%r15,__SF_GPRS(%r15) # load gprs of next task
s390/lpp: use assembler alternatives for the LPP instruction With the new macros for CPU alternatives the MACHINE_FLAG_LPP check around the LPP instruction can be optimized. After this is done the flag can be removed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:23:33 +03:00
ALTERNATIVE "", ".insn s,0xb2800000,_LPP_OFFSET", 40
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BR_EX %r14
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(__switch_to)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
#if IS_ENABLED(CONFIG_KVM)
/*
* sie64a calling convention:
* %r2 pointer to sie control block
* %r3 guest register save area
*/
ENTRY(sie64a)
stmg %r6,%r14,__SF_GPRS(%r15) # save kernel registers
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
lg %r12,__LC_CURRENT
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
stg %r2,__SF_SIE_CONTROL(%r15) # save control block pointer
stg %r3,__SF_SIE_SAVEAREA(%r15) # save guest register save area
xc __SF_SIE_REASON(8,%r15),__SF_SIE_REASON(%r15) # reason code = 0
mvc __SF_SIE_FLAGS(8,%r15),__TI_flags(%r12) # copy thread flags
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_CPU_FLAGS,_CIF_FPU # load guest fp/vx registers ?
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
jno .Lsie_load_guest_gprs
brasl %r14,load_fpu_regs # load guest fp/vx regs
.Lsie_load_guest_gprs:
lmg %r0,%r13,0(%r3) # load guest gprs 0-13
lg %r14,__LC_GMAP # get gmap pointer
ltgr %r14,%r14
jz .Lsie_gmap
lctlg %c1,%c1,__GMAP_ASCE(%r14) # load primary asce
.Lsie_gmap:
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
lg %r14,__SF_SIE_CONTROL(%r15) # get control block pointer
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
oi __SIE_PROG0C+3(%r14),1 # we are going into SIE now
tm __SIE_PROG20+3(%r14),3 # last exit...
jnz .Lsie_skip
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
jo .Lsie_skip # exit if fp/vx regs changed
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
BPEXIT __SF_SIE_FLAGS(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
s390/nmi: s390: New low level handling for machine check happening in guest Add the logic to check if the machine check happens when the guest is running. If yes, set the exit reason -EINTR in the machine check's interrupt handler. Refactor s390_do_machine_check to avoid panicing the host for some kinds of machine checks which happen when guest is running. Reinject the instruction processing damage's machine checks including Delayed Access Exception instead of damaging the host if it happens in the guest because it could be caused by improper update on TLB entry or other software case and impacts the guest only. Signed-off-by: QingFeng Hao <haoqf@linux.vnet.ibm.com> Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
2017-06-07 12:30:42 +03:00
.Lsie_entry:
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
sie 0(%r14)
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPOFF
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
BPENTER __SF_SIE_FLAGS(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
.Lsie_skip:
ni __SIE_PROG0C+3(%r14),0xfe # no longer in SIE
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__LC_KERNEL_ASCE # load primary asce
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
.Lsie_done:
# some program checks are suppressing. C code (e.g. do_protection_exception)
s390/kvm: do not rely on the ILC on kvm host protection fauls For most cases a protection exception in the host (e.g. copy on write or dirty tracking) on the sie instruction will indicate an instruction length of 4. Turns out that there are some corner cases (e.g. runtime instrumentation) where this is not necessarily true and the ILC is unpredictable. Let's replace our 4 byte rewind_pad with 3 byte nops to prepare for all possible ILCs. Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Cc: stable@vger.kernel.org Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-05-15 15:11:03 +03:00
# will rewind the PSW by the ILC, which is often 4 bytes in case of SIE. There
# are some corner cases (e.g. runtime instrumentation) where ILC is unpredictable.
# Other instructions between sie64a and .Lsie_done should not cause program
# interrupts. So lets use 3 nops as a landing pad for all possible rewinds.
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
# See also .Lcleanup_sie
s390/kvm: do not rely on the ILC on kvm host protection fauls For most cases a protection exception in the host (e.g. copy on write or dirty tracking) on the sie instruction will indicate an instruction length of 4. Turns out that there are some corner cases (e.g. runtime instrumentation) where this is not necessarily true and the ILC is unpredictable. Let's replace our 4 byte rewind_pad with 3 byte nops to prepare for all possible ILCs. Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Cc: stable@vger.kernel.org Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-05-15 15:11:03 +03:00
.Lrewind_pad6:
nopr 7
.Lrewind_pad4:
nopr 7
.Lrewind_pad2:
nopr 7
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
.globl sie_exit
sie_exit:
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
lg %r14,__SF_SIE_SAVEAREA(%r15) # load guest register save area
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
stmg %r0,%r13,0(%r14) # save guest gprs 0-13
s390: scrub registers on kernel entry and KVM exit Clear all user space registers on entry to the kernel and all KVM guest registers on KVM guest exit if the register does not contain either a parameter or a result value. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 15:27:30 +03:00
xgr %r0,%r0 # clear guest registers to
xgr %r1,%r1 # prevent speculative use
xgr %r2,%r2
xgr %r3,%r3
xgr %r4,%r4
xgr %r5,%r5
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
lmg %r6,%r14,__SF_GPRS(%r15) # restore kernel registers
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
lg %r2,__SF_SIE_REASON(%r15) # return exit reason code
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BR_EX %r14
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
.Lsie_fault:
lghi %r14,-EFAULT
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
stg %r14,__SF_SIE_REASON(%r15) # set exit reason code
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
j sie_exit
s390/kvm: do not rely on the ILC on kvm host protection fauls For most cases a protection exception in the host (e.g. copy on write or dirty tracking) on the sie instruction will indicate an instruction length of 4. Turns out that there are some corner cases (e.g. runtime instrumentation) where this is not necessarily true and the ILC is unpredictable. Let's replace our 4 byte rewind_pad with 3 byte nops to prepare for all possible ILCs. Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Cc: stable@vger.kernel.org Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-05-15 15:11:03 +03:00
EX_TABLE(.Lrewind_pad6,.Lsie_fault)
EX_TABLE(.Lrewind_pad4,.Lsie_fault)
EX_TABLE(.Lrewind_pad2,.Lsie_fault)
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
EX_TABLE(sie_exit,.Lsie_fault)
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(sie64a)
s390: move exports to definitions Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2016-01-12 21:30:03 +03:00
EXPORT_SYMBOL(sie64a)
EXPORT_SYMBOL(sie_exit)
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* SVC interrupt handler routine. System calls are synchronous events and
s390: fix comment regarding interrupts in svc With the removal of the critical section cleanup, we now enter the svc interrupt handler with interrupts disabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-07-07 15:07:53 +03:00
* are entered with interrupts disabled.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
*/
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(system_call)
[S390] Remove config options. On s390 we always want to run with precise cputime accounting. Remove the config options VIRT_TIMER and VIRT_CPU_ACCOUNTING. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-12-25 15:39:25 +03:00
stpt __LC_SYNC_ENTER_TIMER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r8,%r15,__LC_SAVE_AREA_SYNC
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPOFF
s390: move thread_info into task_struct This is the s390 variant of commit 15f4eae70d36 ("x86: Move thread_info into task_struct"). Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 13:08:26 +03:00
lg %r12,__LC_CURRENT
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
lghi %r14,_PIF_SYSCALL
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_per:
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__LC_KERNEL_ASCE
s390: fix system call single stepping When single stepping an svc instruction on s390, the kernel is entered with a PER program check interruption. The program check handler than jumps to the system call handler by reloading the PSW. The code didn't set GPR13 to the thread pointer in struct task_struct. This made the kernel access invalid memory while trying to fetch the syscall function address. Fix this by always assigned GPR13 after .Lsysc_per. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Reported-and-tested-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2020-06-17 18:30:28 +03:00
lghi %r13,__TASK_thread
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lg %r15,__LC_KERNEL_STACK
la %r11,STACK_FRAME_OVERHEAD(%r15) # pointer to pt_regs
s390: store breaking event address only for program checks The principles of operations specifies that the breaking event address is stored to the address 0x110 in the prefix page only for program checks. The last branch in user space is lost as soon as a branch in kernel space is executed after e.g. an svc. This makes it impossible to accurately maintain the breaking event address for a user space process. Simplify the code, just copy the current breaking event address from 0x110 to the task structure for program checks from user space. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-01-25 14:54:17 +03:00
UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r0,%r7,__PT_R0(%r11)
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
mvc __PT_PSW(16,%r11),__LC_SVC_OLD_PSW
[S390] cleanup trap handling Move the program interruption code and the translation exception identifier to the pt_regs structure as 'int_code' and 'int_parm_long' and make the first level interrupt handler in entry[64].S store the two values. That makes it possible to drop 'prot_addr' and 'trap_no' from the thread_struct and to reduce the number of arguments to a lot of functions. Finally un-inline do_trap. Overall this saves 5812 bytes in the .text section of the 64 bit kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:18 +04:00
mvc __PT_INT_CODE(4,%r11),__LC_SVC_ILC
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
stg %r14,__PT_FLAGS(%r11)
s390: always clear kernel stack backchain before calling functions Clear the kernel stack backchain before potentially calling the lockdep trace_hardirqs_off/on functions. Without this walking the kernel backchain, e.g. during a panic, might stop too early. Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 19:56:57 +03:00
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
ENABLE_INTS
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_do_svc:
s390/entry.S: fix spurious zeroing of r0 when a system call is interrupted we might call the critical section cleanup handler that re-does some of the operations. When we are between .Lsysc_vtime and .Lsysc_do_svc we might also redo the saving of the problem state registers r0-r7: .Lcleanup_system_call: [...] 0: # update accounting time stamp mvc __LC_LAST_UPDATE_TIMER(8),__LC_SYNC_ENTER_TIMER # set up saved register r11 lg %r15,__LC_KERNEL_STACK la %r9,STACK_FRAME_OVERHEAD(%r15) stg %r9,24(%r11) # r11 pt_regs pointer # fill pt_regs mvc __PT_R8(64,%r9),__LC_SAVE_AREA_SYNC ---> stmg %r0,%r7,__PT_R0(%r9) The problem is now, that we might have already zeroed out r0. The fix is to move the zeroing of r0 after sysc_do_svc. Reported-by: Farhan Ali <alifm@linux.vnet.ibm.com> Fixes: 7041d28115e91 ("s390: scrub registers on kernel entry and KVM exit") Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-05 22:18:47 +03:00
# clear user controlled register to prevent speculative use
xgr %r0,%r0
s390: move sys_call_table and last_break from thread_info to thread_struct Move the last two architecture specific fields from the thread_info structure to the thread_struct. All that is left in thread_info is the flags field. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 14:33:38 +03:00
# load address of system call table
lg %r10,__THREAD_sysc_table(%r13,%r12)
[S390] cleanup trap handling Move the program interruption code and the translation exception identifier to the pt_regs structure as 'int_code' and 'int_parm_long' and make the first level interrupt handler in entry[64].S store the two values. That makes it possible to drop 'prot_addr' and 'trap_no' from the thread_struct and to reduce the number of arguments to a lot of functions. Finally un-inline do_trap. Overall this saves 5812 bytes in the .text section of the 64 bit kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:18 +04:00
llgh %r8,__PT_INT_CODE+2(%r11)
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
slag %r8,%r8,3 # shift and test for svc 0
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lsysc_nr_ok
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
# svc 0: system call number in %r1
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
llgfr %r1,%r1 # clear high word in r1
s390/ptrace: pass invalid syscall numbers to tracing tracing expects to see invalid syscalls, so pass it through. The syscall path in entry.S checks the syscall number before looking up the handler, so it is still safe. Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-03-06 15:19:34 +03:00
sth %r1,__PT_INT_CODE+2(%r11)
[S390] add breaking event address for user space Copy the last breaking event address from the lowcore to a new field in the thread_struct on each system entry. Add a new ptrace request PTRACE_GET_LAST_BREAK and a new utrace regset REGSET_LAST_BREAK to query the last breaking event. This is useful for debugging wild branches in user space code. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:05 +04:00
cghi %r1,NR_syscalls
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnl .Lsysc_nr_ok
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
slag %r8,%r1,3
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_nr_ok:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r2,__PT_ORIG_GPR2(%r11)
stg %r7,STACK_FRAME_OVERHEAD(%r15)
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
lg %r9,0(%r8,%r10) # get system call add.
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_TRACE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lsysc_tracesys
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BASR_EX %r14,%r9 # call sys_xxxx
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r2,__PT_R2(%r11) # store return value
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_return:
s390: wire up rseq system call Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-06-30 11:54:15 +03:00
#ifdef CONFIG_DEBUG_RSEQ
lgr %r2,%r11
brasl %r14,rseq_syscall
#endif
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
LOCKDEP_SYS_EXIT
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_tif:
s390: fix system call exit path The system call exit path is running with interrupts enabled while checking for TIF/PIF/CIF bits which require special handling. If all bits have been checked interrupts are disabled and the kernel exits to user space. The problem is that after checking all bits and before interrupts are disabled bits can be set already again, due to interrupt handling. This means that the kernel can exit to user space with some TIF/PIF/CIF bits set, which should never happen. E.g. TIF_NEED_RESCHED might be set, which might lead to additional latencies, since that bit will only be recognized with next exit to user space. Fix this by checking the corresponding bits only when interrupts are disabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Acked-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-03 18:55:43 +03:00
DISABLE_INTS
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __PT_FLAGS(%r11),_PIF_WORK
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lsysc_work
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_WORK
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lsysc_work # check for work
s390/mm: add debug user asce support Verify on exit to user space that always - the primary ASCE (cr1) is set to kernel ASCE - the secondary ASCE (cr7) is set to user ASCE If this is not the case: panic since something went terribly wrong. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:41 +03:00
DEBUG_USER_ASCE
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__LC_USER_ASCE
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
TSTMSK __LC_CPU_FLAGS, _CIF_FPU
jz .Lsysc_skip_fpu
brasl %r14,load_fpu_regs
.Lsysc_skip_fpu:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __LC_RETURN_PSW(16),__PT_PSW(%r11)
stpt __LC_EXIT_TIMER
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
lmg %r0,%r15,__PT_R0(%r11)
b __LC_RETURN_LPSWE
[S390] Fix irq tracing and lockdep_sys_exit calls. Current support for TRACE_IRQFLAGS and lockdep_sys_exit is broken. IRQ flag tracing is broken for program checks. Even worse is that the newly introduced calls to lockdep_sys_exit are in the critical section code which is not supposed to call any C functions. In addition the checks if locks are still held are also done when returning to kernel code which is broken as well. Fix all this by disabling interrupts and machine checks at the exit paths and then do the appropriate checks and calls. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2007-11-20 13:13:32 +03:00
[S390] cleanup sysc_work and io_work code Cleanup the #ifdef mess at io_work in entry[64].S and streamline the TIF work code of the system call and io exit path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:01 +04:00
#
# One of the work bits is on. Find out which one.
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_work:
s390: fix system call exit path The system call exit path is running with interrupts enabled while checking for TIF/PIF/CIF bits which require special handling. If all bits have been checked interrupts are disabled and the kernel exits to user space. The problem is that after checking all bits and before interrupts are disabled bits can be set already again, due to interrupt handling. This means that the kernel can exit to user space with some TIF/PIF/CIF bits set, which should never happen. E.g. TIF_NEED_RESCHED might be set, which might lead to additional latencies, since that bit will only be recognized with next exit to user space. Fix this by checking the corresponding bits only when interrupts are disabled. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Acked-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-03 18:55:43 +03:00
ENABLE_INTS
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lsysc_reschedule
s390/kvm: avoid global config of vm.alloc_pgste=1 The system control vm.alloc_pgste is used to control the size of the page tables, either 2K or 4K. The idea is that a KVM host sets the vm.alloc_pgste control to 1 which causes *all* new processes to run with 4K page tables. For a non-kvm system the control should stay off to save on memory used for page tables. Trouble is that distributions choose to set the control globally to be able to run KVM guests. This wastes memory on non-KVM systems. Introduce the PT_S390_PGSTE ELF segment type to "mark" the qemu executable with it. All executables with this (empty) segment in its ELF phdr array will be started with 4K page tables. Any executable without PT_S390_PGSTE will run with the default 2K page tables. This removes the need to set vm.alloc_pgste=1 for a KVM host and minimizes the waste of memory for page tables. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-06-07 15:10:24 +03:00
TSTMSK __PT_FLAGS(%r11),_PIF_SYSCALL_RESTART
jo .Lsysc_syscall_restart
s390/uprobes: architecture backend for uprobes Signed-off-by: Jan Willeke <willeke@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-09-22 18:39:06 +04:00
#ifdef CONFIG_UPROBES
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_UPROBE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lsysc_uprobe_notify
s390/uprobes: architecture backend for uprobes Signed-off-by: Jan Willeke <willeke@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-09-22 18:39:06 +04:00
#endif
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
TSTMSK __TI_flags(%r12),_TIF_GUARDED_STORAGE
jo .Lsysc_guarded_storage
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __PT_FLAGS(%r11),_PIF_PER_TRAP
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lsysc_singlestep
livepatch/s390: add TIF_PATCH_PENDING thread flag Update a task's patch state when returning from a system call or user space interrupt, or after handling a signal. This greatly increases the chances of a patch operation succeeding. If a task is I/O bound, it can be patched when returning from a system call. If a task is CPU bound, it can be patched when returning from an interrupt. If a task is sleeping on a to-be-patched function, the user can send SIGSTOP and SIGCONT to force it to switch. Since there are two ways the syscall can be restarted on return from a signal handling process, it is important to clear the flag before do_signal() is called. Otherwise we could miss the migration if we used SIGSTOP/SIGCONT procedure or fake signal to migrate patching blocking tasks. If we place our hook to sysc_work label in entry before TIF_SIGPENDING is evaluated we kill two birds with one stone. The task is correctly migrated in all return paths from a syscall. Signed-off-by: Miroslav Benes <mbenes@suse.cz> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2017-02-14 04:42:34 +03:00
#ifdef CONFIG_LIVEPATCH
TSTMSK __TI_flags(%r12),_TIF_PATCH_PENDING
jo .Lsysc_patch_pending # handle live patching just before
# signals and possible syscall restart
#endif
s390/kvm: avoid global config of vm.alloc_pgste=1 The system control vm.alloc_pgste is used to control the size of the page tables, either 2K or 4K. The idea is that a KVM host sets the vm.alloc_pgste control to 1 which causes *all* new processes to run with 4K page tables. For a non-kvm system the control should stay off to save on memory used for page tables. Trouble is that distributions choose to set the control globally to be able to run KVM guests. This wastes memory on non-KVM systems. Introduce the PT_S390_PGSTE ELF segment type to "mark" the qemu executable with it. All executables with this (empty) segment in its ELF phdr array will be started with 4K page tables. Any executable without PT_S390_PGSTE will run with the default 2K page tables. This removes the need to set vm.alloc_pgste=1 for a KVM host and minimizes the waste of memory for page tables. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-06-07 15:10:24 +03:00
TSTMSK __PT_FLAGS(%r11),_PIF_SYSCALL_RESTART
jo .Lsysc_syscall_restart
s390: add support for TIF_NOTIFY_SIGNAL Wire up TIF_NOTIFY_SIGNAL handling for s390. Cc: linux-s390@vger.kernel.org Acked-by: Heiko Carstens <hca@linux.ibm.com> Acked-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-10-10 00:34:12 +03:00
TSTMSK __TI_flags(%r12),(_TIF_SIGPENDING|_TIF_NOTIFY_SIGNAL)
jnz .Lsysc_sigpending
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lsysc_notify_resume
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
j .Lsysc_return
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
# _TIF_NEED_RESCHED is set, call schedule
[S390] Whitespace cleanup. Huge s390 assembly files whitespace cleanup. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-28 18:56:37 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_reschedule:
larl %r14,.Lsysc_return
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
jg schedule
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
signals: s390: renumber TIF_RESTORE_SIGMASK TIF_RESTORE_SIGMASK no longer needs to be in the _TIF_WORK_* masks. Those low bits are scarce, and are all used up now. Renumber TIF_RESTORE_SIGMASK to free one up. Signed-off-by: Roland McGrath <roland@redhat.com> Cc: Oleg Nesterov <oleg@tv-sign.ru> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: "Luck, Tony" <tony.luck@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-04-30 11:53:08 +04:00
# _TIF_SIGPENDING is set, call do_signal
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_sigpending:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
brasl %r14,do_signal
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __PT_FLAGS(%r11),_PIF_SYSCALL
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jno .Lsysc_return
s390: add no-execute support Bit 0x100 of a page table, segment table of region table entry can be used to disallow code execution for the virtual addresses associated with the entry. There is one tricky bit, the system call to return from a signal is part of the signal frame written to the user stack. With a non-executable stack this would stop working. To avoid breaking things the protection fault handler checks the opcode that caused the fault for 0x0a77 (sys_sigreturn) and 0x0aad (sys_rt_sigreturn) and injects a system call. This is preferable to the alternative solution with a stub function in the vdso because it works for vdso=off and statically linked binaries as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-22 12:54:24 +03:00
.Lsysc_do_syscall:
lghi %r13,__TASK_thread
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r2,%r7,__PT_R2(%r11) # load svc arguments
s390: add no-execute support Bit 0x100 of a page table, segment table of region table entry can be used to disallow code execution for the virtual addresses associated with the entry. There is one tricky bit, the system call to return from a signal is part of the signal frame written to the user stack. With a non-executable stack this would stop working. To avoid breaking things the protection fault handler checks the opcode that caused the fault for 0x0a77 (sys_sigreturn) and 0x0aad (sys_rt_sigreturn) and injects a system call. This is preferable to the alternative solution with a stub function in the vdso because it works for vdso=off and statically linked binaries as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-22 12:54:24 +03:00
lghi %r1,0 # svc 0 returns -ENOSYS
j .Lsysc_do_svc
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
#
# _TIF_NOTIFY_RESUME is set, call do_notify_resume
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_notify_resume:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
larl %r14,.Lsysc_return
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
jg do_notify_resume
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
s390/uprobes: architecture backend for uprobes Signed-off-by: Jan Willeke <willeke@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-09-22 18:39:06 +04:00
#
# _TIF_UPROBE is set, call uprobe_notify_resume
#
#ifdef CONFIG_UPROBES
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_uprobe_notify:
s390/uprobes: architecture backend for uprobes Signed-off-by: Jan Willeke <willeke@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-09-22 18:39:06 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
larl %r14,.Lsysc_return
s390/uprobes: architecture backend for uprobes Signed-off-by: Jan Willeke <willeke@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-09-22 18:39:06 +04:00
jg uprobe_notify_resume
#endif
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
#
# _TIF_GUARDED_STORAGE is set, call guarded_storage_load
#
.Lsysc_guarded_storage:
lgr %r2,%r11 # pass pointer to pt_regs
larl %r14,.Lsysc_return
jg gs_load_bc_cb
livepatch/s390: add TIF_PATCH_PENDING thread flag Update a task's patch state when returning from a system call or user space interrupt, or after handling a signal. This greatly increases the chances of a patch operation succeeding. If a task is I/O bound, it can be patched when returning from a system call. If a task is CPU bound, it can be patched when returning from an interrupt. If a task is sleeping on a to-be-patched function, the user can send SIGSTOP and SIGCONT to force it to switch. Since there are two ways the syscall can be restarted on return from a signal handling process, it is important to clear the flag before do_signal() is called. Otherwise we could miss the migration if we used SIGSTOP/SIGCONT procedure or fake signal to migrate patching blocking tasks. If we place our hook to sysc_work label in entry before TIF_SIGPENDING is evaluated we kill two birds with one stone. The task is correctly migrated in all return paths from a syscall. Signed-off-by: Miroslav Benes <mbenes@suse.cz> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2017-02-14 04:42:34 +03:00
#
# _TIF_PATCH_PENDING is set, call klp_update_patch_state
#
#ifdef CONFIG_LIVEPATCH
.Lsysc_patch_pending:
lg %r2,__LC_CURRENT # pass pointer to task struct
larl %r14,.Lsysc_return
jg klp_update_patch_state
#endif
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
# _PIF_PER_TRAP is set, call do_per_trap
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_singlestep:
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
ni __PT_FLAGS+7(%r11),255-_PIF_PER_TRAP
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
larl %r14,.Lsysc_return
[S390] ptrace cleanup Overhaul program event recording and the code dealing with the ptrace user space interface. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-01-05 14:48:10 +03:00
jg do_per_trap
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/kvm: avoid global config of vm.alloc_pgste=1 The system control vm.alloc_pgste is used to control the size of the page tables, either 2K or 4K. The idea is that a KVM host sets the vm.alloc_pgste control to 1 which causes *all* new processes to run with 4K page tables. For a non-kvm system the control should stay off to save on memory used for page tables. Trouble is that distributions choose to set the control globally to be able to run KVM guests. This wastes memory on non-KVM systems. Introduce the PT_S390_PGSTE ELF segment type to "mark" the qemu executable with it. All executables with this (empty) segment in its ELF phdr array will be started with 4K page tables. Any executable without PT_S390_PGSTE will run with the default 2K page tables. This removes the need to set vm.alloc_pgste=1 for a KVM host and minimizes the waste of memory for page tables. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-06-07 15:10:24 +03:00
#
# _PIF_SYSCALL_RESTART is set, repeat the current system call
#
.Lsysc_syscall_restart:
ni __PT_FLAGS+7(%r11),255-_PIF_SYSCALL_RESTART
lmg %r1,%r7,__PT_R1(%r11) # load svc arguments
lg %r2,__PT_ORIG_GPR2(%r11)
j .Lsysc_do_svc
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
# call tracehook_report_syscall_entry/tracehook_report_syscall_exit before
# and after the system call
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_tracesys:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
la %r3,0
[S390] cleanup trap handling Move the program interruption code and the translation exception identifier to the pt_regs structure as 'int_code' and 'int_parm_long' and make the first level interrupt handler in entry[64].S store the two values. That makes it possible to drop 'prot_addr' and 'trap_no' from the thread_struct and to reduce the number of arguments to a lot of functions. Finally un-inline do_trap. Overall this saves 5812 bytes in the .text section of the 64 bit kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:18 +04:00
llgh %r0,__PT_INT_CODE+2(%r11)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r0,__PT_R2(%r11)
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
brasl %r14,do_syscall_trace_enter
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
lghi %r0,NR_syscalls
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
clgr %r0,%r2
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnh .Lsysc_tracenogo
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
sllg %r8,%r2,3
lg %r9,0(%r8,%r10)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r3,%r7,__PT_R3(%r11)
stg %r7,STACK_FRAME_OVERHEAD(%r15)
lg %r2,__PT_ORIG_GPR2(%r11)
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BASR_EX %r14,%r9 # call sys_xxx
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r2,__PT_R2(%r11) # store return value
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lsysc_tracenogo:
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_TRACE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jz .Lsysc_return
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
larl %r14,.Lsysc_return
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
jg do_syscall_trace_exit
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(system_call)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
# a new process exits the kernel with ret_from_fork
#
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(ret_from_fork)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
la %r11,STACK_FRAME_OVERHEAD(%r15)
s390: move thread_info into task_struct This is the s390 variant of commit 15f4eae70d36 ("x86: Move thread_info into task_struct"). Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 13:08:26 +03:00
lg %r12,__LC_CURRENT
s390: fold kernel_thread_helper() into ret_from_fork() ... and don't bother with syscall return path in case of kernel threads. Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2012-09-11 02:03:41 +04:00
brasl %r14,schedule_tail
s390: switch to saner kernel_execve() semantics Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2012-10-11 23:30:14 +04:00
tm __PT_PSW+1(%r11),0x01 # forking a kernel thread ?
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jne .Lsysc_tracenogo
s390: switch to saner kernel_execve() semantics Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2012-10-11 23:30:14 +04:00
# it's a kernel thread
lmg %r9,%r10,__PT_R9(%r11) # load gprs
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
la %r2,0(%r10)
BASR_EX %r14,%r9
j .Lsysc_tracenogo
ENDPROC(ret_from_fork)
s390: fold kernel_thread_helper() into ret_from_fork() ... and don't bother with syscall return path in case of kernel threads. Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2012-09-11 02:03:41 +04:00
ENTRY(kernel_thread_starter)
la %r2,0(%r10)
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BASR_EX %r14,%r9
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lsysc_tracenogo
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(kernel_thread_starter)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* Program check handler routine
*/
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(pgm_check_handler)
[S390] Remove config options. On s390 we always want to run with precise cputime accounting. Remove the config options VIRT_TIMER and VIRT_CPU_ACCOUNTING. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-12-25 15:39:25 +03:00
stpt __LC_SYNC_ENTER_TIMER
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPOFF
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r8,%r15,__LC_SAVE_AREA_SYNC
lg %r10,__LC_LAST_BREAK
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
srag %r11,%r10,12
jnz 0f
/* if __LC_LAST_BREAK is < 4096, it contains one of
* the lpswe addresses in lowcore. Set it to 1 (initial state)
* to prevent leaking that address to userspace.
*/
lghi %r10,1
0: lg %r12,__LC_CURRENT
s390/mm,kvm: improve detection of KVM guest faults The identification of guest fault currently relies on the PF_VCPU flag. This is set in guest_entry_irqoff and cleared in guest_exit_irqoff. Both functions are called by __vcpu_run, the PF_VCPU flag is set for quite a lot of kernel code outside of the guest execution. Replace the PF_VCPU scheme with the PIF_GUEST_FAULT in the pt_regs and make the program check handler code in entry.S set the bit only for exception that occurred between the .Lsie_gmap and .Lsie_done labels. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2017-10-05 09:44:26 +03:00
lghi %r11,0
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r8,%r9,__LC_PGM_OLD_PSW
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
tmhh %r8,0x0001 # coming from user space?
jno .Lpgm_skip_asce
lctlg %c1,%c1,__LC_KERNEL_ASCE
j 3f
.Lpgm_skip_asce:
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
#if IS_ENABLED(CONFIG_KVM)
s390/kvm: fix detection of guest machine checks The new detection code for guest machine checks added a check based on %r11 to .Lcleanup_sie to distinguish between normal asynchronous interrupts and machine checks. But the funtion is called from the program check handler as well with an undefined value in %r11. The effect is that all program exceptions pointing to the SIE instruction will set the CIF_MCCK_GUEST bit. The bit stays set for the CPU until the next machine check comes in which will incorrectly be interpreted as a guest machine check. The simplest fix is to stop using .Lcleanup_sie in the program check handler and duplicate a few instructions. Fixes: c929500d7a5a ("s390/nmi: s390: New low level handling for machine check happening in guest") Cc: <stable@vger.kernel.org> # v4.13+ Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-05 09:29:47 +03:00
# cleanup critical section for program checks in sie64a
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
lgr %r14,%r9
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
larl %r13,.Lsie_gmap
slgr %r14,%r13
lghi %r13,.Lsie_done - .Lsie_gmap
clgr %r14,%r13
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
jhe 1f
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
lg %r14,__SF_SIE_CONTROL(%r15) # get control block pointer
s390/kvm: fix detection of guest machine checks The new detection code for guest machine checks added a check based on %r11 to .Lcleanup_sie to distinguish between normal asynchronous interrupts and machine checks. But the funtion is called from the program check handler as well with an undefined value in %r11. The effect is that all program exceptions pointing to the SIE instruction will set the CIF_MCCK_GUEST bit. The bit stays set for the CPU until the next machine check comes in which will incorrectly be interpreted as a guest machine check. The simplest fix is to stop using .Lcleanup_sie in the program check handler and duplicate a few instructions. Fixes: c929500d7a5a ("s390/nmi: s390: New low level handling for machine check happening in guest") Cc: <stable@vger.kernel.org> # v4.13+ Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-05 09:29:47 +03:00
ni __SIE_PROG0C+3(%r14),0xfe # no longer in SIE
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__LC_KERNEL_ASCE # load primary asce
s390/kvm: fix detection of guest machine checks The new detection code for guest machine checks added a check based on %r11 to .Lcleanup_sie to distinguish between normal asynchronous interrupts and machine checks. But the funtion is called from the program check handler as well with an undefined value in %r11. The effect is that all program exceptions pointing to the SIE instruction will set the CIF_MCCK_GUEST bit. The bit stays set for the CPU until the next machine check comes in which will incorrectly be interpreted as a guest machine check. The simplest fix is to stop using .Lcleanup_sie in the program check handler and duplicate a few instructions. Fixes: c929500d7a5a ("s390/nmi: s390: New low level handling for machine check happening in guest") Cc: <stable@vger.kernel.org> # v4.13+ Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-05 09:29:47 +03:00
larl %r9,sie_exit # skip forward to sie_exit
s390/mm,kvm: improve detection of KVM guest faults The identification of guest fault currently relies on the PF_VCPU flag. This is set in guest_entry_irqoff and cleared in guest_exit_irqoff. Both functions are called by __vcpu_run, the PF_VCPU flag is set for quite a lot of kernel code outside of the guest execution. Replace the PF_VCPU scheme with the PIF_GUEST_FAULT in the pt_regs and make the program check handler code in entry.S set the bit only for exception that occurred between the .Lsie_gmap and .Lsie_done labels. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2017-10-05 09:44:26 +03:00
lghi %r11,_PIF_GUEST_FAULT
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
#endif
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
1: tmhh %r8,0x4000 # PER bit set in old PSW ?
jnz 2f # -> enabled, can't be a double fault
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
tm __LC_PGM_ILC+3,0x80 # check for per exception
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lpgm_svcper # -> single stepped svc
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
2: CHECK_STACK __LC_SAVE_AREA_SYNC
s390: lowcore stack pointer offsets Store the stack pointers in the lowcore for the kernel stack, the async stack and the panic stack with the offset required for the first user. This avoids an unnecessary add instruction on the system call path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-04-24 12:20:43 +04:00
aghi %r15,-(STACK_FRAME_OVERHEAD + __PT_SIZE)
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
# CHECK_VMAP_STACK branches to stack_overflow or 5f
CHECK_VMAP_STACK __LC_SAVE_AREA_SYNC,5f
3: UPDATE_VTIME %r14,%r15,__LC_SYNC_ENTER_TIMER
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lg %r15,__LC_KERNEL_STACK
s390: move thread_info into task_struct This is the s390 variant of commit 15f4eae70d36 ("x86: Move thread_info into task_struct"). Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 13:08:26 +03:00
lgr %r14,%r12
s390: adapt entry.S to the move of thread_struct git commit 0c8c0f03e3a292e031596484275c14cf39c0ab7a "x86/fpu, sched: Dynamically allocate 'struct fpu'" moved the thread_struct to the end of the task_struct. This causes some of the offsets used in entry.S to overflow their instruction operand field. To fix this use aghi to create a dedicated pointer for the thread_struct. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-07-20 11:01:46 +03:00
aghi %r14,__TASK_thread # pointer to thread_struct
s390: add support for transactional memory Allow user-space processes to use transactional execution (TX). If the TX facility is available user space programs can use transactions for fine-grained serialization based on the data objects that are referenced during a transaction. This is useful for lockless data structures and speculative compiler optimizations. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-07-31 13:03:04 +04:00
lghi %r13,__LC_PGM_TDB
tm __LC_PGM_ILC+2,0x02 # check for transaction abort
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
jz 4f
s390: add support for transactional memory Allow user-space processes to use transactional execution (TX). If the TX facility is available user space programs can use transactions for fine-grained serialization based on the data objects that are referenced during a transaction. This is useful for lockless data structures and speculative compiler optimizations. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-07-31 13:03:04 +04:00
mvc __THREAD_trap_tdb(256,%r14),0(%r13)
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
4: stg %r10,__THREAD_last_break(%r14)
5: lgr %r13,%r11
s390/mm,kvm: improve detection of KVM guest faults The identification of guest fault currently relies on the PF_VCPU flag. This is set in guest_entry_irqoff and cleared in guest_exit_irqoff. Both functions are called by __vcpu_run, the PF_VCPU flag is set for quite a lot of kernel code outside of the guest execution. Replace the PF_VCPU scheme with the PIF_GUEST_FAULT in the pt_regs and make the program check handler code in entry.S set the bit only for exception that occurred between the .Lsie_gmap and .Lsie_done labels. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2017-10-05 09:44:26 +03:00
la %r11,STACK_FRAME_OVERHEAD(%r15)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r0,%r7,__PT_R0(%r11)
s390: scrub registers on kernel entry and KVM exit Clear all user space registers on entry to the kernel and all KVM guest registers on KVM guest exit if the register does not contain either a parameter or a result value. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 15:27:30 +03:00
# clear user controlled registers to prevent speculative use
xgr %r0,%r0
xgr %r1,%r1
xgr %r2,%r2
xgr %r3,%r3
xgr %r4,%r4
xgr %r5,%r5
xgr %r6,%r6
xgr %r7,%r7
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
stmg %r8,%r9,__PT_PSW(%r11)
[S390] cleanup trap handling Move the program interruption code and the translation exception identifier to the pt_regs structure as 'int_code' and 'int_parm_long' and make the first level interrupt handler in entry[64].S store the two values. That makes it possible to drop 'prot_addr' and 'trap_no' from the thread_struct and to reduce the number of arguments to a lot of functions. Finally un-inline do_trap. Overall this saves 5812 bytes in the .text section of the 64 bit kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:18 +04:00
mvc __PT_INT_CODE(4,%r11),__LC_PGM_ILC
mvc __PT_INT_PARM_LONG(8,%r11),__LC_TRANS_EXC_CODE
s390/mm,kvm: improve detection of KVM guest faults The identification of guest fault currently relies on the PF_VCPU flag. This is set in guest_entry_irqoff and cleared in guest_exit_irqoff. Both functions are called by __vcpu_run, the PF_VCPU flag is set for quite a lot of kernel code outside of the guest execution. Replace the PF_VCPU scheme with the PIF_GUEST_FAULT in the pt_regs and make the program check handler code in entry.S set the bit only for exception that occurred between the .Lsie_gmap and .Lsie_done labels. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2017-10-05 09:44:26 +03:00
stg %r13,__PT_FLAGS(%r11)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r10,__PT_ARGS(%r11)
tm __LC_PGM_ILC+3,0x80 # check for per exception
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
jz 6f
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
tmhh %r8,0x0001 # kernel per event ?
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jz .Lpgm_kprobe
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
oi __PT_FLAGS+7(%r11),_PIF_PER_TRAP
s390: add support for transactional memory Allow user-space processes to use transactional execution (TX). If the TX facility is available user space programs can use transactions for fine-grained serialization based on the data objects that are referenced during a transaction. This is useful for lockless data structures and speculative compiler optimizations. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-07-31 13:03:04 +04:00
mvc __THREAD_per_address(8,%r14),__LC_PER_ADDRESS
s390: rename and split lowcore field per_perc_atmid per_perc_atmid is currently a two-byte field that combines two fields, the PER code and the PER Addressing-and-Translation-Mode Identification (ATMID) Let's make them accessible indepently and also rename per_cause to per_code. Signed-off-by: Jens Freimann <jfrei@linux.vnet.ibm.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
2014-02-26 19:32:46 +04:00
mvc __THREAD_per_cause(2,%r14),__LC_PER_CODE
mvc __THREAD_per_paid(1,%r14),__LC_PER_ACCESS_ID
s390: always clear kernel stack backchain before calling functions Clear the kernel stack backchain before potentially calling the lockdep trace_hardirqs_off/on functions. Without this walking the kernel backchain, e.g. during a panic, might stop too early. Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 19:56:57 +03:00
6: xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
RESTORE_SM_CLEAR_PER
[S390] Fix IRQ tracing in case of PER In case user space is single stepped (PER) the program check handler claims too early that IRQs are enabled on the return path. Subsequent checks will notice that the IRQ mask in the PSW and what lockdep thinks the IRQ mask should be do not correlate and therefore will print a warning to the console and disable lockdep. Fix this by doing all the work within the correct context. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-07-27 21:29:37 +04:00
larl %r1,pgm_check_table
[S390] cleanup trap handling Move the program interruption code and the translation exception identifier to the pt_regs structure as 'int_code' and 'int_parm_long' and make the first level interrupt handler in entry[64].S store the two values. That makes it possible to drop 'prot_addr' and 'trap_no' from the thread_struct and to reduce the number of arguments to a lot of functions. Finally un-inline do_trap. Overall this saves 5812 bytes in the .text section of the 64 bit kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:18 +04:00
llgh %r10,__PT_INT_CODE+2(%r11)
nill %r10,0x007f
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
sll %r10,3
s390/kernel: squeeze a few more cycles out of the system call handler Reorder the instructions of UPDATE_VTIME to improve superscalar execution, remove duplicate checks for problem-state from the asynchronous interrupt handlers, and move the check for problem-state from the synchronous exit path to the program check path as it is only needed for program checks inside the kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:27:48 +03:00
je .Lpgm_return
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
lg %r9,0(%r10,%r1) # load address of handler routine
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BASR_EX %r14,%r9 # branch to interrupt-handler
s390/kernel: squeeze a few more cycles out of the system call handler Reorder the instructions of UPDATE_VTIME to improve superscalar execution, remove duplicate checks for problem-state from the asynchronous interrupt handlers, and move the check for problem-state from the synchronous exit path to the program check path as it is only needed for program checks inside the kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:27:48 +03:00
.Lpgm_return:
LOCKDEP_SYS_EXIT
tm __PT_PSW+1(%r11),0x01 # returning to user ?
s390: add separate program check exit path System call and program check handler both use the system call exit path when returning to previous context. However the program check handler jumps right to the end of the system call exit path if the previous context is kernel context. This lead to the quite odd double disabling of interrupts in the system call exit path introduced with commit ce9dfafe29be ("s390: fix system call exit path"). To avoid that have a separate program check handler exit path if the previous context is kernel context. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-11 20:46:26 +03:00
jno .Lpgm_restore
s390: add no-execute support Bit 0x100 of a page table, segment table of region table entry can be used to disallow code execution for the virtual addresses associated with the entry. There is one tricky bit, the system call to return from a signal is part of the signal frame written to the user stack. With a non-executable stack this would stop working. To avoid breaking things the protection fault handler checks the opcode that caused the fault for 0x0a77 (sys_sigreturn) and 0x0aad (sys_rt_sigreturn) and injects a system call. This is preferable to the alternative solution with a stub function in the vdso because it works for vdso=off and statically linked binaries as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-22 12:54:24 +03:00
TSTMSK __PT_FLAGS(%r11),_PIF_SYSCALL
jo .Lsysc_do_syscall
s390/kernel: squeeze a few more cycles out of the system call handler Reorder the instructions of UPDATE_VTIME to improve superscalar execution, remove duplicate checks for problem-state from the asynchronous interrupt handlers, and move the check for problem-state from the synchronous exit path to the program check path as it is only needed for program checks inside the kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:27:48 +03:00
j .Lsysc_tif
s390: add separate program check exit path System call and program check handler both use the system call exit path when returning to previous context. However the program check handler jumps right to the end of the system call exit path if the previous context is kernel context. This lead to the quite odd double disabling of interrupts in the system call exit path introduced with commit ce9dfafe29be ("s390: fix system call exit path"). To avoid that have a separate program check handler exit path if the previous context is kernel context. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-11 20:46:26 +03:00
.Lpgm_restore:
DISABLE_INTS
TSTMSK __LC_CPU_FLAGS, _CIF_FPU
jz .Lpgm_skip_fpu
brasl %r14,load_fpu_regs
.Lpgm_skip_fpu:
mvc __LC_RETURN_PSW(16),__PT_PSW(%r11)
stpt __LC_EXIT_TIMER
lmg %r0,%r15,__PT_R0(%r11)
b __LC_RETURN_LPSWE
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
# PER event in supervisor state, must be kprobes
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lpgm_kprobe:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
s390: always clear kernel stack backchain before calling functions Clear the kernel stack backchain before potentially calling the lockdep trace_hardirqs_off/on functions. Without this walking the kernel backchain, e.g. during a panic, might stop too early. Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 19:56:57 +03:00
RESTORE_SM_CLEAR_PER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
brasl %r14,do_per_trap
s390/kernel: squeeze a few more cycles out of the system call handler Reorder the instructions of UPDATE_VTIME to improve superscalar execution, remove duplicate checks for problem-state from the asynchronous interrupt handlers, and move the check for problem-state from the synchronous exit path to the program check path as it is only needed for program checks inside the kernel. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:27:48 +03:00
j .Lpgm_return
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] add kprobes support. Signed-off-by: Michael Grundy <grundym@us.ibm.com> Signed-off-by: David Wilder <dwilder@us.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-20 17:58:39 +04:00
#
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
# single stepped system call
[S390] add kprobes support. Signed-off-by: Michael Grundy <grundym@us.ibm.com> Signed-off-by: David Wilder <dwilder@us.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-20 17:58:39 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lpgm_svcper:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __LC_RETURN_PSW(8),__LC_SVC_NEW_PSW
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
larl %r14,.Lsysc_per
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stg %r14,__LC_RETURN_PSW+8
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
lghi %r14,_PIF_SYSCALL | _PIF_PER_TRAP
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
lpswe __LC_RETURN_PSW # branch to .Lsysc_per
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(pgm_check_handler)
[S390] add kprobes support. Signed-off-by: Michael Grundy <grundym@us.ibm.com> Signed-off-by: David Wilder <dwilder@us.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-20 17:58:39 +04:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* IO interrupt handler routine
*/
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(io_int_handler)
s390/time: always use stckf instead of stck if available The store clock fast instruction saves a couple of instructions compared to the store clock instruction. Always use stckf instead of stck if it is available. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Reviewed-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-09 18:27:39 +04:00
STCK __LC_INT_CLOCK
[PATCH] improve idle cputime accounting Distinguish the cputime of the idle process where idle is actually using cpu cycles from the cputime where idle is sleeping on an enabled wait psw. The former is accounted as system time, the later as idle time. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-12-31 17:11:41 +03:00
stpt __LC_ASYNC_ENTER_TIMER
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPOFF
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r8,%r15,__LC_SAVE_AREA_ASYNC
s390: move thread_info into task_struct This is the s390 variant of commit 15f4eae70d36 ("x86: Move thread_info into task_struct"). Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 13:08:26 +03:00
lg %r12,__LC_CURRENT
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r8,%r9,__LC_IO_OLD_PSW
s390/idle: fix accounting with machine checks When a machine check interrupt is triggered during idle, the code is using the async timer/clock for idle time calculation. It should use the machine check enter timer/clock which is passed to the macro. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-06 12:47:47 +03:00
SWITCH_ASYNC __LC_SAVE_AREA_ASYNC,__LC_ASYNC_ENTER_TIMER,__LC_INT_CLOCK
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r0,%r7,__PT_R0(%r11)
s390: scrub registers on kernel entry and KVM exit Clear all user space registers on entry to the kernel and all KVM guest registers on KVM guest exit if the register does not contain either a parameter or a result value. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 15:27:30 +03:00
# clear user controlled registers to prevent speculative use
xgr %r0,%r0
xgr %r1,%r1
xgr %r2,%r2
xgr %r3,%r3
xgr %r4,%r4
xgr %r5,%r5
xgr %r6,%r6
xgr %r7,%r7
xgr %r10,%r10
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
stmg %r8,%r9,__PT_PSW(%r11)
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
tm __PT_PSW+1(%r11),0x01 # coming from user space?
jno .Lio_skip_asce
lctlg %c1,%c1,__LC_KERNEL_ASCE
.Lio_skip_asce:
s390/irq: store interrupt information in pt_regs Copy the interrupt parameters from the lowcore to the pt_regs structure in entry[64].S and reduce the arguments of the low level interrupt handler to the pt_regs pointer only. In addition move the test-pending-interrupt loop from do_IRQ to entry[64].S to make sure that interrupt information is always delivered via pt_regs. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-17 16:54:02 +04:00
mvc __PT_INT_CODE(12,%r11),__LC_SUBCHANNEL_ID
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
s390: make calls to TRACE_IRQS_OFF/TRACE_IRQS_ON balanced In case of udelay CIF_IGNORE_IRQ is set. This leads to an unbalanced call of TRACE_IRQS_OFF and TRACE_IRQS_ON. That is: from lockdep's point of view TRACE_IRQS_ON is called one time too often. This doesn't fix any real bug, just makes the calls balanced. Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 20:01:22 +03:00
TRACE_IRQS_OFF
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_loop:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: convert interrupt handling to use generic hardirq With the introduction of PCI it became apparent that s390 should convert to generic hardirqs as too many drivers do not have the correct dependency for GENERIC_HARDIRQS. On the architecture level s390 does not have irq lines. It has external interrupts, I/O interrupts and adapter interrupts. This patch hard-codes all external interrupts as irq #1, all I/O interrupts as irq #2 and all adapter interrupts as irq #3. The additional information from the lowcore associated with the interrupt is stored in the pt_regs of the interrupt frame, where the interrupt handler can pick it up. For PCI/MSI interrupts the adapter interrupt handler scans the relevant bit fields and calls generic_handle_irq with the virtual irq number for the MSI interrupt. Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-27 11:01:09 +04:00
lghi %r3,IO_INTERRUPT
tm __PT_INT_CODE+8(%r11),0x80 # adapter interrupt ?
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jz .Lio_call
s390: convert interrupt handling to use generic hardirq With the introduction of PCI it became apparent that s390 should convert to generic hardirqs as too many drivers do not have the correct dependency for GENERIC_HARDIRQS. On the architecture level s390 does not have irq lines. It has external interrupts, I/O interrupts and adapter interrupts. This patch hard-codes all external interrupts as irq #1, all I/O interrupts as irq #2 and all adapter interrupts as irq #3. The additional information from the lowcore associated with the interrupt is stored in the pt_regs of the interrupt frame, where the interrupt handler can pick it up. For PCI/MSI interrupts the adapter interrupt handler scans the relevant bit fields and calls generic_handle_irq with the virtual irq number for the MSI interrupt. Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-27 11:01:09 +04:00
lghi %r3,THIN_INTERRUPT
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_call:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
brasl %r14,do_IRQ
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_LPAR
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jz .Lio_return
s390/irq: store interrupt information in pt_regs Copy the interrupt parameters from the lowcore to the pt_regs structure in entry[64].S and reduce the arguments of the low level interrupt handler to the pt_regs pointer only. In addition move the test-pending-interrupt loop from do_IRQ to entry[64].S to make sure that interrupt information is always delivered via pt_regs. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-17 16:54:02 +04:00
tpi 0
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jz .Lio_return
s390/irq: store interrupt information in pt_regs Copy the interrupt parameters from the lowcore to the pt_regs structure in entry[64].S and reduce the arguments of the low level interrupt handler to the pt_regs pointer only. In addition move the test-pending-interrupt loop from do_IRQ to entry[64].S to make sure that interrupt information is always delivered via pt_regs. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-17 16:54:02 +04:00
mvc __PT_INT_CODE(12,%r11),__LC_SUBCHANNEL_ID
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_loop
.Lio_return:
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
LOCKDEP_SYS_EXIT
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_WORK
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lio_work # there is work to do (signals etc.)
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_CPU_FLAGS,_CIF_WORK
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lio_work
.Lio_restore:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
TRACE_IRQS_ON
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __LC_RETURN_PSW(16),__PT_PSW(%r11)
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
tm __PT_PSW+1(%r11),0x01 # returning to user ?
jno .Lio_exit_kernel
s390/mm: add debug user asce support Verify on exit to user space that always - the primary ASCE (cr1) is set to kernel ASCE - the secondary ASCE (cr7) is set to user ASCE If this is not the case: panic since something went terribly wrong. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:41 +03:00
DEBUG_USER_ASCE
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__LC_USER_ASCE
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stpt __LC_EXIT_TIMER
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
.Lio_exit_kernel:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
lmg %r0,%r15,__PT_R0(%r11)
b __LC_RETURN_LPSWE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_done:
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] s390: Optimize user and work TIF check On return from syscall or interrupt, we have to check if we return to userspace (likely) and if there is work todo (less likely) to decide if we handle the work. We can optimize this check: we first check for the less likely work case and then check for userspace. This patch is also a preparation for an additional patch, that fixes a bug in KVM dealing with cpu bound guests. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-05-07 11:22:52 +04:00
#
[S390] cleanup sysc_work and io_work code Cleanup the #ifdef mess at io_work in entry[64].S and streamline the TIF work code of the system call and io exit path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:01 +04:00
# There is work todo, find out in which context we have been interrupted:
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
# 1) if we return to user space we can do all _TIF_WORK work
[S390] cleanup sysc_work and io_work code Cleanup the #ifdef mess at io_work in entry[64].S and streamline the TIF work code of the system call and io exit path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:01 +04:00
# 2) if we return to kernel code and kvm is enabled check if we need to
# modify the psw to leave SIE
# 3) if we return to kernel code and preemptive scheduling is enabled check
# the preemption counter and if it is zero call preempt_schedule_irq
# Before any work can be done, a switch to the kernel stack is required.
[S390] s390: Optimize user and work TIF check On return from syscall or interrupt, we have to check if we return to userspace (likely) and if there is work todo (less likely) to decide if we handle the work. We can optimize this check: we first check for the less likely work case and then check for userspace. This patch is also a preparation for an additional patch, that fixes a bug in KVM dealing with cpu bound guests. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-05-07 11:22:52 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_work:
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
tm __PT_PSW+1(%r11),0x01 # returning to user ?
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lio_work_user # yes -> do resched & signal
sched/rt, s390: Use CONFIG_PREEMPTION CONFIG_PREEMPTION is selected by CONFIG_PREEMPT and by CONFIG_PREEMPT_RT. Both PREEMPT and PREEMPT_RT require the same functionality which today depends on CONFIG_PREEMPT. Switch the preemption and entry code over to use CONFIG_PREEMPTION. Add PREEMPT_RT output to die(). [bigeasy: +Kconfig, dumpstack.c] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Christian Borntraeger <borntraeger@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: linux-s390@vger.kernel.org Link: https://lore.kernel.org/r/20191015191821.11479-18-bigeasy@linutronix.de Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-10-15 22:18:04 +03:00
#ifdef CONFIG_PREEMPTION
[S390] s390: Optimize user and work TIF check On return from syscall or interrupt, we have to check if we return to userspace (likely) and if there is work todo (less likely) to decide if we handle the work. We can optimize this check: we first check for the less likely work case and then check for userspace. This patch is also a preparation for an additional patch, that fixes a bug in KVM dealing with cpu bound guests. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-05-07 11:22:52 +04:00
# check for preemptive scheduling
s390/preempt: move preempt_count to the lowcore Convert s390 to use a field in the struct lowcore for the CPU preemption count. It is a bit cheaper to access a lowcore field compared to a thread_info variable and it removes the depencency on a task related structure. bloat-o-meter on the vmlinux image for the default configuration (CONFIG_PREEMPT_NONE=y) reports a small reduction in text size: add/remove: 0/0 grow/shrink: 18/578 up/down: 228/-5448 (-5220) A larger improvement is achieved with the default configuration but with CONFIG_PREEMPT=y and CONFIG_DEBUG_PREEMPT=n: add/remove: 2/6 grow/shrink: 59/4477 up/down: 1618/-228762 (-227144) Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-10-25 13:21:44 +03:00
icm %r0,15,__LC_PREEMPT_COUNT
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jnz .Lio_restore # preemption is disabled
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jno .Lio_restore
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
# switch to kernel stack
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lg %r1,__PT_R15(%r11)
aghi %r1,-(STACK_FRAME_OVERHEAD + __PT_SIZE)
mvc STACK_FRAME_OVERHEAD(__PT_SIZE,%r1),0(%r11)
xc __SF_BACKCHAIN(8,%r1),__SF_BACKCHAIN(%r1)
la %r11,STACK_FRAME_OVERHEAD(%r1)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
lgr %r15,%r1
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
brasl %r14,preempt_schedule_irq
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_return
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
#else
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_restore
[S390] improve irq tracing code in entry[64].S The system call path in entry[64].S is run with interrupts enabled. Remove the irq tracing check from the system call exit code. If a program check interrupted a context enabled for interrupts do a call to trace_irq_off_caller in the program check handler before branching to the system call exit code. Restructure the system call and io interrupt return code to avoid avoid the lpsw[e] to disable machine checks. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:02 +04:00
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] cleanup sysc_work and io_work code Cleanup the #ifdef mess at io_work in entry[64].S and streamline the TIF work code of the system call and io exit path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:01 +04:00
#
# Need to do work before returning to userspace, switch to kernel stack
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_work_user:
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
lg %r1,__LC_KERNEL_STACK
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc STACK_FRAME_OVERHEAD(__PT_SIZE,%r1),0(%r11)
xc __SF_BACKCHAIN(8,%r1),__SF_BACKCHAIN(%r1)
la %r11,STACK_FRAME_OVERHEAD(%r1)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
lgr %r15,%r1
[S390] cleanup sysc_work and io_work code Cleanup the #ifdef mess at io_work in entry[64].S and streamline the TIF work code of the system call and io exit path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:01 +04:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
# One of the work bits is on. Find out which one.
#
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lio_reschedule
livepatch/s390: add TIF_PATCH_PENDING thread flag Update a task's patch state when returning from a system call or user space interrupt, or after handling a signal. This greatly increases the chances of a patch operation succeeding. If a task is I/O bound, it can be patched when returning from a system call. If a task is CPU bound, it can be patched when returning from an interrupt. If a task is sleeping on a to-be-patched function, the user can send SIGSTOP and SIGCONT to force it to switch. Since there are two ways the syscall can be restarted on return from a signal handling process, it is important to clear the flag before do_signal() is called. Otherwise we could miss the migration if we used SIGSTOP/SIGCONT procedure or fake signal to migrate patching blocking tasks. If we place our hook to sysc_work label in entry before TIF_SIGPENDING is evaluated we kill two birds with one stone. The task is correctly migrated in all return paths from a syscall. Signed-off-by: Miroslav Benes <mbenes@suse.cz> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2017-02-14 04:42:34 +03:00
#ifdef CONFIG_LIVEPATCH
TSTMSK __TI_flags(%r12),_TIF_PATCH_PENDING
jo .Lio_patch_pending
#endif
s390: add support for TIF_NOTIFY_SIGNAL Wire up TIF_NOTIFY_SIGNAL handling for s390. Cc: linux-s390@vger.kernel.org Acked-by: Heiko Carstens <hca@linux.ibm.com> Acked-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-10-10 00:34:12 +03:00
TSTMSK __TI_flags(%r12),(_TIF_SIGPENDING|_TIF_NOTIFY_SIGNAL)
jnz .Lio_sigpending
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lio_notify_resume
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
TSTMSK __TI_flags(%r12),_TIF_GUARDED_STORAGE
jo .Lio_guarded_storage
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
jo .Lio_vxrs
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_return
[PATCH] s390: improved machine check handling Improved machine check handling. Kernel is now able to receive machine checks while in kernel mode (system call, interrupt and program check handling). Also register validation is now performed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-26 01:55:30 +04:00
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
#
# CIF_FPU is set, restore floating-point controls and floating-point registers.
#
.Lio_vxrs:
larl %r14,.Lio_return
jg load_fpu_regs
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
#
# _TIF_GUARDED_STORAGE is set, call guarded_storage_load
#
.Lio_guarded_storage:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
ENABLE_INTS_TRACE
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
lgr %r2,%r11 # pass pointer to pt_regs
brasl %r14,gs_load_bc_cb
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
DISABLE_INTS_TRACE
s390: add a system call for guarded storage This adds a new system call to enable the use of guarded storage for user space processes. The system call takes two arguments, a command and pointer to a guarded storage control block: s390_guarded_storage(int command, struct gs_cb *gs_cb); The second argument is relevant only for the GS_SET_BC_CB command. The commands in detail: 0 - GS_ENABLE Enable the guarded storage facility for the current task. The initial content of the guarded storage control block will be all zeros. After the enablement the user space code can use load-guarded-storage-controls instruction (LGSC) to load an arbitrary control block. While a task is enabled the kernel will save and restore the current content of the guarded storage registers on context switch. 1 - GS_DISABLE Disables the use of the guarded storage facility for the current task. The kernel will cease to save and restore the content of the guarded storage registers, the task specific content of these registers is lost. 2 - GS_SET_BC_CB Set a broadcast guarded storage control block. This is called per thread and stores a specific guarded storage control block in the task struct of the current task. This control block will be used for the broadcast event GS_BROADCAST. 3 - GS_CLEAR_BC_CB Clears the broadcast guarded storage control block. The guarded- storage control block is removed from the task struct that was established by GS_SET_BC_CB. 4 - GS_BROADCAST Sends a broadcast to all thread siblings of the current task. Every sibling that has established a broadcast guarded storage control block will load this control block and will be enabled for guarded storage. The broadcast guarded storage control block is used up, a second broadcast without a refresh of the stored control block with GS_SET_BC_CB will not have any effect. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-01-26 16:10:34 +03:00
j .Lio_return
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
# _TIF_NEED_RESCHED is set, call schedule
[S390] Whitespace cleanup. Huge s390 assembly files whitespace cleanup. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-28 18:56:37 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_reschedule:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
ENABLE_INTS_TRACE
[S390] Whitespace cleanup. Huge s390 assembly files whitespace cleanup. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-09-28 18:56:37 +04:00
brasl %r14,schedule # call scheduler
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
DISABLE_INTS_TRACE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_return
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
livepatch/s390: add TIF_PATCH_PENDING thread flag Update a task's patch state when returning from a system call or user space interrupt, or after handling a signal. This greatly increases the chances of a patch operation succeeding. If a task is I/O bound, it can be patched when returning from a system call. If a task is CPU bound, it can be patched when returning from an interrupt. If a task is sleeping on a to-be-patched function, the user can send SIGSTOP and SIGCONT to force it to switch. Since there are two ways the syscall can be restarted on return from a signal handling process, it is important to clear the flag before do_signal() is called. Otherwise we could miss the migration if we used SIGSTOP/SIGCONT procedure or fake signal to migrate patching blocking tasks. If we place our hook to sysc_work label in entry before TIF_SIGPENDING is evaluated we kill two birds with one stone. The task is correctly migrated in all return paths from a syscall. Signed-off-by: Miroslav Benes <mbenes@suse.cz> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2017-02-14 04:42:34 +03:00
#
# _TIF_PATCH_PENDING is set, call klp_update_patch_state
#
#ifdef CONFIG_LIVEPATCH
.Lio_patch_pending:
lg %r2,__LC_CURRENT # pass pointer to task struct
larl %r14,.Lio_return
jg klp_update_patch_state
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
signals: s390: renumber TIF_RESTORE_SIGMASK TIF_RESTORE_SIGMASK no longer needs to be in the _TIF_WORK_* masks. Those low bits are scarce, and are all used up now. Renumber TIF_RESTORE_SIGMASK to free one up. Signed-off-by: Roland McGrath <roland@redhat.com> Cc: Oleg Nesterov <oleg@tv-sign.ru> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: "Luck, Tony" <tony.luck@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-04-30 11:53:08 +04:00
# _TIF_SIGPENDING or is set, call do_signal
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_sigpending:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
ENABLE_INTS_TRACE
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
brasl %r14,do_signal
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
DISABLE_INTS_TRACE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_return
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
#
# _TIF_NOTIFY_RESUME or is set, call do_notify_resume
#
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lio_notify_resume:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
ENABLE_INTS_TRACE
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
brasl %r14,do_notify_resume
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
DISABLE_INTS_TRACE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_return
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(io_int_handler)
[S390] ptrace changes * System call parameter and result access functions * Add tracehook calls * Split syscall_trace into two functions do_syscall_trace_enter and do_syscall_trace_exit Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-10-10 23:33:20 +04:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* External interrupt handler routine
*/
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(ext_int_handler)
s390/time: always use stckf instead of stck if available The store clock fast instruction saves a couple of instructions compared to the store clock instruction. Always use stckf instead of stck if it is available. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Reviewed-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-09 18:27:39 +04:00
STCK __LC_INT_CLOCK
[PATCH] improve idle cputime accounting Distinguish the cputime of the idle process where idle is actually using cpu cycles from the cputime where idle is sleeping on an enabled wait psw. The former is accounted as system time, the later as idle time. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2008-12-31 17:11:41 +03:00
stpt __LC_ASYNC_ENTER_TIMER
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPOFF
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r8,%r15,__LC_SAVE_AREA_ASYNC
s390: move thread_info into task_struct This is the s390 variant of commit 15f4eae70d36 ("x86: Move thread_info into task_struct"). Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 13:08:26 +03:00
lg %r12,__LC_CURRENT
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r8,%r9,__LC_EXT_OLD_PSW
s390/idle: fix accounting with machine checks When a machine check interrupt is triggered during idle, the code is using the async timer/clock for idle time calculation. It should use the machine check enter timer/clock which is passed to the macro. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-06 12:47:47 +03:00
SWITCH_ASYNC __LC_SAVE_AREA_ASYNC,__LC_ASYNC_ENTER_TIMER,__LC_INT_CLOCK
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r0,%r7,__PT_R0(%r11)
s390: scrub registers on kernel entry and KVM exit Clear all user space registers on entry to the kernel and all KVM guest registers on KVM guest exit if the register does not contain either a parameter or a result value. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 15:27:30 +03:00
# clear user controlled registers to prevent speculative use
xgr %r0,%r0
xgr %r1,%r1
xgr %r2,%r2
xgr %r3,%r3
xgr %r4,%r4
xgr %r5,%r5
xgr %r6,%r6
xgr %r7,%r7
xgr %r10,%r10
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
stmg %r8,%r9,__PT_PSW(%r11)
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
tm __PT_PSW+1(%r11),0x01 # coming from user space?
jno .Lext_skip_asce
lctlg %c1,%c1,__LC_KERNEL_ASCE
.Lext_skip_asce:
s390/irq: store interrupt information in pt_regs Copy the interrupt parameters from the lowcore to the pt_regs structure in entry[64].S and reduce the arguments of the low level interrupt handler to the pt_regs pointer only. In addition move the test-pending-interrupt loop from do_IRQ to entry[64].S to make sure that interrupt information is always delivered via pt_regs. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-17 16:54:02 +04:00
lghi %r1,__LC_EXT_PARAMS2
mvc __PT_INT_CODE(4,%r11),__LC_EXT_CPU_ADDR
mvc __PT_INT_PARM(4,%r11),__LC_EXT_PARAMS
mvc __PT_INT_PARM_LONG(8,%r11),0(%r1)
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
s390: initialize backchain for ext_int_handler() To allow correct stack backtraces the backchain for the external interrupt handler is now initialized with zero like it is already done for example by io_int_handler(). Signed-off-by: Michael Holzheu <holzheu@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-09 18:27:35 +04:00
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
s390: make calls to TRACE_IRQS_OFF/TRACE_IRQS_ON balanced In case of udelay CIF_IGNORE_IRQ is set. This leads to an unbalanced call of TRACE_IRQS_OFF and TRACE_IRQS_ON. That is: from lockdep's point of view TRACE_IRQS_ON is called one time too often. This doesn't fix any real bug, just makes the calls balanced. Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-04 20:01:22 +03:00
TRACE_IRQS_OFF
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lgr %r2,%r11 # pass pointer to pt_regs
s390: convert interrupt handling to use generic hardirq With the introduction of PCI it became apparent that s390 should convert to generic hardirqs as too many drivers do not have the correct dependency for GENERIC_HARDIRQS. On the architecture level s390 does not have irq lines. It has external interrupts, I/O interrupts and adapter interrupts. This patch hard-codes all external interrupts as irq #1, all I/O interrupts as irq #2 and all adapter interrupts as irq #3. The additional information from the lowcore associated with the interrupt is stored in the pt_regs of the interrupt frame, where the interrupt handler can pick it up. For PCI/MSI interrupts the adapter interrupt handler scans the relevant bit fields and calls generic_handle_irq with the virtual irq number for the MSI interrupt. Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-06-27 11:01:09 +04:00
lghi %r3,EXT_INTERRUPT
brasl %r14,do_IRQ
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lio_return
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(ext_int_handler)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] rework idle code Whenever the cpu loads an enabled wait PSW it will appear as idle to the underlying host system. The code in default_idle calls vtime_stop_cpu which does the necessary voodoo to get the cpu time accounting right. The udelay code just loads an enabled wait PSW. To correct this rework the vtime_stop_cpu/vtime_start_cpu logic and move the difficult parts to entry[64].S, vtime_stop_cpu can now be called from anywhere and vtime_start_cpu is gone. The correction of the cpu time during wakeup from an enabled wait PSW is done with a critical section in entry[64].S. As vtime_start_cpu is gone, s390_idle_check can be removed as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:27 +04:00
/*
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
* Load idle PSW.
[S390] rework idle code Whenever the cpu loads an enabled wait PSW it will appear as idle to the underlying host system. The code in default_idle calls vtime_stop_cpu which does the necessary voodoo to get the cpu time accounting right. The udelay code just loads an enabled wait PSW. To correct this rework the vtime_stop_cpu/vtime_start_cpu logic and move the difficult parts to entry[64].S, vtime_stop_cpu can now be called from anywhere and vtime_start_cpu is gone. The correction of the cpu time during wakeup from an enabled wait PSW is done with a critical section in entry[64].S. As vtime_start_cpu is gone, s390_idle_check can be removed as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:27 +04:00
*/
ENTRY(psw_idle)
s390/vtimer: rework virtual timer interface The current virtual timer interface is inherently per-cpu and hard to use. The sole user of the interface is appldata which uses it to execute a function after a specific amount of cputime has been used over all cpus. Rework the virtual timer interface to hook into the cputime accounting. This makes the interface independent from the CPU timer interrupts, and makes the virtual timers global as opposed to per-cpu. Overall the code is greatly simplified. The downside is that the accuracy is not as good as the original implementation, but it is still good enough for appldata. Reviewed-by: Jan Glauber <jang@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-07-20 13:15:08 +04:00
stg %r3,__SF_EMPTY(%r15)
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
larl %r1,.Lpsw_idle_exit
[S390] rework idle code Whenever the cpu loads an enabled wait PSW it will appear as idle to the underlying host system. The code in default_idle calls vtime_stop_cpu which does the necessary voodoo to get the cpu time accounting right. The udelay code just loads an enabled wait PSW. To correct this rework the vtime_stop_cpu/vtime_start_cpu logic and move the difficult parts to entry[64].S, vtime_stop_cpu can now be called from anywhere and vtime_start_cpu is gone. The correction of the cpu time during wakeup from an enabled wait PSW is done with a critical section in entry[64].S. As vtime_start_cpu is gone, s390_idle_check can be removed as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:27 +04:00
stg %r1,__SF_EMPTY+8(%r15)
s390/vtime: correct scaled cputime of partially idle CPUs The calculation for the SMT scaling factor for a hardware thread which has been partially idle needs to disregard the cycles spent by the other threads of the core while the thread is idle. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-09-18 17:41:36 +03:00
larl %r1,smp_cpu_mtid
llgf %r1,0(%r1)
ltgr %r1,%r1
jz .Lpsw_idle_stcctm
.insn rsy,0xeb0000000017,%r1,5,__SF_EMPTY+16(%r15)
.Lpsw_idle_stcctm:
s390/spinlock: do not yield to a CPU in udelay/mdelay It does not make sense to try to relinquish the time slice with diag 0x9c to a CPU in a state that does not allow to schedule the CPU. The scenario where this can happen is a CPU waiting in udelay/mdelay while holding a spin-lock. Add a CIF bit to tag a CPU in enabled wait and use it to detect that the yield of a CPU will not be successful and skip the diagnose call. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-11-19 13:09:45 +03:00
oi __LC_CPU_FLAGS+7,_CIF_ENABLED_WAIT
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPON
s390/vtimer: rework virtual timer interface The current virtual timer interface is inherently per-cpu and hard to use. The sole user of the interface is appldata which uses it to execute a function after a specific amount of cputime has been used over all cpus. Rework the virtual timer interface to hook into the cputime accounting. This makes the interface independent from the CPU timer interrupts, and makes the virtual timers global as opposed to per-cpu. Overall the code is greatly simplified. The downside is that the accuracy is not as good as the original implementation, but it is still good enough for appldata. Reviewed-by: Jan Glauber <jang@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-07-20 13:15:08 +04:00
STCK __CLOCK_IDLE_ENTER(%r2)
stpt __TIMER_IDLE_ENTER(%r2)
[S390] rework idle code Whenever the cpu loads an enabled wait PSW it will appear as idle to the underlying host system. The code in default_idle calls vtime_stop_cpu which does the necessary voodoo to get the cpu time accounting right. The udelay code just loads an enabled wait PSW. To correct this rework the vtime_stop_cpu/vtime_start_cpu logic and move the difficult parts to entry[64].S, vtime_stop_cpu can now be called from anywhere and vtime_start_cpu is gone. The correction of the cpu time during wakeup from an enabled wait PSW is done with a critical section in entry[64].S. As vtime_start_cpu is gone, s390_idle_check can be removed as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:27 +04:00
lpswe __SF_EMPTY(%r15)
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
.Lpsw_idle_exit:
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BR_EX %r14
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(psw_idle)
[S390] rework idle code Whenever the cpu loads an enabled wait PSW it will appear as idle to the underlying host system. The code in default_idle calls vtime_stop_cpu which does the necessary voodoo to get the cpu time accounting right. The udelay code just loads an enabled wait PSW. To correct this rework the vtime_stop_cpu/vtime_start_cpu logic and move the difficult parts to entry[64].S, vtime_stop_cpu can now be called from anywhere and vtime_start_cpu is gone. The correction of the cpu time during wakeup from an enabled wait PSW is done with a critical section in entry[64].S. As vtime_start_cpu is gone, s390_idle_check can be removed as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:27 +04:00
s390/fpu: always enable the vector facility if it is available If the kernel detects that the s390 hardware supports the vector facility, it is enabled by default at an early stage. To force it off, use the novx kernel parameter. Note that there is a small time window, where the vector facility is enabled before it is forced to be off. With enabling the vector facility by default, the FPU save and restore functions can be improved. They do not longer require to manage expensive control register updates to enable or disable the vector enablement control for particular processes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-09-29 11:04:41 +03:00
/*
* Store floating-point controls and floating-point or vector register
* depending whether the vector facility is available. A critical section
* cleanup assures that the registers are stored even if interrupted for
* some other work. The CIF_FPU flag is set to trigger a lazy restore
* of the register contents at return from io or a system call.
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
*/
ENTRY(save_fpu_regs)
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
stnsm __SF_EMPTY(%r15),0xfc
s390/kernel: remove save_fpu_regs() parameter and use __LC_CURRENT instead All calls to save_fpu_regs() specify the fpu structure of the current task pointer as parameter. The task pointer of the current task can also be retrieved from the CPU lowcore directly. Remove the parameter definition, load the __LC_CURRENT task pointer from the CPU lowcore, and rebase the FPU structure onto the task structure. Apply the same approach for the load_fpu_regs() function. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-29 17:43:06 +03:00
lg %r2,__LC_CURRENT
aghi %r2,__TASK_thread
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
s390: introduce execute-trampolines for branches Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and -mfunction_return= compiler options to create a kernel fortified against the specte v2 attack. With CONFIG_EXPOLINE=y all indirect branches will be issued with an execute type instruction. For z10 or newer the EXRL instruction will be used, for older machines the EX instruction. The typical indirect call basr %r14,%r1 is replaced with a PC relative call to a new thunk brasl %r14,__s390x_indirect_jump_r1 The thunk contains the EXRL/EX instruction to the indirect branch __s390x_indirect_jump_r1: exrl 0,0f j . 0: br %r1 The detour via the execute type instruction has a performance impact. To get rid of the detour the new kernel parameter "nospectre_v2" and "spectre_v2=[on,off,auto]" can be used. If the parameter is specified the kernel and module code will be patched at runtime. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-26 14:46:47 +03:00
jo .Lsave_fpu_regs_exit
s390/kernel: remove save_fpu_regs() parameter and use __LC_CURRENT instead All calls to save_fpu_regs() specify the fpu structure of the current task pointer as parameter. The task pointer of the current task can also be retrieved from the CPU lowcore directly. Remove the parameter definition, load the __LC_CURRENT task pointer from the CPU lowcore, and rebase the FPU structure onto the task structure. Apply the same approach for the load_fpu_regs() function. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-29 17:43:06 +03:00
stfpc __THREAD_FPU_fpc(%r2)
lg %r3,__THREAD_FPU_regs(%r2)
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_VX
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
jz .Lsave_fpu_regs_fp # no -> store FP regs
VSTM %v0,%v15,0,%r3 # vstm 0,15,0(3)
VSTM %v16,%v31,256,%r3 # vstm 16,31,256(3)
j .Lsave_fpu_regs_done # -> set CIF_FPU flag
.Lsave_fpu_regs_fp:
std 0,0(%r3)
std 1,8(%r3)
std 2,16(%r3)
std 3,24(%r3)
std 4,32(%r3)
std 5,40(%r3)
std 6,48(%r3)
std 7,56(%r3)
std 8,64(%r3)
std 9,72(%r3)
std 10,80(%r3)
std 11,88(%r3)
std 12,96(%r3)
std 13,104(%r3)
std 14,112(%r3)
std 15,120(%r3)
.Lsave_fpu_regs_done:
oi __LC_CPU_FLAGS+7,_CIF_FPU
s390: introduce execute-trampolines for branches Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and -mfunction_return= compiler options to create a kernel fortified against the specte v2 attack. With CONFIG_EXPOLINE=y all indirect branches will be issued with an execute type instruction. For z10 or newer the EXRL instruction will be used, for older machines the EX instruction. The typical indirect call basr %r14,%r1 is replaced with a PC relative call to a new thunk brasl %r14,__s390x_indirect_jump_r1 The thunk contains the EXRL/EX instruction to the indirect branch __s390x_indirect_jump_r1: exrl 0,0f j . 0: br %r1 The detour via the execute type instruction has a performance impact. To get rid of the detour the new kernel parameter "nospectre_v2" and "spectre_v2=[on,off,auto]" can be used. If the parameter is specified the kernel and module code will be patched at runtime. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-26 14:46:47 +03:00
.Lsave_fpu_regs_exit:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
ssm __SF_EMPTY(%r15)
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BR_EX %r14
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
.Lsave_fpu_regs_end:
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(save_fpu_regs)
s390: move exports to definitions Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2016-01-12 21:30:03 +03:00
EXPORT_SYMBOL(save_fpu_regs)
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
s390/fpu: always enable the vector facility if it is available If the kernel detects that the s390 hardware supports the vector facility, it is enabled by default at an early stage. To force it off, use the novx kernel parameter. Note that there is a small time window, where the vector facility is enabled before it is forced to be off. With enabling the vector facility by default, the FPU save and restore functions can be improved. They do not longer require to manage expensive control register updates to enable or disable the vector enablement control for particular processes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-09-29 11:04:41 +03:00
/*
* Load floating-point controls and floating-point or vector registers.
* A critical section cleanup assures that the register contents are
* loaded even if interrupted for some other work.
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
*
* There are special calling conventions to fit into sysc and io return work:
* %r15: <kernel stack>
* The function requires:
s390/fpu: always enable the vector facility if it is available If the kernel detects that the s390 hardware supports the vector facility, it is enabled by default at an early stage. To force it off, use the novx kernel parameter. Note that there is a small time window, where the vector facility is enabled before it is forced to be off. With enabling the vector facility by default, the FPU save and restore functions can be improved. They do not longer require to manage expensive control register updates to enable or disable the vector enablement control for particular processes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-09-29 11:04:41 +03:00
* %r4
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
*/
load_fpu_regs:
s390: fix fpu restore in entry.S We need to disable interrupts in load_fpu_regs(). Otherwise an interrupt might come in after the registers are loaded, but before CIF_FPU is cleared in load_fpu_regs(). When the interrupt returns, CIF_FPU will be cleared and the registers will never be restored. The entry.S code usually saves the interrupt state in __SF_EMPTY on the stack when disabling/restoring interrupts. sie64a however saves the pointer to the sie control block in __SF_SIE_CONTROL, which references the same location. This is non-obvious to the reader. To avoid thrashing the sie control block pointer in load_fpu_regs(), move the __SIE_* offsets eight bytes after __SF_EMPTY on the stack. Cc: <stable@vger.kernel.org> # 5.8 Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Reported-by: Pierre Morel <pmorel@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-20 16:17:52 +03:00
stnsm __SF_EMPTY(%r15),0xfc
s390/kernel: remove save_fpu_regs() parameter and use __LC_CURRENT instead All calls to save_fpu_regs() specify the fpu structure of the current task pointer as parameter. The task pointer of the current task can also be retrieved from the CPU lowcore directly. Remove the parameter definition, load the __LC_CURRENT task pointer from the CPU lowcore, and rebase the FPU structure onto the task structure. Apply the same approach for the load_fpu_regs() function. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-29 17:43:06 +03:00
lg %r4,__LC_CURRENT
aghi %r4,__TASK_thread
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
s390: introduce execute-trampolines for branches Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and -mfunction_return= compiler options to create a kernel fortified against the specte v2 attack. With CONFIG_EXPOLINE=y all indirect branches will be issued with an execute type instruction. For z10 or newer the EXRL instruction will be used, for older machines the EX instruction. The typical indirect call basr %r14,%r1 is replaced with a PC relative call to a new thunk brasl %r14,__s390x_indirect_jump_r1 The thunk contains the EXRL/EX instruction to the indirect branch __s390x_indirect_jump_r1: exrl 0,0f j . 0: br %r1 The detour via the execute type instruction has a performance impact. To get rid of the detour the new kernel parameter "nospectre_v2" and "spectre_v2=[on,off,auto]" can be used. If the parameter is specified the kernel and module code will be patched at runtime. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-26 14:46:47 +03:00
jno .Lload_fpu_regs_exit
s390/kernel: remove save_fpu_regs() parameter and use __LC_CURRENT instead All calls to save_fpu_regs() specify the fpu structure of the current task pointer as parameter. The task pointer of the current task can also be retrieved from the CPU lowcore directly. Remove the parameter definition, load the __LC_CURRENT task pointer from the CPU lowcore, and rebase the FPU structure onto the task structure. Apply the same approach for the load_fpu_regs() function. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-29 17:43:06 +03:00
lfpc __THREAD_FPU_fpc(%r4)
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_VX
s390/kernel: remove save_fpu_regs() parameter and use __LC_CURRENT instead All calls to save_fpu_regs() specify the fpu structure of the current task pointer as parameter. The task pointer of the current task can also be retrieved from the CPU lowcore directly. Remove the parameter definition, load the __LC_CURRENT task pointer from the CPU lowcore, and rebase the FPU structure onto the task structure. Apply the same approach for the load_fpu_regs() function. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-29 17:43:06 +03:00
lg %r4,__THREAD_FPU_regs(%r4) # %r4 <- reg save area
s390/fpu: always enable the vector facility if it is available If the kernel detects that the s390 hardware supports the vector facility, it is enabled by default at an early stage. To force it off, use the novx kernel parameter. Note that there is a small time window, where the vector facility is enabled before it is forced to be off. With enabling the vector facility by default, the FPU save and restore functions can be improved. They do not longer require to manage expensive control register updates to enable or disable the vector enablement control for particular processes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-09-29 11:04:41 +03:00
jz .Lload_fpu_regs_fp # -> no VX, load FP regs
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
VLM %v0,%v15,0,%r4
VLM %v16,%v31,256,%r4
j .Lload_fpu_regs_done
.Lload_fpu_regs_fp:
ld 0,0(%r4)
ld 1,8(%r4)
ld 2,16(%r4)
ld 3,24(%r4)
ld 4,32(%r4)
ld 5,40(%r4)
ld 6,48(%r4)
ld 7,56(%r4)
ld 8,64(%r4)
ld 9,72(%r4)
ld 10,80(%r4)
ld 11,88(%r4)
ld 12,96(%r4)
ld 13,104(%r4)
ld 14,112(%r4)
ld 15,120(%r4)
.Lload_fpu_regs_done:
ni __LC_CPU_FLAGS+7,255-_CIF_FPU
s390: introduce execute-trampolines for branches Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and -mfunction_return= compiler options to create a kernel fortified against the specte v2 attack. With CONFIG_EXPOLINE=y all indirect branches will be issued with an execute type instruction. For z10 or newer the EXRL instruction will be used, for older machines the EX instruction. The typical indirect call basr %r14,%r1 is replaced with a PC relative call to a new thunk brasl %r14,__s390x_indirect_jump_r1 The thunk contains the EXRL/EX instruction to the indirect branch __s390x_indirect_jump_r1: exrl 0,0f j . 0: br %r1 The detour via the execute type instruction has a performance impact. To get rid of the detour the new kernel parameter "nospectre_v2" and "spectre_v2=[on,off,auto]" can be used. If the parameter is specified the kernel and module code will be patched at runtime. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-26 14:46:47 +03:00
.Lload_fpu_regs_exit:
s390: fix fpu restore in entry.S We need to disable interrupts in load_fpu_regs(). Otherwise an interrupt might come in after the registers are loaded, but before CIF_FPU is cleared in load_fpu_regs(). When the interrupt returns, CIF_FPU will be cleared and the registers will never be restored. The entry.S code usually saves the interrupt state in __SF_EMPTY on the stack when disabling/restoring interrupts. sie64a however saves the pointer to the sie control block in __SF_SIE_CONTROL, which references the same location. This is non-obvious to the reader. To avoid thrashing the sie control block pointer in load_fpu_regs(), move the __SIE_* offsets eight bytes after __SF_EMPTY on the stack. Cc: <stable@vger.kernel.org> # 5.8 Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Reported-by: Pierre Morel <pmorel@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Acked-by: Christian Borntraeger <borntraeger@de.ibm.com> Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-20 16:17:52 +03:00
ssm __SF_EMPTY(%r15)
s390: move expoline assembler macros to a header To be able to use the expoline branches in different assembler files move the associated macros from entry.S to a new header nospec-insn.h. While we are at it make the macros a bit nicer to use. Cc: stable@vger.kernel.org # 4.16 Fixes: f19fbd5ed6 ("s390: introduce execute-trampolines for branches") Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-04-20 17:49:46 +03:00
BR_EX %r14
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
.Lload_fpu_regs_end:
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(load_fpu_regs)
s390/kernel: lazy restore fpu registers Improve the save and restore behavior of FPU register contents to use the vector extension within the kernel. The kernel does not use floating-point or vector registers and, therefore, saving and restoring the FPU register contents are performed for handling signals or switching processes only. To prepare for using vector instructions and vector registers within the kernel, enhance the save behavior and implement a lazy restore at return to user space from a system call or interrupt. To implement the lazy restore, the save_fpu_regs() sets a CPU information flag, CIF_FPU, to indicate that the FPU registers must be restored. Saving and setting CIF_FPU is performed in an atomic fashion to be interrupt-safe. When the kernel wants to use the vector extension or wants to change the FPU register state for a task during signal handling, the save_fpu_regs() must be called first. The CIF_FPU flag is also set at process switch. At return to user space, the FPU state is restored. In particular, the FPU state includes the floating-point or vector register contents, as well as, vector-enablement and floating-point control. The FPU state restore and clearing CIF_FPU is also performed in an atomic fashion. For KVM, the restore of the FPU register state is performed when restoring the general-purpose guest registers before the SIE instructions is started. Because the path towards the SIE instruction is interruptible, the CIF_FPU flag must be checked again right before going into SIE. If set, the guest registers must be reloaded again by re-entering the outer SIE loop. This is the same behavior as if the SIE critical section is interrupted. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-10 13:53:42 +03:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* Machine check handler routines
*/
[S390] fix s390 assembler code alignments The alignment is missing for various global symbols in s390 assembly code. With a recent gcc and an instruction like stgrl this can lead to a specification exception if the instruction uses such a mis-aligned address. Specify the alignment explicitely and while add it define __ALIGN for s390 and use the ENTRY define to save some lines of code. Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:19 +04:00
ENTRY(mcck_int_handler)
s390/time: always use stckf instead of stck if available The store clock fast instruction saves a couple of instructions compared to the store clock instruction. Always use stckf instead of stck if it is available. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Reviewed-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-05-09 18:27:39 +04:00
STCK __LC_MCCK_CLOCK
s390: add options to change branch prediction behaviour for the kernel Add the PPA instruction to the system entry and exit path to switch the kernel to a different branch prediction behaviour. The instructions are added via CPU alternatives and can be disabled with the "nospec" or the "nobp=0" kernel parameter. If the default behaviour selected with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be used to enable the changed kernel branch prediction. Acked-by: Cornelia Huck <cohuck@redhat.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:11:45 +03:00
BPOFF
s390/nmi: do register validation as early as possible The validation of the CPU registers in the machine check handler is currently split into two parts. The first part is done at the start of the low level mcck_int_handler function, this includes the CPU timer register and the general purpose registers. The second part is done a bit later in s390_do_machine_check for all the other registers, including the control registers, floating pointer control, vector or floating pointer registers, the access registers, the guarded storage registers, the TOD programmable registers and the clock comparator. This is working fine to far but in theory a future extensions could cause the C code to use registers that are not validated yet. A better approach is to validate all CPU registers in "safe" assembler code before any C function is called. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-12 14:24:48 +03:00
la %r1,4095 # validate r1
spt __LC_CPU_TIMER_SAVE_AREA-4095(%r1) # validate cpu timer
sckc __LC_CLOCK_COMPARATOR # validate comparator
lam %a0,%a15,__LC_AREGS_SAVE_AREA-4095(%r1) # validate acrs
lmg %r0,%r15,__LC_GPREGS_SAVE_AREA-4095(%r1)# validate gprs
s390: move thread_info into task_struct This is the s390 variant of commit 15f4eae70d36 ("x86: Move thread_info into task_struct"). Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-11-08 13:08:26 +03:00
lg %r12,__LC_CURRENT
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r8,%r9,__LC_MCK_OLD_PSW
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_MCCK_CODE,MCCK_CODE_SYSTEM_DAMAGE
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
jo .Lmcck_panic # yes -> rest of mcck code invalid
s390/nmi: do register validation as early as possible The validation of the CPU registers in the machine check handler is currently split into two parts. The first part is done at the start of the low level mcck_int_handler function, this includes the CPU timer register and the general purpose registers. The second part is done a bit later in s390_do_machine_check for all the other registers, including the control registers, floating pointer control, vector or floating pointer registers, the access registers, the guarded storage registers, the TOD programmable registers and the clock comparator. This is working fine to far but in theory a future extensions could cause the C code to use registers that are not validated yet. A better approach is to validate all CPU registers in "safe" assembler code before any C function is called. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-12 14:24:48 +03:00
TSTMSK __LC_MCCK_CODE,MCCK_CODE_CR_VALID
jno .Lmcck_panic # control registers invalid -> panic
la %r14,4095
lctlg %c0,%c15,__LC_CREGS_SAVE_AREA-4095(%r14) # validate ctl regs
ptlb
s390/nmi: avoid using long-displacement facility __LC_MCESAD is currently 4528 /* offsetof(struct lowcore, mcesad) */ that would require long-displacement facility for lg, which we don't have on z900. Fixes: 3037a52f9846 ("s390/nmi: do register validation as early as possible") Signed-off-by: Vasily Gorbik <gor@linux.vnet.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-27 13:44:48 +03:00
lg %r11,__LC_MCESAD-4095(%r14) # extended machine check save area
s390/nmi: do register validation as early as possible The validation of the CPU registers in the machine check handler is currently split into two parts. The first part is done at the start of the low level mcck_int_handler function, this includes the CPU timer register and the general purpose registers. The second part is done a bit later in s390_do_machine_check for all the other registers, including the control registers, floating pointer control, vector or floating pointer registers, the access registers, the guarded storage registers, the TOD programmable registers and the clock comparator. This is working fine to far but in theory a future extensions could cause the C code to use registers that are not validated yet. A better approach is to validate all CPU registers in "safe" assembler code before any C function is called. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-12 14:24:48 +03:00
nill %r11,0xfc00 # MCESA_ORIGIN_MASK
TSTMSK __LC_CREGS_SAVE_AREA+16-4095(%r14),CR2_GUARDED_STORAGE
jno 0f
TSTMSK __LC_MCCK_CODE,MCCK_CODE_GS_VALID
jno 0f
.insn rxy,0xe3000000004d,0,__MCESA_GS_SAVE_AREA(%r11) # LGSC
0: l %r14,__LC_FP_CREG_SAVE_AREA-4095(%r14)
TSTMSK __LC_MCCK_CODE,MCCK_CODE_FC_VALID
jo 0f
sr %r14,%r14
0: sfpc %r14
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_VX
jo 0f
lghi %r14,__LC_FPREGS_SAVE_AREA
ld %f0,0(%r14)
ld %f1,8(%r14)
ld %f2,16(%r14)
ld %f3,24(%r14)
ld %f4,32(%r14)
ld %f5,40(%r14)
ld %f6,48(%r14)
ld %f7,56(%r14)
ld %f8,64(%r14)
ld %f9,72(%r14)
ld %f10,80(%r14)
ld %f11,88(%r14)
ld %f12,96(%r14)
ld %f13,104(%r14)
ld %f14,112(%r14)
ld %f15,120(%r14)
j 1f
0: VLM %v0,%v15,0,%r11
VLM %v16,%v31,256,%r11
1: lghi %r14,__LC_CPU_TIMER_SAVE_AREA
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc __LC_MCCK_ENTER_TIMER(8),0(%r14)
s390/entry: add assembler macro to conveniently tests under mask Various functions in entry.S perform test-under-mask instructions to test for particular bits in memory. Because test-under-mask uses a mask value of one byte, the mask value and the offset into the memory must be calculated manually. This easily introduces errors and is hard to review and read. Introduce the TSTMSK assembler macro to specify a mask constant and let the macro calculate the offset and the byte mask to generate a test-under-mask instruction. The benefit is that existing symbolic constants can now be used for tests. Also the macro checks for zero mask values and mask values that consist of multiple bytes. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-01 18:02:48 +03:00
TSTMSK __LC_MCCK_CODE,MCCK_CODE_CPU_TIMER_VALID
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
jo 3f
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
la %r14,__LC_SYNC_ENTER_TIMER
clc 0(8,%r14),__LC_ASYNC_ENTER_TIMER
jl 0f
la %r14,__LC_ASYNC_ENTER_TIMER
0: clc 0(8,%r14),__LC_EXIT_TIMER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
jl 1f
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
la %r14,__LC_EXIT_TIMER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
1: clc 0(8,%r14),__LC_LAST_UPDATE_TIMER
jl 2f
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
la %r14,__LC_LAST_UPDATE_TIMER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
2: spt 0(%r14)
[S390] idle time accounting vs. machine checks A machine check can interrupt the i/o and external interrupt handler anytime. If the machine check occurs while the interrupt handler is waking up from idle vtime_start_cpu can get executed a second time and the int_clock / async_enter_timer values in the lowcore get clobbered. This can confuse the cpu time accounting. To fix this problem two changes are needed. First the machine check handler has to use its own copies of int_clock and async_enter_timer, named mcck_clock and mcck_enter_timer. Second the nested execution of vtime_start_cpu has to be prevented. This is done in s390_idle_check by checking the wait bit in the program status word. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2010-05-17 12:00:03 +04:00
mvc __LC_MCCK_ENTER_TIMER(8),0(%r14)
s390/nmi: do register validation as early as possible The validation of the CPU registers in the machine check handler is currently split into two parts. The first part is done at the start of the low level mcck_int_handler function, this includes the CPU timer register and the general purpose registers. The second part is done a bit later in s390_do_machine_check for all the other registers, including the control registers, floating pointer control, vector or floating pointer registers, the access registers, the guarded storage registers, the TOD programmable registers and the clock comparator. This is working fine to far but in theory a future extensions could cause the C code to use registers that are not validated yet. A better approach is to validate all CPU registers in "safe" assembler code before any C function is called. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-10-12 14:24:48 +03:00
3: TSTMSK __LC_MCCK_CODE,MCCK_CODE_PSW_MWP_VALID
jno .Lmcck_panic
tmhh %r8,0x0001 # interrupting from user ?
jnz 4f
TSTMSK __LC_MCCK_CODE,MCCK_CODE_PSW_IA_VALID
jno .Lmcck_panic
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
4: ssm __LC_PGM_NEW_PSW # turn dat on, keep irqs off
s390/idle: fix accounting with machine checks When a machine check interrupt is triggered during idle, the code is using the async timer/clock for idle time calculation. It should use the machine check enter timer/clock which is passed to the macro. Fixes: 0b0ed657fe00 ("s390: remove critical section cleanup from entry.S") Cc: <stable@vger.kernel.org> # 5.8 Reviewed-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-12-06 12:47:47 +03:00
SWITCH_ASYNC __LC_GPREGS_SAVE_AREA+64,__LC_MCCK_ENTER_TIMER,__LC_MCCK_CLOCK
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lmcck_skip:
s390: critical section cleanup vs. machine checks The current machine check code uses the registers stored by the machine in the lowcore at __LC_GPREGS_SAVE_AREA as the registers of the interrupted context. The registers 0-7 of a user process can get clobbered if a machine checks interrupts the execution of a critical section in entry[64].S. The reason is that the critical section cleanup code may need to modify the PSW and the registers for the previous context to get to the end of a critical section. If registers 0-7 have to be replaced the relevant copy will be in the registers, which invalidates the copy in the lowcore. The machine check handler needs to explicitly store registers 0-7 to the stack. Cc: stable@vger.kernel.org Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-02-28 19:28:41 +04:00
lghi %r14,__LC_GPREGS_SAVE_AREA+64
stmg %r0,%r7,__PT_R0(%r11)
s390: scrub registers on kernel entry and KVM exit Clear all user space registers on entry to the kernel and all KVM guest registers on KVM guest exit if the register does not contain either a parameter or a result value. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 15:27:30 +03:00
# clear user controlled registers to prevent speculative use
xgr %r0,%r0
xgr %r1,%r1
xgr %r2,%r2
xgr %r3,%r3
xgr %r4,%r4
xgr %r5,%r5
xgr %r6,%r6
xgr %r7,%r7
xgr %r10,%r10
s390: critical section cleanup vs. machine checks The current machine check code uses the registers stored by the machine in the lowcore at __LC_GPREGS_SAVE_AREA as the registers of the interrupted context. The registers 0-7 of a user process can get clobbered if a machine checks interrupts the execution of a critical section in entry[64].S. The reason is that the critical section cleanup code may need to modify the PSW and the registers for the previous context to get to the end of a critical section. If registers 0-7 have to be replaced the relevant copy will be in the registers, which invalidates the copy in the lowcore. The machine check handler needs to explicitly store registers 0-7 to the stack. Cc: stable@vger.kernel.org Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-02-28 19:28:41 +04:00
mvc __PT_R8(64,%r11),0(%r14)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r8,%r9,__PT_PSW(%r11)
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
la %r14,4095
mvc __PT_CR1(8,%r11),__LC_CREGS_SAVE_AREA-4095+8(%r14)
lctlg %c1,%c1,__LC_KERNEL_ASCE
s390: split TIF bits into CIF, PIF and TIF bits The oi and ni instructions used in entry[64].S to set and clear bits in the thread-flags are not guaranteed to be atomic in regard to other CPUs. Split the TIF bits into CPU, pt_regs and thread-info specific bits. Updates on the TIF bits are done with atomic instructions, updates on CPU and pt_regs bits are done with non-atomic instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-04-15 14:55:07 +04:00
xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
lgr %r2,%r11 # pass pointer to pt_regs
[PATCH] s390: improved machine check handling Improved machine check handling. Kernel is now able to receive machine checks while in kernel mode (system call, interrupt and program check handling). Also register validation is now performed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-26 01:55:30 +04:00
brasl %r14,s390_do_machine_check
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
cghi %r2,0
je .Lmcck_return
[PATCH] s390: improved machine check handling Improved machine check handling. Kernel is now able to receive machine checks while in kernel mode (system call, interrupt and program check handling). Also register validation is now performed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-26 01:55:30 +04:00
lg %r1,__LC_KERNEL_STACK # switch to kernel stack
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
mvc STACK_FRAME_OVERHEAD(__PT_SIZE,%r1),0(%r11)
xc __SF_BACKCHAIN(8,%r1),__SF_BACKCHAIN(%r1)
la %r11,STACK_FRAME_OVERHEAD(%r1)
[PATCH] s390: improved machine check handling Improved machine check handling. Kernel is now able to receive machine checks while in kernel mode (system call, interrupt and program check handling). Also register validation is now performed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-26 01:55:30 +04:00
lgr %r15,%r1
[PATCH] lockdep: irqtrace subsystem, s390 support irqtrace support for s390. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Arjan van de Ven <arjan@infradead.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-07-03 11:24:46 +04:00
TRACE_IRQS_OFF
[PATCH] s390: improved machine check handling Improved machine check handling. Kernel is now able to receive machine checks while in kernel mode (system call, interrupt and program check handling). Also register validation is now performed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-26 01:55:30 +04:00
brasl %r14,s390_handle_mcck
[PATCH] lockdep: irqtrace subsystem, s390 support irqtrace support for s390. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Arjan van de Ven <arjan@infradead.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-07-03 11:24:46 +04:00
TRACE_IRQS_ON
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lmcck_return:
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__PT_CR1(%r11)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
lmg %r0,%r10,__PT_R0(%r11)
mvc __LC_RETURN_MCCK_PSW(16),__PT_PSW(%r11) # move return PSW
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
tm __LC_RETURN_MCCK_PSW+1,0x01 # returning to user ?
jno 0f
s390: run user space and KVM guests with modified branch prediction Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary plumbing in entry.S to be able to run user space and KVM guests with limited branch prediction. To switch a user space process to limited branch prediction the s390_isolate_bp() function has to be call, and to run a vCPU of a KVM guest associated with the current task with limited branch prediction call s390_isolate_bp_guest(). Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-01-16 09:36:46 +03:00
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
[S390] virtual cpu accounting vs. machine checks. If a machine checks interrupts the external or the i/o interrupt handler before they have completed the cpu time calculations, the accounting goes wrong. After the cpu returned from the machine check handler to the interrupted interrupt handler, a negative cpu time delta can occur. If the accumulated cpu time in lowcore is small enough this value can get negative as well. The next jiffy interrupt will pick up that negative value, shift it by 12 and add the now huge positive value to the cpu time of the process. To solve this the machine check handler is modified not to change any of the timestamps in the lowcore if the machine check interrupted kernel context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2006-06-29 16:58:05 +04:00
stpt __LC_EXIT_TIMER
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
0: lmg %r11,%r15,__PT_R11(%r11)
s390: prevent leaking kernel address in BEAR When userspace executes a syscall or gets interrupted, BEAR contains a kernel address when returning to userspace. This make it pretty easy to figure out where the kernel is mapped even with KASLR enabled. To fix this, add lpswe to lowcore and always execute it there, so userspace sees only the lowcore address of lpswe. For this we have to extend both critical_cleanup and the SWITCH_ASYNC macro to also check for lpswe addresses in lowcore. Fixes: b2d24b97b2a9 ("s390/kernel: add support for kernel address space layout randomization (KASLR)") Cc: <stable@vger.kernel.org> # v5.2+ Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-01-22 15:38:22 +03:00
b __LC_RETURN_MCCK_LPSWE
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
.Lmcck_panic:
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
lg %r15,__LC_NODAT_STACK
s390: fix machine check panic stack switch For system damage machine checks or machine checks due to invalid PSW fields the system will be stopped. In order to get an oops message out before killing the system the machine check handler branches to .Lmcck_panic, switches to the panic stack and then does the usual machine check handling. The switch to the panic stack is incomplete, the stack pointer in %r15 is replaced, but the pt_regs pointer in %r11 is not. The result is a program check which will kill the system in a slightly different way. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-12-02 15:29:22 +03:00
la %r11,STACK_FRAME_OVERHEAD(%r15)
s390: use local symbol names in entry[64].S To improve the output of the perf tool hide most of the symbols from entry[64].S by using the '.L' prefix. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2014-12-03 19:00:08 +03:00
j .Lmcck_skip
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(mcck_int_handler)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] Add PSW restart shutdown trigger With this patch a new S390 shutdown trigger "restart" is added. If under z/VM "systerm restart" is entered or under the HMC the "PSW restart" button is pressed, the PSW located at 0 (31 bit) or 0x1a0 (64 bit) bit is loaded. Now we execute do_restart() that processes the restart action that is defined under /sys/firmware/shutdown_actions/on_restart. Currently the following actions are possible: reipl (default), stop, vmcmd, dump, and dump_reipl. Signed-off-by: Michael Holzheu <holzheu@linux.vnet.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2011-08-03 18:44:19 +04:00
#
# PSW restart interrupt handler
#
[S390] rework smp code Define struct pcpu and merge some of the NR_CPUS arrays into it, including __cpu_logical_map, current_set and smp_cpu_state. Split smp related functions to those operating on physical cpus and the functions operating on a logical cpu number. Make the functions for physical cpus use a pointer to a struct pcpu. This hides the knowledge about cpu addresses in smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header. The PSW restart mechanism is used to start secondary cpus, calling a function on an online cpu, calling a function on the ipl cpu, and for the nmi signal. Replace the different assembler functions with a single function restart_int_handler. The new entry point calls a function whose pointer is stored in the lowcore of the target cpu and it can wait for the source cpu to stop. This covers all existing use cases. Overall the code is now simpler and there are ~380 lines less code. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:26 +04:00
ENTRY(restart_int_handler)
s390/lpp: use assembler alternatives for the LPP instruction With the new macros for CPU alternatives the MACHINE_FLAG_LPP check around the LPP instruction can be optimized. After this is done the flag can be removed. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-26 16:23:33 +03:00
ALTERNATIVE "", ".insn s,0xb2800000,_LPP_OFFSET", 40
stg %r15,__LC_SAVE_AREA_RESTART
[S390] rework smp code Define struct pcpu and merge some of the NR_CPUS arrays into it, including __cpu_logical_map, current_set and smp_cpu_state. Split smp related functions to those operating on physical cpus and the functions operating on a logical cpu number. Make the functions for physical cpus use a pointer to a struct pcpu. This hides the knowledge about cpu addresses in smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header. The PSW restart mechanism is used to start secondary cpus, calling a function on an online cpu, calling a function on the ipl cpu, and for the nmi signal. Replace the different assembler functions with a single function restart_int_handler. The new entry point calls a function whose pointer is stored in the lowcore of the target cpu and it can wait for the source cpu to stop. This covers all existing use cases. Overall the code is now simpler and there are ~380 lines less code. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:26 +04:00
lg %r15,__LC_RESTART_STACK
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
xc STACK_FRAME_OVERHEAD(__PT_SIZE,%r15),STACK_FRAME_OVERHEAD(%r15)
stmg %r0,%r14,STACK_FRAME_OVERHEAD+__PT_R0(%r15)
mvc STACK_FRAME_OVERHEAD+__PT_R15(8,%r15),__LC_SAVE_AREA_RESTART
mvc STACK_FRAME_OVERHEAD+__PT_PSW(16,%r15),__LC_RST_OLD_PSW
[S390] rework smp code Define struct pcpu and merge some of the NR_CPUS arrays into it, including __cpu_logical_map, current_set and smp_cpu_state. Split smp related functions to those operating on physical cpus and the functions operating on a logical cpu number. Make the functions for physical cpus use a pointer to a struct pcpu. This hides the knowledge about cpu addresses in smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header. The PSW restart mechanism is used to start secondary cpus, calling a function on an online cpu, calling a function on the ipl cpu, and for the nmi signal. Replace the different assembler functions with a single function restart_int_handler. The new entry point calls a function whose pointer is stored in the lowcore of the target cpu and it can wait for the source cpu to stop. This covers all existing use cases. Overall the code is now simpler and there are ~380 lines less code. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:26 +04:00
xc 0(STACK_FRAME_OVERHEAD,%r15),0(%r15)
s390/smp: make absolute lowcore / cpu restart parameter accesses more robust Setting the cpu restart parameters is done in three different fashions: - directly setting the four parameters individually - copying the four parameters with memcpy (using 4 * sizeof(long)) - copying the four parameters using a private structure In addition code in entry*.S relies on a certain order of the restart members of struct _lowcore. Make all of this more robust to future changes by adding a mem_absolute_assign(dest, val) define, which assigns val to dest using absolute addressing mode. Also the load multiple instructions in entry*.S have been split into separate load instruction so the order of the struct _lowcore members doesn't matter anymore. In addition move the prototypes of memcpy_real/absolute from uaccess.h to processor.h. These memcpy* variants are not related to uaccess at all. string.h doesn't seem to match as well, so lets use processor.h. Also replace the eight byte array in struct _lowcore which represents a misaliged u64 with a u64. The compiler will always create code that handles the misaligned u64 correctly. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-06-05 11:59:52 +04:00
lg %r1,__LC_RESTART_FN # load fn, parm & source cpu
lg %r2,__LC_RESTART_DATA
lg %r3,__LC_RESTART_SOURCE
[S390] rework smp code Define struct pcpu and merge some of the NR_CPUS arrays into it, including __cpu_logical_map, current_set and smp_cpu_state. Split smp related functions to those operating on physical cpus and the functions operating on a logical cpu number. Make the functions for physical cpus use a pointer to a struct pcpu. This hides the knowledge about cpu addresses in smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header. The PSW restart mechanism is used to start secondary cpus, calling a function on an online cpu, calling a function on the ipl cpu, and for the nmi signal. Replace the different assembler functions with a single function restart_int_handler. The new entry point calls a function whose pointer is stored in the lowcore of the target cpu and it can wait for the source cpu to stop. This covers all existing use cases. Overall the code is now simpler and there are ~380 lines less code. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:26 +04:00
ltgr %r3,%r3 # test source cpu address
jm 1f # negative -> skip source stop
s390/sigp: use sigp order code defines in assembly code Use sigp order code defines in assembly code as well. With this change all places that use sigp constants should have been converted to use self describing defines instead of directly using constants. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-06-04 17:05:43 +04:00
0: sigp %r4,%r3,SIGP_SENSE # sigp sense to source cpu
[S390] rework smp code Define struct pcpu and merge some of the NR_CPUS arrays into it, including __cpu_logical_map, current_set and smp_cpu_state. Split smp related functions to those operating on physical cpus and the functions operating on a logical cpu number. Make the functions for physical cpus use a pointer to a struct pcpu. This hides the knowledge about cpu addresses in smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header. The PSW restart mechanism is used to start secondary cpus, calling a function on an online cpu, calling a function on the ipl cpu, and for the nmi signal. Replace the different assembler functions with a single function restart_int_handler. The new entry point calls a function whose pointer is stored in the lowcore of the target cpu and it can wait for the source cpu to stop. This covers all existing use cases. Overall the code is now simpler and there are ~380 lines less code. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:26 +04:00
brc 10,0b # wait for status stored
1: basr %r14,%r1 # call function
stap __SF_EMPTY(%r15) # store cpu address
llgh %r3,__SF_EMPTY(%r15)
s390/sigp: use sigp order code defines in assembly code Use sigp order code defines in assembly code as well. With this change all places that use sigp constants should have been converted to use self describing defines instead of directly using constants. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-06-04 17:05:43 +04:00
2: sigp %r4,%r3,SIGP_STOP # sigp stop to current cpu
[S390] rework smp code Define struct pcpu and merge some of the NR_CPUS arrays into it, including __cpu_logical_map, current_set and smp_cpu_state. Split smp related functions to those operating on physical cpus and the functions operating on a logical cpu number. Make the functions for physical cpus use a pointer to a struct pcpu. This hides the knowledge about cpu addresses in smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header. The PSW restart mechanism is used to start secondary cpus, calling a function on an online cpu, calling a function on the ipl cpu, and for the nmi signal. Replace the different assembler functions with a single function restart_int_handler. The new entry point calls a function whose pointer is stored in the lowcore of the target cpu and it can wait for the source cpu to stop. This covers all existing use cases. Overall the code is now simpler and there are ~380 lines less code. Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2012-03-11 19:59:26 +04:00
brc 2,2b
3: j 3b
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(restart_int_handler)
[S390] Add PSW restart shutdown trigger With this patch a new S390 shutdown trigger "restart" is added. If under z/VM "systerm restart" is entered or under the HMC the "PSW restart" button is pressed, the PSW located at 0 (31 bit) or 0x1a0 (64 bit) bit is loaded. Now we execute do_restart() that processes the restart action that is defined under /sys/firmware/shutdown_actions/on_restart. Currently the following actions are possible: reipl (default), stop, vmcmd, dump, and dump_reipl. Signed-off-by: Michael Holzheu <holzheu@linux.vnet.ibm.com> Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
2011-08-03 18:44:19 +04:00
[S390] add kprobes annotations Add kprobes annotations to get the massive 'probe kernel.function("*") {}' stress test working. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-01-05 14:47:25 +03:00
.section .kprobes.text, "ax"
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
#if defined(CONFIG_CHECK_STACK) || defined(CONFIG_VMAP_STACK)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
/*
* The synchronous or the asynchronous stack overflowed. We are dead.
* No need to properly save the registers, we are going to panic anyway.
* Setup a pt_regs so that show_trace can provide a good call trace.
*/
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENTRY(stack_overflow)
s390: add support for virtually mapped kernel stacks With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-09-12 17:37:33 +03:00
lg %r15,__LC_NODAT_STACK # change to panic stack
s390: lowcore stack pointer offsets Store the stack pointers in the lowcore for the kernel stack, the async stack and the panic stack with the offset required for the first user. This avoids an unnecessary add instruction on the system call path. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-04-24 12:20:43 +04:00
la %r11,STACK_FRAME_OVERHEAD(%r15)
[S390] entry[64].S improvements Another round of cleanup for entry[64].S, in particular the program check handler looks more reasonable now. The code size for the 31 bit kernel has been reduced by 616 byte and by 528 byte for the 64 bit version. Even better the code is a bit faster as well. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-12-27 14:27:15 +04:00
stmg %r0,%r7,__PT_R0(%r11)
stmg %r8,%r9,__PT_PSW(%r11)
mvc __PT_R8(64,%r11),0(%r14)
stg %r10,__PT_ORIG_GPR2(%r11) # store last break to orig_gpr2
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
lgr %r2,%r11 # pass pointer to pt_regs
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
jg kernel_stack_overflow
s390: add missing ENDPROC statements to assembler functions The assembler code in arch/s390 misses proper ENDPROC statements to properly end functions in a few places. Add them. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-01-17 12:02:22 +03:00
ENDPROC(stack_overflow)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#endif
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
#if IS_ENABLED(CONFIG_KVM)
.Lcleanup_sie:
s390: remove critical section cleanup from entry.S The current code is rather complex and caused a lot of subtle and hard to debug bugs in the past. Simplify the code by calling the system_call handler with interrupts disabled, save machine state, and re-enable them later. This requires significant changes to the machine check handling code as well. When the machine check interrupt arrived while being in kernel mode the new code will signal pending machine checks with a SIGP external call. When userspace was interrupted, the handler will switch to the kernel stack and directly execute s390_handle_mcck(). Signed-off-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
2020-02-20 14:09:36 +03:00
cghi %r11,__LC_SAVE_AREA_ASYNC #Is this in normal interrupt?
je 1f
larl %r13,.Lsie_entry
slgr %r9,%r13
larl %r13,.Lsie_skip
clgr %r9,%r13
jh 1f
oi __LC_CPU_FLAGS+7, _CIF_MCCK_GUEST
1: BPENTER __SF_SIE_FLAGS(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
s390/kvm: improve stack frame constants in entry.S The code in sie64a uses the stack frame passed to the function to store some temporary data in the empty1 array (see struct stack_frame in asm/processor.h. Replace the __SF_EMPTY+x constants with a properly defined offset: s/__SF_EMPTY/__SF_SIE_CONTROL/, s/__SF_EMPTY+8/__SF_SIE_SAVEAREA/, s/__SF_EMPTY+16/__SF_SIE_REASON/, s/__SF_EMPTY+24/__SF_SIE_FLAGS/. Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-03-20 15:33:43 +03:00
lg %r9,__SF_SIE_CONTROL(%r15) # get control block pointer
s390/cpumf: rework program parameter setting to detect guest samples The program parameter can be used to mark hardware samples with some token. Previously, it was used to mark guest samples only. Improve the program parameter doubleword by combining two parts, the leftmost LPP part and the rightmost PID part. Set the PID part for processes by using the task PID. To distinguish host and guest samples for the kernel (PID part is zero), the guest must always set the program paramater to a non-zero value. Use the leftmost bit in the LPP part of the program parameter to be able to detect guest kernel samples. [brueckner@linux.vnet.ibm.com]: Split __LC_CURRENT and introduced __LC_LPP. Corrected __LC_CURRENT users and adjusted assembler parts. And updated the commit message accordingly. Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Reviewed-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-06 19:06:15 +03:00
ni __SIE_PROG0C+3(%r9),0xfe # no longer in SIE
s390/mm: remove set_fs / rework address space handling Remove set_fs support from s390. With doing this rework address space handling and simplify it. As a result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE ----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel, normal execution | kernel | user | kernel kernel, kvm guest execution | gmap | user | kernel To achieve this the getcpu vdso syscall is removed in order to avoid secondary address mode and a separate vdso address space in for user space. The getcpu vdso syscall will be implemented differently with a subsequent patch. The kernel accesses user space always via secondary address space. This happens in different ways: - with mvcos in home space mode and directly read/write to secondary address space - with mvcs/mvcp in primary space mode and copy from primary space to secondary space or vice versa - with e.g. cs in secondary space mode and access secondary space Switching translation modes happens with sacf before and after instructions which access user space, like before. Lazy handling of control register reloading is removed in the hope to make everything simpler, but at the cost of making kernel entry and exit a bit slower. That is: on kernel entry the primary asce is always changed to contain the kernel asce, and on kernel exit the primary asce is changed again so it contains the user asce. In kernel mode there is only one exception to the primary asce: when kvm guests are executed the primary asce contains the gmap asce (which describes the guest address space). The primary asce is reset to kernel asce whenever kvm guest execution is interrupted, so that this doesn't has to be taken into account for any user space accesses. Reviewed-by: Sven Schnelle <svens@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
2020-11-16 10:06:40 +03:00
lctlg %c1,%c1,__LC_KERNEL_ASCE
s390/kvm: integrate HANDLE_SIE_INTERCEPT into cleanup_critical Currently there are two mechanisms to deal with cleanup work due to interrupts. The HANDLE_SIE_INTERCEPT macro is used to undo the changes required to enter SIE in sie64a. If the SIE instruction causes a program check, or an asynchronous interrupt is received the HANDLE_SIE_INTERCEPT code forwards the program execution to sie_exit. All the other critical sections in entry.S are handled by the code in cleanup_critical that is called by the SWITCH_ASYNC macro. Move the sie64a function to the beginning of the critical section and add the code from HANDLE_SIE_INTERCEPT to cleanup_critical. Add a special case for the sie64a cleanup to the program check handler. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-06-22 18:26:40 +03:00
larl %r9,sie_exit # skip forward to sie_exit
s390: Correct register corruption in critical section cleanup In the critical section cleanup we must not mess with r1. For march=z9 or older, larl + ex (instead of exrl) are used with r1 as a temporary register. This can clobber r1 in several interrupt handlers. Fix this by using r11 as a temp register. r11 is being saved by all callers of cleanup_critical. Fixes: 6dd85fbb87 ("s390: move expoline assembler macros to a header") Cc: stable@vger.kernel.org #v4.16 Reported-by: Oliver Kurz <okurz@suse.com> Reported-by: Petr Tesařík <ptesarik@suse.com> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com> Reviewed-by: Hendrik Brueckner <brueckner@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2018-06-21 15:49:38 +03:00
BR_EX %r14,%r11
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
[S390] move sie code to entry.S The entry to / exit from sie has subtle dependencies to the first level interrupt handler. Move the sie assembler code to entry64.S and replace the SIE_HOOK callback with a test and the new _TIF_SIE bit. In addition this patch fixes several problems in regard to the check for the_TIF_EXIT_SIE bits. The old code checked the TIF bits before executing the interrupt handler and it only modified the instruction address if it pointed directly to the sie instruction. In both cases it could miss a TIF bit that normally would cause an exit from the guest and would reenter the guest context. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2011-07-24 12:48:18 +04:00
#endif
s390: remove 31 bit syscalls Remove the 31 bit syscalls from the syscall table. This is a separate patch just in case I screwed something up so it can be easily reverted. However the conversion was done with a script, so everything should be ok. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-02-13 16:44:29 +03:00
.section .rodata, "a"
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
#define SYSCALL(esame,emu) .quad __s390x_ ## esame
[S390] ftrace: add system call tracer support System call tracer support for s390. Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2009-06-12 12:26:47 +04:00
.globl sys_call_table
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
sys_call_table:
s390/syscalls: use generated syscall_table.h and unistd.h header files Update the uapi/asm/unistd.h to include the generated compat and 64-bit version of the unistd.h and, as well as, the unistd_nr.h header file. Also remove the arch/s390/kernel/syscalls.S file and use the generated system call table, syscall_table.h, instead. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-12-11 16:47:27 +03:00
#include "asm/syscall_table.h"
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#undef SYSCALL
[PATCH] s390: cleanup Kconfig Sanitize some s390 Kconfig options. We have ARCH_S390, ARCH_S390X, ARCH_S390_31, 64BIT, S390_SUPPORT and COMPAT. Replace these 6 options by S390, 64BIT and COMPAT. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-06 11:19:28 +03:00
#ifdef CONFIG_COMPAT
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
s390/kernel: convert SYSCALL and PGM_CHECK handlers to .quad With a relocatable kernel that could reside at any place in memory, the storage size for the SYSCALL and PGM_CHECK handlers needs to be increased from .long to .quad. Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com> Reviewed-by: Philipp Rudo <prudo@linux.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2019-02-03 23:36:13 +03:00
#define SYSCALL(esame,emu) .quad __s390_ ## emu
s390: system call path micro optimization Add a pointer to the system call table to the thread_info structure. The TIF_31BIT bit is set or cleared by SET_PERSONALITY exactly once for the lifetime of a process. With the pointer to the correct system call table in thread_info the system call code in entry64.S path can drop the check for TIF_31BIT which saves a couple of instructions. Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2013-04-24 14:58:39 +04:00
.globl sys_call_table_emu
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
sys_call_table_emu:
s390/syscalls: use generated syscall_table.h and unistd.h header files Update the uapi/asm/unistd.h to include the generated compat and 64-bit version of the unistd.h and, as well as, the unistd_nr.h header file. Also remove the arch/s390/kernel/syscalls.S file and use the generated system call table, syscall_table.h, instead. Signed-off-by: Hendrik Brueckner <brueckner@linux.vnet.ibm.com> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2017-12-11 16:47:27 +03:00
#include "asm/syscall_table.h"
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 02:20:36 +04:00
#undef SYSCALL
#endif