C programs can have problems with return statements that would result in undefined or incorrect program behavior. C compilers typically warn about these problems (in other languages, these would typically be compile-time errors because of the severity of the problem). For Checked C, these problems need to result in errors when they occur in a checked scope or involve bounds or check pointer types. This
change adds tests for these problems.
Reading variables with bounds-safe interfaces works fine. Writing
variables with bounds-interfaces caused a compiler crash. Add a
simple test for writing a variable. We should add more tests
for writes at bounds-safe interfaces eventually.
I placed this test in a new subdirectory runtime_operations. This
will contain tests for checked type operations that are done at
runtime, but don't do dynamic runtime checks.
In this case, there is an implicit cast inserted at bounds-safe interfaces.
We want to make sure those casts operate correctly.
The -fcheckedc-extension flag is now set by default for C files in the Checked C version of clang. Remove the flag from the command lines for the test harness.
Also update a few error messages that changed after fixing bugs in the compiler. The bugs were found after building the clang test suite with -fcheckedc-extension on by default.
This change addresses the suggestion from @mwhicks1 in issue #235 that we relax the bounds checking for null-terminated arrays to allow a write of 0 at the upper bound of a null-terminated array. There will be a matching pull request in the checked-clang repo for compiler change.
- Update the description of bounds checking in the Checked C specification.
- Add tests to nullterm_pointers.c that check this behavior.
- Update the sample code in string-helpers.c to delete some special-case code in the "squeeze" function. This function removes all occurrences of a character from a string. If no characters are removed, it redundantly overwrites the existing null-terminator. Before, this had to be special-cased to avoid an out-of-bounds write. I also updated the sample code to avoid declaring default bounds of count(0) and clarified the comments.
I ran into a subtlety when revising the specification. When we allow a 0 to be written at the upper bound, we have to expand the allowed range of memory to include the memory at the upper bound. It is possible for even this expanded range to be empty. The specification is careful to not allow a write of 0 in this situation., by including a comparison of the pointer address against the lower bound.
We've changed the compiler to use the term "unknown bounds" in error messages
instead of "no bounds" because "no bounds" is ambiguous. It could mean
unbounded or unknown bounds. Update the error messages that are checked
in tests.
The compiler now always warns when it cannot prove a bounds
declaration true or false in a checked scope. This adds warnings to tests
where this is the case. Most of these warnings will disappear when we extend
bounds declaration checking to include facts about equality of
variables/expressions after assignments.
The compiler also understands equivalence of base expressions for
constant-sized regions better. This lets the compiler deduce that
`*(_Dynamic_bounds_cast<array_ptr<int>>(r, 1) + 2) = 4` will always
fault at runtime.
Sometimes, despite _FORTIFY_SOURCE being nonzero, there may not be a #define that overwrites a function definition, so we need to cope with this in these definitions. This does so.
Update dynamic tests of runtime bounds checking to suppress these warnings. We don't expect to see warnings for these tests and turn warnings into errors.
For bounds declaration checking, turn TODO's for expected warnings into pattern matches for the expected warnings.
We are updating the compiler to do bounds declaration checking for constant-sized ranges (https://github.com/Microsoft/checkedc-clang/pull/414). This causes new error messages to be produced for a number of tests. For example, tests for typechecking might pass typechecking (as expected), but then fail bounds declaration checking..
Update the tests with the expected error messages from clang (most of these errors were expected once enough bounds declaration checking came on-line and were flagged by TODO comments).
As an FYI, there are additional bounds declaration checking tests in the Checked C clang repo. These contain detailed checks of the expected explanations, and are not placed in the Checked C repo.
The bounds declaration checking detects and gives errors for the bogus C-style casts described in issue #232.
This adds the fortified declarations, in line with the specifications within clang and the gcc documentation here: https://gcc.gnu.org/onlinedocs/gcc-6.3.0/gcc/Object-Size-Checking.html#Object-Size-Checking
This simplifies the checks for fortify source, and makes the tests more robust.
There is also the addition of a posix header, unistd.h, as a stopgap measure.
Tested: Mac OS X, Linux, Windows.
This change updates our checked wrappers for C standard libraries to use null-terminated array pointers for strings.
- I expanded the list of string functions and had to disable more optimized macros for GCC.
- In cases where I could make some parameters a function checked, but not other parameters checked,
I went ahead and did that. This include functions such as `strcat` and `strncat`. The fact that the destination buffer has an unchecked pointer type should be a warning to programmers trying to use them.
- For `strncat`, we have no way of expressing the bounds requirement in checked code, because the bounds requirement involves a function call.
Testing:
- Passed automated testing, including LNT testing.
The example is adapted from a simple "pattern finding" program from "The C Programming Language, Second Edition", by Kernighan and Ritchie. The example uses only checked code. The example does output using fputs; I updated fputs and the rest of stdio_checked.h with annotations for strings.
Null-terminated array pointers have a default bounds of count(0). This adds tests of bounds checking and type checking for null-terminated array pointers with programmer-specified bounds and default bounds. This change matches a corresponding clang compiler change (https://github.com/Microsoft/checkedc-clang/pull/406).
There need to be additional tests added of struct members with default bounds of count(0).
When bounds are not declared for an nt_array_ptr variable or member, the
bounds are inferred to be count(0). This differs from array_ptr, where the
bounds are assumed to be bounds(unknown).
nt_array_ptr values returned by pointer dereference or array subscript
operations are also assumed to have count(0). nt_array_ptr values
stored via pointer dereference or array_subscript operations must
also imply bounds of count(0).
This change adds tests that check that bounds inference is inferring
default bounds as expected. Because we have weak static checking of bounds
declarations, we only check that expressions involving nt_array_ptr variables
have known bounds as expected.
* Test redeclarations that provides same as default bounds.
* Add missing initializers discovered by setting default bounds earlier.
This adds examples of using null-terminated strings adapted from the "The C Programming Language, Second Edition" by Kernighan and Ritchie. The code illustrates some basic aspects of using strings, as well as the sometimes trickier aspects of writing to a string.
I plan to clean up the code once the compiler change for making count(0) be the default bounds for nt_array_ptr is complete.
I plan to clean up the code once the compiler change for making count(0) be the default bounds for nt_array_ptr is complete.
Checked C allows reads at exactly the upper bound for null-terminated pointers
so that the potential terminating value can be examined.
This change adds new dynamic tests that check that the memory location at
the upper bound is accessible for null-terminated array pointers, and not
accessible for non-null-terminated array pointers.
The bounds expression bounds(none) has a potentially ambiguous English meaning. The bounds could be unbounded or be unknown. This replaces the bounds expression bounds(none) with bounds(unknown) in the specification. This is mostly a search-and-replace of the boundsnone macro. I also updated the text that defines the meaning of the bounds expression.
The Checked C specification proposed omitting the null terminator from the number of elements in a null-terminated array. This would allow the count of elements to be identical for both the array and the bounds for an `nt_array_ptr`. However, this is not how null-terminated string literals work in C. String literals have an array type that includes the null terminator element in the size of the array.
To match existing C behavior, we now include the null terminator in the size of null-terminated checked arrays. This updates the specification accordingly.
This change corresponds to a matching compiler that adds support for string literals with checked types and completes support for initialized checked data. The language rules for string and array literals are discussed in issue #207 (they haven't been added to the Checked C specification yet).
The compiler change also improves error messages for types that occur syntactically in expressions and are used in checked scopes. Those types must be checked and everything they use must be checked too. This change updates checked_scope_basic.c to use new error messages.
Testing:
- Passes automated testing with the updated compiler.
Update the specification to disallow array_ptrs of function types. Bounds checking pointers to function types does not make sense because functions do not have sizes in C. This addiresses issue #34.
Also correct the capitalization of the _Nt_array_ptr keyword in the specification.
This change adds tests for parsing and typechecking the new Checked C null-terminated array and pointer types. This matches a corresponding Checked C clang compiler pull request (https://github.com/Microsoft/checkedc-clang/pull/394). The changes are integrated into existing test cases for checked arrays and array_ptrs.
The tests for parsing are straightforward. The new keywords `nt_checked` and `nt_array_ptr` can be used in place of the `checked` and `array_ptr` keywords. For type checking, we add tests for building the new types. We check that null-terminated arrays and pointers can only be built from integer or pointer types (the assumption is that 0 is the null-terminator value, so only types for which 0 is a valid value can be used).
We also add tests that cover implicit conversions at assignments, function calls, and conditional expressions. Nt_array_ptrs can be converted to array_ptrs and ptrs, but not the reverse. The problem is that array_ptrs or ptrs may not point to null-terminated data, or, if they do, there may be an alias that permits the null-terminator to be overwritten. We also add tests that cover uses of operators with the new types.
The tests include positive cases (where no error is expected) and many more negative cases, where the compiler is expected to issue an error. To avoid unnecessary source code changes that would obscure the new tests, I avoided renumbering temporary variables. I added letter suffixes to existing variables instead.
The compiler change disallows array_ptrs to function types (https://github.com/Microsoft/checkedc/issues/34), so we add tests of that too. Functions have indeterminate size, so bounds checking does not make sense for them. We allow array_ptrs to ptrs to function types.
Variables and members with unchecked types may have bounds-safe interfaces.
When they are used in checked scopes, they should be treated as though they
have checked types. This means that accesses to memory using pointers
stored in those variables or member should be bounds checked.
The changes adds tests of bounds checking of global and local variables with
bounds-safe interfaces and struct members with bounds-safe interfaces. This
completes the work for issue #339.
Add tests of uses of variables and fields with array types where
bounds are also declared. The declared bounds should override the bounds
based on the array type. This matches a corresponding Checked C compiler change.
These tests check that bounds checking is being done against the declared bounds.
For arrays with known size, the declare bounds should be smaller than the bounds
based on the type. Accessing an element above the declared bounds should fail.
Also test variable-length arrays at the ends of structs. Without using the declared
bounds, they have unknown bounds and can't be accessed in checked code.
This change adds tests of type checking uses of declarations with bounds-safe interfaces in checked scopes. The Checked C specification says that such uses shall be treated as having checked types. This change adds tests that check this. It matches the compiler change in https://github.com/Microsoft/checkedc-clang/pull/379.
It adds a new file typechecking\checked_scope_interfaces.c that contains tests of the uses of bounds-safe interfaces in checked scopes. The file systematically combines different kinds of declarations with bounds-safe interfaces. It varies the types, starting with simple types (pointer to scalar types) and then looking at more complex constructed types (pointer to pointers, pointers to function pointers, and arrays of function pointers). It also includes some stress tests that combine bounds-safe interfaces, function pointer types, and typedefs.
We now allow casts from array_ptr<void> to other checked pointer types. This change updates tests to reflect this change.
It refactors checked_scope.c. Most of the code moves to checked_scope_basic.c, which focuses on testing uses of checked and unchecked types in checked scopes. Some tests that are redundant with other existing tests are deleted. The code specific to bounds-safe interfaces move to checked_scope_interfaces.c.
Avoid using unchecked pointers to structs in tests of bounds checking - use checked pointers instead.
When a function parameter has bounds, corresponding call arguments should have bounds. This adds tests that check that. This corresponds to the compiler changes for https://github.com/Microsoft/checkedc-clang/issues/373,
This change adds a new type `nt_array_ptr` that is a variant of `array_ptr` for pointers to null-terminated arrays. The change:
- describes the new type in core-extensions.tex.
- describes a new checked null-terminated array type go with `nt_array_ptr`
- describes the runtime bounds checking rules for nt_array_ptr.
For `nt_array_ptr` we allow inferred bounds if bounds are not declared. Inference can widen bounds when it sees non-null memory access. This raises the problem that memory accesses could depend on bounds that are not written down in the program. The proposal is that any such accesses to memory must be provably in-bounds at compile time.
Most of the rules later in the specification for `array_ptr` apply to `nt_array_ptr`. I initially started down the route of updating all occurrences of `array_ptr` to say "or `nt_array_ptr`". I decided that was resulting in lots of changes that didn't help understanding much and made the text longer. There is a paragraph in the description of `nt_array_ptr` that says that the rules for `array_ptr` usually apply to `nt_array_ptr` and that we mention `nt_array_ptr` only when the rules are different.
This change also cleans up some aspects of the spec:
- Remove the text that says bounds-checking is done for &e, if e is a dereference or subscript operation. This is incorrect and doesn't match what the implementation does.
- Makes the description of bounds checks more precise, by describing the use of temporaries instead of expressions during bounds checking (the expression being bounds checked is not re-evaluated).
- Revise the section describing the deferral of the evaluation of bounds expression.
Still to do (in another pull request):
- Update the rules for conversions between checked pointer types.
- Describe the analysis that widens bounds for 'nt_array_ptr`.
Add tests for returning checked pointers from functions with bounds-safe
interfaces for their return values. Modify an existing test to allow this
behavior.
The important part of `free` is that you're passing an in-bounds pointer to an object to be free-ed. Given how we infer bounds for both array_ptrs and ptrs, require subsumption for function arguments, and that the smallest range possible for a pointer is `byte_count(1)`, this is the correct interface type rather than one that explicitly mentions array_ptr or ptr.
We also change the interoperation type of the pointer parameter of `void *realloc(void*, size_t)` because it can also free memory.
We add a set of tests that ensure these interface types for malloc, free, realloc and alloc_aligned operate with each other for every kind of pointer: unchecked, ptr and array_ptr.
- Update tests to allow conversions from any kind of checked pointer to
any kind of checked void pointer (ptr<void> and array_ptr<void>).
- Test that implicit conversions between void pointers and checked function
pointers are not allowed. This is a common C extension that should
not be allowed for checked function pointers.
Checked C clang pull request https://github.com/Microsoft/checkedc-clang/pull/363 addresses the problem that bounds-safe interfaces on function pointers were not being used during type checking. It substantially revised the error messages for cases where the term "unchecked type" might be unclear to programmers (for example, a function or array type that uses an unchecked type).
This updates the error messages in the existing Checked C tests. It also adds tests of typecheckng function pointer types involving bounds-safe interfaces.
- Test that generic no-prototype functions cannot be declared.
- Test that generic K&R old-style functions cannot be declared.
- Test generic functions applied to an empty type argument list. Make
sure an error is emitted if type arguments are expected.
- Stop capitalizing the first word of some error messages for generic
functions.
- Stop checking notes issued by clang in Checked C language tests. These
are implementation-specific details that don't belong in language tests.
- Improve tests that K&R old-style functions can be declared or used within
checked blocks. We weren't testing that K&R style functions cannot
be declared within checked blocks.
* Adding test files for _For_any specifier and type variable declarations.
* Adding test case to generic Function Call Info
* The comments were wrong in the newly created test case.
* Fixing more errors due to the fact that Type Variables are now incomplete type
* Adding test files for _For_any specifier and type variable declarations.
* Adding test case to generic Function Call Info
* The comments were wrong in the newly created test case.
With the introduction of type variable types in clang, it is important to make sure that generic functions are a FunctionProtoType. We did not allow generic functions to be FunctionNoProtoType. In order for the tests to work correctly, I had to add "void" in () if there are no parameters.
As part of statically checking bounds declarations, we need to check symbolic equality of expressions and to lexicographically order expressions (for sorting commutative operands). This implements language-level tests of bounds declarations that indirectly cover lexical equality/inequality.
* Adding test files for _For_any specifier and type variable declarations.
* Fixing an error in forany_parsing.c This is caused because now the generic function call expects to see type lists
* Adding test files for _For_any specifier and type variable declarations.
* These two files tests for parsing generic function calls and any errors that clang may encounter
* Editing parsing error testing in accordance to coding changes in checkedc-clang
* Moving a couple test cases to checkedc-clang since it tests for specifics of how clang is tested.
For_any parsing is generating LLVM IR files as part of test runs. This causes automation testing to fail because it creates the file in a source directory. The next automation run fails because it finds a file with no RUN line in it in the test directory. The fix is to not generate LLVM IR files.
I should not be trusted with De Morgan's Laws, turns out I'll break them.
`!a && b` is not equivalent to `!(a && b)`.
This should make Ptrdist/anagram work on Linux again, I tested it on my WSL installation
Stdio_checked.h is failing to compile on Windows (issue #169). Bounds-safe
interfaces were added for stdin, stdout, and stderr. These identifiers only
have to be expressions with type FILE *. On Windows, they are function calls,
so we add a bounds-safe interface for the function being called.
A Variety of changes
- Removing some `_Unchecked` annotations
- Adding interop type to `__builtin_object_size`
- Adding interop types for `stdin` `stdout` and `stderr`
- Adding interop declaration for `fputc`
Sam Elliott
David Tarditi
Jay Lim