зеркало из https://github.com/microsoft/pybryt.git
delete files
This commit is contained in:
Chris Pyles
Родитель
9f6ea46567
Коммит
fd0aa15597
|
|
@ -1,74 +0,0 @@
|
|||
Initial Conditions
|
||||
==================
|
||||
|
||||
For some problems, it is necessary for students to choose some configurations or initial conditions
|
||||
that can vary from student to student; handling all possible values of these initial conditions
|
||||
would require writing quite a few references. Instead, PyBryt offers the
|
||||
:py:class:`InitialCondition<pybryt.annotations.initial_condition.InitialCondition>` class to
|
||||
represent a value that will be set when the student's submission is executed for use in writing
|
||||
annotations.
|
||||
|
||||
|
||||
Writing a Reference with Initial Conditions
|
||||
-------------------------------------------
|
||||
|
||||
Using initial conditions in your references is pretty simple. Each
|
||||
:py:class:`InitialCondition<pybryt.annotations.initial_condition.InitialCondition>` has a name,
|
||||
the first argument to its constructor. This name is how the value in the student submission will be
|
||||
identified (but more on that later).
|
||||
|
||||
Once you've instantiated an
|
||||
:py:class:`InitialCondition<pybryt.annotations.initial_condition.InitialCondition>`, you can use it
|
||||
as normal in value and other types of annotations (except attribute annotations).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
ic = pybryt.InitialCondition("foo")
|
||||
pybryt.Value(ic)
|
||||
|
||||
Sometimes, however, you may want to look for a value derived from an initial condition. For this
|
||||
reason, :py:class:`InitialCondition<pybryt.annotations.initial_condition.InitialCondition>`
|
||||
supports all of Python's arithmetic operators, and you can also apply transformations by writing
|
||||
functions:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pybryt.Value(ic + 2)
|
||||
pybryt.Value(2 * ic - 3)
|
||||
|
||||
pybryt.Value(ic.apply(np.transpose).apply(np.linalg.norm))
|
||||
pybryt.Value(ic.apply(lambda v: pow(v, 10, 73)))
|
||||
|
||||
Each statement inside the :py:class:`Value<pybryt.annotations.value.Value>` constructors above
|
||||
returns an :py:class:`InitialCondition<pybryt.annotations.initial_condition.InitialCondition>`.
|
||||
Initial conditions maintain a list of transformations that need to be applied to reach the final
|
||||
value for the annotation, and when a value is supplied from the submission, the transformations
|
||||
are applied in sequence to determine the value that the value annotations should look for.
|
||||
|
||||
|
||||
Collecting Initial Conditions in Submissions
|
||||
--------------------------------------------
|
||||
|
||||
In order to link the initial conditions in the reference implementation to the values in the
|
||||
submission, you must call
|
||||
:py:func:`pybryt.set_initial_conditions<pybryt.execution.set_initial_conditions>` in the
|
||||
submission. This function accepts a dictionary as its only argument mapping strings corresponding
|
||||
to the names of initial conditions to the values of those initial conditions. When PyBryt is not
|
||||
actively tracing, it has no effect; but, when called while PyBryt executes the submission, it tells
|
||||
PyBryt to store the initial conditions passed to it in the memory footprint for later use by the
|
||||
annotations.
|
||||
|
||||
For example, continuing the example from above, you would need to call
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pybryt.set_initial_conditions({
|
||||
"foo": 2,
|
||||
})
|
||||
|
||||
to set the initial condition.
|
||||
|
||||
This function can be called multiple times, but the memory footprint only retains a single store of
|
||||
initial conditions, so calling it multiple times with the same key will overwrite any old values
|
||||
of that key. (For example, calling it with ``{"foo": 1, "bar": 3}`` and then ``{"foo": 2}``
|
||||
will result in initial conditions ``{"foo": 2, "bar": 3}``).
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
.. _invariants:
|
||||
|
||||
Invariants
|
||||
==========
|
||||
|
||||
Invariants provide logic for determining whether a value satisfies a value annotation. They can be
|
||||
used to ensure that annotations can still be satisfied independent of the different ways students
|
||||
format their answers.
|
||||
|
||||
|
||||
Invariant Structure
|
||||
-------------------
|
||||
|
||||
Invariants are subclasses of the abstract base class
|
||||
:py:class:`invariants.invariant<pybryt.annotations.invariants.invariant>`. All subclasses implement
|
||||
the static :py:meth:`run<pybryt.annotations.invariants.invariant.run>` method, which takes in a list of
|
||||
objects and returns a transformed list of objects that contains all versions of every object in the
|
||||
input list which are considered equal.
|
||||
|
||||
Consider an invariant for string capitalization. The ``run`` method of this invariant takes in a list
|
||||
of objects and returns the same list but with every string lowercased. For a matrix transposition
|
||||
invariant, the ``run`` method might return a list with every 2D array's transpose included, as well
|
||||
as the original array.
|
||||
|
||||
Invariants have a custom ``__new__`` method that calls the ``run`` method, so that they function as
|
||||
callables rather than classes.
|
||||
|
||||
PyBryt supports custom invariants. To create your own invariant, subclass
|
||||
:py:class:`invariants.invariant<pybryt.annotations.invariants.invariant>` and implement the ``run``
|
||||
method.
|
||||
|
||||
|
||||
Built-In Invariants
|
||||
-------------------
|
||||
|
||||
A list of built-in invariants can be found :ref:`here<invariants_ref>`.
|
||||
|
|
@ -1,136 +0,0 @@
|
|||
.. _value:
|
||||
|
||||
Value Annotations
|
||||
=================
|
||||
|
||||
Value annotations are the most basic type of annotation. They expect a specific
|
||||
value to appear while executing the student's code. To create a value
|
||||
annotation, create an instance of :py:class:`Value<pybryt.annotations.value.Value>` and pass to
|
||||
its constructor the value expected in the student's code:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
arr = np.linspace(0, 10, 11)
|
||||
pybryt.Value(arr)
|
||||
|
||||
Note that when an instance of :py:class:`Value<pybryt.annotations.value.Value>` is created,
|
||||
:py:meth:`copy.copy` is called on the argument passed to it, so values cannot be
|
||||
affected by mutability.
|
||||
|
||||
|
||||
Attribute Annotations
|
||||
---------------------
|
||||
|
||||
PyBryt supports checking for the presence of an object with a specific attribute value using the
|
||||
:py:class:`Attribute<pybryt.annotations.value.Attribute>` annotation. This annotation takes in an
|
||||
object and one or more strings representing an instance variable, and asserts that the student's
|
||||
memory footprint should contain an object with that attribute such that the value of the attribute
|
||||
equals the value of the attribute in the annotation.
|
||||
|
||||
For example, this can be useful in checking that students have correctly fitted in ``sklearn``
|
||||
regression model by checking that the coefficients are correct:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import sklearn.linear_model as lm
|
||||
|
||||
model = lm.LinearRegression()
|
||||
model.fit(X, y)
|
||||
|
||||
pybryt.Attribute(model, "coef_",
|
||||
failure_message="Your model doesn't have the correct coefficients.")
|
||||
|
||||
Note that, by default, PyBryt doesn't check that the object satisfying the attribute annotation has
|
||||
the same type as the object the annotation was created for. If a student knew the coefficient values
|
||||
in the above example, the following student code would satisfy that annotation:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class Foo:
|
||||
|
||||
coef_ = ... # the array of coefficients
|
||||
|
||||
f = Foo() # f will satisfy the annotation
|
||||
|
||||
To ensure that the annotation is only satisfied when the object is of the same type, set
|
||||
``enforce_type=True`` in the constructor:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pybryt.Attribute(model, "coef_", enforce_type=True,
|
||||
failure_message="Your model doesn't have the correct coefficients.")
|
||||
|
||||
|
||||
Return Value Annotations
|
||||
------------------------
|
||||
|
||||
In addition to checking for the presence of a value, PyBryt also has an annotation that asserts that
|
||||
a value was returned by a student's function. The annotation does not specify anything about the
|
||||
function that returns it except that the function is part of the submission code (i.e. not part of
|
||||
an imported library); it merely checks that the value was seen at least once in a return event
|
||||
passed to the trace function.
|
||||
|
||||
You can create a return value annotation with the
|
||||
:py:class:`ReturnValue<pybryt.annotations.value.ReturnValue>` constructor; it accepts all the same
|
||||
arguments as the :py:class:`Value<pybryt.annotations.value.Value>` constructor:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pybryt.ReturnValue(df, success_message="...", failure_message="...")
|
||||
|
||||
|
||||
Numerical Tolerances
|
||||
---------------------
|
||||
|
||||
For numerical values, or iterables of numerical values that support vectorized
|
||||
math, it is also possible to define an absolute tolerance (``atol``) and/or a
|
||||
relative tolerance (``rtol``) to allow the student's solution to deviate from
|
||||
the reference. Numerical tolerances are computed as with ``numpy.allcose``,
|
||||
where the value is "equal enough" if it is within :math:`v \pm (\texttt{atol}
|
||||
+ \texttt{rtol} \cdot |v|)`, where :math:`v` is the value of the annotation.
|
||||
Both ``atol`` and ``rtol`` default to zero and have to be specified when value
|
||||
annotation is defined:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pybryt.Value(arr, atol=1e-3, rtol=1e-5)
|
||||
|
||||
|
||||
Invariants
|
||||
---------------------
|
||||
|
||||
Similar to numerical tolerances, which allow the student's solution to deviate
|
||||
from the reference value, PyBryt allows specifying conditions when other data
|
||||
types should be considered equal. PyBryt supports defining these conditions
|
||||
using :ref:`invariants<invariants>`. To use invariants on a value, we need to
|
||||
pass the invariant objects as a list to the ``invariants`` argument of the
|
||||
:py:class:`Value<pybryt.Value>` constructor. For instance, let's say we want to
|
||||
allow the student's solution (a string) to be case-insensitive.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
correct_answer = "a CasE-inSensiTiVe stRING"
|
||||
pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
|
||||
|
||||
More information about invariants can be found :ref:`here<invariants>`.
|
||||
|
||||
|
||||
Custom Equivalence Functions
|
||||
----------------------------
|
||||
|
||||
In some cases, the algorithm that value annotations use for checking if two objects are
|
||||
equivalent may not be suitable to the problem at hand. For cases like this, you can provide a
|
||||
custom equivalence function that the value annotation will use instead to determine if two
|
||||
objects are equal. The equivalence function should return ``True`` if the objects are equal and
|
||||
``False`` otherwise. If the equivalence function raises an error, this will be interpeted as
|
||||
``False`` (unless :ref:`debug mode<debugging>` is enabled).
|
||||
|
||||
For example, we could implement the ``string_capitalization`` invariant using a custom
|
||||
equivalence function:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def string_lower_eq(s1, s2):
|
||||
return s1.lower() == s2.lower()
|
||||
|
||||
pybryt.Value(correct_answer, equivalence_fn=string_lower_eq)
|
||||
Загрузка…
Ссылка в новой задаче