diff --git a/docs/annotations/initial_conditions.rst b/docs/annotations/initial_conditions.rst deleted file mode 100644 index 2a8abed..0000000 --- a/docs/annotations/initial_conditions.rst +++ /dev/null @@ -1,74 +0,0 @@ -Initial Conditions -================== - -For some problems, it is necessary for students to choose some configurations or initial conditions -that can vary from student to student; handling all possible values of these initial conditions -would require writing quite a few references. Instead, PyBryt offers the -:py:class:`InitialCondition` class to -represent a value that will be set when the student's submission is executed for use in writing -annotations. - - -Writing a Reference with Initial Conditions -------------------------------------------- - -Using initial conditions in your references is pretty simple. Each -:py:class:`InitialCondition` has a name, -the first argument to its constructor. This name is how the value in the student submission will be -identified (but more on that later). - -Once you've instantiated an -:py:class:`InitialCondition`, you can use it -as normal in value and other types of annotations (except attribute annotations). - -.. code-block:: python - - ic = pybryt.InitialCondition("foo") - pybryt.Value(ic) - -Sometimes, however, you may want to look for a value derived from an initial condition. For this -reason, :py:class:`InitialCondition` -supports all of Python's arithmetic operators, and you can also apply transformations by writing -functions: - -.. code-block:: python - - pybryt.Value(ic + 2) - pybryt.Value(2 * ic - 3) - - pybryt.Value(ic.apply(np.transpose).apply(np.linalg.norm)) - pybryt.Value(ic.apply(lambda v: pow(v, 10, 73))) - -Each statement inside the :py:class:`Value` constructors above -returns an :py:class:`InitialCondition`. -Initial conditions maintain a list of transformations that need to be applied to reach the final -value for the annotation, and when a value is supplied from the submission, the transformations -are applied in sequence to determine the value that the value annotations should look for. - - -Collecting Initial Conditions in Submissions --------------------------------------------- - -In order to link the initial conditions in the reference implementation to the values in the -submission, you must call -:py:func:`pybryt.set_initial_conditions` in the -submission. This function accepts a dictionary as its only argument mapping strings corresponding -to the names of initial conditions to the values of those initial conditions. When PyBryt is not -actively tracing, it has no effect; but, when called while PyBryt executes the submission, it tells -PyBryt to store the initial conditions passed to it in the memory footprint for later use by the -annotations. - -For example, continuing the example from above, you would need to call - -.. code-block:: python - - pybryt.set_initial_conditions({ - "foo": 2, - }) - -to set the initial condition. - -This function can be called multiple times, but the memory footprint only retains a single store of -initial conditions, so calling it multiple times with the same key will overwrite any old values -of that key. (For example, calling it with ``{"foo": 1, "bar": 3}`` and then ``{"foo": 2}`` -will result in initial conditions ``{"foo": 2, "bar": 3}``). diff --git a/docs/annotations/invariants.rst b/docs/annotations/invariants.rst deleted file mode 100644 index 6a69ed6..0000000 --- a/docs/annotations/invariants.rst +++ /dev/null @@ -1,36 +0,0 @@ -.. _invariants: - -Invariants -========== - -Invariants provide logic for determining whether a value satisfies a value annotation. They can be -used to ensure that annotations can still be satisfied independent of the different ways students -format their answers. - - -Invariant Structure -------------------- - -Invariants are subclasses of the abstract base class -:py:class:`invariants.invariant`. All subclasses implement -the static :py:meth:`run` method, which takes in a list of -objects and returns a transformed list of objects that contains all versions of every object in the -input list which are considered equal. - -Consider an invariant for string capitalization. The ``run`` method of this invariant takes in a list -of objects and returns the same list but with every string lowercased. For a matrix transposition -invariant, the ``run`` method might return a list with every 2D array's transpose included, as well -as the original array. - -Invariants have a custom ``__new__`` method that calls the ``run`` method, so that they function as -callables rather than classes. - -PyBryt supports custom invariants. To create your own invariant, subclass -:py:class:`invariants.invariant` and implement the ``run`` -method. - - -Built-In Invariants -------------------- - -A list of built-in invariants can be found :ref:`here`. diff --git a/docs/annotations/value_annotations.rst b/docs/annotations/value_annotations.rst deleted file mode 100644 index e41cd12..0000000 --- a/docs/annotations/value_annotations.rst +++ /dev/null @@ -1,136 +0,0 @@ -.. _value: - -Value Annotations -================= - -Value annotations are the most basic type of annotation. They expect a specific -value to appear while executing the student's code. To create a value -annotation, create an instance of :py:class:`Value` and pass to -its constructor the value expected in the student's code: - -.. code-block:: python - - arr = np.linspace(0, 10, 11) - pybryt.Value(arr) - -Note that when an instance of :py:class:`Value` is created, -:py:meth:`copy.copy` is called on the argument passed to it, so values cannot be -affected by mutability. - - -Attribute Annotations ---------------------- - -PyBryt supports checking for the presence of an object with a specific attribute value using the -:py:class:`Attribute` annotation. This annotation takes in an -object and one or more strings representing an instance variable, and asserts that the student's -memory footprint should contain an object with that attribute such that the value of the attribute -equals the value of the attribute in the annotation. - -For example, this can be useful in checking that students have correctly fitted in ``sklearn`` -regression model by checking that the coefficients are correct: - -.. code-block:: python - - import sklearn.linear_model as lm - - model = lm.LinearRegression() - model.fit(X, y) - - pybryt.Attribute(model, "coef_", - failure_message="Your model doesn't have the correct coefficients.") - -Note that, by default, PyBryt doesn't check that the object satisfying the attribute annotation has -the same type as the object the annotation was created for. If a student knew the coefficient values -in the above example, the following student code would satisfy that annotation: - -.. code-block:: python - - class Foo: - - coef_ = ... # the array of coefficients - - f = Foo() # f will satisfy the annotation - -To ensure that the annotation is only satisfied when the object is of the same type, set -``enforce_type=True`` in the constructor: - -.. code-block:: python - - pybryt.Attribute(model, "coef_", enforce_type=True, - failure_message="Your model doesn't have the correct coefficients.") - - -Return Value Annotations ------------------------- - -In addition to checking for the presence of a value, PyBryt also has an annotation that asserts that -a value was returned by a student's function. The annotation does not specify anything about the -function that returns it except that the function is part of the submission code (i.e. not part of -an imported library); it merely checks that the value was seen at least once in a return event -passed to the trace function. - -You can create a return value annotation with the -:py:class:`ReturnValue` constructor; it accepts all the same -arguments as the :py:class:`Value` constructor: - -.. code-block:: python - - pybryt.ReturnValue(df, success_message="...", failure_message="...") - - -Numerical Tolerances ---------------------- - -For numerical values, or iterables of numerical values that support vectorized -math, it is also possible to define an absolute tolerance (``atol``) and/or a -relative tolerance (``rtol``) to allow the student's solution to deviate from -the reference. Numerical tolerances are computed as with ``numpy.allcose``, -where the value is "equal enough" if it is within :math:`v \pm (\texttt{atol} -+ \texttt{rtol} \cdot |v|)`, where :math:`v` is the value of the annotation. -Both ``atol`` and ``rtol`` default to zero and have to be specified when value -annotation is defined: - -.. code-block:: python - - pybryt.Value(arr, atol=1e-3, rtol=1e-5) - - -Invariants ---------------------- - -Similar to numerical tolerances, which allow the student's solution to deviate -from the reference value, PyBryt allows specifying conditions when other data -types should be considered equal. PyBryt supports defining these conditions -using :ref:`invariants`. To use invariants on a value, we need to -pass the invariant objects as a list to the ``invariants`` argument of the -:py:class:`Value` constructor. For instance, let's say we want to -allow the student's solution (a string) to be case-insensitive. - -.. code-block:: python - - correct_answer = "a CasE-inSensiTiVe stRING" - pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization]) - -More information about invariants can be found :ref:`here`. - - -Custom Equivalence Functions ----------------------------- - -In some cases, the algorithm that value annotations use for checking if two objects are -equivalent may not be suitable to the problem at hand. For cases like this, you can provide a -custom equivalence function that the value annotation will use instead to determine if two -objects are equal. The equivalence function should return ``True`` if the objects are equal and -``False`` otherwise. If the equivalence function raises an error, this will be interpeted as -``False`` (unless :ref:`debug mode` is enabled). - -For example, we could implement the ``string_capitalization`` invariant using a custom -equivalence function: - -.. code-block:: python - - def string_lower_eq(s1, s2): - return s1.lower() == s2.lower() - - pybryt.Value(correct_answer, equivalence_fn=string_lower_eq)