diff --git a/docs/annotations/index.rst b/docs/annotations/index.rst new file mode 100644 index 0000000..775df58 --- /dev/null +++ b/docs/annotations/index.rst @@ -0,0 +1,65 @@ +.. _annotations: + +Annotations +=========== + +.. toctree:: + :maxdepth: 3 + :hidden: + + relational_annotations + invariants + +Annotations are the building blocks out of which reference implementations are constructed. They +annotations represent a single condition that a student's implementation should meet, and define a +set of behaviors for responding to the passing or failing of those conditions. Annotations provide +conditions not only for expecting a specific value but also for combining those expectations to form +conditions on the structure of students' code, including the temporal relationship of values and +complex boolean logic surrounding the presence, or lack thereof, of those values. + +All annotations are created by instaniating subclasses of the abstract +:py:class:`Annotation` class. There are two main types of annotations: value +annotations, described below, and :ref:`relational annotations`. + + +Values +------ + +Consider the most basic kind of annotation: expecting a specific value to appear while executing the +student's code. To create a value annotation, create an instance of :py:class:`Value`. +The constructor takes in the value that you are expecting to see in the student's code: + +.. code-block:: python + + np.random.seed(42) + arr = np.random.normal(3, 2, size=100) + pybryt.Value(arr) + +Note that when an instance of :py:class:`Value` is created, :py:meth:`copy.copy` is +called on the argument passed to it, so values don't need to worry about being affected by +mutability. + + +Numerical Tolerance ++++++++++++++++++++ + +For numerical values, or iterables of numerical values that support vectorized math, it is also +possible to set an absolute tolerance for the acceptance of student values using the ``tol`` +argument, which defaults to zero. + +.. code-block:: python + + pybryt.Value(arr, tol=1e-4) + + +Invariants +++++++++++ + +PyBryt supports :ref:`invariants`, which are conditions that allow for objects with +different structures to be considered "equal" from PyBryt's perspective. To use invariants on a +value, pass the invariant objects in a list to the ``invariants`` argument of the constructor: + +.. code-block:: python + + correct_answer = "a CasE-inSensiTiVe stRING" + pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization]) diff --git a/docs/reference_implementations/invariants.rst b/docs/annotations/invariants.rst similarity index 100% rename from docs/reference_implementations/invariants.rst rename to docs/annotations/invariants.rst diff --git a/docs/reference_implementations/annotations.rst b/docs/annotations/relational_annotations.rst similarity index 65% rename from docs/reference_implementations/annotations.rst rename to docs/annotations/relational_annotations.rst index 52929de..94f0d33 100644 --- a/docs/reference_implementations/annotations.rst +++ b/docs/annotations/relational_annotations.rst @@ -1,63 +1,7 @@ -.. _annotations: - -Annotations -=========== - -Annotations are the building blocks out of which reference implementations are constructed. They -annotations represent a single condition that a student's implementation should meet, and define a -set of behaviors for responding to the passing or failing of those conditions. Annotations provide -conditions not only for expecting a specific value but also for combining those expectations to form -conditions on the structure of students' code, including the temporal relationship of values and -complex boolean logic surrounding the presence, or lack thereof, of those values. - - -Values ------- - -All annotations are created by instaniating subclasses of the abstract -:py:class:`Annotation` class. Consider the most basic kind of annotation: -expecting a specific value to appear while executing the student's code. To create a value -annotation, create an instance of :py:class:`Value`. The constructor takes in the -value that you are expecting to see in the student's code: - -.. code-block:: python - - np.random.seed(42) - arr = np.random.normal(3, 2, size=100) - pybryt.Value(arr) - -Note that when an instance of :py:class:`Value` is created, :py:meth:`copy.copy` is -called on the argument passed to it, so values don't need to worry about being affected by -mutability. - - -Numerical Tolerance -+++++++++++++++++++ - -For numerical values, or iterables of numerical values that support vectorized math, it is also -possible to set an absolute tolerance for the acceptance of student values using the ``tol`` -argument, which defaults to zero. - -.. code-block:: python - - pybryt.Value(arr, tol=1e-4) - - -Invariants -++++++++++ - -PyBryt supports :ref:`invariants`, which are conditions that allow for objects with -different structures to be considered "equal" from PyBryt's perspective. To use invariants on a -value, pass the invariant objects in a list to the ``invariants`` argument of the constructor: - -.. code-block:: python - - correct_answer = "a CasE-inSensiTiVe stRING" - pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization]) - +.. _relational: Relational Annotations ----------------------- +====================== Relational annotations define some kind of relationship between two or more annotations. Currently, PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations. @@ -66,7 +10,7 @@ All relational annotations are subclasses of the abstract defines some helpful defaults for working with annotations that have child annotations. Temporal Annotations -++++++++++++++++++++ +-------------------- Temporal annotations describe *when* variables should appear in students' code relative to one another. For example, consider the problem of a dynamic programming algorithm to compute the @@ -130,7 +74,7 @@ the operands switched. Boolean Annotations -+++++++++++++++++++ +------------------- Boolean annotations define conditions on the presence of different values. For example, in defining a solutions, students may be able to take two different paths, and this logic can be enforced diff --git a/docs/html/_sources/annotations/index.rst.txt b/docs/html/_sources/annotations/index.rst.txt new file mode 100644 index 0000000..775df58 --- /dev/null +++ b/docs/html/_sources/annotations/index.rst.txt @@ -0,0 +1,65 @@ +.. _annotations: + +Annotations +=========== + +.. toctree:: + :maxdepth: 3 + :hidden: + + relational_annotations + invariants + +Annotations are the building blocks out of which reference implementations are constructed. They +annotations represent a single condition that a student's implementation should meet, and define a +set of behaviors for responding to the passing or failing of those conditions. Annotations provide +conditions not only for expecting a specific value but also for combining those expectations to form +conditions on the structure of students' code, including the temporal relationship of values and +complex boolean logic surrounding the presence, or lack thereof, of those values. + +All annotations are created by instaniating subclasses of the abstract +:py:class:`Annotation` class. There are two main types of annotations: value +annotations, described below, and :ref:`relational annotations`. + + +Values +------ + +Consider the most basic kind of annotation: expecting a specific value to appear while executing the +student's code. To create a value annotation, create an instance of :py:class:`Value`. +The constructor takes in the value that you are expecting to see in the student's code: + +.. code-block:: python + + np.random.seed(42) + arr = np.random.normal(3, 2, size=100) + pybryt.Value(arr) + +Note that when an instance of :py:class:`Value` is created, :py:meth:`copy.copy` is +called on the argument passed to it, so values don't need to worry about being affected by +mutability. + + +Numerical Tolerance ++++++++++++++++++++ + +For numerical values, or iterables of numerical values that support vectorized math, it is also +possible to set an absolute tolerance for the acceptance of student values using the ``tol`` +argument, which defaults to zero. + +.. code-block:: python + + pybryt.Value(arr, tol=1e-4) + + +Invariants +++++++++++ + +PyBryt supports :ref:`invariants`, which are conditions that allow for objects with +different structures to be considered "equal" from PyBryt's perspective. To use invariants on a +value, pass the invariant objects in a list to the ``invariants`` argument of the constructor: + +.. code-block:: python + + correct_answer = "a CasE-inSensiTiVe stRING" + pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization]) diff --git a/docs/html/_sources/annotations/invariants.rst.txt b/docs/html/_sources/annotations/invariants.rst.txt new file mode 100644 index 0000000..7f780ee --- /dev/null +++ b/docs/html/_sources/annotations/invariants.rst.txt @@ -0,0 +1,4 @@ +.. _invariants: + +Invariants +========== diff --git a/docs/html/_sources/annotations/relational_annotations.rst.txt b/docs/html/_sources/annotations/relational_annotations.rst.txt new file mode 100644 index 0000000..94f0d33 --- /dev/null +++ b/docs/html/_sources/annotations/relational_annotations.rst.txt @@ -0,0 +1,101 @@ +.. _relational: + +Relational Annotations +====================== + +Relational annotations define some kind of relationship between two or more annotations. Currently, +PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations. +All relational annotations are subclasses of the abstract +:py:class:`RelationalAnnotation` class, which +defines some helpful defaults for working with annotations that have child annotations. + +Temporal Annotations +-------------------- + +Temporal annotations describe *when* variables should appear in students' code relative to one +another. For example, consider the problem of a dynamic programming algorithm to compute the +Fibonacci sequence: the array containing :math:`n-1` first Fibonacci numbers should appear in memory +before the array containing the :math:`n` first Fibonacci numbers. + +To enforce such a constraint, the :py:class:`Annotation` class defines a +``before`` method that asserts that one annotation occurs before another: + +.. code-block:: python + + def fib(n): + """ + Compute and return an array of the first n Fibonacci numbers using dynamic programming. + + Args: + n (``int``): the number of Fibonacci numbers to return + + Returns: + ``np.ndarray``: the first ``n`` Fibonacci numbers + """ + fibs = np.zeros(n, dtype=int) + + fibs[0] = 0 + curr_val = pybryt.Value(fibs) + if n == 1: + return fibs + + fibs[1] = 1 + v = pybryt.Value(fibs) + curr_val.before(v) + curr_val = v + if n == 2: + return fibs + + for i in range(2, n-1): + fibs[i] = fibs[i-1] + fibs[i-2] + + v = pybryt.Value(fibs) # array of first n Fibonacci numbrs + curr_val.before(v) # check that first n-1 Fib numbers come before first n + curr_val = v # update curr_val for next iteration + + return fibs + +In the example above, updating a pointer ``curr_val`` in the loop allows us to create a ``before`` +condition such that we ensure the student followed the correct dynamic programming algorithm by +checking each update to the ``fibs`` array. + +Temporal annotations are satisfied when the student's code satisfies all of the child +:py:class:`Value` annotations and when the first annotation (the one calling +:py:meth:`Annotation.before`) has a timestamp greater than or equal to the +timestamp of the second annotation. + +Note that :py:meth:`Annotation.before` returns an instance of the +:py:class:`BeforeAnnotation` class, which is +itself a subclass of :py:class:`Annotation` and supports all of the same operations. +:py:class:`Annotation` also provides +:py:meth:`Annotation.after`, which also returns an instance of the +:py:class:`BeforeAnnotation` class, but with +the operands switched. + + +Boolean Annotations +------------------- + +Boolean annotations define conditions on the presence of different values. For example, in defining +a solutions, students may be able to take two different paths, and this logic can be enforced +using a :py:class:`XorAnnotation` to ensure that +only one of the two possible values is present. + +Relational annotations can be created either by instantiating the classes directly using the +constructor or, as is more recommended, by using Python's bitwise logical operators, ``&``, ``|``, +``^``, and ``~``, on annotations. The dunder methods for these operators have been overrided with +for the :py:class:`Annotation` class, and return the +:py:class:`RelationalAnnotation` subclass +instance corresponding to the logical operator used. + +To create the xor example above from two values ``v1`` and ``v2``, simply write + +.. code-block:: python + + v1 ^ v2 + +To assert that a student should *not* have a specific value ``v`` in their code, use + +.. code-block:: python + + ~v diff --git a/docs/html/_sources/index.rst.txt b/docs/html/_sources/index.rst.txt index 67b4409..f4c6a84 100644 --- a/docs/html/_sources/index.rst.txt +++ b/docs/html/_sources/index.rst.txt @@ -10,7 +10,8 @@ PyBryt Documentation :maxdepth: 3 :hidden: - reference_implementations/index + annotations/index + reference_implementations api_reference PyBryt is an open source Python auto-assessment library for teaching and learning. Our goal is to empower students and educators to learn about technology through fun, guided, hands-on content aimed at specific learning goals. PyBryt is designed to work with existing autograding solutions and workflows such as `Otter Grader`_, `OkPy`_, and `Autolab`_. @@ -26,6 +27,16 @@ Educators and institutions can leverage PyBryt to integrate auto-assessment and - Plagiarism detection and support for reference implementations - Easy integration into existing organizational or institutional grading infrastructure +PyBryt's core auto-assessment behavior operates by comparing a student's implementation of some +programming problem to a series of reference implementations provided by an instructor. A +**reference implementation** defines a pattern of values, and conditions on those values, to look +for in students' code. + +A reference implementation is created by annotating code written or found by an instructor and +executing this code to create a :py:class:`ReferenceImplementation` +object. Annotations are created by creating instances of subclasses of the abstract +:py:class:`Annotation` class. + .. _Otter Grader: https://otter-grader.readthedocs.io .. _OkPy: https://okpy.org .. _Autolab: https://autolab.readthedocs.io/ diff --git a/docs/reference_implementations/reference_objects.rst b/docs/html/_sources/reference_implementations.rst.txt similarity index 98% rename from docs/reference_implementations/reference_objects.rst rename to docs/html/_sources/reference_implementations.rst.txt index 2460459..d847f22 100644 --- a/docs/reference_implementations/reference_objects.rst +++ b/docs/html/_sources/reference_implementations.rst.txt @@ -1,5 +1,5 @@ -Reference Implementation Objects -================================ +Reference Implementations +========================= The functional unit of PyBryt is a reference implementation. A **reference implenetation** is a set of conditions expected of students' code that determine whether a student has correctly implemented diff --git a/docs/html/annotations/index.html b/docs/html/annotations/index.html new file mode 100644 index 0000000..8389b2e --- /dev/null +++ b/docs/html/annotations/index.html @@ -0,0 +1,258 @@ + + + + + + + Annotations - PyBryt documentation + + + + + + + + + + + + Contents + + + + + + + + + Menu + + + + + + + + Expand + + + + + + + + + + + + + +
+
+
+ +
+
+
PyBryt documentation
 +
+
+ +
+
+ +
+
+
+ +
+

Annotations

+
+
+

Annotations are the building blocks out of which reference implementations are constructed. They +annotations represent a single condition that a student’s implementation should meet, and define a +set of behaviors for responding to the passing or failing of those conditions. Annotations provide +conditions not only for expecting a specific value but also for combining those expectations to form +conditions on the structure of students’ code, including the temporal relationship of values and +complex boolean logic surrounding the presence, or lack thereof, of those values.

+

All annotations are created by instaniating subclasses of the abstract +Annotation class. There are two main types of annotations: value +annotations, described below, and relational annotations.

+
+

Values

+

Consider the most basic kind of annotation: expecting a specific value to appear while executing the +student’s code. To create a value annotation, create an instance of Value. +The constructor takes in the value that you are expecting to see in the student’s code:

+
np.random.seed(42)
+arr = np.random.normal(3, 2, size=100)
+pybryt.Value(arr)
+
+
+

Note that when an instance of Value is created, copy.copy() is +called on the argument passed to it, so values don’t need to worry about being affected by +mutability.

+
+

Numerical Tolerance

+

For numerical values, or iterables of numerical values that support vectorized math, it is also +possible to set an absolute tolerance for the acceptance of student values using the tol +argument, which defaults to zero.

+
pybryt.Value(arr, tol=1e-4)
+
+
+
+
+

Invariants

+

PyBryt supports invariants, which are conditions that allow for objects with +different structures to be considered “equal” from PyBryt’s perspective. To use invariants on a +value, pass the invariant objects in a list to the invariants argument of the constructor:

+
correct_answer = "a CasE-inSensiTiVe stRING"
+pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
+
+
+
+
+
+ +
+ +
+ +
+
+ + + + + + + + \ No newline at end of file diff --git a/docs/html/annotations/invariants.html b/docs/html/annotations/invariants.html new file mode 100644 index 0000000..385a751 --- /dev/null +++ b/docs/html/annotations/invariants.html @@ -0,0 +1,191 @@ + + + + + + + Invariants - PyBryt documentation + + + + + + + + + + + + Contents + + + + + + + + + Menu + + + + + + + + Expand + + + + + + + + + + + + + +
+
+
+ +
+
+
PyBryt documentation
 +
+
+ +
+
+ +
+
+
+ +
+

Invariants

+
+ +
+ +
+ +
+
+ + + + + + + + \ No newline at end of file diff --git a/docs/html/annotations/relational_annotations.html b/docs/html/annotations/relational_annotations.html new file mode 100644 index 0000000..38b8d4b --- /dev/null +++ b/docs/html/annotations/relational_annotations.html @@ -0,0 +1,294 @@ + + + + + + + Relational Annotations - PyBryt documentation + + + + + + + + + + + + Contents + + + + + + + + + Menu + + + + + + + + Expand + + + + + + + + + + + + + +
+
+
+ +
+
+
PyBryt documentation
 +
+
+ +
+
+ +
+
+
+ +
+

Relational Annotations

+

Relational annotations define some kind of relationship between two or more annotations. Currently, +PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations. +All relational annotations are subclasses of the abstract +RelationalAnnotation class, which +defines some helpful defaults for working with annotations that have child annotations.

+
+

Temporal Annotations

+

Temporal annotations describe when variables should appear in students’ code relative to one +another. For example, consider the problem of a dynamic programming algorithm to compute the +Fibonacci sequence: the array containing \(n-1\) first Fibonacci numbers should appear in memory +before the array containing the \(n\) first Fibonacci numbers.

+

To enforce such a constraint, the Annotation class defines a +before method that asserts that one annotation occurs before another:

+
def fib(n):
+    """
+    Compute and return an array of the first n Fibonacci numbers using dynamic programming.
+
+    Args:
+    n (``int``): the number of Fibonacci numbers to return
+
+    Returns:
+    ``np.ndarray``: the first ``n`` Fibonacci numbers
+    """
+    fibs = np.zeros(n, dtype=int)
+
+    fibs[0] = 0
+    curr_val = pybryt.Value(fibs)
+    if n == 1:
+        return fibs
+
+    fibs[1] = 1
+    v = pybryt.Value(fibs)
+    curr_val.before(v)
+    curr_val = v
+    if n == 2:
+        return fibs
+
+    for i in range(2, n-1):
+        fibs[i] = fibs[i-1] + fibs[i-2]
+
+        v = pybryt.Value(fibs) # array of first n Fibonacci numbrs
+        curr_val.before(v)     # check that first n-1 Fib numbers come before first n
+        curr_val = v           # update curr_val for next iteration
+
+    return fibs
+
+
+

In the example above, updating a pointer curr_val in the loop allows us to create a before +condition such that we ensure the student followed the correct dynamic programming algorithm by +checking each update to the fibs array.

+

Temporal annotations are satisfied when the student’s code satisfies all of the child +Value annotations and when the first annotation (the one calling +Annotation.before) has a timestamp greater than or equal to the +timestamp of the second annotation.

+

Note that Annotation.before returns an instance of the +BeforeAnnotation class, which is +itself a subclass of Annotation and supports all of the same operations. +Annotation also provides +Annotation.after, which also returns an instance of the +BeforeAnnotation class, but with +the operands switched.

+
+
+

Boolean Annotations

+

Boolean annotations define conditions on the presence of different values. For example, in defining +a solutions, students may be able to take two different paths, and this logic can be enforced +using a XorAnnotation to ensure that +only one of the two possible values is present.

+

Relational annotations can be created either by instantiating the classes directly using the +constructor or, as is more recommended, by using Python’s bitwise logical operators, &, |, +^, and ~, on annotations. The dunder methods for these operators have been overrided with +for the Annotation class, and return the +RelationalAnnotation subclass +instance corresponding to the logical operator used.

+

To create the xor example above from two values v1 and v2, simply write

+
v1 ^ v2
+
+
+

To assert that a student should not have a specific value v in their code, use

+
~v
+
+
+
+
+ +
+ +
+ +
+
+ + + + + + + + \ No newline at end of file diff --git a/docs/html/api_reference.html b/docs/html/api_reference.html index b17e9eb..aa4375d 100644 --- a/docs/html/api_reference.html +++ b/docs/html/api_reference.html @@ -1,7 +1,7 @@ - + API Reference - PyBryt documentation @@ -107,12 +107,12 @@
    -
  • Reference Implementations @@ -836,14 +836,14 @@ present, this is the maximum satisfying timestamp of all child results

    - +
    Previous
    -
    Reference Implementation Objects
    +
    Reference Implementations
     diff --git a/docs/html/genindex.html b/docs/html/genindex.html index 76023e8..23922a5 100644 --- a/docs/html/genindex.html +++ b/docs/html/genindex.html @@ -106,12 +106,12 @@
      -
    • Reference Implementations diff --git a/docs/html/index.html b/docs/html/index.html index 112eac9..f1e5a50 100644 --- a/docs/html/index.html +++ b/docs/html/index.html @@ -1,7 +1,7 @@ - + Home - PyBryt documentation @@ -107,12 +107,12 @@
      +

      PyBryt’s core auto-assessment behavior operates by comparing a student’s implementation of some +programming problem to a series of reference implementations provided by an instructor. A +reference implementation defines a pattern of values, and conditions on those values, to look +for in students’ code.

      +

      A reference implementation is created by annotating code written or found by an instructor and +executing this code to create a ReferenceImplementation +object. Annotations are created by creating instances of subclasses of the abstract +Annotation class.