diff --git a/docs/api_reference.rst b/docs/api_reference.rst index ce73a7c..42a44c5 100644 --- a/docs/api_reference.rst +++ b/docs/api_reference.rst @@ -1,9 +1,16 @@ +.. _api: + API Reference ============= + Annotations ----------- +.. autoclass:: pybryt.Annotation + :members: + :undoc-members: + .. autoclass:: pybryt.Value :members: :undoc-members: @@ -12,13 +19,15 @@ Annotations :members: :undoc-members: + Annotation Results ++++++++++++++++++ -.. autoclass:: pybryt.annotations.annotation.AnnotationResult +.. autoclass:: pybryt.AnnotationResult :members: :undoc-members: + Reference Implementations ------------------------- @@ -26,6 +35,7 @@ Reference Implementations :members: :undoc-members: + Reference Results +++++++++++++++++ @@ -33,6 +43,7 @@ Reference Results :members: :undoc-members: + Student Implementations ----------------------- diff --git a/docs/conf.py b/docs/conf.py index 6752e1c..e8add5c 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -82,7 +82,6 @@ html_theme = 'furo' html_theme_options = { # 'github_url': 'https://github.com/microsoft/pybryt', # 'repository_url': 'https://github.com/microsoft/pybryt', - 'navigation_depth': 3, } # github_url = 'https://github.com/microsoft' diff --git a/docs/index.rst b/docs/index.rst index 1aa74ea..67b4409 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -6,6 +6,13 @@ PyBryt Documentation ==================== +.. toctree:: + :maxdepth: 3 + :hidden: + + reference_implementations/index + api_reference + PyBryt is an open source Python auto-assessment library for teaching and learning. Our goal is to empower students and educators to learn about technology through fun, guided, hands-on content aimed at specific learning goals. PyBryt is designed to work with existing autograding solutions and workflows such as `Otter Grader`_, `OkPy`_, and `Autolab`_. .. image:: _static/images/pybryt_goals.png @@ -19,13 +26,6 @@ Educators and institutions can leverage PyBryt to integrate auto-assessment and - Plagiarism detection and support for reference implementations - Easy integration into existing organizational or institutional grading infrastructure -.. toctree:: - :maxdepth: 3 - :hidden: - - reference_implementations - api_reference - .. _Otter Grader: https://otter-grader.readthedocs.io .. _OkPy: https://okpy.org .. _Autolab: https://autolab.readthedocs.io/ diff --git a/docs/reference_implementations.rst b/docs/reference_implementations.rst deleted file mode 100644 index 4890a72..0000000 --- a/docs/reference_implementations.rst +++ /dev/null @@ -1,2 +0,0 @@ -Reference Implementations -========================= diff --git a/docs/reference_implementations/annotations.rst b/docs/reference_implementations/annotations.rst new file mode 100644 index 0000000..8dd75eb --- /dev/null +++ b/docs/reference_implementations/annotations.rst @@ -0,0 +1,154 @@ +Annotations +=========== + +Annotations are the building blocks out of which reference implementations are constructed. They +annotations represent a single condition that a student's implementation should meet, and define a +set of behaviors for responding to the passing or failing of those conditions. Annotations provide +conditions not only for expecting a specific value but also for combining those expectations to form +conditions on the structure of students' code, including the temporal relationship of values and +complex boolean logic surrounding the presence, or lack thereof, of those values. + + +Values +------ + +All annotations are created by instaniating subclasses of the abstract :py:class:`pybryt.Annotation` +class. Consider the most basic kind of annotation: expecting a specific value to appear while +executing the student's code. To create a value annotation, create an instance of +:py:class:`pybryt.Value`. The constructor takes in the value that you are expecting to see in the +student's code: + +.. code-block:: python + + np.random.seed(42) + arr = np.random.normal(3, 2, size=100) + pybryt.Value(arr) + +Note that when an instance of :py:class:`pybryt.Value` is created, :py:meth:`copy.copy` is called on +the argument passed to it, so values don't need to worry about being affected by mutability. + + +Numerical Tolerance ++++++++++++++++++++ + +For numerical values, or iterables of numerical values that support vectorized math, it is also +possible to set an absolute tolerance for the acceptance of student values using the ``tol`` argument, +which defaults to zero. + +.. code-block:: python + + pybryt.Value(arr, tol=1e-4) + + +Invariants +++++++++++ + +PyBryt supports :ref:`invariants`, which are conditions that allow for objects with +different structures to be considered "equal" from PyBryt's perspective. To use invariants on a +value, pass the invariant objects in a list to the ``invariants`` argument of the constructor: + +.. code-block:: python + + correct_answer = "a CasE-inSensiTiVe stRING" + pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization]) + + +Relational Annotations +---------------------- + +Relational annotations define some kind of relationship between two or more annotations. Currently, +PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations. +All relational annotations are subclasses of the abstract +:py:class:`pybryt.BeforeAnnotation` class, which +defines some helpful defaults for working with annotations that have child annotations. + +Temporal Annotations +++++++++++++++++++++ + +Temporal annotations describe *when* variables should appear in students' code relative to one +another. For example, consider the problem of a dynamic programming algorithm to compute the +Fibonacci sequence: the array containing :math:`n-1` first Fibonacci numbers should appear in memory +before the array containing the :math:`n` first Fibonacci numbers. + +To enforce such a constraint, the :py:class:`pybryt.Annotation` class defines a ``before`` method +that asserts that one annotation occurs before another: + +.. code-block:: python + + def fib(n): + """ + Compute and return an array of the first n Fibonacci numbers using dynamic programming. + + Args: + n (``int``): the number of Fibonacci numbers to return + + Returns: + ``np.ndarray``: the first ``n`` Fibonacci numbers + """ + fibs = np.zeros(n, dtype=int) + + fibs[0] = 0 + curr_val = pybryt.Value(fibs) + if n == 1: + return fibs + + fibs[1] = 1 + v = pybryt.Value(fibs) + curr_val.before(v) + curr_val = v + if n == 2: + return fibs + + for i in range(2, n-1): + fibs[i] = fibs[i-1] + fibs[i-2] + + v = pybryt.Value(fibs) # array of first n Fibonacci numbrs + curr_val.before(v) # check that first n-1 Fib numbers come before first n + curr_val = v # update curr_val for next iteration + + return fibs + +In the example above, updating a pointer ``curr_val`` in the loop allows us to create a ``before`` +condition such that we ensure the student followed the correct dynamic programming algorithm by +checking each update to the ``fibs`` array. + +Temporal annotations are satisfied when the student's code satisfies all of the child +:py:class:`pybryt.Value` annotations and when the first annotation (the one calling +:py:meth:`pybryt.Annotation.before`) has a timestamp greater than or equal to the timestamp of the +second annotation. + +Note that :py:meth:`pybryt.Annotation.before` returns an instance of the +:py:class:`pybryt.BeforeAnnotation` class, which is +itself a subclass of :py:class:`pybryt.Annotation` and supports all of the same operations. +:py:class:`pybryt.Annotation` also provides :py:meth:`pybryt.Annotation.after`, which also returns +and instance of the +:py:class:`pybryt.BeforeAnnotation` class, but with +the operands switched. + + +Boolean Annotations ++++++++++++++++++++ + +Boolean annotations define conditions on the presence of different values. For example, in defining +a solutions, students may be able to take two different paths, and this logic can be enforced +using a :py:class:`pybryt.XorAnnotation` to ensure that +only one of the two possible values is present. + +Relational annotations can be created either by instantiating the classes directly using the +constructor or, as is more recommended, by using Python's bitwise logical operators, ``&``, ``|``, +``^``, and ``~``, on annotations. The dunder methods for these operators have been overrided with +for the :py:class:`pybryt.Annotation` class, and return the +:py:class:`pybryt.RelationalAnnotation` subclass +instance corresponding to the logical operator used. + +To create the xor example above from two values ``v1`` and ``v2``, simply write + +.. code-block:: python + + v1 ^ v2 + +To assert that a student should *not* have a specific value ``v`` in their code, use + +.. code-block:: python + + ~v diff --git a/docs/reference_implementations/index.rst b/docs/reference_implementations/index.rst new file mode 100644 index 0000000..f3b612b --- /dev/null +++ b/docs/reference_implementations/index.rst @@ -0,0 +1,23 @@ +Reference Implementations +========================= + +.. toctree:: + :hidden: + :maxdepth: 3 + + annotations + invariants + reference_objects + +PyBryt's core auto-assessment behavior operates by comparing a student's implementation of some +programming problem to a series of reference implementations provided by an instructor. A +**reference implementation** defines a pattern of values, and conditions on those values, to look +for in students' code. + +A reference implementation is created by annotating code written or found by an instructor and +executing this code to create a :py:class:`pybryt.ReferenceImplementation` object. Annotations are +created by creating instances of subclasses of the abstract :py:class:`pybryt.Annotation` class. + +This section details the creation and behavior of the different annotation classes that PyBryt +provides and describes how reference implementations as a single unit are created, manipulated, +stored, and run. diff --git a/docs/reference_implementations/invariants.rst b/docs/reference_implementations/invariants.rst new file mode 100644 index 0000000..7f780ee --- /dev/null +++ b/docs/reference_implementations/invariants.rst @@ -0,0 +1,4 @@ +.. _invariants: + +Invariants +========== diff --git a/docs/reference_implementations/reference_objects.rst b/docs/reference_implementations/reference_objects.rst new file mode 100644 index 0000000..46a23b7 --- /dev/null +++ b/docs/reference_implementations/reference_objects.rst @@ -0,0 +1,9 @@ +Reference Implementation Objects +================================ + +The ``__init__`` methods of these subclasses automatically add the instances created to a +singleton list that PyBryt maintains, so assigning them to variables or tracking them further is +unnecessary unless more advanced reference implementations are being built. This means that when +marking up code, as below, creating new variables is unnecessary unless further conditions are to +be made later down the line. + diff --git a/pybryt/annotations/__init__.py b/pybryt/annotations/__init__.py index 9b40719..c85b72b 100644 --- a/pybryt/annotations/__init__.py +++ b/pybryt/annotations/__init__.py @@ -2,6 +2,7 @@ """ from . import invariants +from .annotation import Annotation, AnnotationResult from .function import Function from .reference import ReferenceImplementation, ReferenceResult from .relation import *