updated docs structure

docs/annotations/index.rst

@@ -0,0 +1,65 @@
+.. _annotations:
+
+Annotations
+===========
+
+.. toctree::
+   :maxdepth: 3
+   :hidden:
+
+   relational_annotations
+   invariants
+
+Annotations are the building blocks out of which reference implementations are constructed. They 
+annotations represent a single condition that a student's implementation should meet, and define a 
+set of behaviors for responding to the passing or failing of those conditions. Annotations provide
+conditions not only for expecting a specific value but also for combining those expectations to form
+conditions on the structure of students' code, including the temporal relationship of values and
+complex boolean logic surrounding the presence, or lack thereof, of those values.
+
+All annotations are created by instaniating subclasses of the abstract 
+:py:class:`Annotation<pybryt.Annotation>` class. There are two main types of annotations: value 
+annotations, described below, and :ref:`relational annotations<relational>`.
+
+
+Values
+------
+
+Consider the most basic kind of annotation: expecting a specific value to appear while executing the 
+student's code. To create a value annotation, create an instance of :py:class:`Value<pybryt.Value>`. 
+The constructor takes in the value that you are expecting to see in the student's code:
+
+.. code-block:: python
+
+   np.random.seed(42)
+   arr = np.random.normal(3, 2, size=100)
+   pybryt.Value(arr)
+
+Note that when an instance of :py:class:`Value<pybryt.Value>` is created, :py:meth:`copy.copy` is 
+called on the argument passed to it, so values don't need to worry about being affected by 
+mutability.
+
+
+Numerical Tolerance
++++++++++++++++++++
+
+For numerical values, or iterables of numerical values that support vectorized math, it is also 
+possible to set an absolute tolerance for the acceptance of student values using the ``tol`` 
+argument, which defaults to zero.
+
+.. code-block:: python
+
+   pybryt.Value(arr, tol=1e-4)
+
+
+Invariants
+++++++++++
+
+PyBryt supports :ref:`invariants<invariants>`, which are conditions that allow for objects with 
+different structures to be considered "equal" from PyBryt's perspective. To use invariants on a 
+value, pass the invariant objects in a list to the ``invariants`` argument of the constructor:
+
+.. code-block:: python
+
+   correct_answer = "a CasE-inSensiTiVe stRING"
+   pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])

docs/annotations/relational_annotations.rst

@@ -1,63 +1,7 @@
-.. _annotations:
+.. _relational:
-
-Annotations
-===========
-
-Annotations are the building blocks out of which reference implementations are constructed. They 
-annotations represent a single condition that a student's implementation should meet, and define a 
-set of behaviors for responding to the passing or failing of those conditions. Annotations provide
-conditions not only for expecting a specific value but also for combining those expectations to form
-conditions on the structure of students' code, including the temporal relationship of values and
-complex boolean logic surrounding the presence, or lack thereof, of those values.
-
-
-Values
-------
-
-All annotations are created by instaniating subclasses of the abstract 
-:py:class:`Annotation<pybryt.Annotation>` class. Consider the most basic kind of annotation: 
-expecting a specific value to appear while executing the student's code. To create a value 
-annotation, create an instance of :py:class:`Value<pybryt.Value>`. The constructor takes in the 
-value that you are expecting to see in the student's code:
-
-.. code-block:: python
-
-   np.random.seed(42)
-   arr = np.random.normal(3, 2, size=100)
-   pybryt.Value(arr)
-
-Note that when an instance of :py:class:`Value<pybryt.Value>` is created, :py:meth:`copy.copy` is 
-called on the argument passed to it, so values don't need to worry about being affected by 
-mutability.
-
-
-Numerical Tolerance
-+++++++++++++++++++
-
-For numerical values, or iterables of numerical values that support vectorized math, it is also 
-possible to set an absolute tolerance for the acceptance of student values using the ``tol`` 
-argument, which defaults to zero.
-
-.. code-block:: python
-
-   pybryt.Value(arr, tol=1e-4)
-
-
-Invariants
-++++++++++
-
-PyBryt supports :ref:`invariants<invariants>`, which are conditions that allow for objects with 
-different structures to be considered "equal" from PyBryt's perspective. To use invariants on a 
-value, pass the invariant objects in a list to the ``invariants`` argument of the constructor:
-
-.. code-block:: python
-
-   correct_answer = "a CasE-inSensiTiVe stRING"
-   pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
-
 
 Relational Annotations
-----------------------
+======================
 
 Relational annotations define some kind of relationship between two or more annotations. Currently,
 PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations. 
@@ -66,7 +10,7 @@ All relational annotations are subclasses of the abstract
 defines some helpful defaults for working with annotations that have child annotations.
 
 Temporal Annotations
-++++++++++++++++++++
+--------------------
 
 Temporal annotations describe *when* variables should appear in students' code relative to one 
 another. For example, consider the problem of a dynamic programming algorithm to compute the 
@@ -130,7 +74,7 @@ the operands switched.
 
 
 Boolean Annotations
-+++++++++++++++++++
+-------------------
 
 Boolean annotations define conditions on the presence of different values. For example, in defining
 a solutions, students may be able to take two different paths, and this logic can be enforced 

docs/html/_sources/annotations/index.rst.txt

@@ -0,0 +1,65 @@
+.. _annotations:
+
+Annotations
+===========
+
+.. toctree::
+   :maxdepth: 3
+   :hidden:
+
+   relational_annotations
+   invariants
+
+Annotations are the building blocks out of which reference implementations are constructed. They 
+annotations represent a single condition that a student's implementation should meet, and define a 
+set of behaviors for responding to the passing or failing of those conditions. Annotations provide
+conditions not only for expecting a specific value but also for combining those expectations to form
+conditions on the structure of students' code, including the temporal relationship of values and
+complex boolean logic surrounding the presence, or lack thereof, of those values.
+
+All annotations are created by instaniating subclasses of the abstract 
+:py:class:`Annotation<pybryt.Annotation>` class. There are two main types of annotations: value 
+annotations, described below, and :ref:`relational annotations<relational>`.
+
+
+Values
+------
+
+Consider the most basic kind of annotation: expecting a specific value to appear while executing the 
+student's code. To create a value annotation, create an instance of :py:class:`Value<pybryt.Value>`. 
+The constructor takes in the value that you are expecting to see in the student's code:
+
+.. code-block:: python
+
+   np.random.seed(42)
+   arr = np.random.normal(3, 2, size=100)
+   pybryt.Value(arr)
+
+Note that when an instance of :py:class:`Value<pybryt.Value>` is created, :py:meth:`copy.copy` is 
+called on the argument passed to it, so values don't need to worry about being affected by 
+mutability.
+
+
+Numerical Tolerance
++++++++++++++++++++
+
+For numerical values, or iterables of numerical values that support vectorized math, it is also 
+possible to set an absolute tolerance for the acceptance of student values using the ``tol`` 
+argument, which defaults to zero.
+
+.. code-block:: python
+
+   pybryt.Value(arr, tol=1e-4)
+
+
+Invariants
+++++++++++
+
+PyBryt supports :ref:`invariants<invariants>`, which are conditions that allow for objects with 
+different structures to be considered "equal" from PyBryt's perspective. To use invariants on a 
+value, pass the invariant objects in a list to the ``invariants`` argument of the constructor:
+
+.. code-block:: python
+
+   correct_answer = "a CasE-inSensiTiVe stRING"
+   pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])

docs/html/_sources/annotations/invariants.rst.txt

@@ -0,0 +1,4 @@
+.. _invariants:
+
+Invariants
+==========

docs/html/_sources/annotations/relational_annotations.rst.txt

@@ -0,0 +1,101 @@
+.. _relational:
+
+Relational Annotations
+======================
+
+Relational annotations define some kind of relationship between two or more annotations. Currently,
+PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations. 
+All relational annotations are subclasses of the abstract 
+:py:class:`RelationalAnnotation<pybryt.annotations.relation.RelationalAnnotation>` class, which 
+defines some helpful defaults for working with annotations that have child annotations.
+
+Temporal Annotations
+--------------------
+
+Temporal annotations describe *when* variables should appear in students' code relative to one 
+another. For example, consider the problem of a dynamic programming algorithm to compute the 
+Fibonacci sequence: the array containing :math:`n-1` first Fibonacci numbers should appear in memory 
+before the array containing the :math:`n` first Fibonacci numbers. 
+
+To enforce such a constraint, the :py:class:`Annotation<pybryt.Annotation>` class defines a 
+``before`` method that asserts that one annotation occurs before another:
+
+.. code-block:: python
+
+    def fib(n):
+        """
+        Compute and return an array of the first n Fibonacci numbers using dynamic programming.
+
+        Args:
+        n (``int``): the number of Fibonacci numbers to return
+
+        Returns:
+        ``np.ndarray``: the first ``n`` Fibonacci numbers
+        """
+        fibs = np.zeros(n, dtype=int)
+
+        fibs[0] = 0
+        curr_val = pybryt.Value(fibs)
+        if n == 1:
+            return fibs
+
+        fibs[1] = 1
+        v = pybryt.Value(fibs)
+        curr_val.before(v)
+        curr_val = v
+        if n == 2:
+            return fibs
+
+        for i in range(2, n-1):
+            fibs[i] = fibs[i-1] + fibs[i-2]
+
+            v = pybryt.Value(fibs) # array of first n Fibonacci numbrs
+            curr_val.before(v)     # check that first n-1 Fib numbers come before first n
+            curr_val = v           # update curr_val for next iteration
+
+        return fibs
+
+In the example above, updating a pointer ``curr_val`` in the loop allows us to create a ``before`` 
+condition such that we ensure the student followed the correct dynamic programming algorithm by 
+checking each update to the ``fibs`` array.
+
+Temporal annotations are satisfied when the student's code satisfies all of the child 
+:py:class:`Value<pybryt.Value>` annotations and when the first annotation (the one calling 
+:py:meth:`Annotation.before<pybryt.Annotation.before>`) has a timestamp greater than or equal to the 
+timestamp of the second annotation.
+
+Note that :py:meth:`Annotation.before<pybryt.Annotation.before>` returns an instance of the 
+:py:class:`BeforeAnnotation<pybryt.annotations.relation.BeforeAnnotation>` class, which is 
+itself a subclass of :py:class:`Annotation<pybryt.Annotation>` and supports all of the same operations. 
+:py:class:`Annotation<pybryt.Annotation>` also provides 
+:py:meth:`Annotation.after<pybryt.Annotation.after>`, which also returns an instance of the 
+:py:class:`BeforeAnnotation<pybryt.annotations.relation.BeforeAnnotation>` class, but with 
+the operands switched.
+
+
+Boolean Annotations
+-------------------
+
+Boolean annotations define conditions on the presence of different values. For example, in defining
+a solutions, students may be able to take two different paths, and this logic can be enforced 
+using a :py:class:`XorAnnotation<pybryt.annotations.relation.XorAnnotation>` to ensure that
+only one of the two possible values is present.
+
+Relational annotations can be created either by instantiating the classes directly using the 
+constructor or, as is more recommended, by using Python's bitwise logical operators, ``&``, ``|``, 
+``^``, and ``~``, on annotations. The dunder methods for these operators have been overrided with 
+for the :py:class:`Annotation<pybryt.Annotation>` class, and return the 
+:py:class:`RelationalAnnotation<pybryt.annotations.relation.RelationalAnnotation>` subclass
+instance corresponding to the logical operator used.
+
+To create the xor example above from two values ``v1`` and ``v2``, simply write
+
+.. code-block:: python
+
+   v1 ^ v2
+
+To assert that a student should *not* have a specific value ``v`` in their code, use
+
+.. code-block:: python
+
+   ~v

docs/html/_sources/index.rst.txt

@@ -10,7 +10,8 @@ PyBryt Documentation
    :maxdepth: 3
    :hidden:
 
-   reference_implementations/index
+   annotations/index
+   reference_implementations
    api_reference
 
 PyBryt is an open source  Python auto-assessment library for teaching and learning. Our goal is to empower students and educators to learn about technology through fun, guided, hands-on content aimed at specific learning goals. PyBryt is designed to work with existing autograding solutions and workflows such as `Otter Grader`_, `OkPy`_, and `Autolab`_.
@@ -26,6 +27,16 @@ Educators and institutions can leverage PyBryt to integrate auto-assessment and
 - Plagiarism detection and support for reference implementations
 - Easy integration into existing organizational or institutional grading infrastructure
 
+PyBryt's core auto-assessment behavior operates by comparing a student's implementation of some 
+programming problem to a series of reference implementations provided by an instructor. A 
+**reference implementation** defines a pattern of values, and conditions on those values, to look 
+for in students' code.
+
+A reference implementation is created by annotating code written or found by an instructor and 
+executing this code to create a :py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>` 
+object. Annotations are created by creating instances of subclasses of the abstract 
+:py:class:`Annotation<pybryt.Annotation>` class.
+
 .. _Otter Grader: https://otter-grader.readthedocs.io
 .. _OkPy: https://okpy.org
 .. _Autolab: https://autolab.readthedocs.io/

docs/html/_sources/reference_implementations.rst.txt

@@ -1,5 +1,5 @@
-Reference Implementation Objects
+Reference Implementations
-================================
+=========================
 
 The functional unit of PyBryt is a reference implementation. A **reference implenetation** is a set 
 of conditions expected of students' code that determine whether a student has correctly implemented

docs/html/annotations/index.html

@@ -0,0 +1,258 @@
+<!doctype html>
+<html class="no-js">
+  <head><meta charset="utf-8"/>
+    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="../genindex.html" /><link rel="search" title="Search" href="../search.html" /><link rel="next" title="Relational Annotations" href="relational_annotations.html" /><link rel="prev" title="PyBryt Documentation" href="../index.html" />
+
+    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+        <title>Annotations - PyBryt documentation</title>
+      <link rel="stylesheet" href="../_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
+    <link rel="stylesheet" href="../_static/pygments.css">
+    <link media="(prefers-color-scheme: dark)" rel="stylesheet" href="../_static/pygments_dark.css">
+    
+
+
+<style>
+  :root {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  @media (prefers-color-scheme: dark) {
+    :root {
+      --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+    }
+  }
+
+  /* For allowing end-user-specific overrides */
+  .override-light {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  .override-dark {
+    --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+  }
+</style><link rel="stylesheet" href="../_static/styles/furo-extensions.css?digest=d391b54134226e4196576da3bdb6dddb7e05ba2b"></head>
+  <body dir="">
+    
+<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
+  <symbol id="svg-toc" viewBox="0 0 24 24">
+    <title>Contents</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke-width="1.5" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round">
+      <path stroke="none" d="M0 0h24v24H0z"/>
+      <line x1="4" y1="6" x2="20" y2="6" />
+      <line x1="10" y1="12" x2="20" y2="12" />
+      <line x1="6" y1="18" x2="20" y2="18" />
+    </svg>
+  </symbol>
+  <symbol id="svg-menu" viewBox="0 0 24 24">
+    <title>Menu</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-menu">
+      <line x1="3" y1="12" x2="21" y2="12"></line>
+      <line x1="3" y1="6" x2="21" y2="6"></line>
+      <line x1="3" y1="18" x2="21" y2="18"></line>
+    </svg>
+  </symbol>
+  <symbol id="svg-arrow-right" viewBox="0 0 24 24">
+    <title>Expand</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-chevron-right">
+      <polyline points="9 18 15 12 9 6"></polyline>
+    </svg>
+  </symbol>
+</svg>
+
+<input type="checkbox" class="sidebar-toggle" name="__navigation" id="__navigation">
+<input type="checkbox" class="sidebar-toggle" name="__toc" id="__toc">
+<label class="overlay sidebar-overlay" for="__navigation"></label>
+<label class="overlay toc-overlay" for="__toc"></label>
+
+
+
+<div class="page">
+  <header class="mobile-header">
+    <div class="header-left">
+      <label class="nav-overlay-icon" for="__navigation">
+        <i class="icon"><svg><use href="#svg-menu"></use></svg></i>
+      </label>
+    </div>
+    <div class="header-center">
+      <a href="../index.html"><div class="brand">PyBryt  documentation</div></a>
+    </div>
+    <div class="header-right">
+      <label class="toc-overlay-icon toc-header-icon" for="__toc">
+        <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+      </label>
+    </div>
+  </header>
+  <aside class="sidebar-drawer">
+    <div class="sidebar-container">
+      
+      <div class="sidebar-sticky"><a class="sidebar-brand" href="../index.html">
+  
+  
+  <span class="sidebar-brand-text">PyBryt  documentation</span>
+  
+</a><form class="sidebar-search-container" method="get" action="../search.html">
+  <input class="sidebar-search" placeholder=Search name="q">
+  <input type="hidden" name="check_keywords" value="yes">
+  <input type="hidden" name="area" value="default">
+</form><div class="sidebar-scroll"><div class="sidebar-tree">
+  <ul class="current">
+<li class="toctree-l1 current has-children current-page"><a class="current reference internal" href="#">Annotations</a><input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
+<li class="toctree-l2"><a class="reference internal" href="relational_annotations.html">Relational Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="invariants.html">Invariants</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../reference_implementations.html">Reference Implementations</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api_reference.html">API Reference</a></li>
+</ul>
+
+</div>
+</div>
+      </div>
+      
+    </div>
+  </aside>
+  <main class="main">
+    <div class="content">
+      <article role="main">
+        <label class="toc-overlay-icon toc-content-icon" for="__toc">
+          <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+        </label>
+        <div class="section" id="annotations">
+<span id="id1"></span><h1>Annotations<a class="headerlink" href="#annotations" title="Permalink to this headline">¶</a></h1>
+<div class="toctree-wrapper compound">
+</div>
+<p>Annotations are the building blocks out of which reference implementations are constructed. They
+annotations represent a single condition that a student’s implementation should meet, and define a
+set of behaviors for responding to the passing or failing of those conditions. Annotations provide
+conditions not only for expecting a specific value but also for combining those expectations to form
+conditions on the structure of students’ code, including the temporal relationship of values and
+complex boolean logic surrounding the presence, or lack thereof, of those values.</p>
+<p>All annotations are created by instaniating subclasses of the abstract
+<a class="reference internal" href="../api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> class. There are two main types of annotations: value
+annotations, described below, and <a class="reference internal" href="relational_annotations.html#relational"><span class="std std-ref">relational annotations</span></a>.</p>
+<div class="section" id="values">
+<h2>Values<a class="headerlink" href="#values" title="Permalink to this headline">¶</a></h2>
+<p>Consider the most basic kind of annotation: expecting a specific value to appear while executing the
+student’s code. To create a value annotation, create an instance of <code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code>.
+The constructor takes in the value that you are expecting to see in the student’s code:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
+<span class="n">arr</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
+<span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">arr</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Note that when an instance of <code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code> is created, <code class="xref py py-meth docutils literal notranslate"><span class="pre">copy.copy()</span></code> is
+called on the argument passed to it, so values don’t need to worry about being affected by
+mutability.</p>
+<div class="section" id="numerical-tolerance">
+<h3>Numerical Tolerance<a class="headerlink" href="#numerical-tolerance" title="Permalink to this headline">¶</a></h3>
+<p>For numerical values, or iterables of numerical values that support vectorized math, it is also
+possible to set an absolute tolerance for the acceptance of student values using the <code class="docutils literal notranslate"><span class="pre">tol</span></code>
+argument, which defaults to zero.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">arr</span><span class="p">,</span> <span class="n">tol</span><span class="o">=</span><span class="mf">1e-4</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="invariants">
+<h3>Invariants<a class="headerlink" href="#invariants" title="Permalink to this headline">¶</a></h3>
+<p>PyBryt supports <a class="reference internal" href="invariants.html#invariants"><span class="std std-ref">invariants</span></a>, which are conditions that allow for objects with
+different structures to be considered “equal” from PyBryt’s perspective. To use invariants on a
+value, pass the invariant objects in a list to the <code class="docutils literal notranslate"><span class="pre">invariants</span></code> argument of the constructor:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">correct_answer</span> <span class="o">=</span> <span class="s2">"a CasE-inSensiTiVe stRING"</span>
+<span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">correct_answer</span><span class="p">,</span> <span class="n">invariants</span><span class="o">=</span><span class="p">[</span><span class="n">pybryt</span><span class="o">.</span><span class="n">invariants</span><span class="o">.</span><span class="n">string_capitalization</span><span class="p">])</span>
+</pre></div>
+</div>
+</div>
+</div>
+</div>
+
+      </article>
+      <footer>
+        
+        <div class="related-pages">
+          <a class="next-page" href="relational_annotations.html">
+              <div class="page-info">
+                <div class="context">
+                  <span>Next</span>
+                </div>
+                <div class="title">Relational Annotations</div>
+              </div>
+              <svg><use href="#svg-arrow-right"></use></svg>
+            </a>
+          <a class="prev-page" href="../index.html">
+              <svg><use href="#svg-arrow-right"></use></svg>
+              <div class="page-info">
+                <div class="context">
+                  <span>Previous</span>
+                </div>
+                
+                <div class="title">Home</div>
+                
+              </div>
+            </a>
+        </div>
+
+        <div class="related-information">
+              Copyright &#169; 2021, Microsoft
+            |
+            Built with <a href="https://www.sphinx-doc.org/">Sphinx</a>
+              and
+              <a class="muted-link" href="https://pradyunsg.me">@pradyunsg</a>'s
+              <a href="https://github.com/pradyunsg/furo">Furo theme</a>.
+            |
+            <a class="muted-link" href="../_sources/annotations/index.rst.txt"
+               rel="nofollow">
+              Show Source
+            </a>
+        </div>
+        
+      </footer>
+    </div>
+    <aside class="toc-drawer">
+      
+      
+      <div class="toc-sticky toc-scroll">
+        <div class="toc-title-container">
+          <span class="toc-title">
+            Contents
+          </span>
+        </div>
+        <div class="toc-tree-container">
+          <div class="toc-tree">
+            <ul>
+<li><a class="reference internal" href="#">Annotations</a><ul>
+<li><a class="reference internal" href="#values">Values</a><ul>
+<li><a class="reference internal" href="#numerical-tolerance">Numerical Tolerance</a></li>
+<li><a class="reference internal" href="#invariants">Invariants</a></li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+
+          </div>
+        </div>
+      </div>
+      
+      
+    </aside>
+  </main>
+</div>
+    <script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
+    <script src="../_static/jquery.js"></script>
+    <script src="../_static/underscore.js"></script>
+    <script src="../_static/doctools.js"></script>
+    <script src="../_static/language_data.js"></script>
+    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
+    <script src="../_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
+</html>

docs/html/annotations/invariants.html

@@ -0,0 +1,191 @@
+<!doctype html>
+<html class="no-js">
+  <head><meta charset="utf-8"/>
+    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="../genindex.html" /><link rel="search" title="Search" href="../search.html" /><link rel="next" title="Reference Implementations" href="../reference_implementations.html" /><link rel="prev" title="Relational Annotations" href="relational_annotations.html" />
+
+    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+        <title>Invariants - PyBryt documentation</title>
+      <link rel="stylesheet" href="../_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
+    <link rel="stylesheet" href="../_static/pygments.css">
+    <link media="(prefers-color-scheme: dark)" rel="stylesheet" href="../_static/pygments_dark.css">
+    
+
+
+<style>
+  :root {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  @media (prefers-color-scheme: dark) {
+    :root {
+      --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+    }
+  }
+
+  /* For allowing end-user-specific overrides */
+  .override-light {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  .override-dark {
+    --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+  }
+</style><link rel="stylesheet" href="../_static/styles/furo-extensions.css?digest=d391b54134226e4196576da3bdb6dddb7e05ba2b"></head>
+  <body dir="">
+    
+<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
+  <symbol id="svg-toc" viewBox="0 0 24 24">
+    <title>Contents</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke-width="1.5" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round">
+      <path stroke="none" d="M0 0h24v24H0z"/>
+      <line x1="4" y1="6" x2="20" y2="6" />
+      <line x1="10" y1="12" x2="20" y2="12" />
+      <line x1="6" y1="18" x2="20" y2="18" />
+    </svg>
+  </symbol>
+  <symbol id="svg-menu" viewBox="0 0 24 24">
+    <title>Menu</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-menu">
+      <line x1="3" y1="12" x2="21" y2="12"></line>
+      <line x1="3" y1="6" x2="21" y2="6"></line>
+      <line x1="3" y1="18" x2="21" y2="18"></line>
+    </svg>
+  </symbol>
+  <symbol id="svg-arrow-right" viewBox="0 0 24 24">
+    <title>Expand</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-chevron-right">
+      <polyline points="9 18 15 12 9 6"></polyline>
+    </svg>
+  </symbol>
+</svg>
+
+<input type="checkbox" class="sidebar-toggle" name="__navigation" id="__navigation">
+<input type="checkbox" class="sidebar-toggle" name="__toc" id="__toc">
+<label class="overlay sidebar-overlay" for="__navigation"></label>
+<label class="overlay toc-overlay" for="__toc"></label>
+
+
+
+<div class="page">
+  <header class="mobile-header">
+    <div class="header-left">
+      <label class="nav-overlay-icon" for="__navigation">
+        <i class="icon"><svg><use href="#svg-menu"></use></svg></i>
+      </label>
+    </div>
+    <div class="header-center">
+      <a href="../index.html"><div class="brand">PyBryt  documentation</div></a>
+    </div>
+    <div class="header-right">
+      <label class="toc-overlay-icon toc-header-icon no-toc" for="__toc">
+        <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+      </label>
+    </div>
+  </header>
+  <aside class="sidebar-drawer">
+    <div class="sidebar-container">
+      
+      <div class="sidebar-sticky"><a class="sidebar-brand" href="../index.html">
+  
+  
+  <span class="sidebar-brand-text">PyBryt  documentation</span>
+  
+</a><form class="sidebar-search-container" method="get" action="../search.html">
+  <input class="sidebar-search" placeholder=Search name="q">
+  <input type="hidden" name="check_keywords" value="yes">
+  <input type="hidden" name="area" value="default">
+</form><div class="sidebar-scroll"><div class="sidebar-tree">
+  <ul class="current">
+<li class="toctree-l1 current has-children"><a class="reference internal" href="index.html">Annotations</a><input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="relational_annotations.html">Relational Annotations</a></li>
+<li class="toctree-l2 current current-page"><a class="current reference internal" href="#">Invariants</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../reference_implementations.html">Reference Implementations</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api_reference.html">API Reference</a></li>
+</ul>
+
+</div>
+</div>
+      </div>
+      
+    </div>
+  </aside>
+  <main class="main">
+    <div class="content">
+      <article role="main">
+        <label class="toc-overlay-icon toc-content-icon no-toc" for="__toc">
+          <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+        </label>
+        <div class="section" id="invariants">
+<span id="id1"></span><h1>Invariants<a class="headerlink" href="#invariants" title="Permalink to this headline">¶</a></h1>
+</div>
+
+      </article>
+      <footer>
+        
+        <div class="related-pages">
+          <a class="next-page" href="../reference_implementations.html">
+              <div class="page-info">
+                <div class="context">
+                  <span>Next</span>
+                </div>
+                <div class="title">Reference Implementations</div>
+              </div>
+              <svg><use href="#svg-arrow-right"></use></svg>
+            </a>
+          <a class="prev-page" href="relational_annotations.html">
+              <svg><use href="#svg-arrow-right"></use></svg>
+              <div class="page-info">
+                <div class="context">
+                  <span>Previous</span>
+                </div>
+                
+                <div class="title">Relational Annotations</div>
+                
+              </div>
+            </a>
+        </div>
+
+        <div class="related-information">
+              Copyright &#169; 2021, Microsoft
+            |
+            Built with <a href="https://www.sphinx-doc.org/">Sphinx</a>
+              and
+              <a class="muted-link" href="https://pradyunsg.me">@pradyunsg</a>'s
+              <a href="https://github.com/pradyunsg/furo">Furo theme</a>.
+            |
+            <a class="muted-link" href="../_sources/annotations/invariants.rst.txt"
+               rel="nofollow">
+              Show Source
+            </a>
+        </div>
+        
+      </footer>
+    </div>
+    <aside class="toc-drawer no-toc">
+      
+      
+      
+    </aside>
+  </main>
+</div>
+    <script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
+    <script src="../_static/jquery.js"></script>
+    <script src="../_static/underscore.js"></script>
+    <script src="../_static/doctools.js"></script>
+    <script src="../_static/language_data.js"></script>
+    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
+    <script src="../_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
+</html>

docs/html/annotations/relational_annotations.html

@@ -0,0 +1,294 @@
+<!doctype html>
+<html class="no-js">
+  <head><meta charset="utf-8"/>
+    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="../genindex.html" /><link rel="search" title="Search" href="../search.html" /><link rel="next" title="Invariants" href="invariants.html" /><link rel="prev" title="Annotations" href="index.html" />
+
+    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+        <title>Relational Annotations - PyBryt documentation</title>
+      <link rel="stylesheet" href="../_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
+    <link rel="stylesheet" href="../_static/pygments.css">
+    <link media="(prefers-color-scheme: dark)" rel="stylesheet" href="../_static/pygments_dark.css">
+    
+
+
+<style>
+  :root {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  @media (prefers-color-scheme: dark) {
+    :root {
+      --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+    }
+  }
+
+  /* For allowing end-user-specific overrides */
+  .override-light {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  .override-dark {
+    --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+  }
+</style><link rel="stylesheet" href="../_static/styles/furo-extensions.css?digest=d391b54134226e4196576da3bdb6dddb7e05ba2b"></head>
+  <body dir="">
+    
+<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
+  <symbol id="svg-toc" viewBox="0 0 24 24">
+    <title>Contents</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke-width="1.5" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round">
+      <path stroke="none" d="M0 0h24v24H0z"/>
+      <line x1="4" y1="6" x2="20" y2="6" />
+      <line x1="10" y1="12" x2="20" y2="12" />
+      <line x1="6" y1="18" x2="20" y2="18" />
+    </svg>
+  </symbol>
+  <symbol id="svg-menu" viewBox="0 0 24 24">
+    <title>Menu</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-menu">
+      <line x1="3" y1="12" x2="21" y2="12"></line>
+      <line x1="3" y1="6" x2="21" y2="6"></line>
+      <line x1="3" y1="18" x2="21" y2="18"></line>
+    </svg>
+  </symbol>
+  <symbol id="svg-arrow-right" viewBox="0 0 24 24">
+    <title>Expand</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-chevron-right">
+      <polyline points="9 18 15 12 9 6"></polyline>
+    </svg>
+  </symbol>
+</svg>
+
+<input type="checkbox" class="sidebar-toggle" name="__navigation" id="__navigation">
+<input type="checkbox" class="sidebar-toggle" name="__toc" id="__toc">
+<label class="overlay sidebar-overlay" for="__navigation"></label>
+<label class="overlay toc-overlay" for="__toc"></label>
+
+
+
+<div class="page">
+  <header class="mobile-header">
+    <div class="header-left">
+      <label class="nav-overlay-icon" for="__navigation">
+        <i class="icon"><svg><use href="#svg-menu"></use></svg></i>
+      </label>
+    </div>
+    <div class="header-center">
+      <a href="../index.html"><div class="brand">PyBryt  documentation</div></a>
+    </div>
+    <div class="header-right">
+      <label class="toc-overlay-icon toc-header-icon" for="__toc">
+        <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+      </label>
+    </div>
+  </header>
+  <aside class="sidebar-drawer">
+    <div class="sidebar-container">
+      
+      <div class="sidebar-sticky"><a class="sidebar-brand" href="../index.html">
+  
+  
+  <span class="sidebar-brand-text">PyBryt  documentation</span>
+  
+</a><form class="sidebar-search-container" method="get" action="../search.html">
+  <input class="sidebar-search" placeholder=Search name="q">
+  <input type="hidden" name="check_keywords" value="yes">
+  <input type="hidden" name="area" value="default">
+</form><div class="sidebar-scroll"><div class="sidebar-tree">
+  <ul class="current">
+<li class="toctree-l1 current has-children"><a class="reference internal" href="index.html">Annotations</a><input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul class="current">
+<li class="toctree-l2 current current-page"><a class="current reference internal" href="#">Relational Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="invariants.html">Invariants</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../reference_implementations.html">Reference Implementations</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api_reference.html">API Reference</a></li>
+</ul>
+
+</div>
+</div>
+      </div>
+      
+    </div>
+  </aside>
+  <main class="main">
+    <div class="content">
+      <article role="main">
+        <label class="toc-overlay-icon toc-content-icon" for="__toc">
+          <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+        </label>
+        <div class="section" id="relational-annotations">
+<span id="relational"></span><h1>Relational Annotations<a class="headerlink" href="#relational-annotations" title="Permalink to this headline">¶</a></h1>
+<p>Relational annotations define some kind of relationship between two or more annotations. Currently,
+PyBryt supports two kinds of relational annotations: temporal annotations and boolean annotations.
+All relational annotations are subclasses of the abstract
+<a class="reference internal" href="../api_reference.html#pybryt.annotations.relation.RelationalAnnotation" title="pybryt.annotations.relation.RelationalAnnotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">RelationalAnnotation</span></code></a> class, which
+defines some helpful defaults for working with annotations that have child annotations.</p>
+<div class="section" id="temporal-annotations">
+<h2>Temporal Annotations<a class="headerlink" href="#temporal-annotations" title="Permalink to this headline">¶</a></h2>
+<p>Temporal annotations describe <em>when</em> variables should appear in students’ code relative to one
+another. For example, consider the problem of a dynamic programming algorithm to compute the
+Fibonacci sequence: the array containing <span class="math notranslate nohighlight">\(n-1\)</span> first Fibonacci numbers should appear in memory
+before the array containing the <span class="math notranslate nohighlight">\(n\)</span> first Fibonacci numbers.</p>
+<p>To enforce such a constraint, the <a class="reference internal" href="../api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> class defines a
+<code class="docutils literal notranslate"><span class="pre">before</span></code> method that asserts that one annotation occurs before another:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
+    <span class="sd">"""</span>
+<span class="sd">    Compute and return an array of the first n Fibonacci numbers using dynamic programming.</span>
+
+<span class="sd">    Args:</span>
+<span class="sd">    n (``int``): the number of Fibonacci numbers to return</span>
+
+<span class="sd">    Returns:</span>
+<span class="sd">    ``np.ndarray``: the first ``n`` Fibonacci numbers</span>
+<span class="sd">    """</span>
+    <span class="n">fibs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
+
+    <span class="n">fibs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="n">curr_val</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+    <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+        <span class="k">return</span> <span class="n">fibs</span>
+
+    <span class="n">fibs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
+    <span class="n">v</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+    <span class="n">curr_val</span><span class="o">.</span><span class="n">before</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>
+    <span class="n">curr_val</span> <span class="o">=</span> <span class="n">v</span>
+    <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span>
+        <span class="k">return</span> <span class="n">fibs</span>
+
+    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
+        <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
+
+        <span class="n">v</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span> <span class="c1"># array of first n Fibonacci numbrs</span>
+        <span class="n">curr_val</span><span class="o">.</span><span class="n">before</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>     <span class="c1"># check that first n-1 Fib numbers come before first n</span>
+        <span class="n">curr_val</span> <span class="o">=</span> <span class="n">v</span>           <span class="c1"># update curr_val for next iteration</span>
+
+    <span class="k">return</span> <span class="n">fibs</span>
+</pre></div>
+</div>
+<p>In the example above, updating a pointer <code class="docutils literal notranslate"><span class="pre">curr_val</span></code> in the loop allows us to create a <code class="docutils literal notranslate"><span class="pre">before</span></code>
+condition such that we ensure the student followed the correct dynamic programming algorithm by
+checking each update to the <code class="docutils literal notranslate"><span class="pre">fibs</span></code> array.</p>
+<p>Temporal annotations are satisfied when the student’s code satisfies all of the child
+<code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code> annotations and when the first annotation (the one calling
+<a class="reference internal" href="../api_reference.html#pybryt.Annotation.before" title="pybryt.Annotation.before"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Annotation.before</span></code></a>) has a timestamp greater than or equal to the
+timestamp of the second annotation.</p>
+<p>Note that <a class="reference internal" href="../api_reference.html#pybryt.Annotation.before" title="pybryt.Annotation.before"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Annotation.before</span></code></a> returns an instance of the
+<a class="reference internal" href="../api_reference.html#pybryt.annotations.relation.BeforeAnnotation" title="pybryt.annotations.relation.BeforeAnnotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">BeforeAnnotation</span></code></a> class, which is
+itself a subclass of <a class="reference internal" href="../api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> and supports all of the same operations.
+<a class="reference internal" href="../api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> also provides
+<a class="reference internal" href="../api_reference.html#pybryt.Annotation.after" title="pybryt.Annotation.after"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Annotation.after</span></code></a>, which also returns an instance of the
+<a class="reference internal" href="../api_reference.html#pybryt.annotations.relation.BeforeAnnotation" title="pybryt.annotations.relation.BeforeAnnotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">BeforeAnnotation</span></code></a> class, but with
+the operands switched.</p>
+</div>
+<div class="section" id="boolean-annotations">
+<h2>Boolean Annotations<a class="headerlink" href="#boolean-annotations" title="Permalink to this headline">¶</a></h2>
+<p>Boolean annotations define conditions on the presence of different values. For example, in defining
+a solutions, students may be able to take two different paths, and this logic can be enforced
+using a <a class="reference internal" href="../api_reference.html#pybryt.annotations.relation.XorAnnotation" title="pybryt.annotations.relation.XorAnnotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">XorAnnotation</span></code></a> to ensure that
+only one of the two possible values is present.</p>
+<p>Relational annotations can be created either by instantiating the classes directly using the
+constructor or, as is more recommended, by using Python’s bitwise logical operators, <code class="docutils literal notranslate"><span class="pre">&amp;</span></code>, <code class="docutils literal notranslate"><span class="pre">|</span></code>,
+<code class="docutils literal notranslate"><span class="pre">^</span></code>, and <code class="docutils literal notranslate"><span class="pre">~</span></code>, on annotations. The dunder methods for these operators have been overrided with
+for the <a class="reference internal" href="../api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> class, and return the
+<a class="reference internal" href="../api_reference.html#pybryt.annotations.relation.RelationalAnnotation" title="pybryt.annotations.relation.RelationalAnnotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">RelationalAnnotation</span></code></a> subclass
+instance corresponding to the logical operator used.</p>
+<p>To create the xor example above from two values <code class="docutils literal notranslate"><span class="pre">v1</span></code> and <code class="docutils literal notranslate"><span class="pre">v2</span></code>, simply write</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">v1</span> <span class="o">^</span> <span class="n">v2</span>
+</pre></div>
+</div>
+<p>To assert that a student should <em>not</em> have a specific value <code class="docutils literal notranslate"><span class="pre">v</span></code> in their code, use</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">~</span><span class="n">v</span>
+</pre></div>
+</div>
+</div>
+</div>
+
+      </article>
+      <footer>
+        
+        <div class="related-pages">
+          <a class="next-page" href="invariants.html">
+              <div class="page-info">
+                <div class="context">
+                  <span>Next</span>
+                </div>
+                <div class="title">Invariants</div>
+              </div>
+              <svg><use href="#svg-arrow-right"></use></svg>
+            </a>
+          <a class="prev-page" href="index.html">
+              <svg><use href="#svg-arrow-right"></use></svg>
+              <div class="page-info">
+                <div class="context">
+                  <span>Previous</span>
+                </div>
+                
+                <div class="title">Annotations</div>
+                
+              </div>
+            </a>
+        </div>
+
+        <div class="related-information">
+              Copyright &#169; 2021, Microsoft
+            |
+            Built with <a href="https://www.sphinx-doc.org/">Sphinx</a>
+              and
+              <a class="muted-link" href="https://pradyunsg.me">@pradyunsg</a>'s
+              <a href="https://github.com/pradyunsg/furo">Furo theme</a>.
+            |
+            <a class="muted-link" href="../_sources/annotations/relational_annotations.rst.txt"
+               rel="nofollow">
+              Show Source
+            </a>
+        </div>
+        
+      </footer>
+    </div>
+    <aside class="toc-drawer">
+      
+      
+      <div class="toc-sticky toc-scroll">
+        <div class="toc-title-container">
+          <span class="toc-title">
+            Contents
+          </span>
+        </div>
+        <div class="toc-tree-container">
+          <div class="toc-tree">
+            <ul>
+<li><a class="reference internal" href="#">Relational Annotations</a><ul>
+<li><a class="reference internal" href="#temporal-annotations">Temporal Annotations</a></li>
+<li><a class="reference internal" href="#boolean-annotations">Boolean Annotations</a></li>
+</ul>
+</li>
+</ul>
+
+          </div>
+        </div>
+      </div>
+      
+      
+    </aside>
+  </main>
+</div>
+    <script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
+    <script src="../_static/jquery.js"></script>
+    <script src="../_static/underscore.js"></script>
+    <script src="../_static/doctools.js"></script>
+    <script src="../_static/language_data.js"></script>
+    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
+    <script src="../_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
+</html>

docs/html/api_reference.html

@@ -1,7 +1,7 @@
 <!doctype html>
 <html class="no-js">
   <head><meta charset="utf-8"/>
-    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="prev" title="Reference Implementation Objects" href="reference_implementations/reference_objects.html" />
+    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="prev" title="Reference Implementations" href="reference_implementations.html" />
 
     <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
         <title>API Reference - PyBryt documentation</title>
@@ -107,12 +107,12 @@
   <input type="hidden" name="area" value="default">
 </form><div class="sidebar-scroll"><div class="sidebar-tree">
   <ul class="current">
-<li class="toctree-l1 has-children"><a class="reference internal" href="reference_implementations/index.html">Reference Implementations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
+<li class="toctree-l1 has-children"><a class="reference internal" href="annotations/index.html">Annotations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/annotations.html">Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/relational_annotations.html">Relational Annotations</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/invariants.html">Invariants</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/invariants.html">Invariants</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/reference_objects.html">Reference Implementation Objects</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="reference_implementations.html">Reference Implementations</a></li>
 <li class="toctree-l1 current current-page"><a class="current reference internal" href="#">API Reference</a></li>
 </ul>
 
@@ -836,14 +836,14 @@ present, this is the maximum satisfying timestamp of all child results</p>
         
         <div class="related-pages">
           
-          <a class="prev-page" href="reference_implementations/reference_objects.html">
+          <a class="prev-page" href="reference_implementations.html">
               <svg><use href="#svg-arrow-right"></use></svg>
               <div class="page-info">
                 <div class="context">
                   <span>Previous</span>
                 </div>
                 
-                <div class="title">Reference Implementation Objects</div>
+                <div class="title">Reference Implementations</div>
                 
               </div>
             </a>

docs/html/genindex.html

@@ -106,12 +106,12 @@
   <input type="hidden" name="area" value="default">
 </form><div class="sidebar-scroll"><div class="sidebar-tree">
   <ul>
-<li class="toctree-l1 has-children"><a class="reference internal" href="reference_implementations/index.html">Reference Implementations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
+<li class="toctree-l1 has-children"><a class="reference internal" href="annotations/index.html">Annotations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/annotations.html">Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/relational_annotations.html">Relational Annotations</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/invariants.html">Invariants</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/invariants.html">Invariants</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/reference_objects.html">Reference Implementation Objects</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="reference_implementations.html">Reference Implementations</a></li>
 <li class="toctree-l1"><a class="reference internal" href="api_reference.html">API Reference</a></li>
 </ul>
 

docs/html/index.html

@@ -1,7 +1,7 @@
 <!doctype html>
 <html class="no-js">
   <head><meta charset="utf-8"/>
-    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="Reference Implementations" href="reference_implementations/index.html" />
+    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="Annotations" href="annotations/index.html" />
 
     <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
         <title>Home - PyBryt documentation</title>
@@ -107,12 +107,12 @@
   <input type="hidden" name="area" value="default">
 </form><div class="sidebar-scroll"><div class="sidebar-tree">
   <ul>
-<li class="toctree-l1 has-children"><a class="reference internal" href="reference_implementations/index.html">Reference Implementations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
+<li class="toctree-l1 has-children"><a class="reference internal" href="annotations/index.html">Annotations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/annotations.html">Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/relational_annotations.html">Relational Annotations</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/invariants.html">Invariants</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/invariants.html">Invariants</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/reference_objects.html">Reference Implementation Objects</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="reference_implementations.html">Reference Implementations</a></li>
 <li class="toctree-l1"><a class="reference internal" href="api_reference.html">API Reference</a></li>
 </ul>
 
@@ -143,18 +143,26 @@
 <li><p>Plagiarism detection and support for reference implementations</p></li>
 <li><p>Easy integration into existing organizational or institutional grading infrastructure</p></li>
 </ul>
+<p>PyBryt’s core auto-assessment behavior operates by comparing a student’s implementation of some
+programming problem to a series of reference implementations provided by an instructor. A
+<strong>reference implementation</strong> defines a pattern of values, and conditions on those values, to look
+for in students’ code.</p>
+<p>A reference implementation is created by annotating code written or found by an instructor and
+executing this code to create a <a class="reference internal" href="api_reference.html#pybryt.ReferenceImplementation" title="pybryt.ReferenceImplementation"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceImplementation</span></code></a>
+object. Annotations are created by creating instances of subclasses of the abstract
+<a class="reference internal" href="api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> class.</p>
 </div>
 
       </article>
       <footer>
         
         <div class="related-pages">
-          <a class="next-page" href="reference_implementations/index.html">
+          <a class="next-page" href="annotations/index.html">
               <div class="page-info">
                 <div class="context">
                   <span>Next</span>
                 </div>
-                <div class="title">Reference Implementations</div>
+                <div class="title">Annotations</div>
               </div>
               <svg><use href="#svg-arrow-right"></use></svg>
             </a>

docs/html/reference_implementations.html

@@ -0,0 +1,342 @@
+<!doctype html>
+<html class="no-js">
+  <head><meta charset="utf-8"/>
+    <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="API Reference" href="api_reference.html" /><link rel="prev" title="Invariants" href="annotations/invariants.html" />
+
+    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+        <title>Reference Implementations - PyBryt documentation</title>
+      <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
+    <link rel="stylesheet" href="_static/pygments.css">
+    <link media="(prefers-color-scheme: dark)" rel="stylesheet" href="_static/pygments_dark.css">
+    
+
+
+<style>
+  :root {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  @media (prefers-color-scheme: dark) {
+    :root {
+      --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+    }
+  }
+
+  /* For allowing end-user-specific overrides */
+  .override-light {
+    --color-code-background: #eeffcc;
+  --color-code-foreground: black;
+  
+  }
+  .override-dark {
+    --color-code-background: #202020;
+  --color-code-foreground: #d0d0d0;
+  
+  }
+</style><link rel="stylesheet" href="_static/styles/furo-extensions.css?digest=d391b54134226e4196576da3bdb6dddb7e05ba2b"></head>
+  <body dir="">
+    
+<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
+  <symbol id="svg-toc" viewBox="0 0 24 24">
+    <title>Contents</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke-width="1.5" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round">
+      <path stroke="none" d="M0 0h24v24H0z"/>
+      <line x1="4" y1="6" x2="20" y2="6" />
+      <line x1="10" y1="12" x2="20" y2="12" />
+      <line x1="6" y1="18" x2="20" y2="18" />
+    </svg>
+  </symbol>
+  <symbol id="svg-menu" viewBox="0 0 24 24">
+    <title>Menu</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-menu">
+      <line x1="3" y1="12" x2="21" y2="12"></line>
+      <line x1="3" y1="6" x2="21" y2="6"></line>
+      <line x1="3" y1="18" x2="21" y2="18"></line>
+    </svg>
+  </symbol>
+  <symbol id="svg-arrow-right" viewBox="0 0 24 24">
+    <title>Expand</title>
+    <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none"
+      stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
+      class="feather feather-chevron-right">
+      <polyline points="9 18 15 12 9 6"></polyline>
+    </svg>
+  </symbol>
+</svg>
+
+<input type="checkbox" class="sidebar-toggle" name="__navigation" id="__navigation">
+<input type="checkbox" class="sidebar-toggle" name="__toc" id="__toc">
+<label class="overlay sidebar-overlay" for="__navigation"></label>
+<label class="overlay toc-overlay" for="__toc"></label>
+
+
+
+<div class="page">
+  <header class="mobile-header">
+    <div class="header-left">
+      <label class="nav-overlay-icon" for="__navigation">
+        <i class="icon"><svg><use href="#svg-menu"></use></svg></i>
+      </label>
+    </div>
+    <div class="header-center">
+      <a href="index.html"><div class="brand">PyBryt  documentation</div></a>
+    </div>
+    <div class="header-right">
+      <label class="toc-overlay-icon toc-header-icon" for="__toc">
+        <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+      </label>
+    </div>
+  </header>
+  <aside class="sidebar-drawer">
+    <div class="sidebar-container">
+      
+      <div class="sidebar-sticky"><a class="sidebar-brand" href="index.html">
+  
+  
+  <span class="sidebar-brand-text">PyBryt  documentation</span>
+  
+</a><form class="sidebar-search-container" method="get" action="search.html">
+  <input class="sidebar-search" placeholder=Search name="q">
+  <input type="hidden" name="check_keywords" value="yes">
+  <input type="hidden" name="area" value="default">
+</form><div class="sidebar-scroll"><div class="sidebar-tree">
+  <ul class="current">
+<li class="toctree-l1 has-children"><a class="reference internal" href="annotations/index.html">Annotations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
+<li class="toctree-l2"><a class="reference internal" href="annotations/relational_annotations.html">Relational Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/invariants.html">Invariants</a></li>
+</ul>
+</li>
+<li class="toctree-l1 current current-page"><a class="current reference internal" href="#">Reference Implementations</a></li>
+<li class="toctree-l1"><a class="reference internal" href="api_reference.html">API Reference</a></li>
+</ul>
+
+</div>
+</div>
+      </div>
+      
+    </div>
+  </aside>
+  <main class="main">
+    <div class="content">
+      <article role="main">
+        <label class="toc-overlay-icon toc-content-icon" for="__toc">
+          <i class="icon"><svg><use href="#svg-toc"></use></svg></i>
+        </label>
+        <div class="section" id="reference-implementations">
+<h1>Reference Implementations<a class="headerlink" href="#reference-implementations" title="Permalink to this headline">¶</a></h1>
+<p>The functional unit of PyBryt is a reference implementation. A <strong>reference implenetation</strong> is a set
+of conditions expected of students’ code that determine whether a student has correctly implemented
+some program. They are constructed by creating a series of <a class="reference internal" href="annotations/index.html#annotations"><span class="std std-ref">annotations</span></a> that are
+tracked in a <a class="reference internal" href="api_reference.html#pybryt.ReferenceImplementation" title="pybryt.ReferenceImplementation"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceImplementation</span></code></a> object.</p>
+<div class="section" id="creating-reference-implementations">
+<h2>Creating Reference Implementations<a class="headerlink" href="#creating-reference-implementations" title="Permalink to this headline">¶</a></h2>
+<p>Reference implemenations can be created by compiling Jupyter Notebooks that have been marked-up
+with annotations. To compile a reference implementation, use
+<a class="reference internal" href="api_reference.html#pybryt.ReferenceImplementation.compile" title="pybryt.ReferenceImplementation.compile"><code class="xref py py-meth docutils literal notranslate"><span class="pre">ReferenceImplementation.compile</span></code></a>, which takes in
+the path to a notebook file:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">pybryt</span><span class="o">.</span><span class="n">ReferenceImplementation</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s2">"reference.ipynb"</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>There are two ways to mark-up a notebook: by creating annotations and having PyBryt track them
+automatically (for a single reference), or by tracking annotations in a list (for creating multiple
+reference implementations from a single notebook).</p>
+<p>When compiling a notebook, PyBryt executes all of the code in the notebook and searches the
+resulting global environment for instances of the
+<a class="reference internal" href="api_reference.html#pybryt.ReferenceImplementation" title="pybryt.ReferenceImplementation"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceImplementation</span></code></a> class. If it finds them, it
+collects them into a list and returns the list of reference implementations (or a single reference
+implementation if only one is found). If it doesn’t find any reference implementations, it takes all
+of the annotations <a class="reference internal" href="#tracked-annotations"><span class="std std-ref">tracked by</span></a> PyBryt and turns those into a single
+reference implementation.</p>
+<div class="section" id="automatic-reference-creation">
+<h3>Automatic Reference Creation<a class="headerlink" href="#automatic-reference-creation" title="Permalink to this headline">¶</a></h3>
+<p>To create a single reference implementation from a notebook, create
+<a class="reference internal" href="api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> instances (assigning them to variables is not necessary).
+After annotating the notebook, when PyBryt compiles the notebook, it will find all of the
+annotations.</p>
+<p id="tracked-annotations">PyBryt finds the annotations because the <code class="docutils literal notranslate"><span class="pre">__init__</span></code> method automatically adds the instances
+created to a singleton list that PyBryt maintains, so assigning them to variables or tracking them
+further is unnecessary unless more advanced reference implementations are being built. This means
+that when marking up code, as below, creating new variables is unnecessary unless further conditions
+are to be made later down the line.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">fibs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
+
+<span class="n">fibs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span>
+<span class="n">curr_val</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+<span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+    <span class="k">return</span> <span class="n">fibs</span>
+
+<span class="n">fibs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
+<span class="n">v</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+<span class="n">curr_val</span><span class="o">.</span><span class="n">before</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>         <span class="c1"># not assigned to a variable, but still tracked</span>
+<span class="n">curr_val</span> <span class="o">=</span> <span class="n">v</span>
+<span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span>
+    <span class="k">return</span> <span class="n">fibs</span>
+
+<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
+    <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
+
+    <span class="n">v</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+    <span class="n">curr_val</span><span class="o">.</span><span class="n">before</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>     <span class="c1"># not assigned to a variable, but still tracked</span>
+    <span class="n">curr_val</span> <span class="o">=</span> <span class="n">v</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="custom-reference-creation">
+<h3>Custom Reference Creation<a class="headerlink" href="#custom-reference-creation" title="Permalink to this headline">¶</a></h3>
+<p>To create multiple reference implementations from a single notebook, begin by creating
+<a class="reference internal" href="api_reference.html#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> instances and grouping them into lists. These lists will
+be passed to the <a class="reference internal" href="api_reference.html#pybryt.ReferenceImplementation" title="pybryt.ReferenceImplementation"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceImplementation</span></code></a> constructor
+to create the reference implementations. <em>These objects must be assigned to global variables, or
+PyBryt will not find them.</em></p>
+<p>As an example, consider the code below, which creates two reference implementations for a Fibonacci
+sequence generator:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">n_fibs</span> <span class="o">=</span> <span class="mi">50</span>
+<span class="n">first_ref</span> <span class="o">=</span> <span class="p">[]</span>
+<span class="n">second_ref</span> <span class="o">=</span>  <span class="p">[]</span>
+
+
+<span class="c1"># first implementation: dynamic programming</span>
+<span class="n">fibs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">n_fibs</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
+
+<span class="n">fibs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span>
+<span class="n">first_ref</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">))</span>
+<span class="k">if</span> <span class="n">n_fibs</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+    <span class="k">return</span> <span class="n">fibs</span>
+
+<span class="n">fibs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
+<span class="n">v</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+<span class="n">first_ref</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">curr_val</span><span class="o">.</span><span class="n">before</span><span class="p">(</span><span class="n">v</span><span class="p">))</span>
+<span class="n">curr_val</span> <span class="o">=</span> <span class="n">v</span>
+<span class="k">if</span> <span class="n">n_fibs</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span>
+    <span class="k">return</span> <span class="n">fibs</span>
+
+<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">n_fibs</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
+    <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">fibs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
+
+    <span class="n">v</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fibs</span><span class="p">)</span>
+    <span class="n">first_ref</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">curr_val</span><span class="o">.</span><span class="n">before</span><span class="p">(</span><span class="n">v</span><span class="p">))</span>
+    <span class="n">curr_val</span> <span class="o">=</span> <span class="n">v</span>
+
+<span class="n">final_answer</span> <span class="o">=</span> <span class="n">fibs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+
+
+<span class="c1"># second implementation: hash map</span>
+<span class="n">fib_map</span> <span class="o">=</span> <span class="p">{}</span>
+<span class="k">def</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+        <span class="k">return</span> <span class="mi">0</span>
+
+    <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+        <span class="k">return</span> <span class="mi">1</span>
+
+    <span class="k">if</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">fib_map</span><span class="p">:</span>
+        <span class="k">return</span> <span class="n">fib_map</span><span class="p">[</span><span class="n">n</span><span class="p">]</span>
+
+    <span class="n">ans</span> <span class="o">=</span> <span class="n">fib</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="n">fib</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span>
+    <span class="n">fib_map</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">ans</span>
+    <span class="n">second_ref</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">pybryt</span><span class="o">.</span><span class="n">Value</span><span class="p">(</span><span class="n">fib_map</span><span class="p">))</span>
+
+    <span class="k">return</span> <span class="n">ans</span>
+
+<span class="n">final_answer</span> <span class="o">=</span> <span class="n">fib</span><span class="p">(</span><span class="n">n_fibs</span><span class="p">)</span>
+
+
+<span class="c1"># create references</span>
+<span class="n">ref1</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">ReferenceImplementation</span><span class="p">(</span><span class="n">first_ref</span><span class="p">)</span>
+<span class="n">ref2</span> <span class="o">=</span> <span class="n">pybryt</span><span class="o">.</span><span class="n">ReferenceImplementation</span><span class="p">(</span><span class="n">second_ref</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="interacting-with-reference-implementations">
+<h2>Interacting with Reference Implementations<a class="headerlink" href="#interacting-with-reference-implementations" title="Permalink to this headline">¶</a></h2>
+</div>
+</div>
+
+      </article>
+      <footer>
+        
+        <div class="related-pages">
+          <a class="next-page" href="api_reference.html">
+              <div class="page-info">
+                <div class="context">
+                  <span>Next</span>
+                </div>
+                <div class="title">API Reference</div>
+              </div>
+              <svg><use href="#svg-arrow-right"></use></svg>
+            </a>
+          <a class="prev-page" href="annotations/invariants.html">
+              <svg><use href="#svg-arrow-right"></use></svg>
+              <div class="page-info">
+                <div class="context">
+                  <span>Previous</span>
+                </div>
+                
+                <div class="title">Invariants</div>
+                
+              </div>
+            </a>
+        </div>
+
+        <div class="related-information">
+              Copyright &#169; 2021, Microsoft
+            |
+            Built with <a href="https://www.sphinx-doc.org/">Sphinx</a>
+              and
+              <a class="muted-link" href="https://pradyunsg.me">@pradyunsg</a>'s
+              <a href="https://github.com/pradyunsg/furo">Furo theme</a>.
+            |
+            <a class="muted-link" href="_sources/reference_implementations.rst.txt"
+               rel="nofollow">
+              Show Source
+            </a>
+        </div>
+        
+      </footer>
+    </div>
+    <aside class="toc-drawer">
+      
+      
+      <div class="toc-sticky toc-scroll">
+        <div class="toc-title-container">
+          <span class="toc-title">
+            Contents
+          </span>
+        </div>
+        <div class="toc-tree-container">
+          <div class="toc-tree">
+            <ul>
+<li><a class="reference internal" href="#">Reference Implementations</a><ul>
+<li><a class="reference internal" href="#creating-reference-implementations">Creating Reference Implementations</a><ul>
+<li><a class="reference internal" href="#automatic-reference-creation">Automatic Reference Creation</a></li>
+<li><a class="reference internal" href="#custom-reference-creation">Custom Reference Creation</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#interacting-with-reference-implementations">Interacting with Reference Implementations</a></li>
+</ul>
+</li>
+</ul>
+
+          </div>
+        </div>
+      </div>
+      
+      
+    </aside>
+  </main>
+</div>
+    <script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
+    <script src="_static/jquery.js"></script>
+    <script src="_static/underscore.js"></script>
+    <script src="_static/doctools.js"></script>
+    <script src="_static/language_data.js"></script>
+    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
+    <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
+</html>

docs/html/search.html

@@ -105,12 +105,12 @@
   <input type="hidden" name="area" value="default">
 </form><div class="sidebar-scroll"><div class="sidebar-tree">
   <ul>
-<li class="toctree-l1 has-children"><a class="reference internal" href="reference_implementations/index.html">Reference Implementations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
+<li class="toctree-l1 has-children"><a class="reference internal" href="annotations/index.html">Annotations</a><input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/><label for="toctree-checkbox-1"><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/annotations.html">Annotations</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/relational_annotations.html">Relational Annotations</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/invariants.html">Invariants</a></li>
+<li class="toctree-l2"><a class="reference internal" href="annotations/invariants.html">Invariants</a></li>
-<li class="toctree-l2"><a class="reference internal" href="reference_implementations/reference_objects.html">Reference Implementation Objects</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="reference_implementations.html">Reference Implementations</a></li>
 <li class="toctree-l1"><a class="reference internal" href="api_reference.html">API Reference</a></li>
 </ul>
 

docs/index.rst

@@ -10,7 +10,8 @@ PyBryt Documentation
    :maxdepth: 3
    :hidden:
 
-   reference_implementations/index
+   annotations/index
+   reference_implementations
    api_reference
 
 PyBryt is an open source  Python auto-assessment library for teaching and learning. Our goal is to empower students and educators to learn about technology through fun, guided, hands-on content aimed at specific learning goals. PyBryt is designed to work with existing autograding solutions and workflows such as `Otter Grader`_, `OkPy`_, and `Autolab`_.
@@ -26,6 +27,16 @@ Educators and institutions can leverage PyBryt to integrate auto-assessment and
 - Plagiarism detection and support for reference implementations
 - Easy integration into existing organizational or institutional grading infrastructure
 
+PyBryt's core auto-assessment behavior operates by comparing a student's implementation of some 
+programming problem to a series of reference implementations provided by an instructor. A 
+**reference implementation** defines a pattern of values, and conditions on those values, to look 
+for in students' code.
+
+A reference implementation is created by annotating code written or found by an instructor and 
+executing this code to create a :py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>` 
+object. Annotations are created by creating instances of subclasses of the abstract 
+:py:class:`Annotation<pybryt.Annotation>` class.
+
 .. _Otter Grader: https://otter-grader.readthedocs.io
 .. _OkPy: https://okpy.org
 .. _Autolab: https://autolab.readthedocs.io/

docs/reference_implementations.rst

@@ -0,0 +1,146 @@
+Reference Implementations
+=========================
+
+The functional unit of PyBryt is a reference implementation. A **reference implenetation** is a set 
+of conditions expected of students' code that determine whether a student has correctly implemented
+some program. They are constructed by creating a series of :ref:`annotations<annotations>` that are
+tracked in a :py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>` object.
+
+
+Creating Reference Implementations
+----------------------------------
+
+Reference implemenations can be created by compiling Jupyter Notebooks that have been marked-up
+with annotations. To compile a reference implementation, use 
+:py:meth:`ReferenceImplementation.compile<pybryt.ReferenceImplementation.compile>`, which takes in
+the path to a notebook file:
+
+.. code-block:: python
+
+    pybryt.ReferenceImplementation.compile("reference.ipynb")
+
+There are two ways to mark-up a notebook: by creating annotations and having PyBryt track them 
+automatically (for a single reference), or by tracking annotations in a list (for creating multiple
+reference implementations from a single notebook).
+
+When compiling a notebook, PyBryt executes all of the code in the notebook and searches the 
+resulting global environment for instances of the 
+:py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>` class. If it finds them, it 
+collects them into a list and returns the list of reference implementations (or a single reference
+implementation if only one is found). If it doesn't find any reference implementations, it takes all
+of the annotations :ref:`tracked by<tracked_annotations>` PyBryt and turns those into a single
+reference implementation.
+
+
+Automatic Reference Creation
+++++++++++++++++++++++++++++
+
+To create a single reference implementation from a notebook, create
+:py:class:`Annotation<pybryt.Annotation>` instances (assigning them to variables is not necessary).
+After annotating the notebook, when PyBryt compiles the notebook, it will find all of the 
+annotations.
+
+.. _tracked_annotations:
+
+PyBryt finds the annotations because the ``__init__`` method automatically adds the instances 
+created to a singleton list that PyBryt maintains, so assigning them to variables or tracking them 
+further is unnecessary unless more advanced reference implementations are being built. This means 
+that when marking up code, as below, creating new variables is unnecessary unless further conditions
+are to be made later down the line.
+
+.. code-block:: python
+
+    fibs = np.zeros(n, dtype=int)
+
+    fibs[0] = 0
+    curr_val = pybryt.Value(fibs)
+    if n == 1:
+        return fibs
+
+    fibs[1] = 1
+    v = pybryt.Value(fibs)
+    curr_val.before(v)         # not assigned to a variable, but still tracked
+    curr_val = v
+    if n == 2:
+        return fibs
+
+    for i in range(2, n-1):
+        fibs[i] = fibs[i-1] + fibs[i-2]
+        
+        v = pybryt.Value(fibs)
+        curr_val.before(v)     # not assigned to a variable, but still tracked
+        curr_val = v
+
+
+Custom Reference Creation
++++++++++++++++++++++++++
+
+To create multiple reference implementations from a single notebook, begin by creating 
+:py:class:`Annotation<pybryt.Annotation>` instances and grouping them into lists. These lists will 
+be passed to the :py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>` constructor
+to create the reference implementations. *These objects must be assigned to global variables, or 
+PyBryt will not find them.*
+
+As an example, consider the code below, which creates two reference implementations for a Fibonacci
+sequence generator:
+
+.. code-block:: python
+
+    n_fibs = 50
+    first_ref = []
+    second_ref =  []
+
+
+    # first implementation: dynamic programming
+    fibs = np.zeros(n_fibs, dtype=int)
+
+    fibs[0] = 0
+    first_ref.append(pybryt.Value(fibs))
+    if n_fibs == 1:
+        return fibs
+
+    fibs[1] = 1
+    v = pybryt.Value(fibs)
+    first_ref.append(curr_val.before(v))
+    curr_val = v
+    if n_fibs == 2:
+        return fibs
+
+    for i in range(2, n_fibs-1):
+        fibs[i] = fibs[i-1] + fibs[i-2]
+        
+        v = pybryt.Value(fibs)
+        first_ref.append(curr_val.before(v))
+        curr_val = v
+
+    final_answer = fibs[-1]
+
+
+    # second implementation: hash map
+    fib_map = {}
+    def fib(n):
+        if n == 0:
+            return 0
+        
+        if n == 1:
+            return 1
+        
+        if n in fib_map:
+            return fib_map[n]
+        
+        ans = fib(n-1) + fib(n-2)
+        fib_map[n] = ans
+        second_ref.append(pybryt.Value(fib_map))
+        
+        return ans
+
+    final_answer = fib(n_fibs)
+
+
+    # create references
+    ref1 = pybryt.ReferenceImplementation(first_ref)
+    ref2 = pybryt.ReferenceImplementation(second_ref)
+
+
+Interacting with Reference Implementations
+------------------------------------------

docs/reference_implementations/index.rst

@@ -1,24 +0,0 @@
-Reference Implementations
-=========================
-
-.. toctree::
-   :hidden:
-   :maxdepth: 3
-
-   annotations
-   invariants
-   reference_objects
-
-PyBryt's core auto-assessment behavior operates by comparing a student's implementation of some 
-programming problem to a series of reference implementations provided by an instructor. A 
-**reference implementation** defines a pattern of values, and conditions on those values, to look 
-for in students' code.
-
-A reference implementation is created by annotating code written or found by an instructor and 
-executing this code to create a :py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>` 
-object. Annotations are created by creating instances of subclasses of the abstract 
-:py:class:`Annotation<pybryt.Annotation>` class.
-
-This section details the creation and behavior of the different annotation classes that PyBryt 
-provides and describes how reference implementations as a single unit are created, manipulated, 
-stored, and run.