Π·Π΅ΡΠΊΠ°Π»ΠΎ ΠΈΠ· https://github.com/github/scientist.git
This commit is contained in:
John Barnette
ΠΠΎΠΌΠΌΠΈΡ
37071bc02b
|
|
@ -0,0 +1,4 @@
|
|||
/*.gem
|
||||
/.bundle
|
||||
/.ruby-version
|
||||
/Gemfile.lock
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
source "https://rubygems.org"
|
||||
|
||||
gemspec
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
Copyright 2013, 2014 GitHub, Inc.
|
||||
|
||||
MIT License
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of this software and associated documentation files (the
|
||||
"Software"), to deal in the Software without restriction, including
|
||||
without limitation the rights to use, copy, modify, merge, publish,
|
||||
distribute, sublicense, and/or sell copies of the Software, and to
|
||||
permit persons to whom the Software is furnished to do so, subject to
|
||||
the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be
|
||||
included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
||||
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
||||
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
|
|
@ -0,0 +1,420 @@
|
|||
# Scientist!
|
||||
|
||||
A Ruby library for carefully refactoring critical paths.
|
||||
|
||||
## How do I do science?
|
||||
|
||||
Let's pretend you're changing the way you handle permissions in a large web app. Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load.
|
||||
|
||||
```ruby
|
||||
require "scientist"
|
||||
|
||||
class MyWidget
|
||||
def allows?(user)
|
||||
experiment = Scientist::Default.new "widget-permissions" do |e|
|
||||
e.use { model.check_user?(user).valid? } # old way
|
||||
e.try { user.can?(:read, model) } # new way
|
||||
end
|
||||
|
||||
experiment.run
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
Wrap a `use` block around the code's original behavior, and wrap `try` around the new behavior. `experiment.run` will always return whatever the `use` block returns, but it does a bunch of stuff behind the scenes:
|
||||
|
||||
* It decides whether or not to run the `try` block,
|
||||
* Randomizes the order in which `use` and `try` blocks are run,
|
||||
* Measures the durations of all behaviors,
|
||||
* Compares the result of `try` to the result of `use`,
|
||||
* Swallows (but records) any exceptions raise in the `try` block, and
|
||||
* Publishes all this information.
|
||||
|
||||
The `use` block is called the **control**. The `try` block is called the **candidate**.
|
||||
|
||||
Creating an experiment is wordy, but when you include the `Scientist` module, the `science` helper will instantiate an experiment and call `run` for you:
|
||||
|
||||
```ruby
|
||||
require "scientist"
|
||||
|
||||
class MyWidget
|
||||
include Scientist
|
||||
|
||||
def allows?(user)
|
||||
science "widget-permissions" do |e|
|
||||
e.use { model.check_user(user).valid? } # old way
|
||||
e.try { user.can?(:read, model) } # new way
|
||||
end # returns the control value
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
If you don't declare any `try` blocks, none of the Scientist machinery is invoked and the control value is always returned.
|
||||
|
||||
## Making science useful
|
||||
|
||||
The examples above will run, but they're not really *doing* anything. The `try` blocks run every time and none of the results get published. Replace the default experiment implementation to control execution and reporting:
|
||||
|
||||
```ruby
|
||||
require "scientist"
|
||||
|
||||
class MyExperiment < ActiveRecord::Base
|
||||
include Scientist::Experiment
|
||||
|
||||
def enabled?
|
||||
# see "Ramping up experiments" below
|
||||
super
|
||||
end
|
||||
|
||||
def publish(result)
|
||||
# see "Publishing results" below
|
||||
super
|
||||
end
|
||||
end
|
||||
|
||||
# replace `Scientist::Default` as the default implementation
|
||||
def Scientist::Experiment.new(name)
|
||||
MyExperiment.find_or_initialize_by(name: name)
|
||||
end
|
||||
```
|
||||
|
||||
Now calls to the `science` helper will load instances of `MyExperiment`.
|
||||
|
||||
### Controlling comparison
|
||||
|
||||
Scientist compares control and candidate values using `==`. To override this behavior, use `compare` to define how to compare observed values instead:
|
||||
|
||||
```ruby
|
||||
class MyWidget
|
||||
include Scientist
|
||||
|
||||
def users
|
||||
science "users" do |e|
|
||||
e.use { User.all } # returns User instances
|
||||
e.try { UserService.list } # returns UserService::User instances
|
||||
|
||||
e.compare do |control, candidate|
|
||||
control.map(&:login) == candidate.map(&:login)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Adding context
|
||||
|
||||
Results aren't very useful without some way to identify them. Use the `context` method to add to or retrieve the context for an experiment:
|
||||
|
||||
```ruby
|
||||
science "widget-permissions" do |e|
|
||||
e.context :user => user
|
||||
|
||||
e.use { model.check_user(user).valid? }
|
||||
e.try { user.can?(:read, model) }
|
||||
end
|
||||
```
|
||||
|
||||
`context` takes a Symbol-keyed Hash of extra data. The data is available in `Experiment#publish` via the `context` method. If you're using the `science` helper a lot in a class, you can provide a default context:
|
||||
|
||||
```ruby
|
||||
class MyWidget
|
||||
include Scientist
|
||||
|
||||
def allows?(user)
|
||||
science "widget-permissions" do |e|
|
||||
e.context :user => user
|
||||
|
||||
e.use { model.check_user(user).valid? }
|
||||
e.try { user.can?(:read, model) }
|
||||
end
|
||||
end
|
||||
|
||||
def destroy
|
||||
science "widget-destruction" do |e|
|
||||
e.use { old_scary_destroy }
|
||||
e.try { new_safe_destroy }
|
||||
end
|
||||
end
|
||||
|
||||
def default_scientist_context
|
||||
{ :widget => self }
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
The `widget-permissions` and `widget-destruction` experiments will both have a `:widget` key in their contexts.
|
||||
|
||||
### Keeping it clean
|
||||
|
||||
Sometimes you don't want to store the full value for later analysis. For example, an experiment may return `User` instances, but when researching a mismatch, all you care about is the logins. You can define how to clean these values in an experiment:
|
||||
|
||||
```ruby
|
||||
class MyWidget
|
||||
include Scientist
|
||||
|
||||
def users
|
||||
science "users" do |e|
|
||||
e.use { User.all }
|
||||
e.try { UserService.list }
|
||||
|
||||
e.clean do |value|
|
||||
value.map(&:login).sort
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
And this cleaned value is available in observations in the final published result:
|
||||
|
||||
```ruby
|
||||
class MyExperiment < ActiveRecord::Base
|
||||
include Scientist::Experiment
|
||||
|
||||
def publish(result)
|
||||
result.control.value # [<User alice>, <User bob>, <User carol>]
|
||||
result.control.cleaned_value # ["alice", "bob", "carol"]
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Ignoring mismatches
|
||||
|
||||
During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell an experiment whether or not to ignore a mismatch using the `ignore` method. You may include more than one block if needed:
|
||||
|
||||
```ruby
|
||||
def admin?(user)
|
||||
science "widget-permissions" do |e|
|
||||
e.use { model.check_user(user).admin? }
|
||||
e.try { user.can?(:admin, model) }
|
||||
|
||||
e.ignore { user.staff? } # user is staff, always an admin in the new system
|
||||
e.ignore do |control, candidate|
|
||||
# new system doesn't handle unconfirmed users yet:
|
||||
control && !candidate && !user.confirmed_email?
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
The ignore blocks are only called if the *values* don't match. If one observation raises an exception and the other doesn't, it's always considered a mismatch. If both observations raise different exceptions, that is also considered a mismatch.
|
||||
|
||||
### Enabling/disabling experiments
|
||||
|
||||
Sometimes you don't want an experiment to run. Say, disabling a new codepath for anyone who isn't staff. You can disable an experiment by setting a `run_if` block. If this returns `false`, the experiment will merely return the control value. Otherwise, it defers to the experiment's configured `enabled?` method.
|
||||
|
||||
```ruby
|
||||
class DashboardController
|
||||
include Scientist
|
||||
|
||||
def dashboard_items
|
||||
science "dashboard-items" do |e|
|
||||
# only run this experiment for staff members
|
||||
e.run_if { current_user.staff? }
|
||||
# ...
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Ramping up experiments
|
||||
|
||||
As a scientist, you know it's always important to be able to turn your experiment off, lest it run amok and result in villagers with pitchforks on your doorstep. In order to control whether or not an experiment is enabled, you must include the `enabled?` method in your `Scientist::Experiment` implementation.
|
||||
|
||||
```ruby
|
||||
class MyExperiment < ActiveRecord::Base
|
||||
include Scientist::Experiment
|
||||
def enabled?
|
||||
percent_enabled > 0 && rand(100) < percent_enabled
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
This code will be invoked for every method with an experiment every time, so be sensitive about its performance. For example, you can store an experiment in the database but wrap it in various levels of caching such as memcache or per-request thread-locals.
|
||||
|
||||
### Publishing results
|
||||
|
||||
What good is science if you can't publish your results?
|
||||
|
||||
You must implement the `publish(result)` method, and can publish data however you like. For example, timing data can be sent to graphite, and mismatches can be placed in a capped collection in redis for debugging later.
|
||||
|
||||
The `publish` method is given a `Scientist::Result` instance with its associated `Scientist::Observation`s:
|
||||
|
||||
```ruby
|
||||
class MyExperiment
|
||||
include Scientist::Experiment
|
||||
|
||||
# ...
|
||||
|
||||
def publish(result)
|
||||
|
||||
# Store the timing for the control value,
|
||||
$statsd.timing "science.#{name}.control", result.control.duration
|
||||
# for the candidate (only the first, see "Breaking the rules" below,
|
||||
$statsd.timing "science.#{name}.candidate", result.candidates.first.duration
|
||||
|
||||
# and counts for match/ignore/mismatch:
|
||||
if result.matched?
|
||||
$statsd.increment "science.#{name}.matched"
|
||||
elsif result.ignored?
|
||||
$statsd.increment "science.#{name}.ignored"
|
||||
else
|
||||
$statsd.increment "science.#{name}.mismatched"
|
||||
# Finally, store mismatches in redis so they can be retrieved and examined
|
||||
# later on, for debugging and research.
|
||||
store_mismatch_data(result)
|
||||
end
|
||||
end
|
||||
|
||||
def store_mismatch_data(result)
|
||||
payload = {
|
||||
:name => name,
|
||||
:context => context,
|
||||
:control => observation_payload(result.control),
|
||||
:candidate => observation_payload(result.candidates.first)
|
||||
:execution_order => result.observations.map(&:name),
|
||||
}
|
||||
|
||||
key = "science.#{name}.mismatch"
|
||||
$redis.lpush key, payload
|
||||
$redis.ltrim key, 0, 1000
|
||||
end
|
||||
|
||||
def observation_payload(observation)
|
||||
if observation.raised?
|
||||
{
|
||||
:exception => observation.exeception.class,
|
||||
:message => observation.exeception.message,
|
||||
:backtrace => observation.exception.backtrace
|
||||
}
|
||||
else
|
||||
{
|
||||
# see "Keeping it clean" below
|
||||
:value => observation.cleaned_value
|
||||
}
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
When running your test suite, it's helpful to know that the experimental results always match. To help with testing, Scientist defines a `raise_on_mismatches` class attribute when you include `Scientist::Experiment`. Only do this in your test suite!
|
||||
|
||||
To raise on mismatches:
|
||||
|
||||
```ruby
|
||||
class MyExperiment
|
||||
include Scientist::Experiment
|
||||
# ... implementation
|
||||
end
|
||||
|
||||
MyExperiment.raise_on_mismatches = true
|
||||
```
|
||||
|
||||
Scientist will raise a `Scientist::Experiment::MismatchError` exception if any observations don't match.
|
||||
|
||||
### Handling errors
|
||||
|
||||
If an exception is raised within any of scientist's internal helpers, like `publish`, `compare`, or `clean`, the `raised` method is called with the symbol name of the internal operation that failed and the exception that was raised. The default behavior of `Scientist::Default` is to simply re-raise the exception. Since this halts the experiment entirely, it's often a better idea to handle this error and continue so the experiment as a whole isn't canceled entirely:
|
||||
|
||||
```ruby
|
||||
class MyExperiment
|
||||
include Scientist::Experiment
|
||||
|
||||
# ...
|
||||
|
||||
def raised(operation, error)
|
||||
InternalErrorTracker.track! "science failure in #{name}: #{operation}", error
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
The operations that may be handled here are:
|
||||
|
||||
* `:clean` - an exception is raised in a `clean` block
|
||||
* `:compare` - an exception is raised in a `compare` block
|
||||
* `:enabled` - an exception is raised in the `enabled?` method
|
||||
* `:ignore` - an exception is raised in an `ignore` block
|
||||
* `:publish` - an exception is raised in the `publish` method
|
||||
* `:run_if` - an exception is raised in a `run_if` block
|
||||
|
||||
### Designing an experiment
|
||||
|
||||
Because `enabled?` and `run_if` determine when a candidate runs, it's impossible to guarantee that it will run every time. For this reason, Scientist is only safe for wrapping methods that aren't changing data.
|
||||
|
||||
When using Scientist, we've found it most useful to modify both the existing and new systems simultaneously anywhere writes happen, and verify the results at read time with `science`. `raise_on_mismatches` has also been useful to ensure that the correct data was written during tests, and reviewing published mismatches has helped us find any situations we overlooked with our production data at runtime. When writing to and reading from two systems, it's also useful to write some data reconciliation scripts to verify and clean up production data alongside any running experiments.
|
||||
|
||||
### Finishing an experiment
|
||||
|
||||
As your candidate behavior converges on the controls, you'll start thinking about removing an experiment and using the new behavior.
|
||||
|
||||
* If there are any ignore blocks, the candidate behavior is *guaranteed* to be different. If this is unacceptable, you'll need to remove the ignore blocks and resolve any ongoing mismatches in behavior until the observations match perfectly every time.
|
||||
* When removing a read-behavior experiment, it's a good idea to keep any write-side duplication between an old and new system in place until well after the new behavior has been in production, in case you need to roll back.
|
||||
|
||||
## Breaking the rules
|
||||
|
||||
Sometimes scientists just gotta do weird stuff. We understand.
|
||||
|
||||
### Ignoring results entirely
|
||||
|
||||
Science is useful even when all you care about is the timing data or even whether or not a new code path blew up. If you have the ability to incrementally control how often an experiment runs via your `enabled?` method, you can use it to silently and carefully test new code paths and ignore the results altogether. You can do this by setting `ignore { true }`, or for greater efficiency, `compare { true }`.
|
||||
|
||||
This will still log mismatches if any exceptions are raised, but will disregard the values entirely.
|
||||
|
||||
### Trying more than one thing
|
||||
|
||||
It's not usually a good idea to try more than one alternative simultaneously. Behavior isn't guaranteed to be isolated and reporting + visualization get quite a bit harder. Still, it's sometimes useful.
|
||||
|
||||
To try more than one alternative at once, add names to some `try` blocks:
|
||||
|
||||
```ruby
|
||||
require "scientist"
|
||||
|
||||
class MyWidget
|
||||
include Scientist
|
||||
|
||||
def allows?(user)
|
||||
science "widget-permissions" do |e|
|
||||
e.use { model.check_user(user).valid? } # old way
|
||||
|
||||
e.try("api") { user.can?(:read, model) } # new service API
|
||||
e.try("raw-sql") { user.can_sql?(:read, model) } # raw query
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
When the experiment runs, all candidate behaviors are tested and each candidate observation is compared with the control in turn.
|
||||
|
||||
### No control, just candidates
|
||||
|
||||
Define the candidates with named `try` blocks, omit a `use`, and pass a candidate name to `run`:
|
||||
|
||||
```ruby
|
||||
experiment = MyExperiment.new("various-ways") do |e|
|
||||
e.try("first-way") { ... }
|
||||
e.try("second-way") { ... }
|
||||
end
|
||||
|
||||
experiment.run("second-way")
|
||||
```
|
||||
|
||||
The `science` helper also knows this trick:
|
||||
|
||||
```ruby
|
||||
science "various-ways", run: "first-way" do |e|
|
||||
e.try("first-way") { ... }
|
||||
e.try("second-way") { ... }
|
||||
end
|
||||
```
|
||||
|
||||
## Hacking
|
||||
|
||||
Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs the unit tests. All development dependencies are installed automatically. Science requires Ruby 2.1.
|
||||
|
||||
## Maintainers
|
||||
|
||||
[@jbarnette](https://github.com/jbarnette),
|
||||
[@jesseplusplus](https://github.com/jesseplusplus),
|
||||
[@rick](https://github.com/rick),
|
||||
and [@zerowidth](https://github.com/zerowidth)
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
# Include this module into any class which requires science experiments in its
|
||||
# methods. Provides the `science` and `default_scientist_context` methods for
|
||||
# defining and running experiments.
|
||||
#
|
||||
# If you need to run science on class methods, extend this module instead.
|
||||
module Scientist
|
||||
# Define and run a science experiment.
|
||||
#
|
||||
# name - a String name for this experiment.
|
||||
# run: - optional argument for which named test to run instead of "control".
|
||||
#
|
||||
# Yields an object which implements the Scientist::Experiment interface.
|
||||
# See `Scientist::Experiment.new` for how this is defined.
|
||||
#
|
||||
# Returns the calculated value of the control experiment, or raises if an
|
||||
# exception was raised.
|
||||
def science(name, run: nil)
|
||||
experiment = Experiment.new(name)
|
||||
experiment.context(default_scientist_context)
|
||||
|
||||
yield experiment
|
||||
|
||||
experiment.run(run)
|
||||
end
|
||||
|
||||
# Public: the default context data for an experiment created and run via the
|
||||
# `science` helper method. Override this in any class that includes Scientist
|
||||
# to define your own behavior.
|
||||
#
|
||||
# Returns a Hash.
|
||||
def default_scientist_context
|
||||
{}
|
||||
end
|
||||
end
|
||||
|
||||
require "scientist/default"
|
||||
require "scientist/errors"
|
||||
require "scientist/experiment"
|
||||
require "scientist/observation"
|
||||
require "scientist/result"
|
||||
require "scientist/version"
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
require "scientist/experiment"
|
||||
|
||||
# A null experiment.
|
||||
class Scientist::Default
|
||||
include Scientist::Experiment
|
||||
|
||||
attr_reader :name
|
||||
|
||||
def initialize(name)
|
||||
@name = name
|
||||
end
|
||||
|
||||
# Run everything every time.
|
||||
def enabled?
|
||||
true
|
||||
end
|
||||
|
||||
# Don't publish anything.
|
||||
def publish(result)
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
module Scientist
|
||||
|
||||
# Smoking in the bathroom and/or sassing.
|
||||
class BadBehavior < StandardError
|
||||
attr_reader :experiment
|
||||
attr_reader :name
|
||||
|
||||
def initialize(experiment, name, message)
|
||||
@experiment = experiment
|
||||
@name = name
|
||||
|
||||
super message
|
||||
end
|
||||
end
|
||||
|
||||
class BehaviorMissing < BadBehavior
|
||||
def initialize(experiment, name)
|
||||
super experiment, name,
|
||||
"#{experiment.name} missing #{name} behavior"
|
||||
end
|
||||
end
|
||||
|
||||
class BehaviorNotUnique < BadBehavior
|
||||
def initialize(experiment, name)
|
||||
super experiment, name,
|
||||
"#{experiment.name} alread has #{name} behavior"
|
||||
end
|
||||
end
|
||||
|
||||
class NoValue < StandardError
|
||||
attr_reader :observation
|
||||
|
||||
def initialize(observation)
|
||||
@observation = observation
|
||||
super "#{observation.name} didn't return a value"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,233 @@
|
|||
# This mixin provides shared behavior for experiments. Includers must implement
|
||||
# `enabled?` and `publish(result)`.
|
||||
#
|
||||
# Override Scientist::Experiment.new to set your own class which includes and
|
||||
# implements Scientist::Experiment's interface.
|
||||
module Scientist::Experiment
|
||||
|
||||
# Create a new instance of a class that implements the Scientist::Experiment
|
||||
# interface.
|
||||
#
|
||||
# Override this method directly to change the default implementation.
|
||||
def self.new(name)
|
||||
Scientist::Default.new(name)
|
||||
end
|
||||
|
||||
# A mismatch, raised when raise_on_mismatches is enabled.
|
||||
class MismatchError < StandardError
|
||||
def initialize(name, result)
|
||||
super "#{name}: control #{result.control.inspect}, candidates #{result.candidates.map(&:inspect)}"
|
||||
end
|
||||
end
|
||||
|
||||
module RaiseOnMismatch
|
||||
# Set this flag to raise on experiment mismatches.
|
||||
#
|
||||
# This causes all science mismatches to raise a MismatchError. This is
|
||||
# intended for test environments and should not be enabled in a production
|
||||
# environment.
|
||||
#
|
||||
# bool - true/false - whether to raise when the control and candidate mismatch.
|
||||
def raise_on_mismatches=(bool)
|
||||
@raise_on_mismatches = bool
|
||||
end
|
||||
|
||||
# Whether or not to raise a mismatch error when a mismatch occurs.
|
||||
def raise_on_mismatches?
|
||||
@raise_on_mismatches
|
||||
end
|
||||
end
|
||||
|
||||
def self.included(base)
|
||||
base.extend RaiseOnMismatch
|
||||
end
|
||||
|
||||
# A Hash of behavior blocks, keyed by String name. Register behavior blocks
|
||||
# with the `try` and `use` methods.
|
||||
def behaviors
|
||||
@_scientist_behaviors ||= {}
|
||||
end
|
||||
|
||||
# A block to clean an observed value for publishing or storing.
|
||||
#
|
||||
# The block takes one argument, the observed value which will be cleaned.
|
||||
#
|
||||
# Returns the configured block.
|
||||
def clean(&block)
|
||||
@_scientist_cleaner = block
|
||||
end
|
||||
|
||||
# Internal: Clean a value with the configured clean block, or return the value
|
||||
# if no clean block is configured.
|
||||
#
|
||||
# Rescues and reports exceptions in the clean block if they occur.
|
||||
def clean_value(value)
|
||||
if @_scientist_cleaner
|
||||
@_scientist_cleaner.call value
|
||||
else
|
||||
value
|
||||
end
|
||||
rescue StandardError => ex
|
||||
raised :clean, ex
|
||||
value
|
||||
end
|
||||
|
||||
# A block which compares two experimental values.
|
||||
#
|
||||
# The block must take two arguments, the control value and a candidate value,
|
||||
# and return true or false.
|
||||
#
|
||||
# Returns the block.
|
||||
def compare(*args, &block)
|
||||
@_scientist_comparator = block
|
||||
end
|
||||
|
||||
# A Symbol-keyed Hash of extra experiment data.
|
||||
def context(context = nil)
|
||||
@_scientist_context ||= {}
|
||||
@_scientist_context.merge!(context) if !context.nil?
|
||||
@_scientist_context
|
||||
end
|
||||
|
||||
# Configure this experiment to ignore an observation with the given block.
|
||||
#
|
||||
# The block takes two arguments, the control observation and the candidate
|
||||
# observation which didn't match the control. If the block returns true, the
|
||||
# mismatch is disregarded.
|
||||
#
|
||||
# This can be called more than once with different blocks to use.
|
||||
def ignore(&block)
|
||||
@_scientist_ignores ||= []
|
||||
@_scientist_ignores << block
|
||||
end
|
||||
|
||||
# Internal: ignore a mismatched observation?
|
||||
#
|
||||
# Iterates through the configured ignore blocks and calls each of them with
|
||||
# the given control and mismatched candidate observations.
|
||||
#
|
||||
# Returns true or false.
|
||||
def ignore_mismatched_observation?(control, candidate)
|
||||
return false unless @_scientist_ignores
|
||||
@_scientist_ignores.any? do |ignore|
|
||||
begin
|
||||
ignore.call control.value, candidate.value
|
||||
rescue StandardError => ex
|
||||
raised :ignore, ex
|
||||
false
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# The String name of this experiment. Default is "experiment". See
|
||||
# Scientist::Default for an example of how to override this default.
|
||||
def name
|
||||
"experiment"
|
||||
end
|
||||
|
||||
# Internal: compare two observations, using the configured compare block if present.
|
||||
def observations_are_equivalent?(a, b)
|
||||
if @_scientist_comparator
|
||||
a.equivalent_to?(b, &@_scientist_comparator)
|
||||
else
|
||||
a.equivalent_to? b
|
||||
end
|
||||
rescue StandardError => ex
|
||||
raised :compare, ex
|
||||
false
|
||||
end
|
||||
|
||||
# Called when an exception is raised while running an internal operation,
|
||||
# like :publish. Override this method to track these exceptions. The
|
||||
# default implementation re-raises the exception.
|
||||
def raised(operation, error)
|
||||
raise error
|
||||
end
|
||||
|
||||
# Internal: Run all the behaviors for this experiment, observing each and
|
||||
# publishing the results. Return the result of the named behavior, default
|
||||
# "control".
|
||||
def run(name = nil)
|
||||
behaviors.freeze
|
||||
context.freeze
|
||||
|
||||
name = (name || "control").to_s
|
||||
block = behaviors[name]
|
||||
|
||||
if block.nil?
|
||||
raise Scientist::BehaviorMissing.new(self, name)
|
||||
end
|
||||
|
||||
return block.call unless should_experiment_run?
|
||||
|
||||
observations = []
|
||||
|
||||
behaviors.keys.shuffle.each do |key|
|
||||
block = behaviors[key]
|
||||
observations << Scientist::Observation.new(key, self, &block)
|
||||
end
|
||||
|
||||
control = observations.detect { |o| o.name == name }
|
||||
|
||||
result = Scientist::Result.new self,
|
||||
observations: observations,
|
||||
control: control
|
||||
|
||||
begin
|
||||
publish(result)
|
||||
rescue StandardError => ex
|
||||
raised :publish, ex
|
||||
end
|
||||
|
||||
if control.raised?
|
||||
raise control.exception
|
||||
end
|
||||
|
||||
if self.class.raise_on_mismatches? && result.mismatched?
|
||||
raise MismatchError.new(name, result)
|
||||
end
|
||||
|
||||
control.value
|
||||
end
|
||||
|
||||
# Define a block that determines whether or not the experiment should run.
|
||||
def run_if(&block)
|
||||
@_scientist_run_if_block = block
|
||||
end
|
||||
|
||||
# Internal: does a run_if block allow the experiment to run?
|
||||
#
|
||||
# Rescues and reports exceptions in a run_if block if they occur.
|
||||
def run_if_block_allows?
|
||||
(@_scientist_run_if_block ? @_scientist_run_if_block.call : true)
|
||||
rescue StandardError => ex
|
||||
raised :run_if, ex
|
||||
return false
|
||||
end
|
||||
|
||||
# Internal: determine whether or not an experiment should run.
|
||||
#
|
||||
# Rescues and reports exceptions in the enabled method if they occur.
|
||||
def should_experiment_run?
|
||||
behaviors.size > 1 && enabled? && run_if_block_allows?
|
||||
rescue StandardError => ex
|
||||
raised :enabled, ex
|
||||
return false
|
||||
end
|
||||
|
||||
# Register a named behavior for this experiment, default "candidate".
|
||||
def try(name = nil, &block)
|
||||
name = (name || "candidate").to_s
|
||||
|
||||
if behaviors.include?(name)
|
||||
raise Scientist::BehaviorNotUnique.new(self, name)
|
||||
end
|
||||
|
||||
behaviors[name] = block
|
||||
end
|
||||
|
||||
# Register the control behavior for this experiment.
|
||||
def use(&block)
|
||||
try "control", &block
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
# What happened when this named behavior was executed? Immutable.
|
||||
class Scientist::Observation
|
||||
|
||||
# The experiment this observation is for
|
||||
attr_reader :experiment
|
||||
|
||||
# The instant observation began.
|
||||
attr_reader :now
|
||||
|
||||
# The String name of the behavior.
|
||||
attr_reader :name
|
||||
|
||||
# The value returned, if any.
|
||||
attr_reader :value
|
||||
|
||||
# The raised exception, if any.
|
||||
attr_reader :exception
|
||||
|
||||
# The Float seconds elapsed.
|
||||
attr_reader :duration
|
||||
|
||||
def initialize(name, experiment, &block)
|
||||
@name = name
|
||||
@experiment = experiment
|
||||
@now = Time.now
|
||||
|
||||
begin
|
||||
@value = block.call
|
||||
rescue Object => e
|
||||
@exception = e
|
||||
end
|
||||
|
||||
@duration = (Time.now - @now).to_f
|
||||
|
||||
freeze
|
||||
end
|
||||
|
||||
# Return a cleaned value suitable for publishing. Uses the experiment's
|
||||
# defined cleaner block to clean the observed value.
|
||||
def cleaned_value
|
||||
if value
|
||||
experiment.clean_value value
|
||||
end
|
||||
end
|
||||
|
||||
# Is this observation equivalent to another?
|
||||
#
|
||||
# other - the other Observation in question
|
||||
# comparator - an optional comparison block. This observation's value and the
|
||||
# other observation's value are yielded to this to determine
|
||||
# their equivalency. Block should return true/false.
|
||||
#
|
||||
# Returns true if:
|
||||
#
|
||||
# * The values of the observation are equal (using `==`)
|
||||
# * The values of the observations are equal according to a comparison
|
||||
# block, if given
|
||||
# * Both observations raised an exception with the same class and message.
|
||||
#
|
||||
# Returns false otherwise.
|
||||
def equivalent_to?(other, &comparator)
|
||||
return false unless other.is_a?(Scientist::Observation)
|
||||
|
||||
values_are_equal = false
|
||||
both_raised = other.raised? && raised?
|
||||
neither_raised = !other.raised? && !raised?
|
||||
|
||||
if neither_raised
|
||||
if block_given?
|
||||
values_are_equal = yield value, other.value
|
||||
else
|
||||
values_are_equal = value == other.value
|
||||
end
|
||||
end
|
||||
|
||||
exceptions_are_equivalent = # backtraces will differ, natch
|
||||
both_raised &&
|
||||
other.exception.class == exception.class &&
|
||||
other.exception.message == exception.message
|
||||
|
||||
(neither_raised && values_are_equal) ||
|
||||
(both_raised && exceptions_are_equivalent)
|
||||
end
|
||||
|
||||
def hash
|
||||
[value, exception, self.class].compact.map(&:hash).inject(:^)
|
||||
end
|
||||
|
||||
def raised?
|
||||
!exception.nil?
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
# The immutable result of running an experiment.
|
||||
class Scientist::Result
|
||||
|
||||
# An Array of candidate Observations.
|
||||
attr_reader :candidates
|
||||
|
||||
# The control Observation to which the rest are compared.
|
||||
attr_reader :control
|
||||
|
||||
# An Experiment.
|
||||
attr_reader :experiment
|
||||
|
||||
# An Array of observations which didn't match the control, but were ignored.
|
||||
attr_reader :ignored
|
||||
|
||||
# An Array of observations which didn't match the control.
|
||||
attr_reader :mismatched
|
||||
|
||||
# An Array of Observations in execution order.
|
||||
attr_reader :observations
|
||||
|
||||
# Internal: Create a new result.
|
||||
#
|
||||
# experiment - the Experiment this result is for
|
||||
# observations: - an Array of Observations, in execution order
|
||||
# control: - the control Observation
|
||||
#
|
||||
def initialize(experiment, observations:, control:)
|
||||
@experiment = experiment
|
||||
@observations = observations
|
||||
@control = control
|
||||
@candidates = observations - [control]
|
||||
evaluate_candidates
|
||||
|
||||
freeze
|
||||
end
|
||||
|
||||
# Public: the experiment's context
|
||||
def context
|
||||
experiment.context
|
||||
end
|
||||
|
||||
# Public: the name of the experiment
|
||||
def experiment_name
|
||||
experiment.name
|
||||
end
|
||||
|
||||
# Public: was the result a match between all behaviors?
|
||||
def matched?
|
||||
mismatched.empty?
|
||||
end
|
||||
|
||||
# Public: were there mismatches in the behaviors?
|
||||
def mismatched?
|
||||
mismatched.any?
|
||||
end
|
||||
|
||||
# Public: were there any ignored mismatches?
|
||||
def ignored?
|
||||
ignored.any?
|
||||
end
|
||||
|
||||
# Internal: evaluate the candidates to find mismatched and ignored results
|
||||
#
|
||||
# Sets @ignored and @mismatched with the ignored and mismatched candidates.
|
||||
def evaluate_candidates
|
||||
mismatched = candidates.reject do |candidate|
|
||||
experiment.observations_are_equivalent?(control, candidate)
|
||||
end
|
||||
|
||||
@ignored = mismatched.select do |candidate|
|
||||
experiment.ignore_mismatched_observation? control, candidate
|
||||
end
|
||||
|
||||
@mismatched = mismatched - @ignored
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
module Scientist
|
||||
VERSION = "0.0.1"
|
||||
end
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
$: << "lib" and require "scientist/version"
|
||||
|
||||
Gem::Specification.new do |gem|
|
||||
gem.name = "scientist"
|
||||
gem.description = "A Ruby library for carefully refactoring critical paths"
|
||||
gem.version = Scientist::VERSION
|
||||
gem.authors = ["John Barnette", "Rick Bradley"]
|
||||
gem.email = ["jbarnette@github.com", "rick@github.com"]
|
||||
gem.summary = "Carefully test, measure, and track refactored code."
|
||||
gem.homepage = "https://github.com/github/scientist"
|
||||
gem.license = "MIT"
|
||||
|
||||
gem.required_ruby_version = ">= 2.1.0"
|
||||
|
||||
gem.files = `git ls-files`.split($/)
|
||||
gem.executables = []
|
||||
gem.test_files = gem.files.grep(/^test/)
|
||||
gem.require_paths = ["lib"]
|
||||
|
||||
gem.add_development_dependency "minitest", "~> 5.2"
|
||||
end
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
#!/bin/sh
|
||||
# Ensure local dependencies are available.
|
||||
|
||||
set -e
|
||||
|
||||
cd $(dirname "$0")/..
|
||||
rm -f .bundle/config
|
||||
|
||||
bundle install --path .bundle --quiet "$@"
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
#!/bin/sh
|
||||
# Tag and push a release.
|
||||
|
||||
set -e
|
||||
|
||||
# Make sure we're in the project root.
|
||||
|
||||
cd $(dirname "$0")/..
|
||||
|
||||
# Build a new gem archive.
|
||||
|
||||
rm -rf scientist-*.gem
|
||||
gem build -q scientist.gemspec
|
||||
|
||||
# Make sure we're on the master branch.
|
||||
|
||||
(git branch | grep -q '* master') || {
|
||||
echo "Only release from the master branch."
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Figure out what version we're releasing.
|
||||
|
||||
tag=v`ls scientist-*.gem | sed 's/^scientist-\(.*\)\.gem$/\1/'`
|
||||
|
||||
# Make sure we haven't released this version before.
|
||||
|
||||
git fetch -t origin
|
||||
|
||||
(git tag -l | grep -q "$tag") && {
|
||||
echo "Whoops, there's already a '${tag}' tag."
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Tag it and bag it.
|
||||
|
||||
gem push scientist-*.gem && git tag "$tag" &&
|
||||
git push origin master && git push origin "$tag"
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
#!/bin/sh
|
||||
# Run the unit tests.
|
||||
|
||||
set -e
|
||||
|
||||
cd $(dirname "$0")/..
|
||||
script/bootstrap && ruby -I lib \
|
||||
-e 'require "bundler/setup"' \
|
||||
-e 'require "minitest/autorun"' \
|
||||
-e 'require "scientist"' \
|
||||
-e '(ARGV.empty? ? Dir["test/**/*_test.rb"] : ARGV).each { |f| load f }' -- "$@"
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
describe Scientist::Default do
|
||||
before do
|
||||
@ex = Scientist::Default.new "default"
|
||||
end
|
||||
|
||||
it "is always enabled" do
|
||||
assert @ex.enabled?
|
||||
end
|
||||
|
||||
it "noops publish" do
|
||||
assert_nil @ex.publish("data")
|
||||
end
|
||||
|
||||
it "is an experiment" do
|
||||
assert Scientist::Default < Scientist::Experiment
|
||||
end
|
||||
|
||||
it "reraises when an internal action raises" do
|
||||
assert_raises RuntimeError do
|
||||
@ex.raised :publish, RuntimeError.new("kaboom")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,398 @@
|
|||
describe Scientist::Experiment do
|
||||
class Fake
|
||||
include Scientist::Experiment
|
||||
|
||||
def initialize(*args)
|
||||
end
|
||||
|
||||
def enabled?
|
||||
true
|
||||
end
|
||||
|
||||
attr_reader :published_result
|
||||
|
||||
def exceptions
|
||||
@exceptions ||= []
|
||||
end
|
||||
|
||||
def raised(op, exception)
|
||||
exceptions << [op, exception]
|
||||
end
|
||||
|
||||
def publish(result)
|
||||
@published_result = result
|
||||
end
|
||||
end
|
||||
|
||||
before do
|
||||
@ex = Fake.new
|
||||
end
|
||||
|
||||
it "has a default implementation" do
|
||||
ex = Scientist::Experiment.new("hello")
|
||||
assert_kind_of Scientist::Default, ex
|
||||
assert_equal "hello", ex.name
|
||||
end
|
||||
|
||||
it "provides a static default name" do
|
||||
assert_equal "experiment", Fake.new.name
|
||||
end
|
||||
|
||||
it "requires includers to implement enabled?" do
|
||||
obj = Object.new
|
||||
obj.extend Scientist::Experiment
|
||||
|
||||
assert_raises NoMethodError do
|
||||
obj.enabled?
|
||||
end
|
||||
end
|
||||
|
||||
it "requires includers to implement publish" do
|
||||
obj = Object.new
|
||||
obj.extend Scientist::Experiment
|
||||
|
||||
assert_raises NoMethodError do
|
||||
obj.publish("result")
|
||||
end
|
||||
end
|
||||
|
||||
it "can't be run without a control behavior" do
|
||||
e = assert_raises Scientist::BehaviorMissing do
|
||||
@ex.run
|
||||
end
|
||||
|
||||
assert_equal "control", e.name
|
||||
end
|
||||
|
||||
it "is a straight pass-through with only a control behavior" do
|
||||
@ex.use { "control" }
|
||||
assert_equal "control", @ex.run
|
||||
end
|
||||
|
||||
it "runs other behaviors but always returns the control" do
|
||||
@ex.use { "control" }
|
||||
@ex.try { "candidate" }
|
||||
|
||||
assert_equal "control", @ex.run
|
||||
end
|
||||
|
||||
it "complains about duplicate behavior names" do
|
||||
@ex.use { "control" }
|
||||
|
||||
e = assert_raises Scientist::BehaviorNotUnique do
|
||||
@ex.use { "control-again" }
|
||||
end
|
||||
|
||||
assert_equal @ex, e.experiment
|
||||
assert_equal "control", e.name
|
||||
end
|
||||
|
||||
it "swallows exceptions raised by candidate behaviors" do
|
||||
@ex.use { "control" }
|
||||
@ex.try { raise "candidate" }
|
||||
|
||||
assert_equal "control", @ex.run
|
||||
end
|
||||
|
||||
it "passes through exceptions raised by the control behavior" do
|
||||
@ex.use { raise "control" }
|
||||
@ex.try { "candidate" }
|
||||
|
||||
exception = assert_raises RuntimeError do
|
||||
@ex.run
|
||||
end
|
||||
|
||||
assert_equal "control", exception.message
|
||||
end
|
||||
|
||||
it "shuffles behaviors before running" do
|
||||
last = nil
|
||||
runs = []
|
||||
|
||||
@ex.use { last = "control" }
|
||||
@ex.try { last = "candidate" }
|
||||
|
||||
10000.times do
|
||||
@ex.run
|
||||
runs << last
|
||||
end
|
||||
|
||||
assert runs.uniq.size > 1
|
||||
end
|
||||
|
||||
it "re-raises exceptions raised during publish by default" do
|
||||
ex = Scientist::Experiment.new("hello")
|
||||
assert_kind_of Scientist::Default, ex
|
||||
def ex.publish(result)
|
||||
raise "boomtown"
|
||||
end
|
||||
|
||||
ex.use { "control" }
|
||||
ex.try { "candidate" }
|
||||
|
||||
exception = assert_raises RuntimeError do
|
||||
ex.run
|
||||
end
|
||||
|
||||
assert_equal "boomtown", exception.message
|
||||
end
|
||||
|
||||
it "reports publishing errors" do
|
||||
def @ex.publish(result)
|
||||
raise "boomtown"
|
||||
end
|
||||
|
||||
@ex.use { "control" }
|
||||
@ex.try { "candidate" }
|
||||
|
||||
assert_equal "control", @ex.run
|
||||
|
||||
op, exception = @ex.exceptions.pop
|
||||
|
||||
assert_equal :publish, op
|
||||
assert_equal "boomtown", exception.message
|
||||
end
|
||||
|
||||
it "publishes results" do
|
||||
@ex.use { 1 }
|
||||
@ex.try { 1 }
|
||||
assert_equal 1, @ex.run
|
||||
assert @ex.published_result
|
||||
end
|
||||
|
||||
it "does not publish results when there is only a control value" do
|
||||
@ex.use { 1 }
|
||||
assert_equal 1, @ex.run
|
||||
assert_nil @ex.published_result
|
||||
end
|
||||
|
||||
it "compares results with a comparator block if provided" do
|
||||
@ex.compare { |a, b| a == b.to_s }
|
||||
@ex.use { "1" }
|
||||
@ex.try { 1 }
|
||||
|
||||
assert_equal "1", @ex.run
|
||||
assert @ex.published_result.matched?
|
||||
end
|
||||
|
||||
it "knows how to compare two experiments" do
|
||||
a = Scientist::Observation.new(@ex, "a") { 1 }
|
||||
b = Scientist::Observation.new(@ex, "b") { 2 }
|
||||
|
||||
assert @ex.observations_are_equivalent?(a, a)
|
||||
refute @ex.observations_are_equivalent?(a, b)
|
||||
end
|
||||
|
||||
it "uses a compare block to determine if observations are equivalent" do
|
||||
a = Scientist::Observation.new(@ex, "a") { "1" }
|
||||
b = Scientist::Observation.new(@ex, "b") { 1 }
|
||||
@ex.compare { |x, y| x == y.to_s }
|
||||
assert @ex.observations_are_equivalent?(a, b)
|
||||
end
|
||||
|
||||
it "reports errors in a compare block" do
|
||||
@ex.compare { raise "boomtown" }
|
||||
@ex.use { "control" }
|
||||
@ex.try { "candidate" }
|
||||
|
||||
assert_equal "control", @ex.run
|
||||
|
||||
op, exception = @ex.exceptions.pop
|
||||
|
||||
assert_equal :compare, op
|
||||
assert_equal "boomtown", exception.message
|
||||
end
|
||||
|
||||
it "reports errors in the enabled? method" do
|
||||
def @ex.enabled?
|
||||
raise "kaboom"
|
||||
end
|
||||
|
||||
@ex.use { "control" }
|
||||
@ex.try { "candidate" }
|
||||
assert_equal "control", @ex.run
|
||||
|
||||
op, exception = @ex.exceptions.pop
|
||||
|
||||
assert_equal :enabled, op
|
||||
assert_equal "kaboom", exception.message
|
||||
end
|
||||
|
||||
it "reports errors in a run_if block" do
|
||||
@ex.run_if { raise "kaboom" }
|
||||
@ex.use { "control" }
|
||||
@ex.try { "candidate" }
|
||||
assert_equal "control", @ex.run
|
||||
|
||||
op, exception = @ex.exceptions.pop
|
||||
|
||||
assert_equal :run_if, op
|
||||
assert_equal "kaboom", exception.message
|
||||
end
|
||||
|
||||
it "returns the given value when no clean block is configured" do
|
||||
assert_equal 10, @ex.clean_value(10)
|
||||
end
|
||||
|
||||
it "calls the configured clean block with a value when configured" do
|
||||
@ex.clean do |value|
|
||||
value.upcase
|
||||
end
|
||||
|
||||
assert_equal "TEST", @ex.clean_value("test")
|
||||
end
|
||||
|
||||
it "reports an error and returns the original value when an error is raised in a clean block" do
|
||||
@ex.clean { |value| raise "kaboom" }
|
||||
|
||||
@ex.use { "control" }
|
||||
@ex.try { "candidate" }
|
||||
assert_equal "control", @ex.run
|
||||
|
||||
assert_equal "control", @ex.published_result.control.cleaned_value
|
||||
|
||||
op, exception = @ex.exceptions.pop
|
||||
|
||||
assert_equal :clean, op
|
||||
assert_equal "kaboom", exception.message
|
||||
end
|
||||
|
||||
describe "#run_if" do
|
||||
it "does not run the experiment if the given block returns false" do
|
||||
candidate_ran = false
|
||||
run_check_ran = false
|
||||
|
||||
@ex.use { 1 }
|
||||
@ex.try { candidate_ran = true; 1 }
|
||||
|
||||
@ex.run_if { run_check_ran = true; false }
|
||||
|
||||
@ex.run
|
||||
|
||||
assert run_check_ran
|
||||
refute candidate_ran
|
||||
end
|
||||
|
||||
it "runs the experiment if the given block returns true" do
|
||||
candidate_ran = false
|
||||
run_check_ran = false
|
||||
|
||||
@ex.use { true }
|
||||
@ex.try { candidate_ran = true }
|
||||
|
||||
@ex.run_if { run_check_ran = true }
|
||||
|
||||
@ex.run
|
||||
|
||||
assert run_check_ran
|
||||
assert candidate_ran
|
||||
end
|
||||
end
|
||||
|
||||
describe "#ignore_mismatched_observation?" do
|
||||
before do
|
||||
@a = Scientist::Observation.new(@ex, "a") { 1 }
|
||||
@b = Scientist::Observation.new(@ex, "b") { 2 }
|
||||
end
|
||||
|
||||
it "does not ignore an observation if no ignores are configured" do
|
||||
refute @ex.ignore_mismatched_observation?(@a, @b)
|
||||
end
|
||||
|
||||
it "calls a configured ignore block with the given observed values" do
|
||||
called = false
|
||||
@ex.ignore do |a, b|
|
||||
called = true
|
||||
assert_equal @a.value, a
|
||||
assert_equal @b.value, b
|
||||
true
|
||||
end
|
||||
|
||||
assert @ex.ignore_mismatched_observation?(@a, @b)
|
||||
assert called
|
||||
end
|
||||
|
||||
it "calls multiple ignore blocks to see if any match" do
|
||||
called_one = called_two = called_three = false
|
||||
@ex.ignore { |a, b| called_one = true; false }
|
||||
@ex.ignore { |a, b| called_two = true; false }
|
||||
@ex.ignore { |a, b| called_three = true; false }
|
||||
refute @ex.ignore_mismatched_observation?(@a, @b)
|
||||
assert called_one
|
||||
assert called_two
|
||||
assert called_three
|
||||
end
|
||||
|
||||
it "only calls ignore blocks until one matches" do
|
||||
called_one = called_two = called_three = false
|
||||
@ex.ignore { |a, b| called_one = true; false }
|
||||
@ex.ignore { |a, b| called_two = true; true }
|
||||
@ex.ignore { |a, b| called_three = true; false }
|
||||
assert @ex.ignore_mismatched_observation?(@a, @b)
|
||||
assert called_one
|
||||
assert called_two
|
||||
refute called_three
|
||||
end
|
||||
|
||||
it "reports exceptions raised in an ignore block and returns false" do
|
||||
def @ex.exceptions
|
||||
@exceptions ||= []
|
||||
end
|
||||
|
||||
def @ex.raised(op, exception)
|
||||
exceptions << [op, exception]
|
||||
end
|
||||
|
||||
@ex.ignore { raise "kaboom" }
|
||||
|
||||
refute @ex.ignore_mismatched_observation?(@a, @b)
|
||||
|
||||
op, exception = @ex.exceptions.pop
|
||||
assert_equal :ignore, op
|
||||
assert_equal "kaboom", exception.message
|
||||
end
|
||||
|
||||
it "skips ignore blocks that raise and tests any remaining blocks if an exception is swallowed" do
|
||||
def @ex.exceptions
|
||||
@exceptions ||= []
|
||||
end
|
||||
|
||||
# this swallows the exception rather than re-raising
|
||||
def @ex.raised(op, exception)
|
||||
exceptions << [op, exception]
|
||||
end
|
||||
|
||||
@ex.ignore { raise "kaboom" }
|
||||
@ex.ignore { true }
|
||||
|
||||
assert @ex.ignore_mismatched_observation?(@a, @b)
|
||||
assert_equal 1, @ex.exceptions.size
|
||||
end
|
||||
end
|
||||
|
||||
describe "raising on mismatches" do
|
||||
before do
|
||||
@old_raise_on_mismatches = Fake.raise_on_mismatches?
|
||||
end
|
||||
|
||||
after do
|
||||
Fake.raise_on_mismatches = @old_raise_on_mismatches
|
||||
end
|
||||
|
||||
it "raises when there is a mismatch if raise on mismatches is enabled" do
|
||||
Fake.raise_on_mismatches = true
|
||||
@ex.use { "fine" }
|
||||
@ex.try { "not fine" }
|
||||
|
||||
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
|
||||
end
|
||||
|
||||
it "doesn't raise when there is a mismatch if raise on mismatches is disabled" do
|
||||
Fake.raise_on_mismatches = false
|
||||
@ex.use { "fine" }
|
||||
@ex.try { "not fine" }
|
||||
|
||||
@ex.run
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
describe Scientist::Observation do
|
||||
|
||||
before do
|
||||
@experiment = Scientist::Experiment.new "test"
|
||||
end
|
||||
|
||||
it "observes and records the execution of a block" do
|
||||
ob = Scientist::Observation.new("test", @experiment) do
|
||||
sleep 0.1
|
||||
"ret"
|
||||
end
|
||||
|
||||
assert_equal "ret", ob.value
|
||||
refute ob.raised?
|
||||
assert_in_delta 0.1, ob.duration, 0.01
|
||||
end
|
||||
|
||||
it "stashes exceptions" do
|
||||
ob = Scientist::Observation.new("test", @experiment) do
|
||||
raise "exception"
|
||||
end
|
||||
|
||||
assert ob.raised?
|
||||
assert_equal "exception", ob.exception.message
|
||||
assert_nil ob.value
|
||||
end
|
||||
|
||||
it "compares values" do
|
||||
a = Scientist::Observation.new("test", @experiment) { 1 }
|
||||
b = Scientist::Observation.new("test", @experiment) { 1 }
|
||||
|
||||
assert a.equivalent_to?(b)
|
||||
|
||||
x = Scientist::Observation.new("test", @experiment) { 1 }
|
||||
y = Scientist::Observation.new("test", @experiment) { 2 }
|
||||
|
||||
refute x.equivalent_to?(y)
|
||||
end
|
||||
|
||||
it "compares exception messages" do
|
||||
a = Scientist::Observation.new("test", @experiment) { raise "error" }
|
||||
b = Scientist::Observation.new("test", @experiment) { raise "error" }
|
||||
|
||||
assert a.equivalent_to?(b)
|
||||
|
||||
x = Scientist::Observation.new("test", @experiment) { raise "error" }
|
||||
y = Scientist::Observation.new("test", @experiment) { raise "ERROR" }
|
||||
|
||||
refute x.equivalent_to?(y)
|
||||
end
|
||||
|
||||
FirstErrror = Class.new(StandardError)
|
||||
SecondError = Class.new(StandardError)
|
||||
|
||||
it "compares exception classes" do
|
||||
x = Scientist::Observation.new("test", @experiment) { raise FirstError, "error" }
|
||||
y = Scientist::Observation.new("test", @experiment) { raise SecondError, "error" }
|
||||
z = Scientist::Observation.new("test", @experiment) { raise FirstError, "error" }
|
||||
|
||||
assert x.equivalent_to?(z)
|
||||
refute x.equivalent_to?(y)
|
||||
end
|
||||
|
||||
it "compares values using a comparator block" do
|
||||
a = Scientist::Observation.new("test", @experiment) { 1 }
|
||||
b = Scientist::Observation.new("test", @experiment) { "1" }
|
||||
|
||||
refute a.equivalent_to?(b)
|
||||
assert a.equivalent_to?(b) { |x, y| x.to_s == y.to_s }
|
||||
|
||||
yielded = []
|
||||
a.equivalent_to?(b) do |x, y|
|
||||
yielded << x
|
||||
yielded << y
|
||||
true
|
||||
end
|
||||
assert_equal [a.value, b.value], yielded
|
||||
end
|
||||
|
||||
describe "#cleaned_value" do
|
||||
it "returns the observation's value by default" do
|
||||
a = Scientist::Observation.new("test", @experiment) { 1 }
|
||||
assert_equal 1, a.cleaned_value
|
||||
end
|
||||
|
||||
it "uses the experiment's clean block to clean a value when configured" do
|
||||
@experiment.clean { |value| value.upcase }
|
||||
a = Scientist::Observation.new("test", @experiment) { "test" }
|
||||
assert_equal "TEST", a.cleaned_value
|
||||
end
|
||||
end
|
||||
|
||||
end
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
describe Scientist::Result do
|
||||
before do
|
||||
@experiment = Scientist::Experiment.new "experiment"
|
||||
end
|
||||
|
||||
it "is immutable" do
|
||||
control = Scientist::Observation.new("control", @experiment)
|
||||
candidate = Scientist::Observation.new("candidate", @experiment)
|
||||
|
||||
result = Scientist::Result.new @experiment,
|
||||
observations: [control, candidate], control: control
|
||||
|
||||
assert result.frozen?
|
||||
end
|
||||
|
||||
it "evaluates its observations" do
|
||||
a = Scientist::Observation.new("a", @experiment) { 1 }
|
||||
b = Scientist::Observation.new("b", @experiment) { 1 }
|
||||
|
||||
assert a.equivalent_to?(b)
|
||||
|
||||
result = Scientist::Result.new @experiment, observations: [a, b], control: a
|
||||
assert result.matched?
|
||||
refute result.mismatched?
|
||||
assert_equal [], result.mismatched
|
||||
|
||||
x = Scientist::Observation.new("x", @experiment) { 1 }
|
||||
y = Scientist::Observation.new("y", @experiment) { 2 }
|
||||
z = Scientist::Observation.new("z", @experiment) { 3 }
|
||||
|
||||
result = Scientist::Result.new @experiment, observations: [x, y, z], control: x
|
||||
refute result.matched?
|
||||
assert result.mismatched?
|
||||
assert_equal [y, z], result.mismatched
|
||||
end
|
||||
|
||||
it "has no mismatches if there is only a control observation" do
|
||||
a = Scientist::Observation.new("a", @experiment) { 1 }
|
||||
result = Scientist::Result.new @experiment, observations: [a], control: a
|
||||
assert result.matched?
|
||||
end
|
||||
|
||||
it "evaluates observations using the experiment's compare block" do
|
||||
a = Scientist::Observation.new("a", @experiment) { "1" }
|
||||
b = Scientist::Observation.new("b", @experiment) { 1 }
|
||||
|
||||
@experiment.compare { |x, y| x == y.to_s }
|
||||
|
||||
result = Scientist::Result.new @experiment, observations: [a, b], control: a
|
||||
|
||||
assert result.matched?, result.mismatched
|
||||
end
|
||||
|
||||
it "does not ignore any mismatches when nothing's ignored" do
|
||||
x = Scientist::Observation.new("x", @experiment) { 1 }
|
||||
y = Scientist::Observation.new("y", @experiment) { 2 }
|
||||
|
||||
result = Scientist::Result.new @experiment, observations: [x, y], control: x
|
||||
|
||||
assert result.mismatched?
|
||||
refute result.ignored?
|
||||
end
|
||||
|
||||
it "uses the experiment's ignore block to ignore mismatched observations" do
|
||||
x = Scientist::Observation.new("x", @experiment) { 1 }
|
||||
y = Scientist::Observation.new("y", @experiment) { 2 }
|
||||
called = false
|
||||
@experiment.ignore { called = true }
|
||||
|
||||
result = Scientist::Result.new @experiment, observations: [x, y], control: x
|
||||
|
||||
refute result.mismatched?
|
||||
assert result.ignored?
|
||||
assert_equal [], result.mismatched
|
||||
assert_equal [y], result.ignored
|
||||
assert called
|
||||
end
|
||||
|
||||
it "partitions observations into mismatched and ignored when applicable" do
|
||||
x = Scientist::Observation.new("x", @experiment) { :x }
|
||||
y = Scientist::Observation.new("y", @experiment) { :y }
|
||||
z = Scientist::Observation.new("z", @experiment) { :z }
|
||||
|
||||
@experiment.ignore { |control, candidate| candidate == :y }
|
||||
|
||||
result = Scientist::Result.new @experiment, observations: [x, y, z], control: x
|
||||
|
||||
assert result.mismatched?
|
||||
assert result.ignored?
|
||||
assert_equal [y], result.ignored
|
||||
assert_equal [z], result.mismatched
|
||||
end
|
||||
|
||||
it "knows the experiment's name" do
|
||||
a = Scientist::Observation.new("a", @experiment) { 1 }
|
||||
b = Scientist::Observation.new("b", @experiment) { 1 }
|
||||
result = Scientist::Result.new @experiment, observations: [a, b], control: a
|
||||
|
||||
assert_equal @experiment.name, result.experiment_name
|
||||
end
|
||||
|
||||
it "has the context from an experiment" do
|
||||
@experiment.context :foo => :bar
|
||||
a = Scientist::Observation.new("a", @experiment) { 1 }
|
||||
b = Scientist::Observation.new("b", @experiment) { 1 }
|
||||
result = Scientist::Result.new @experiment, observations: [a, b], control: a
|
||||
|
||||
assert_equal({:foo => :bar}, result.context)
|
||||
end
|
||||
|
||||
end
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
describe Scientist do
|
||||
it "has a version or whatever" do
|
||||
assert Scientist::VERSION
|
||||
end
|
||||
|
||||
it "provides a helper to instantiate and run experiments" do
|
||||
obj = Object.new
|
||||
obj.extend(Scientist)
|
||||
|
||||
r = obj.science "test" do |e|
|
||||
e.use { :control }
|
||||
e.try { :candidate }
|
||||
end
|
||||
|
||||
assert_equal :control, r
|
||||
end
|
||||
|
||||
it "provides an empty default_scientist_context" do
|
||||
obj = Object.new
|
||||
obj.extend(Scientist)
|
||||
|
||||
assert_equal Hash.new, obj.default_scientist_context
|
||||
end
|
||||
|
||||
it "respects default_scientist_context" do
|
||||
obj = Object.new
|
||||
obj.extend(Scientist)
|
||||
|
||||
def obj.default_scientist_context
|
||||
{ :default => true }
|
||||
end
|
||||
|
||||
experiment = nil
|
||||
|
||||
obj.science "test" do |e|
|
||||
experiment = e
|
||||
e.context :inline => true
|
||||
e.use { }
|
||||
end
|
||||
|
||||
refute_nil experiment
|
||||
assert_equal true, experiment.context[:default]
|
||||
assert_equal true, experiment.context[:inline]
|
||||
end
|
||||
end
|
||||
ΠΠ°Π³ΡΡΠ·ΠΊΠ°β¦
Π‘ΡΡΠ»ΠΊΠ° Π² Π½ΠΎΠ²ΠΎΠΉ Π·Π°Π΄Π°ΡΠ΅