github
/
dat-science
зеркало из https://github.com/github/dat-science.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Drop dat-analysis libs and tests. 

These will be in a separate `dat-analysis` project.
This commit is contained in:
Rick Bradley 2013-04-23 17:53:19 -05:00
Родитель 7b76b7ac4e
Коммит 5350f3838a
24 изменённых файлов: 0 добавлений и 1773 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

446
lib/dat/analysis.rb
Просмотреть файл

@ -1,446 +0,0 @@
module Dat
# Public: Analyze the findings of an Experiment
#
# Typically implementors will wish to subclass this to provide their own
# implementations of the following methods suited to the environment where
# `dat-science` is being used: `#read`, `#count`, `#cook`.
#
# Example:
#
# class AnalyzeThis < Dat::Analysis
# # Read a result out of our redis stash
# def read
# RedisHandle.rpop "scienceness.#{experiment_name}.results"
# end
#
# # Query our redis stash to see how many new results are pending
# def count
# RedisHandle.llen("scienceness.#{experiment_name}.results")
# end
#
# # Deserialize a JSON-encoded result from redis
# def cook(raw_result)
# return nil unless raw_result
# JSON.parse raw_result
# end
# end
class Analysis
# Public: Returns the name of the experiment
attr_reader :experiment_name
# Public: Returns the current science mismatch result
attr_reader :current
# Public: an alias for #current
alias_method :result, :current
# Public: Returns a raw ("un-cooked") version of the current science mismatch result
attr_reader :raw
# Public: Gets/Sets the base path for loading matcher and wrapper classes.
# Note that the base path will be appended with the experiment name
# before searching for wrappers and matchers.
attr_accessor :path
# Public: Create a new Dat::Analysis object. Will load any matcher and
# wrapper classes for this experiment if `#path` is non-nil.
#
# experiment_name - The String naming the experiment to analyze.
#
# Examples
#
# analyzer = Dat::Analysis.new('bcrypt-passwords')
# => #<Dat::Analysis:...>
def initialize(experiment_name)
@experiment_name = experiment_name
@wrappers = []
load_classes unless path.nil? rescue nil
end
# Public: process a raw science mismatch result to make it usable in analysis.
# This is typically overridden by subclasses to do any sort of unmarshalling
# or deserialization required.
#
# raw_result - a raw science mismatch result, typically, as returned by `#read`
#
# Returns a "cooked" science mismatch result.
def cook(raw_result)
raw_result
end
# Public: fetch and summarize pending science mismatch results until an
# an unrecognized result is found. Outputs summaries to STDOUT. May
# modify current mismatch result.
#
# Returns nil. Leaves current mismatch result set to first unknown result,
# if one is found.
def analyze
track do
while true
unless more?
fetch # clear current result
return summarize_unknown_result
end
fetch
break if unknown?
summarize
count_as_seen identify
end
print "\n"
summarize_unknown_result
end
end
# Public: skip pending mismatch results not satisfying the provided block.
# May modify current mismatch result.
#
# &block - block accepting a prepared mismatch result and returning true
# or false.
#
# Examples:
#
# skip do |result|
# result.user.staff?
# end
#
# skip do |result|
# result['group']['id'] > 100 && result['url'] =~ %r{/admin}
# end
#
# skip do |result|
# result['timestamp'].to_i > 1.hour.ago
# end
#
# Returns nil if no satisfying results are found. Current result will be nil.
# Returns count of remaining results if a satisfying result found. Leaves
# current result set to first result for which block returns a truthy value.
def skip(&block)
raise ArgumentError, "a block is required" unless block_given?
while more?
fetch
return count if yield(current)
end
# clear current result since nothing of interest was found.
@current = @identified = nil
end
# Public: Are additional science mismatch results available?
#
# Returns true if more results can be fetched.
# Returns false if no more results can be fetched.
def more?
count != 0
end
# Public: retrieve a new science mismatch result, as returned by `#read`.
#
# Returns nil if no new science mismatch results are available.
# Returns a cooked and wrapped science mismatch result if available.
# Raises NoMethodError if `#read` is not defined on this class.
def fetch
@identified = nil
@raw = read
@current = raw ? prepare(raw) : nil
end
# Public: Return a readable representation of the current science mismatch
# result. This will utilize the `#readable` methods declared on a matcher
# which identifies the current result.
#
# Returns a string containing a readable representation of the current
# science mismatch result.
# Returns nil if there is no current result.
def summary
return nil unless current
recognizer = identify
return readable unless recognizer && recognizer.respond_to?(:readable)
recognizer.readable
end
# Public: Print a readable summary for the current science mismatch result
# to STDOUT.
#
# Returns nil.
def summarize
puts summary
end
# Public: Is the current science mismatch result unidentifiable?
#
# Returns nil if current result is nil.
# Returns true if no matcher can identify current result.
# Returns false if a single matcher can identify the current result.
# Raises RuntimeError if multiple matchers can identify the current result.
def unknown?
return nil if current.nil?
!identify
end
# Public: Find a matcher which can identify the current science mismatch result.
#
# Returns nil if current result is nil.
# Returns matcher class if a single matcher can identify current result.
# Returns false if no matcher can identify the current result.
# Raises RuntimeError if multiple matchers can identify the current result.
def identify
return @identified if @identified
results = registry.identify(current)
if results.size > 1
report_multiple_matchers(results)
end
@identified = results.first
end
# Internal: Output failure message about duplicate matchers for a science
# mismatch result.
#
# dupes - Array of Dat::Analysis::Matcher instances, initialized with a result
#
# Raises RuntimeError.
def report_multiple_matchers(dupes)
puts "\n\nMultiple matchers identified result:"
puts
dupes.each_with_index do |matcher, i|
print " #{i+1}. "
if matcher.respond_to?(:readable)
puts matcher.readable
else
puts readable
end
end
puts
raise "Result cannot be uniquely identified."
end
# Internal: cook and wrap a raw science mismatch result.
#
# raw_result - an unmodified result, typically, as returned by `#read`
#
# Returns the science mismatch result processed by `#cook` and then by `#wrap`.
def prepare(raw_result)
wrap(cook(raw_result))
end
# Internal: wrap a "cooked" science mismatch result with any known wrapper methods
#
# cooked_result - a "cooked" mismatch result, as returned by `#cook`
#
# Returns the cooked science mismatch result, which will now respond to any
# instance methods found on our known wrapper classes
def wrap(cooked_result)
cooked_result.extend Dat::Analysis::Result::DefaultMethods
if !wrappers.empty?
cooked_result.send(:instance_variable_set, '@analyzer', self)
class << cooked_result
define_method(:method_missing) do |meth, *args|
found = nil
@analyzer.wrappers.each do |wrapper|
next unless wrapper.public_instance_methods.detect {|m| m.to_s == meth.to_s }
found = wrapper.new(self).send(meth, *args)
break
end
found
end
end
end
cooked_result
end
# Internal: Return the *default* readable representation of the current science
# mismatch result. This method is typically overridden by subclasses or defined
# in matchers which wish to customize the readable representation of a science
# mismatch result. This implementation is provided as a default.
#
# Returns a string containing a readable representation of the current
# science mismatch result.
def readable
synopsis = []
synopsis << "Experiment %-20s first: %10s @ %s" % [
"[#{current['experiment']}]", current['first'], current['timestamp']
]
synopsis << "Duration: control (%6.2f) | candidate (%6.2f)" % [
current['control']['duration'], current['candidate']['duration']
]
synopsis << ""
if current['control']['exception']
synopsis << "Control raised exception:\n\t#{current['control']['exception'].inspect}"
else
synopsis << "Control value: [#{current['control']['value']}]"
end
if current['candidate']['exception']
synopsis << "Candidate raised exception:\n\t#{current['candidate']['exception'].inspect}"
else
synopsis << "Candidate value: [#{current['candidate']['value']}]"
end
synopsis << ""
remaining = current.keys - ['control', 'candidate', 'experiment', 'first', 'timestamp']
remaining.sort.each do |key|
if current[key].respond_to?(:keys)
# do ordered sorting of hash keys
subkeys = key_sort(current[key].keys)
synopsis << "\t%15s => {" % [ key ]
subkeys.each do |subkey|
synopsis << "\t%15s %15s => %-20s" % [ '', subkey, current[key][subkey].inspect ]
end
synopsis << "\t%15s }" % [ '' ]
else
synopsis << "\t%15s => %-20s" % [ key, current[key] ]
end
end
synopsis.join "\n"
end
def preferred_fields
%w(id name title owner description login username)
end
def key_sort(keys)
str_keys = keys.map {|k| k.to_s }
(preferred_fields & str_keys) + (str_keys - preferred_fields)
end
# Public: Which matcher classes are known?
#
# Returns: list of Dat::Analysis::Matcher classes known to this analyzer.
def matchers
registry.matchers
end
# Public: Which wrapper classes are known?
#
# Returns: list of Dat::Analysis::Result classes known to this analyzer.
def wrappers
registry.wrappers
end
# Public: Add a matcher or wrapper class to this analyzer.
#
# klass - a subclass of either Dat::Analysis::Matcher or Dat::Analysis::Result
# to be registered with this analyzer.
#
# Returns the list of known matchers and wrappers for this analyzer.
def add(klass)
klass.add_to_analyzer(self)
end
# Public: Load matcher and wrapper classes from the library for our experiment.
#
# Returns: a list of loaded matcher and wrapper classes.
def load_classes
new_classes = library.select_classes do
experiment_files.each { |file| load file }
end
new_classes.map {|klass| add klass }
end
# Internal: Print to STDOUT a readable summary of the current (unknown) science
# mismatch result, as well a summary of the tally of identified science mismatch
# results analyzed to this point.
#
# Returns nil if there are no pending science mismatch results.
# Returns the number of pending science mismatch results.
def summarize_unknown_result
tally.summarize
if current
puts "\nFirst unidentifiable result:\n\n"
summarize
else
puts "\nNo unidentifiable results found. \\m/\n"
end
more? ? count : nil
end
# Internal: keep a tally of analyzed science mismatch results.
#
# &block: block which will presumably call `#count_as_seen` to update
# tallies of identified science mismatch results.
#
# Returns: value returned by &block.
def track(&block)
@tally = Tally.new
yield
end
# Internal: Increment count for an object in an ongoing tally.
#
# obj - an Object for which we are recording occurrence counts
#
# Returns updated tally count for obj.
def count_as_seen(obj)
tally.count(obj.class.name || obj.class.inspect)
end
# Internal: The current Tally instance. Cached between calls to `#track`.
#
# Returns the current Tally instance object.
def tally
@tally ||= Tally.new
end
# Internal: handle to the library, used for collecting newly discovered
# matcher and wrapper classes.
#
# Returns: handle to the library class.
def library
Dat::Analysis::Library
end
# Internal: registry of wrapper and matcher classes known to this analyzer.
#
# Returns a (cached between calls) handle to our registry instance.
def registry
@registry ||= Dat::Analysis::Registry.new
end
# Internal: which class files are candidates for loading matchers and wrappers
# for this experiment?
#
# Returns: sorted Array of paths to ruby files which may contain declarations
# of matcher and wrapper classes for this experiment.
def experiment_files
Dir[File.join(path, experiment_name, '*.rb')].sort
end
# Internal: Add a matcher class to this analyzer's registry.
# (Intended to be called only by Dat::Analysis::Matcher and subclasses)
def add_matcher(matcher_class)
puts "Loading matcher class [#{matcher_class}]"
registry.add matcher_class
end
# Internal: Add a wrapper class to this analyzer's registry.
# (Intended to be called only by Dat::Analysis::Result and its subclasses)
def add_wrapper(wrapper_class)
puts "Loading results wrapper class [#{wrapper_class}]"
registry.add wrapper_class
end
end
end
require 'dat/analysis/library'
require 'dat/analysis/matcher'
require 'dat/analysis/result'
require 'dat/analysis/registry'
require 'dat/analysis/tally'

30
lib/dat/analysis/library.rb
Просмотреть файл

@ -1,30 +0,0 @@
module Dat
# Internal: Keep a registry of Dat::Analysis::Matcher and
# Dat::Analysis::Result subclasses for use by an Dat::Analysis::Analysis
# instance.
class Analysis::Library
@@known_classes = []
# Public: Collect matcher and results classes created by the
# provided block.
#
# &block - Block which instantiates matcher and results classes.
#
# Returns the newly-instantiated matcher and results classes.
def self.select_classes(&block)
@@known_classes = [] # prepare for registering new classes
yield
@@known_classes # return all the newly-registered classes
end
# Public: register a matcher or results class.
#
# klass - a Dat::Analysis::Matcher or Dat::Analysis::Result subclass.
#
# Returns the current list of registered classes.
def self.add(klass)
@@known_classes << klass
end
end
end

43
lib/dat/analysis/matcher.rb
Просмотреть файл

@ -1,43 +0,0 @@
module Dat
# Public: Base class for science mismatch results matchers. Subclasses
# implement the `#match?` instance method, which returns true when
# a provided science mismatch result is recognized by the matcher.
#
# Subclasses are expected to define `#match?`.
#
# Subclasses may optionally define `#readable` to return an alternative
# readable String representation of a cooked science mismatch result. The
# default implementation is defined in Dat::Analysis#readable.
class Analysis::Matcher
# Public: The science mismatch result to be matched.
attr_reader :result
# Internal: Called at subclass instantiation time to register the subclass
# with Dat::Analysis::Library.
#
# subclass - The Dat::Analysis::Matcher subclass being instantiated.
#
# Not intended to be called directly.
def self.inherited(subclass)
Dat::Analysis::Library.add subclass
end
# Internal: Add this class to a Dat::Analysis instance. Intended to be
# called from Dat::Analysis to dispatch registration.
#
# analyzer - a Dat::Analysis instance for an experiment
#
# Returns the analyzer's updated list of known matcher classes.
def self.add_to_analyzer(analyzer)
analyzer.add_matcher self
end
# Public: create a new Matcher.
#
# result - a science mismatch result, to be tested via `#match?`
def initialize(result)
@result = result
end
end
end

50
lib/dat/analysis/registry.rb
Просмотреть файл

@ -1,50 +0,0 @@
module Dat
# Internal: Registry of Dat::Analysis::Matcher and Dat::Analysis::Result
# classes. This is used to maintain the mapping of matchers and
# results wrappers for a particular Dat::Analysis instance.
class Analysis::Registry
# Public: Create a new Registry instance.
def initialize
@known_classes = []
end
# Public: Add a matcher or results wrapper class to the registry
#
# klass - a Dat::Analysis::Matcher subclass or a Dat::Analysis::Result
# subclass, to be added to the registry.
#
# Returns the list of currently registered classes.
def add(klass)
@known_classes << klass
end
# Public: Get the list of known Dat::Analysis::Matcher subclasses
#
# Returns the list of currently known matcher classes.
def matchers
@known_classes.select {|c| c <= ::Dat::Analysis::Matcher }
end
# Public: Get the list of known Dat::Analysis::Result subclasses
#
# Returns the list of currently known result wrapper classes.
def wrappers
@known_classes.select {|c| c <= ::Dat::Analysis::Result }
end
# Public: Get list of Dat::Analysis::Matcher subclasses for which
# `#match?` is truthy for the given result.
#
# result - a cooked science mismatch result
#
# Returns a list of matchers initialized with the provided result.
def identify(result)
matchers.inject([]) do |hits, matcher|
instance = matcher.new(result)
hits << instance if instance.match?
hits
end
end
end
end

78
lib/dat/analysis/result.rb
Просмотреть файл

@ -1,78 +0,0 @@
require "time"
module Dat
# Public: Base class for wrappers around science mismatch results.
#
# Instance methods defined on subclasses will be added as instance methods
# on science mismatch results handled by Dat::Analysis instances which
# add the wrapper subclass via Dat::Analysis#add or Dat::Analysis#load_classes.
class Analysis::Result
# Public: return the current science mismatch result
attr_reader :result
# Internal: Called at subclass instantiation time to register the subclass
# with Dat::Analysis::Library.
#
# subclass - The Dat::Analysis::Result subclass being instantiated.
#
# Not intended to be called directly.
def self.inherited(subclass)
Dat::Analysis::Library.add subclass
end
# Internal: Add this class to a Dat::Analysis instance. Intended to be
# called from Dat::Analysis to dispatch registration.
#
# analyzer - a Dat::Analysis instance for an experiment
#
# Returns the analyzer's updated list of known result wrapper classes.
def self.add_to_analyzer(analyzer)
analyzer.add_wrapper self
end
# Public: create a new Result wrapper.
#
# result - a science mismatch result, to be wrapped with our instance methods.
def initialize(result)
@result = result
end
end
module Analysis::Result::DefaultMethods
# Public: Get the result data for the 'control' code path.
#
# Returns the 'control' field of the result hash.
def control
self['control']
end
# Public: Get the result data for the 'candidate' code path.
#
# Returns the 'candidate' field of the result hash.
def candidate
self['candidate']
end
# Public: Get the timestamp when the result was recorded.
#
# Returns a Time object for the timestamp for this result.
def timestamp
@timestamp ||= Time.parse(self['timestamp'])
end
# Public: Get which code path was run first.
#
# Returns the 'first' field of the result hash.
def first
self['first']
end
# Public: Get the experiment name
#
# Returns the 'experiment' field of the result hash.
def experiment_name
self['experiment']
end
end
end

59
lib/dat/analysis/tally.rb
Просмотреть файл

@ -1,59 +0,0 @@
module Dat
# Internal: Track and summarize counts of occurrences of mismatch objects.
#
# Examples
#
# tally = Dat::Analysis::Tally.new
# tally.count('foo')
# => 1
# tally.count('bar')
# => 1
# tally.count('foo')
# => 2
# puts tally.summary
# Summary of known mismatches found:
# foo 2
# bar 1
# TOTAL: 3
# => nil
#
class Analysis::Tally
# Public: Returns the hash of recorded mismatches.
attr_reader :tally
def initialize
@tally = {}
end
# Public: record an occurrence of a mismatch class.
def count(klass)
tally[klass] ||= 0
tally[klass] += 1
end
# Public: Return a String summary of mismatches seen so far.
#
# Returns a printable String summarizing the counts of mismatches seen,
# sorted in descending count order.
def summary
return "\nNo results identified.\n" if tally.keys.empty?
result = [ "\nSummary of identified results:\n" ]
sum = 0
tally.keys.sort_by {|k| -1*tally[k] }.each do |k|
sum += tally[k]
result << "%30s: %6d" % [k, tally[k]]
end
result << "%30s: %6d" % ['TOTAL', sum]
result.join "\n"
end
# Public: prints a summary of mismatches seen so far to STDOUT (see
# `#summary` above).
#
# Returns nil.
def summarize
puts summary
end
end
end

119
test/dat_analysis_subclassing_test.rb
Просмотреть файл

@ -1,119 +0,0 @@
require "minitest/autorun"
require "mocha/setup"
require "dat/analysis"
# helper class to provide mismatch results
class TestCookedAnalyzer < Dat::Analysis
attr_accessor :mismatches
def initialize(experiment_name)
super
@mismatches = [] # use a simple array for a mismatch store
end
# load data files from our fixtures path
def path
File.expand_path('fixtures/', __FILE__)
end
def cook(raw_result)
return "cooked" unless raw_result
"cooked-#{raw_result}"
end
def count
mismatches.size
end
def read
mismatches.pop
end
# neuter formatter to take simple non-structured results
def readable
current.inspect
end
# neuter calls to `puts`, make it possible to test them.
def puts(*args)
@last_printed = args.join('')
nil
end
attr_reader :last_printed # for tests: last call to puts
# neuter calls to 'print' to eliminate test output clutter
def print(*args) end
end
class DatAnalysisSubclassingTest < MiniTest::Unit::TestCase
def setup
@experiment_name = 'test-suite-experiment'
@analyzer = ::TestCookedAnalyzer.new @experiment_name
end
def test_is_0_when_count_is_overridden_and_there_are_no_mismatches
assert_equal 0, @analyzer.count
end
def test_returns_the_count_of_mismatches_when_count_is_overridden
@analyzer.mismatches.push 'mismatch'
@analyzer.mismatches.push 'mismatch'
assert_equal 2, @analyzer.count
end
def test_fetch_returns_nil_when_read_is_overridden_and_read_returns_no_mismatches
assert_nil @analyzer.fetch
end
def test_fetch_returns_the_cooked_version_of_the_next_mismatch_from_read_when_read_is_overridden
@analyzer.mismatches.push 'mismatch'
assert_equal 'cooked-mismatch', @analyzer.fetch
end
def test_raw_returns_nil_when_no_mismatches_have_been_fetched_and_cook_is_overridden
assert_nil @analyzer.raw
end
def test_current_returns_nil_when_no_mismatches_have_been_fetch_and_cook_is_overridden
assert_nil @analyzer.current
end
def test_raw_returns_nil_when_last_fetched_returns_no_results_and_cook_is_overridden
@analyzer.fetch
assert_nil @analyzer.raw
end
def test_current_returns_nil_when_last_fetched_returns_no_results_and_cook_is_overridden
@analyzer.fetch
assert_nil @analyzer.current
end
def test_raw_returns_unprocess_mismatch_when_cook_is_overridden
@analyzer.mismatches.push 'mismatch-1'
result = @analyzer.fetch
assert_equal 'mismatch-1', @analyzer.raw
end
def test_current_returns_a_cooked_mismatch_when_cook_is_overridden
@analyzer.mismatches.push 'mismatch-1'
result = @analyzer.fetch
assert_equal 'cooked-mismatch-1', @analyzer.current
end
def test_raw_updates_with_later_fetches_when_cook_is_overridden
@analyzer.mismatches.push 'mismatch-1'
@analyzer.mismatches.push 'mismatch-2'
@analyzer.fetch # discard the first one
@analyzer.fetch
assert_equal 'mismatch-1', @analyzer.raw
end
def test_current_updates_with_later_fetches_when_cook_is_overridden
@analyzer.mismatches.push 'mismatch-1'
@analyzer.mismatches.push 'mismatch-2'
@analyzer.fetch # discard the first one
@analyzer.fetch
assert_equal 'cooked-mismatch-1', @analyzer.current
end
end

822
test/dat_analysis_test.rb
Просмотреть файл

@ -1,822 +0,0 @@
require "minitest/autorun"
require "mocha/setup"
require "dat/analysis"
require "time"
# helper class to provide mismatch results
class TestMismatchAnalysis < Dat::Analysis
attr_accessor :mismatches
def initialize(experiment_name)
super
@mismatches = [] # use a simple array for a mismatch store
end
# load data files from our fixtures path
def path
File.expand_path(File.join(File.dirname(__FILE__), 'fixtures'))
end
def count
mismatches.size
end
def read
mismatches.pop
end
# neuter formatter to take simple non-structured results
def readable
current.inspect
end
# neuter calls to `puts`, make it possible to test them.
def puts(*args)
@last_printed = args.join('')
nil
end
attr_reader :last_printed # for tests: last call to puts
# neuter calls to 'print' to eliminate test output clutter
def print(*args) end
end
# for testing that a non-registered Recognizer class can still
# supply a default `#readable` method to subclasses
class TestSubclassRecognizer < Dat::Analysis::Matcher
def readable
"experiment-formatter: #{result['extra']}"
end
end
class DatAnalysisTest < MiniTest::Unit::TestCase
def setup
Dat::Analysis::Tally.any_instance.stubs(:puts)
@experiment_name = 'test-suite-experiment'
@analyzer = TestMismatchAnalysis.new @experiment_name
@timestamp = Time.now
@result = {
'experiment' => @experiment_name,
'control' => {
'duration' => 0.03,
'exception' => nil,
'value' => true,
},
'candidate' => {
'duration' => 1.03,
'exception' => nil,
'value' => false,
},
'first' => 'candidate',
'extra' => 'bacon',
'timestamp' => @timestamp.to_s
}
end
def test_preserves_the_experiment_name
assert_equal @experiment_name, @analyzer.experiment_name
end
def test_analyze_returns_nil_if_there_is_no_current_result_and_no_additional_results
assert_nil @analyzer.analyze
end
def test_analyze_leaves_tallies_empty_if_there_is_no_current_result_and_no_additional_results
@analyzer.analyze
assert_equal({}, @analyzer.tally.tally)
end
def test_analyze_returns_nil_if_there_is_a_current_result_but_no_additional_results
@analyzer.mismatches.push @result
@analyzer.fetch
assert @analyzer.current
assert_nil @analyzer.analyze
end
def test_analyze_leaves_tallies_empty_if_there_is_a_current_result_but_no_additional_results
@analyzer.mismatches.push @result
@analyzer.fetch
assert @analyzer.current
@analyzer.analyze
assert_equal({}, @analyzer.tally.tally)
end
def test_analyze_outputs_default_result_summary_and_tally_summary_when_one_unrecognized_result_is_present
@analyzer.expects(:summarize_unknown_result)
@analyzer.mismatches.push @result
@analyzer.analyze
end
def test_analyze_returns_nil_when_one_unrecognized_result_is_present
@analyzer.mismatches.push @result
assert_nil @analyzer.analyze
end
def test_analyze_leaves_current_result_set_to_first_result_when_one_unrecognized_result_is_present
@analyzer.mismatches.push @result
@analyzer.analyze
assert_equal @result, @analyzer.current
end
def test_analyze_leaves_tallies_empty_when_one_unrecognized_result_is_present
@analyzer.mismatches.push @result
@analyzer.analyze
assert_equal({}, @analyzer.tally.tally)
end
def test_analyze_outputs_default_results_summary_for_first_unrecognized_result_and_tally_summary_when_recognized_and_unrecognized_results_are_present
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
result['extra'] =~ /^known-/
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'known-1')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-1')
@analyzer.mismatches.push @result.merge('extra' => 'known-2')
@analyzer.expects(:summarize_unknown_result)
@analyzer.analyze
end
def test_analyze_returns_number_of_unanalyzed_results_when_recognized_and_unrecognized_results_are_present
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
result['extra'] =~ /^known-/
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'known-1')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-1')
@analyzer.mismatches.push @result.merge('extra' => 'known-2')
assert_equal 1, @analyzer.analyze
end
def test_analyze_leaves_current_result_set_to_first_unrecognized_result_when_recognized_and_unrecognized_results_are_present
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
result['extra'] =~ /^known/
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'known-1')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-1')
@analyzer.mismatches.push @result.merge('extra' => 'known-2')
@analyzer.analyze
assert_equal 'unknown-1', @analyzer.current['extra']
end
def test_analyze_leaves_recognized_result_counts_in_tally_when_recognized_and_unrecognized_results_are_present
matcher1 = Class.new(Dat::Analysis::Matcher) do
def self.name() "RecognizerOne" end
def match?
result['extra'] =~ /^known-1/
end
end
matcher2 = Class.new(Dat::Analysis::Matcher) do
def self.name() "RecognizerTwo" end
def match?
result['extra'] =~ /^known-2/
end
end
@analyzer.add matcher1
@analyzer.add matcher2
@analyzer.mismatches.push @result.merge('extra' => 'known-1-last')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-1')
@analyzer.mismatches.push @result.merge('extra' => 'known-10')
@analyzer.mismatches.push @result.merge('extra' => 'known-20')
@analyzer.mismatches.push @result.merge('extra' => 'known-11')
@analyzer.mismatches.push @result.merge('extra' => 'known-21')
@analyzer.mismatches.push @result.merge('extra' => 'known-12')
@analyzer.analyze
tally = @analyzer.tally.tally
assert_equal [ 'RecognizerOne', 'RecognizerTwo' ], tally.keys.sort
assert_equal 3, tally['RecognizerOne']
assert_equal 2, tally['RecognizerTwo']
end
def test_analyze_proceeds_from_stop_point_when_analyzing_with_more_results
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
result['extra'] =~ /^known-/
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'known-1')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-1')
@analyzer.mismatches.push @result.merge('extra' => 'known-2')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-2')
@analyzer.mismatches.push @result.merge('extra' => 'known-3')
assert_equal 3, @analyzer.analyze
assert_equal 'unknown-2', @analyzer.current['extra']
assert_equal 1, @analyzer.analyze
assert_equal 'unknown-1', @analyzer.current['extra']
assert_equal @analyzer.readable, @analyzer.last_printed
end
def test_analyze_resets_tally_between_runs_when_analyzing_later_results_after_a_stop
matcher1 = Class.new(Dat::Analysis::Matcher) do
def self.name() "RecognizerOne" end
def match?
result['extra'] =~ /^known-1/
end
end
matcher2 = Class.new(Dat::Analysis::Matcher) do
def self.name() "RecognizerTwo" end
def match?
result['extra'] =~ /^known-2/
end
end
@analyzer.add matcher1
@analyzer.add matcher2
@analyzer.mismatches.push @result.merge('extra' => 'known-1-last')
@analyzer.mismatches.push @result.merge('extra' => 'unknown-1')
@analyzer.mismatches.push @result.merge('extra' => 'known-10')
@analyzer.mismatches.push @result.merge('extra' => 'known-20')
@analyzer.mismatches.push @result.merge('extra' => 'known-11')
@analyzer.mismatches.push @result.merge('extra' => 'known-21')
@analyzer.mismatches.push @result.merge('extra' => 'known-12')
@analyzer.analyze # proceed to first stop point
@analyzer.analyze # and continue analysis
assert_equal({'RecognizerOne' => 1}, @analyzer.tally.tally)
end
def test_skip_fails_if_no_block_is_provided
assert_raises(ArgumentError) do
@analyzer.skip
end
end
def test_skip_returns_nil_if_there_is_no_current_result
remaining = @analyzer.skip do |result|
true
end
assert_nil remaining
end
def test_skip_leaves_current_alone_if_the_current_result_satisfies_the_block
@analyzer.mismatches.push @result
@analyzer.skip do |result|
true
end
end
def test_skip_returns_0_if_the_current_result_satisfies_the_block_and_no_other_results_are_available
@analyzer.mismatches.push @result
remaining = @analyzer.skip do |result|
true
end
assert_equal 0, remaining
end
def test_skip_returns_the_number_of_additional_results_if_the_current_result_satisfies_the_block_and_other_results_are_available
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
remaining = @analyzer.skip do |result|
true
end
assert_equal 2, remaining
end
def test_skip_returns_nil_if_no_results_are_satisfying
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
remaining = @analyzer.skip do |result|
false
end
assert_nil remaining
end
def test_skip_skips_all_results_if_no_results_are_satisfying
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
remaining = @analyzer.skip do |result|
false
end
assert !@analyzer.more?
end
def test_skip_leaves_current_as_nil_if_no_results_are_satisfying
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
remaining = @analyzer.skip do |result|
false
end
assert_nil @analyzer.current
end
def test_more_is_false_when_there_are_no_mismatches
assert !@analyzer.more?
end
def test_more_is_true_when_there_are_mismatches
@analyzer.mismatches.push @result
assert @analyzer.more?
end
def test_count_fails
assert_raises(NoMethodError) do
Dat::Analysis.new(@experiment_name).count
end
end
def test_fetch_fails_unless_read_is_implemented_by_a_subclass
assert_raises(NameError) do
Dat::Analysis.new(@experiment_name).fetch
end
end
def test_current_returns_nil_when_no_mismmatches_have_been_fetched
assert_nil @analyzer.current
end
def test_current_returns_nil_when_last_fetch_returned_no_results
@analyzer.fetch
assert_nil @analyzer.current
end
def test_current_returns_the_most_recent_mismatch_when_one_has_been_fetched
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal @result, @analyzer.current
end
def test_current_updates_with_later_fetches
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
@analyzer.fetch
result = @analyzer.fetch
assert_equal result, @analyzer.current
end
def test_result_is_an_alias_for_current
@analyzer.mismatches.push @result
@analyzer.mismatches.push @result
@analyzer.fetch
result = @analyzer.fetch
assert_equal result, @analyzer.result
end
def test_raw_returns_nil_when_no_mismatches_have_been_fetched
assert_nil @analyzer.raw
end
def test_raw_returns_nil_when_last_fetched_returned_no_results
@analyzer.fetch
assert_nil @analyzer.raw
end
def test_raw_returns_an_unprocessed_version_of_the_most_recent_mismatch
@analyzer.mismatches.push @result
result = @analyzer.fetch
assert_equal @result, @analyzer.raw
end
def test_raw_updates_with_later_fetches
@analyzer.mismatches.push 'mismatch-1'
@analyzer.mismatches.push 'mismatch-2'
@analyzer.fetch # discard the first one
@analyzer.fetch
assert_equal 'mismatch-1', @analyzer.raw
end
def test_when_loading_support_classes_loads_no_matchers_if_no_matcher_files_exist_on_load_path
analyzer = TestMismatchAnalysis.new('experiment-with-no-classes')
analyzer.load_classes
assert_equal [], analyzer.matchers
assert_equal [], analyzer.wrappers
end
def test_when_loading_support_classes_loads_matchers_and_wrappers_if_they_exist_on_load_path
analyzer = TestMismatchAnalysis.new('experiment-with-classes')
analyzer.load_classes
assert_equal ["MatcherA", "MatcherB", "MatcherC"], analyzer.matchers.map(&:name)
assert_equal ["WrapperA", "WrapperB", "WrapperC"], analyzer.wrappers.map(&:name)
end
def test_when_loading_support_classes_ignores_extraneous_classes_on_load_path
analyzer = TestMismatchAnalysis.new('experiment-with-good-and-extraneous-classes')
analyzer.load_classes
assert_equal ["MatcherX", "MatcherY", "MatcherZ"], analyzer.matchers.map(&:name)
assert_equal ["WrapperX", "WrapperY", "WrapperZ"], analyzer.wrappers.map(&:name)
end
def test_when_loading_support_classes_loads_classes_at_initialization_time_if_they_are_available
analyzer = TestMismatchAnalysis.new('initialize-classes')
assert_equal ["MatcherM", "MatcherN"], analyzer.matchers.map(&:name)
assert_equal ["WrapperM", "WrapperN"], analyzer.wrappers.map(&:name)
end
def test_when_loading_support_classes_does_not_load_classes_at_initialization_time_if_they_cannot_be_loaded
analyzer = TestMismatchAnalysis.new('invalid-matcher')
assert_equal [], analyzer.matchers
end
def test_loading_classes_post_initialization_fails_if_loading_has_errors
# fails at #load_classes time since we define #path later
analyzer = Dat::Analysis.new('invalid-matcher')
analyzer.path = File.expand_path(File.join(File.dirname(__FILE__), 'fixtures'))
assert_raises(Errno::EACCES) do
analyzer.load_classes
end
end
def test_result_has_an_useful_timestamp
@analyzer.mismatches.push(@result)
result = @analyzer.fetch
assert_equal @timestamp.to_i, result.timestamp.to_i
end
def test_result_has_a_method_for_first
@analyzer.mismatches.push(@result)
result = @analyzer.fetch
assert_equal @result['first'], result.first
end
def test_result_has_a_method_for_control
@analyzer.mismatches.push(@result)
result = @analyzer.fetch
assert_equal @result['control'], result.control
end
def test_result_has_a_method_for_candidate
@analyzer.mismatches.push(@result)
result = @analyzer.fetch
assert_equal @result['candidate'], result.candidate
end
def test_result_has_a_method_for_experiment_name
@analyzer.mismatches.push(@result)
result = @analyzer.fetch
assert_equal @result['experiment'], result.experiment_name
end
def test_results_helper_methods_are_not_available_on_results_unless_loaded
@analyzer.mismatches.push @result
result = @analyzer.fetch
assert_raises(NoMethodError) do
result.repository
end
end
def test_results_helper_methods_are_made_available_on_returned_results
wrapper = Class.new(Dat::Analysis::Result) do
def repository
'github/dat-science'
end
end
@analyzer.add wrapper
@analyzer.mismatches.push @result
result = @analyzer.fetch
assert_equal 'github/dat-science', result.repository
end
def test_results_helper_methods_can_be_loaded_from_multiple_classes
wrapper1 = Class.new(Dat::Analysis::Result) do
def repository
'github/dat-science'
end
end
wrapper2 = Class.new(Dat::Analysis::Result) do
def user
:rick
end
end
@analyzer.add wrapper1
@analyzer.add wrapper2
@analyzer.mismatches.push @result
result = @analyzer.fetch
assert_equal 'github/dat-science', result.repository
assert_equal :rick, result.user
end
def test_results_helper_methods_are_made_available_in_the_order_loaded
wrapper1 = Class.new(Dat::Analysis::Result) do
def repository
'github/dat-science'
end
end
wrapper2 = Class.new(Dat::Analysis::Result) do
def repository
'github/linguist'
end
def user
:rick
end
end
@analyzer.add wrapper1
@analyzer.add wrapper2
@analyzer.mismatches.push @result
result = @analyzer.fetch
assert_equal 'github/dat-science', result.repository
assert_equal :rick, result.user
end
def test_results_helper_methods_do_not_hide_existing_result_methods
wrapper = Class.new(Dat::Analysis::Result) do
def size
'huge'
end
end
@analyzer.add wrapper
@analyzer.mismatches.push 'mismatch-1'
result = @analyzer.fetch
assert_equal 10, result.size
end
def test_methods_can_access_the_result_using_the_result_method
wrapper = Class.new(Dat::Analysis::Result) do
def esrever
result.reverse
end
end
@analyzer.add wrapper
@analyzer.mismatches.push 'mismatch-1'
result = @analyzer.fetch
assert_equal 'mismatch-1'.reverse, result.esrever
end
def test_summarize_returns_nil_and_prints_the_empty_string_if_no_result_is_current
assert_nil @analyzer.summarize
assert_equal "", @analyzer.last_printed
end
def test_summarize_returns_nil_and_prints_the_default_readable_result_if_a_result_is_current_but_no_matchers_are_known
@analyzer.mismatches.push @result
@analyzer.fetch
assert_nil @analyzer.summarize
assert_equal @analyzer.readable, @analyzer.last_printed
end
def test_summarize_returns_nil_and_prints_the_default_readable_result_if_a_result_is_current_but_not_matched_by_any_known_matchers
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
false
end
def readable
'this should never run'
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_nil @analyzer.summarize
assert_equal @analyzer.readable, @analyzer.last_printed
end
def test_summarize_returns_nil_and_prints_the_matchers_readable_result_when_a_result_is_current_and_matched_by_a_matcher
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
def readable
"recognized: #{result['extra']}"
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'mismatch-1')
@analyzer.fetch
assert_nil @analyzer.summarize
assert_equal "recognized: mismatch-1", @analyzer.last_printed
end
def test_summarize_returns_nil_and_prints_the_default_readable_result_when_a_result_is_matched_by_a_matcher_with_no_formatter
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_nil @analyzer.summarize
assert_equal @analyzer.readable, @analyzer.last_printed
end
def test_summarize_supports_use_of_a_matcher_base_class_for_shared_formatting
matcher = Class.new(TestSubclassRecognizer) do
def match?
true
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'mismatch-1')
@analyzer.fetch
assert_nil @analyzer.summarize
assert_equal "experiment-formatter: mismatch-1", @analyzer.last_printed
end
def test_summary_returns_nil_if_no_result_is_current
assert_nil @analyzer.summary
end
def test_summary_returns_the_default_readable_result_if_a_result_is_current_but_no_matchers_are_known
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal @analyzer.readable, @analyzer.summary
end
def test_summary_returns_the_default_readable_result_if_a_result_is_current_but_not_matched_by_any_known_matchers
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
false
end
def readable
'this should never run'
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal @analyzer.readable, @analyzer.summary
end
def test_summary_returns_the_matchers_readable_result_when_a_result_is_current_and_matched_by_a_matcher
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
def readable
"recognized: #{result['extra']}"
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'mismatch-1')
@analyzer.fetch
assert_equal "recognized: mismatch-1", @analyzer.summary
end
def test_summary_formats_with_the_default_formatter_if_a_matching_matcher_does_not_define_a_formatter
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal @analyzer.readable, @analyzer.summary
end
def test_summary_supports_use_of_a_matcher_base_class_for_shared_formatting
matcher = Class.new(TestSubclassRecognizer) do
def match?
true
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result.merge('extra' => 'mismatch-1')
@analyzer.fetch
assert_equal "experiment-formatter: mismatch-1", @analyzer.summary
end
def test_unknown_returns_nil_if_no_result_is_current
assert_nil @analyzer.unknown?
end
def test_unknown_returns_true_if_a_result_is_current_but_no_matchers_are_known
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal true, @analyzer.unknown?
end
def test_unknown_returns_true_if_current_result_is_not_matched_by_any_known_matchers
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
false
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal true, @analyzer.unknown?
end
def test_unknown_returns_false_if_a_matcher_class_matches_the_current_result
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal false, @analyzer.unknown?
end
def test_identify_returns_nil_if_no_result_is_current
assert_nil @analyzer.identify
end
def test_identify_returns_nil_if_a_result_is_current_but_no_matchers_are_known
@analyzer.mismatches.push @result
@analyzer.fetch
assert_nil @analyzer.identify
end
def test_identify_returns_nil_if_current_result_is_not_matched_by_any_known_matchers
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
false
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_nil @analyzer.identify
end
def test_identify_returns_the_matcher_class_which_matches_the_current_result
matcher = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
end
@analyzer.add matcher
@analyzer.mismatches.push @result
@analyzer.fetch
assert_equal matcher, @analyzer.identify.class
end
def test_identify_fails_if_more_than_one_matcher_class_matches_the_current_result
matcher1 = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
end
matcher2 = Class.new(Dat::Analysis::Matcher) do
def match?
true
end
end
@analyzer.add matcher1
@analyzer.add matcher2
@analyzer.mismatches.push @result
@analyzer.fetch
assert_raises(RuntimeError) do
@analyzer.identify
end
end
end

7
test/fixtures/analysis/test-suite-experiment/matcher.rb поставляемый
Просмотреть файл

@ -1,7 +0,0 @@
class MatcherA < Dat::Analysis::Matcher
end
class MatcherB < Dat::Analysis::Matcher
end

5
test/fixtures/experiment-with-classes/matcher_a.rb поставляемый
Просмотреть файл

@ -1,5 +0,0 @@
class MatcherA < Dat::Analysis::Matcher
def match?
result =~ /^known/
end
end

11
test/fixtures/experiment-with-classes/matcher_b.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class MatcherB < Dat::Analysis::Matcher
def match?
result =~ /b/
end
end
class MatcherC < Dat::Analysis::Matcher
def match?
result =~ /c/
end
end

5
test/fixtures/experiment-with-classes/wrapper_a.rb поставляемый
Просмотреть файл

@ -1,5 +0,0 @@
class WrapperA < Dat::Analysis::Result
def repository
'wrapper-a-repository'
end
end

11
test/fixtures/experiment-with-classes/wrapper_b.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class WrapperB < Dat::Analysis::Result
def repository
'wrapper-b-repository'
end
end
class WrapperC < Dat::Analysis::Result
def repository
'wrapper-c-repository'
end
end

5
test/fixtures/experiment-with-good-and-extraneous-classes/matcher_w.rb поставляемый
Просмотреть файл

@ -1,5 +0,0 @@
class MatcherW
def match?
true
end
end

11
test/fixtures/experiment-with-good-and-extraneous-classes/matcher_y.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class MatcherX < Dat::Analysis::Matcher
def match?
result =~ /b/
end
end
class MatcherY < Dat::Analysis::Matcher
def match?
result =~ /c/
end
end

11
test/fixtures/experiment-with-good-and-extraneous-classes/matcher_z.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class MatcherZ < Dat::Analysis::Matcher
def match?
result =~ /^known/
end
end
class MatcherV
def match?
true
end
end

5
test/fixtures/experiment-with-good-and-extraneous-classes/wrapper_w.rb поставляемый
Просмотреть файл

@ -1,5 +0,0 @@
class WrapperW
def match?
true
end
end

11
test/fixtures/experiment-with-good-and-extraneous-classes/wrapper_y.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class WrapperX < Dat::Analysis::Result
def user
'wrapper-x-user'
end
end
class WrapperY < Dat::Analysis::Result
def user
'wrapper-y-user'
end
end

11
test/fixtures/experiment-with-good-and-extraneous-classes/wrapper_z.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class WrapperZ < Dat::Analysis::Result
def user
'wrapper-z-user'
end
end
class WrapperV
def user
'wrapper-v-user'
end
end

5
test/fixtures/initialize-classes/matcher_m.rb поставляемый
Просмотреть файл

@ -1,5 +0,0 @@
class MatcherM < Dat::Analysis::Matcher
def match?
result =~ /^known/
end
end

11
test/fixtures/initialize-classes/matcher_n.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class MatcherN < Dat::Analysis::Matcher
def match?
result =~ /n/
end
end
class MatcherO
def match?
true
end
end

5
test/fixtures/initialize-classes/wrapper_m.rb поставляемый
Просмотреть файл

@ -1,5 +0,0 @@
class WrapperM < Dat::Analysis::Result
def repository
'wrapper-m-repository'
end
end

11
test/fixtures/initialize-classes/wrapper_n.rb поставляемый
Просмотреть файл

@ -1,11 +0,0 @@
class WrapperN < Dat::Analysis::Result
def repository
'wrapper-n-repository'
end
end
class WrapperO
def repository
'wrapper-o-repository'
end
end

1
test/fixtures/invalid-matcher/matcher.rb поставляемый
Просмотреть файл

@ -1 +0,0 @@
raise Errno::EACCES