pjs/mstone/ToDo

Mstone TODO List

Updated:
	3-20-2000	Dan Christian

======================================================================
Minor improvements

Data reduction at each testbed client

Test message generator (plain and MIME styles)

More graphs for multiple runs

Option to drop a fraction of connections

IMAP delete without reading
	Just get the message list and delete everything.  This could
be the fastest way to drain out the store.


Display MIN/MAX or standard deviation on graphs
	Gnuplot can do data points with error bars.  You could either
use MIN/MAX or the standard deviation for the error bars.  There are
issues with calculating the standard deviation numbers throughout the
graph that need to be addressed.


Statistics reset
	At least the MIN and MAX statistics could be easily reset
during the test (after ramp up).  This keeps the transients during
startup and shutdown from dominating these numbers.  The standard
deviation statistics are much trickier to reset during the run.  It
may be better to isolate sections in post processing.


Perl web server
	Sometimes it would be nice to include our own web server to
provide the results (instead of using file: URLs).  This would also be
a gateway to form based test configuration editing and allow results
to be interactively updated during tests.  Perl with a socket library
could handle this without too much trouble.


Dynamic test loading
	Finalize an API for dynamically loading tests.  This would
allow tests to be added or updated separately from the core
functionality.  This may be needed for some types of security testing.


Link graphs to the results tables
	There are already tags by each graph.  You should be able to
link the appropriate results table entry to each graph.  This sort of
tricky since graphs often combine multiple entries.


Show statistics for every block
	Statistics are actually kept for every block in every thread.
There should be a way to view information at this detail.


Man pages
	The online docs are nice, but good 'ol man pages would be
great in a different way.


Reduce namespace polution
	Scripts names like setup and cleanup are too general.  They

should be part of the main 'mstone' script (e.g. mstone setup).

Examples of script series
	Include example scripts to run entire series of tests in a
sane way.


Fix FORMAT clash
	At the start of a test, each client process outputs the
information needed to report all its protocols.  When there are
multiple processes on one client, these FORMAT lines can intermix and
cause parsing errors.


Set connection drop rate
	Drop some percentage of the connections without a proper
shutdown.  This tests how well a server can detect and recover from
hard disconnects.


Improve randomness
	The way that we generate random numbers in a range may be not
generate the proper randomness.  We are using lrand48()%range we
should use (lrand48/RAND_RANGE)*range.  There are some end conditions
that need to be thought about.  All of this is in the sequence code;
one change should fix (or break :) everything.

	Also, we may be generating numbers that are never used.  This
may be costly, and can create holes in the proper sequence.


Improve printing
	The color graphs are great on screen, but tend to print
poorly.  Either we need a better way to generate a printable version
(maybe through a CGI button), or the seperate protocols need to be
printed individually.  Also, Communicator does a lousy job of keeping
title with tables or graphs.  Hopefully, Mozilla will do better.


======================================================================
Whole new protocol tests:

ICQ
	Test high volume instant messaging.  Very useful for the
bridges and gateways that people are considering using.


WAP
	WAP is the emerging standard for mobile phones.


WCAP
	Web based calendar services


LDAP
	Use the LDAP SDK to do basic LDAP testing.  The SDK probably
isn't fast enough to call this a real performance test, but you can at
least test performance degredation due to load from a real application
(like a mail server).


DNS
	Mail servers use DNS a lot.  You should at least be able to
see if performance is degrading due to load.


Disk/filesystem
	Test read, write, sync, link, un-link, and append performance under
multiple threads.


Cert servers
	Test certificate authenticity checking performance


======================================================================
Possible dummy servers:

SMTP
	Receive mail via SMTP and ignore it.  Usefull for SMTP relay
testing.


DNS
	Simulate slow DNS server lookups.  Usefull for SMTP relay testing.


======================================================================
Major changes

Throttling
	Monitor and control transaction rates so that specific load
levels can be easily specified.  The rates should be able to vary to
simulate peek hour usage and disconnect-restore.


NSPR threading
	Use NSPR for threading and sockets.  This may allow other OSes
to be used as clients.  This might be easy, since mstone does not need
any locking, just simple thread creation and harvesting.  NSPR
argument parsing and hashes may also be useful.


SSL
	Support SSL on the protocols that allow it.  May require NSPR.


Line speed emulation
	Simulate the variable delays and limited throughput of dial up
connections.


Scripting
	Allow more detailed control of protocol tests.  It looks
difficult to make this scalable and fast.


Combined tests
	Deliver mail over SMTP and then see when it arrives using IMAP.