Mailstone Utility

This document describes how to use Mailstone 4.1 for capacity planning and testing on Netscape Messaging Server and other mail servers.

This document contains the following sections:


About the Mailstone Utility
The Mailstone utility is a stress-testing tool that system administrators can use to determine Netscape Messaging Server capacity by testing how the server performs under load. You can use Mailstone on all Unix platforms supported by Netscape Messaging Server and also on Windows NT.

Mailstone can also be used to similarly test and compare other mail servers that support SMTP, POP, and IMAP protocols.

Mailstone simulates tasks that users might perform. It lets you simulate worst-case loads--for example, Monday at 9 a.m., when most users are checking their mail and sending and receiving messages--so you can determine what maximum load your messaging server can take.

Mailstone can be used to test the load-carrying capabilities of a system configuration before it is put into use. Based on how well a mail server handles the loads generated by Mailstone, you can determine whether your current deployment configuration is adequate, or whether you need additional or more powerful server machines.

Mailstone measures both throughput and response time of the mail server being tested. Mailstone reports the number and size of messages sent, the rate at which messages (and bytes) pass through the system, and the average time taken to both transmit a message to the server and receive a message from the server. Mailstone can present its results in both text and graphical format.

Mailstone is run on a client machine (referred to as testmaster in these instructions).

The Mailstone testbed consists of the testmaster machine, any testclient machines, the Messaging Server to be tested, plus the network configuration and Directory Server. (See Figure 0.1.)

Figure 0.1 Mailstone Testbed (Unix environment)

When Mailstone is run on the testmaster machine using the mstone command the following happens:

  1. Command line arguments are combined with configuration files to define the test.
  2. The test processes are executed across the testbed. (In Unix environments rsh is used to run the processes on the other client machines.)
  3. Results are collected and displayed on the HTML results page.

Installing Mailstone
You can install Mailstone at the same time as Messaging Server by choosing Mailstone as one of the Messaging Server components to be installed. See your Messaging Server installation instructions for details on how to do this. The Mailstone software that is installed at the same time as Messaging Server is a version for the same operating system used by Messaging Server.

The instructions in this section describe how to install an all-platforms version of Mailstone separate from a Messaging Server installation. This Mailstone version can be used to test Messaging Server or any other Mailstone-compatible email server. (If you initially install Mailstone as part of Messaging Server installation and later want to install the all-platforms version of Mailstone, see Reinstalling Mailstone.)

To transfer, extract, and install the all-platforms version of the Mailstone archive files follow these steps:

  1. Locate the Mailstone archive file for your platform.
  2. (The contents of both files are identical. Different file types are provided for convenience.)

    For Unix the file is mailstone-4.15.tar.gz.

    For Windows NT the file is mailstone-4.15.zip.

  3. Choose the testmaster machine on which you will run Mailstone and copy the archive file to a temporary installation directory, such as /tmp, on that machine.
  4. The testmaster machine can be the same machine as the server being tested, or a different machine.

  5. Go to the testmaster directory immediately above where you want the Mailstone software to be installed.
  6. For example, if you go to /usr/netscape Mailstone will be installed in
    /usr/netscape/mailstone.

  7. Extract the Mailstone files.
  8. In Unix environments:

    gunzip -c /tmp/mailstone-4.15.tar.gz | tar xf -

    In Windows NT environments:

    unzip c:\temp\mailstone-4.15.zip

    The extraction process creates the required Mailstone files and places them in a mailstone directory and associated subdirectories.

    See Table 0.2 for a description of the main Mailstone files.

Once Mailstone is installed, you set up (configure) each test as described in Setting Up Mailstone Tests and then run the mstone program as described in Synchronizing Client Machines.

Reinstalling Mailstone

If you initially install Mailstone as part of Messaging Server installation and later want to install the all-platforms version of Mailstone, or you want to reinstall Mailstone for any other reason, you must first perform the following steps before using the installation procedure described above.

  1. Go to the mailstone directory.
  2. For example: cd /usr/netscape/mailstone.

  3. Run the cleanup program to remove installed files.
  4. Type: ./cleanup.

  5. Rename the mailstone/conf directory.
  6. Type: mv conf conf.save.

After you install the new version of Mailstone, you can copy any needed files from conf.save into the new conf directory.


Setting Up Mailstone Tests
To configure for a Mailstone test run, you typically perform the following steps:

  1. Go to the mailstone directory (e.g. cd /usr/netscape/mailstone).
  2. Run setup config. This will display the LICENSE and ask if you agree to it.
  3. Answer yes to set up an initial configuration.
  4. Enter the name of the mail server.
  5. Enter the test user account name and password. The user account names should be separate from any actual users and should be unique to the server being tested (e.g. mailhost-test).
  6. By default, the user number will be appended to the end of the user account name. If the name contains %ld, then this will be replaced with the user number. If the password contains %ld, then it will be replaced with the user number (otherwise, no numbering will be used in the password).

  7. Enter the test client machine names. For NT, only localhost is allowed.
  8. Answer yes to create an LDIF file for users and a broadcast account.
  9. Answer yes to view the new configuration file. Make any additional changes to conf/general.html with a text editor. See Configuring Test Execution for details about adding clients and controlling test execution. To re-run the simple configuration editor, run setup config.
  10. Edit the contents of your System Data file to specify the system configuration data you want appended to test results.
  11. By default, this file is named conf/general.html. See System Data File (general.html).

  12. On UNIX systems, make sure you can rsh to each client machine. You may have to edit the ~/.rhosts files to enable permission.
  13. Run setup to copy messages and executables to test clients machines. If you encounter problems with configuration or access permissions, correct them and re-run setup.
  14. (Even though there are no additional testclient machines in Windows environments, you must still run setup because the testmaster machine acts as a testclient machine.)

  15. Important. If you are using more than one testclient machine, run checktime as described in Synchronizing Client Machines to verify that the system clocks are within 2 seconds of each other.
If you add/change client machines or change test execution parameters, run setup again to copy the necessary files to each test client machine.

Establishing the Test User Base

In order to test a messaging server, you need to create a set of test users. During set up, you created an LDIF file that will create these users in the Directory Server used by Netscape Messaging Server. If you are testing another kind of mail server, you will have to use other means to create these accounts.

You then add the test LDIF file's entries to your directory. You can use either of the following to update the LDAP Directory from the conf/mailhost.ldif file:

Caution: You should backup the Directory Server before making any changes.

Adding Test Users with Database Manager

You can use the Database Manager feature to add the test user account information in the conf/mailhost.ldif file to the LDAP Directory on your instance of Directory Server.

Note: This process will go faster if you first disable access logging on Directory Server.

Directory Server 4.x. To add test users with Directory Server version 4.x, follow these steps:

  1. Start up Netscape Console and log in.
  2. Open the Server Group containing the instance of Directory Server you want to use.
  3. Double-click the name of the Directory Server.
  4. Click the `Configuration' tab.
  5. Right-click Database and select the `Import' option.
  6. Enter the name of the file containing your LDIF modifications in the dialog box.
  7. For example, /usr/netscape/mailstone/conf/mailhost.ldif.

  8. Select `Append Data to Database'.
  9. Select `Add Only'.
  10. Select `Continue on Error'.
  11. Click `OK'.
See Appendix for steps for adding users under Directory 3.x or using ldapmodify.

Synchronizing Client Machines

For accurate test results it is important that the testmaster and all of the testclient machines show exactly the same time on their system clocks. If at all possible, the times should be identical to less than a second.

Windows NT Note: Since you only have one machine in a Windows NT environment, synchronization as described in this section is not necessary.

To check for time consistency run the checktime utility. The checktime utility uses the following syntax:

checktime [-w machine.tbd]

By default, the checktime utility uses testbed data contained in the machine configuration file (all.tbd). If you wish to use a different machine configuration file, you specify that file on the command line with the -w option.

The checktime utility reports the time for each system. If there are discrepancies of more than a second you need to adjust the system clocks.

The timesync utility is stored in the mailstone directory. You must run timesync as root, and you must have root rsh permissions. The timesync utility uses the following syntax:

timesync [-w machine.tbd]

By default, the timesync utility uses testbed data contained in the machine configuration file named all.tbd. If you wish to use a different machine configuration file, you specify that file on the command line with the -w option.


Running Mailstone
To run a Mailstone test:

  1. Go to the mailstone directory (e.g. cd /usr/netscape/mailstone).
  2. If you have additional testclient machines in a Unix environment, synchronize them as described in Synchronizing Client Machines.
  3. Run the mstone command using the syntax:
  4. mstone testname [options]

    where testname is the test workload file. For example:

    mstone popsmtp

  5. View the results by copying the display URL into a browser.
You can specify command line options as needed. Running mstone with no arguments will show the most common options.

Options specified on the command line override any corresponding configuration parameters in a configuration file. For example, if the configuration file sets the test run time to two hours, but you only want to make a brief trial run of 10 minutes to make sure that the configuration is correct, you would enter this:

mstone popsmtp -t 10m

Table 0.1 describes the command line options you can use with the mstone command.

Table 0.1 mstone command line options

Option
Description
-a addgraphs
Specify additional graph sections.
-b banner
Specifies the banner (title) to be displayed as part of test results, where banner is the actual text you want displayed enclosed by quotation marks. For example:
-b 'POP reads (full store)'

-c configfile
Back compatibility with Mailstone 4.0/4.1.
-d
Debug mode. Large amounts of information is output to the stderr log.
-l load
Specify the number of clients to use.
-m machinefile
Back compatibility with Mailstone 4.0/4.1.
-n `notes'
Specify additional notes (`comments' in the CONFIG section).
-r ramptime
Specifies the ramp time, where ramptime is a number and time unit designation. For example, to set a ramp time of 10 seconds: -r 10s
-s sysconfig
Specify the system data file.
-t time
Specifies the length of time the test is to run, where time is a number and time unit designation. For example, to set a test run of 20 minutes: -t 20m
-v
Be more verbose during report processing.
-w workfile
Specify additional workload files.
PARAM=value
Set attribute PARAM in the CONFIG section.

Setting Up Test Mailboxes

Even though you have created test users, the Messaging Server does not create each mailbox until it is accessed. One way to create mailboxes for the test users in your LDAP directory is by simply running a Mailstone test.

To create mailboxes for your test users using the script, follow these steps:

  1. Run the pop test as follows:
  2. ./mstone pop -t 1m

    This will access each login and force the mailbox to be created.

  3. Look at how many logins were achieved in one minute and re-run with enough time to get to all mailboxes.
Checking Results in Mid Test

During a long test run, you can display the current results by running the process command. For example:

process

The process command does not stop or abort a test run; it simply displays the results of the current test up to the point the command was run. In other words, if you run process after 45 minutes, you will see results based on the first 45 minutes of data.

The process command can be used early in a long test run to make sure that it is functioning properly with the correct configuration parameters.

Note: Do not run process when the test run has less than a few minutes to go before finishing.

Redisplaying Old Test Results

You can also use the process command to reprocess and display older test runs by specifying the data directory on the command line. For example, to reprocess the results of a test that was run on July 14, 1999 at 14:31 (24 hour clock) which are stored in the results/19990714.1431 directory, you would enter this:

process 19990714.1431

Aborting a Test Run

To stop a test run before it has completed, enter Control-C or use any other standard method of halting an ongoing process. Then run process to generate a report up to the point of termination.


Reviewing Test Results
All of the results files from each Mailstone test run are placed in a time-stamped subdirectory of the results directory. For example, the results of a test run on July 14, 1999 at 14:31 (24 hour clock) are stored in a subdirectory named: results/19990714.1431.

The Results Page

Mailstone displays the results of a test run as an HTML page named results.html. There is one results file in each test run subdirectory.

An HTML-formatted index of all test runs is created and maintained in the results/index.html file. This file contains live links to the results.html files from every test run.

The top portion of a typical results file is shown in Figure 0.2. (Your results will vary depending on how you configured your test run.)

Figure 0.2 Example Mailstone test result page (top portion)

In this illustration:

Following the test result tables, the results file displays the tests results as a series of graphs.

About the Report Columns

Mailstone breaks each client session into timers. Each timer tracks these variables:

The bytes written and read represent data bytes for the retrieve and submit timers, and protocol bytes for all others. For readability, values may be displayed in K bytes (1024), M bytes (1048576), and G bytes (1073741824) and rounded off.

The number of successful operations is tries minus errors.

Note that actual precision of the time values depends on the precision of the system clock and on statistical sampling. The times are displayed in seconds unless other units are shown.

About the Report Rows

Each row in the report is a timer for a specific aspect of the protocol test. The one exception is "Total", which is a sum of the other timers. Details of how each timer is used in each protocol are found in Detailed Protocol Test Steps. Here is the current list of possible timers:

Some stages may be bypasses depending on the protocol and workload. For example, the SMTP protocol will only do a "Login" if useEHLO and useAuthLogin are enabled and the server supports AUTHLOGIN.

There are also some special timers. The block timer counts the number of workload command blocks that have been executed. This can help indicate problems with the workload files.

The mailstone results tables contain two sections:

Advanced Reports

You may want to extract additional information from the data that Mailstone collects. Summaries of all the information are available in two additional formats for further processing.

A text version of the results is also produced in results/timestamp/results.txt. This is the same information as the HTML report, but in a machine friendly format. All byte counts are in bytes. All time values are in seconds. The protocol name and timer are listed on each line of the result tables. The operation rate entries end in `/s'.

If advanced customization of test reporting is needed, import the results/timestamp/clients-*.csv files into a spreadsheet. The sum of all the clients-*.csv files should equal the results shown in Mailstone's report.

Further graphing can be done by importing the results/timestamp/time-*.csv files into a spreadsheet. Each file contains the sum of all client's timers over the duration of the test for one protocol.

Additional Details

At the end of the results.html file under the heading "Details" is a description of the test environment. This is the text you specified in the system data file (conf/general.html). This allows you to capture all the configuration that is beyond the scope of Mailstone, but critical to what you are testing. Typical information to document includes: software version, operating system version, hardware capacities, network configuration, disk organization, caching configuration, and so on. A simple description is shown in Figure 0.3.

Figure 0.3 Example Mailstone test results page (test description)

Test and Configuration Files

Mailstone files are stored in a directory named mailstone with several subdirectories. All programs must be run from the mailstone directory. These directories are created when the Mailstone files are extracted with tar (Unix) or are unzip (NT). File names are the same for both Unix and NT platforms, except that in some cases NT files have filename extensions such as .exe or .bat.

The main Mailstone files are listed in Table 0.2 (ancillary files are not listed).

Table 0.2 Important Mailstone Files

File
Description
mstone
The command script that you run to perform a Mailstone test as described in Synchronizing Client Machines.
cleanup
A script that removes Mailstone software and files from testclient machines (Unix environments only).
INSTALL
A text file giving additional information on installation procedures.
process
A script to reprocess test results with different labels and descriptions. This does not rerun the test.
README
A text file providing additional information about using Mailstone.
setup
A script that prepares your systems for a test series by copying the correct version of Mailstone software and the specified test messages to the /var/tmp directories in Unix environments or the %TEMP% directory in NT environments. See Synchronizing Client Machines.
checktime
A utility that checks the current system time on all machines that are part of the test. (Unix environments only). See Synchronizing Client Machines.
timesync
A utility the synchronizes the system time on all machines that are part of the test. (Unix environments only). See Synchronizing Client Machines.
conf/general.html
The system data file that describes the system configuration. The contents of general.html are appended to the test result output. See System Data File (general.html).
conf/general.wld
The general configuration file you use to specify configuration parameters for the test as a whole. See General Configuration File (general.wld).
conf/*.wld
The workload configuration files you use to specify tests. See Workload Configuration File (testname.wld).
conf/sample.wld
A workload file with examples of all sections and attributes.
data/*.msg
The "filler" message bodies that Mailstone uses when it sends messages during testing. See Test Messages.

Configuring Test Execution

Configuration Files

Configuration files describe both how the test will be run (e.g. what are the client machines and client counts) and the test itself. A number of test configurations are supplied and you can add more test configurations as needed. All configuration files are stored in the conf directory.

Each configuration files is structured into sections using common XML notation. A section begins with a type tag and ends with a matching closing tag. Each section can appear multiple times.

Within a section, each line specifies an attribute name and then its value. Some attributes can appear more than once. A `#' (pound sign) indicates a comment to the end of the line. Lines can be continued by ending in a `\' (back slash). Initial whitespace is ignored (except for line continuations).

Examples of all section types and all attributes are in conf/sample.wld. Here is an example CONFIG section demonstrating the line formatting:

<CONFIG>
title \ # title of this test
POP reads

# blank lines and comment lines don't matter
#telemetry 1 # uncomment this to enable telemetry logs
</CONFIG>

All the details of test execution are performed on the `testmaster' machine. The following section types control test execution: CONFIG, CLIENT, PRETEST, MONITOR, POSTTEST, and GRAPH. All other section types are protocol tests.

Configuration files can include other configuration files using
<include conf/filename.wld>. This allows all the general configuration information to be kept in one file and not to have to be repeated in each test workload. A section must be completely contained in a single file.

There are different kinds of configuration files:

Specifying Times in Configuration Files

Times are specified in configuration commands (or on the command line) by a number followed by one of these time unit designations:

For example, 4m specifies a time of four minutes. No spaces are permitted between the number and the time unit.

General Configuration File (general.wld)

The general configuration file specifies general configuration parameters that apply to all Mailstone tests across this testbed. The general configuration file is named conf/general.wld.

The initial general.wld file contains the following lines:

<Default>
server servername
smtpMailFrom from-address
addressFormat addressformat
loginFormat loginformat
passwdFormat password
numAddresses N
numLogins N
firstAddress N
firstLogin N
</Default>

<CLIENT HOSTS=localhost>
</CLIENT>

System Data File (general.html)

The system data file specifies additional data to be added to the test report. That is, at the end of each test run the information in the system data file is appended to the HTML-formatted test results report. By default, this file is named general.html. Typically, the system data file describes the Messaging Server being tested. However, you can use this file to add any information you wish to the test report.

The system data file is a block of HTML-formatted text that can be inserted into an HTML file.

A typical system data file looks like this:

<PRE>
<B> Mailhost.domain.com </B>
Netscape Messaging Server 4.1 (2/18/99)
E6000
26x250MHz UltraSPARC-II
6.5GB RAM
A3500 w/ 60x9GB 7200 RPM
store: 5 x (10x9 RAID-0)
queue: 1 x (10x9 RAID-0)
</PRE>

Workload Configuration File (testname.wld)

The workload configuration file specifies the services and commands to be tested, their weights, and the test message files to be used. There are sample workload files for different kinds of tests: imap.wld, pop.wld, smtp.wld, popimapsmtp.wld, popsmtp.wld, and wmap.wld. You can create additional workload files as necessary.

For example, the operative portions of the sample popimapsmtp.wld file look like this:

<include conf/general.wld> 

<CONFIG>                         # test specific config
title POP, IMAP, SMTP combined load
clientCount 100 # set an appropriate client count
</CONFIG>

<SMTP HOSTS=client1>
file en-1k.msg
weight 100
blockTime 1s
</SMTP>

<POP3 HOSTS=client1>
weight 50
blockTime 1s
#leaveMailOnServer 1
</POP3>

# Since IMAP connections are much longer lived than POP or SMTP,
# run them on a separate host for clearer results.
<IMAP4 HOSTS=client2,client3>
weight 100
loopTime 60s
numLoops 10
</IMAP4>

This example illustrates most of the features of a test workload. The first line includes the general configuration file to get all the specifics of the system being tested. This allows the test to be independent of most of the system details. We'll see in a minute that we still have to put some specific machine names in for this unusual test.

Next there is a CONFIG section for the test. Since CONFIG is a special section, the attributes override the values in the general configuration file. This makes the general configuration act as test defaults. The command line can further override any value in CONFIG at test run time. Note that a final copy of the test with all attributes specified will be saved with the test report.

Now a note about the pop-imap-smtp test and why it is tricky. The SMTP portion brings new mail into the system. The SMTP connections are short lived. The POP section reads and deletes all the mail in a mailbox. Again, connections are short lived. The IMAP section logs in, and then spends 10 minutes watching for new mail. This large difference in execution times means that mixing IMAP with POP and SMTP will produce difficult to understand results. A short lived section like SMTP might be waiting 10 minutes to run again. This makes it hard to predict delivery rates.

To make things more predictable, the HOSTS attribute has been used to assign different sections to specific machines. The short-lived protocols (SMTP and POP) have been assigned to client1 and the longer-lived IMAP section has been assigned to client2 and client3. Note an alternate way to do this is to set useGroups in the CONFIG section and then group names would have been used instead of machine names. This would keep all machine names out of the test workload.

In addition to splitting sections by machine, the weight section has been used to control the balance between SMTP and POP. The SMTP section which is weighted 100 and will run twice as often as the POP section which is weighted 50. Note that weights only matter for sections running on the same machine.

Description of the CONFIG section

The CONFIG section is used to control the test startup and report generation. Nothing in CONFIG directly affects a particular protocol test.

There is a general CONFIG section in conf/general.wld and an additional CONFIG in each test. All CONFIG sections will be merged into one when the test is run. Later values will override earlier ones.

Every parameter in CONFIG can be overridden from the command line using either a predefined switch, or an ATTRIBUTE=value definition. For example: mstone pop -b `Initialize store' would run the pop test with a title (banner) of `Initialize store'.

Table 0.3 describes the various attributes you can use in the CONFIG section.

Table 0.3 CONFIG section attributes

Attribute
Description
title string
Specify the title of the test. This will appear on the second line of the report. Same as command line switch `-b title'.
comments string
Specify the command for the test. This will appear on the third line of the report. Same as command line switch `-n notes'.
time duration
Specify the test duration. Same as command line switch `-t time'.
rampTime duration
Specifies the duration of the test load ramp up. This is part of the test time. Defaults to 0. Same as command line switch `-r time'.
clientCount N
Specifies how many clients to use for the test. Same as command line switch `-l clients'.
maxErrors N
Specifies the maximum number of total errors to allow before terminating the test.
maxBlocks N
Specifies the maximum number of blocks to allow before terminating the test. If there is only one client, then this number of blocks will be run.
If there is more than one client, then the test will stop some time after this count is reached.

sysConfig file
Specifies the file to be used for the `Details' portion of the report. Same as command line switch `-s filename'.
useGroups N
If 1, use the group name set for each client, instead of its hostname.
telemetry N
If 1, record a log of all transactions. This log is written into the /var/tmp directory on each client machine.
This is intended for protocol debugging and will probably affect performance measurements.

Description of the CLIENT section

The CLIENT section defines one or more machines to be used to run the test. The machines are specified using the HOSTS=client1,client2 section attribute. There are often multiple CLIENT sections.

Note that the machine name specified in the HOSTS= section attribute must be consistent with any other HOSTS= sections. Just because client1 and client1.example.com resolve to the same machine, does not mean that Mailstone understands that they are the same machine.

Mailstone is designed to simulate very large client counts. In order to do this, it can use multiple client machines, multiple processes on each machine, and multiple threads in each process.

Normally, you specify the number of clients using clientCount or the -l count command line parameter. Mailstone then figures out how to assign those clients across machines. If clientCount is lower than the number of CLIENT sections, then some sections will not be used. The machines used, the process counts, and the thread counts are all shown when the test is started.

Because machines may have differing capabilities, and you may need to define the limits of what each machine can handle, the maximum processes and thread counts will depend on operating system, CPU count, memory, and other system loads. The CLIENT section defines the capabilities of the machine, so that an appropriate client count can be assigned to it during test execution. If more than one machine is specified for a CLIENT section, then all machines will run the same client sub-count.

The example below shows two machines that should always run identical loads. The operating system is SunOS5.6 (i.e. Solaris 2.6). This tells Mailstone not to assign more than 500 threads per process or more that 2000 clients total.

<CLIENT HOSTS=client1,client2>
arch SunOS5.6
maxThreads 500
maxClients 2000
</CLIENT>

Note that on NT, only one process is allowed. If the testmaster is running on NT, then only localhost is allowed as a CLIENT.

Table 0.4 describes the various attributes you can use in the CLIENT section.

Table 0.4 CLIENT section attributes

Attribute
Description
arch string
Specify the operation system version for this client. By default, this is the same as the testmaster machine. The subdirectories under bin define the supported arch types. For example: Linux2.2_x86.
group string
Specify a group to assign the client to if useGroups is defined.
command string
Specify the Mailstone client command to run. Defaults to mailclient.
tempDir string
Specify the directory to run from on this machine.
maxClients N
Specifies the maximum number of clients to assign to this machine.
maxThreads N
Specifies the maximum number of threads to run in a single client process.
processes N
Specifies the number of processes to run. If this is set, then Mailstone will try to pick a thread count so that the right clientCount is run.
threads N
Specifies the threads to run. If processes and threads are both set, then this machine will run processes*threads clients regardless of what clientCount is set to.

Description of the PRETEST Section

The PRETEST section defines commands to be run before the test starts. This is often used to check system state before the test (e.g. disk usage, server configuration, etc). The machine(s) are specified using the HOSTS=client1 section attribute.

The results of PRETEST will be linked to the final report.

Table 0.5 describes the various attributes you can use in the PRETEST section.

Table 0.5 PRETEST section attributes

Attribute
Description
command string
Specify the command to run. For example: df -k
rsh string
Specify the remote shell command used to run command. For example: rsh -l mailuser would cause the command to run as mailuser on the remote machine (of course permissions must be setup in ~mailuser/ .rhosts to allow this).

Description of the POSTTEST section

The POSTTEST section defines commands to be run after the test ends. This is often used to check system state after the test (e.g. disk usage, error logs, etc.). The machine(s) are specified using the HOSTS=client1 section attribute.

The results of POSTTEST will be linked to the final report.

Table 0.6 describes the various attributes you can use in the POSTTEST section.

Table 0.6 POSTTEST section attributes

Attribute
Description
command string
Specify the command to run. For example: df -k
rsh string
Specify the remote shell command used to run command. For example: rsh -l mailuser would cause the command to run as mailuser on the remote machine (of course permissions must be setup in ~mailuser/ .rhosts to allow this).

Description of the MONITOR section

The MONITOR section defines commands to be run during the test. This is often used to monitor system load. The machine(s) are specified using the HOSTS=client1 section attribute.

The results of MONITOR will be linked to the final report.

On NT, any MONITOR sections will be ignored.

Table 0.7 describes the various attributes you can use in the PRETEST section.

Table 0.7 PRETEST section attributes

Attribute
Description
command string
Specify the command to run. If there is a %f in the string, it will be replaced by the reporting frequency used my Mailstone (in seconds). If there is a %c in the string, it will be replaced by the number of reports needed for the test duration. If the string does not contain %c, then it will be shutdown when the first client exits. If the string does contain %c, then it will be allowed to run to completion. The default is vmstat %f.
rsh string
Specify the remote shell command used to run command. For example: rsh -l mailuser would cause the command to run as mailuser on the remote machine (of course permissions must be setup in ~mailuser/ .rhosts to allow this).

Description of the GRAPH section

The GRAPH section defines each graph to be generated for the final report. If no GRAPH sections are defined, then Mailstone will use a pre-defined set of graphs. Additional graphs can be added after the test is complete, by running process timestamp -a moregraphs.wld.

In addition to the protocol timers, there is an internally generated "timer" called "connections". This contains the current server connection count, taking into account connects, logouts, and errors. To graph this timer, specify a variable of "connection".

Table 0.8 describes the various attributes you can use in the GRAPH section.

Table 0.8 GRAPH section attributes

Attribute
Description
title string
Specify the title to appear above the graph. For example: "Time to get fully logged in".
label string
Specify the label to be used on the left side of the graph (the other axis is always time). For example: "Login time".
variables string
Specify the timers to be added together to form the graph. For example: "connect, banner, login".
field string
Specify the field in each timer to be used for the graph. For example: "Time".
If the timer does not contain fields (e.g. blocks or connections), then field should not be set.

Description of the DEFAULT section

The DEFAULT section defines attributes that may be used by all the protocol sections. Note that attributes in DEFAULT that are never used are silently ignored. If there is more than one DEFAULT section, they will be merged into one when the test is run.

For example: the conf/general.wld file might contain the following default section:

<Default>
server mailserver.example.com
smtpMailFrom test0@mailserver.example.com
addressFormat test%ld@mailserver.example.com
loginFormat test%ld
passwdFormat mozilla
numAddresses 1000
numLogins 1000
firstLogin 0
firstAddress 0
</Default>

Test Messages

The standard test messages used by Mailstone during a test run are stored in the data subdirectory. Five message files of varying sizes are provided for you: en-1k.msg, en-5k.msg, en-17k.msg, en-20k.msg, and en-32k.msg. The number indicates the size of the message in kilobytes. For example, the en-17k.msg file is about 17K long.

You can create additional message files as needed. Each file should be a properly formatted message. Mailstone does not use or alter anything in the message. Run setup to copy the new files to the testclients.


Detailed Protocol Test Steps
Mailstone implements a series of protocol tests. The following sections detail what is done in each protocol test. Timers are shown in ALL CAPS. Attributes are shown in italics.

This top level loop is use to run the protocol tests:

Select a section using a random number and the weight of each section.

Run the protocol initialization.

IDLE for idleTime.

Repeat this part numLoops times (or until test ends)
{

   Run the protocol loop body.

   IDLE for loopDelay

}

Run the protocol conclusion.

IDLE for blockTime

Note that an error will cause the section to be aborted and the connection closed. However, the final IDLE for blockTime will still be performed. If the test time is up, then all IDLE calls will return immediately and no more loops will be performed.

Table 0.9 describes attributes that all protocol sections support.

Table 0.9 General protocol section attributes

Command
Description
HOSTS=name
Command section attribute specifying that the commands are to be run only on specific testclient(s). For example, <IMAP4 HOSTS=client1,client3> specifies that this IMAP4 command is to be run on machines client1 and client3. Note that no spaces or tabs are allowed.
blockTime time
Specifies the number of seconds to delay after completing the current section. The delay is taken even if an error was encountered. For example, `1m' means delay one minute. Defaults to 0.
idleTime time
Specifies the number of seconds to delay after connecting before starting the loop. For example, `5s' means delay five seconds after connecting. Defaults to 0.
loopDelay time
Specifies seconds between loops (e.g. download checks).
For example: `60s' creates a delay of 60 seconds between checks. Defaults to 0.
numLoops N
Specifies the number of loops to perform before closing the connection. See Detailed Protocol Test Steps for what is done in each loop. Defaults to 1.
weight N
Specifies the relative frequency of Mailstone running that command. See Weights in Configuration Files for details. Defaults to 100.

Weights in Configuration Files

Commands in workload files are assigned a weight factor that determines the relative frequency of the command's occurrence. For example, if the numeric weights you have assigned to all commands called by the test add up to 25, and a particular command has a weight of 15, then that command has a 60 percent chance of being chosen every time a command is executed, while a command with a weight of 5 has only a 20 percent chance of being chosen. By adjusting the weight factors for the different commands, you can balance them according to your needs.

Weight commands use the syntax:

weight nn

where nn is a number representing the weight factor of that command, or command block. For example:

weight 9

assigns the numeric weight of nine to that command.

Note that weights only affect the ballance on one machine. If some sections are only run on specific machines, then only the sections on that machine matter.

Common substitutions

Wherever loginFormat is used, the first %ld will be replaced by a user number and the second %ld will be replaced by a domain number. The user number is controlled by firstLogin, numLogins, and sequentialLogins. The domain number is controlled by numDomains, firstDomain, and sequentialDomains. Wherever passwordFormat is used, the first %ld will be replaced by the user number.

Wherever addressFormat is used, the first %ld will be replaced by a user number and the second %ld will be replaced by a domain number. The user number is controlled by firstAddress, numAddresses, and sequentialAddresses. The domain number is controlled by numDomains, firstDomain, and sequentialDomains.

The purpose sequentialDomains, sequentialLogins, and sequentialAddresses attributes allow a range of numbers to be covered in a uniform and predictable way. However, the exact interpretation of these attributes may not be obvious when multiple machines are used.

By default, domain, login, and user numbers are picked randomly from the range specified by the appropriate first_ and num_ attributes. The range is divided among all the threads within a process so that there is an efficient sharing of the total range.

When sequential mode is specified, then the numbers proceed in ascending sequence within the range assigned to each thread. If there is more than one process, then each process will cover the same total range. Multiple machines can be assigned non-overlapping total ranges to properly distribute a desired overall test range.

For example: if you specify one machine, two threads, and a range of 100; then one thread will start at 0 and go to 49, and the other thread will start at 50 and go to 99.

IMAP4 Protocol Steps

The IMAP protocol stores mail on the mail server and allows message management without downloading the whole message. Clients are usually connected for long periods of time and periodically check for new messages.

Table 0.10 describes the various attributes you can use in the IMAP section.

Table 0.10 IMAP section attributes

Attribute
Description
loopDelay time
Specifies how often to check for new mail. The default is every 10 minutes (10m). For example, `5s' means check for mail every five seconds.
firstDomain N
Specify the first domain number to use. The domain number will be inserted into the second %ld in the loginFormat. Defaults to 0.
firstLogin N
The first (lowest) login number. The login number will be inserted into the first %ld in the loginFormat. Defaults to 0.
leaveMailOnServer N
If 1, don't delete messages after reading them.
loginFormat format
Specifies the login string. The first %ld is replaced by the login number, the second %ld (if present) is replaced by the domain number. For example: test%ld or server%ld@dom%ld.com.
numDomains N
Specifies the number of domains. Defaults to 0.
numlogins N
Specifies the number of logins.
passwdFormat password
Specifies the password string. If the string contains %ld, it will be replaced by the login number. For example: test%ld.
portNum N
Specifies a non-default port number for a service. Defaults to the standard port for IMAP4.
sequentialDomains N
If 1, domain numbers should proceed in order. By default, use random domain numbers.
sequentialLogins N
If 1, logins should proceed in order. By default, use random login numbers.
server name
The name of the mail server being tested. For example: mailhost.example.com.

CONNECT by TCP to portNum on server.

Read the BANNER.

LOGIN with loginFormat and passwordFormat. Get response.

IDLE for idleTime.

Repeat this part numLoops times (or until test ends)
{

   COMMAND: select INBOX folder. Get response.

   For each new messages
{

      COMMAND: Fetch the message size. Get response.
      RETRIEVE message. Get data.

      COMMAND: Send noop. Get response.

      COMMAND: Mark message as deleted and seen. Get response. 

   }

   COMMAND: Expunge deleted messages. Get response. 

   IDLE for loopDelay

}

LOGOUT

IDLE for blockTime

POP3 protocol steps:

The POP protocol transfers messages to the client for viewing and storage. The client connects long enough to download (and usually delete) messages and then disconnects.

Table 0.11 describes the various attributes you can use in the POP3 section.

Table 0.11 POP3 section attributes

Attribute
Description
firstDomain N
Specify the first domain number to use. The domain number will be inserted into the second %ld in the loginFormat. Defaults to 0.
firstLogin N
The first (lowest) login number. The login number will be inserted into the first %ld in the loginFormat. Defaults to 0.
leaveMailOnServer N
If 1, don't delete messages after reading them.
loginFormat format
Specifies the login string. The first %ld is replaced by the login number, the second %ld (if present) is replaced by the domain number. For example: test%ld or server%ld@dom%ld.com.
numDomains N
Specifies the number of domains. Defaults to 0.
numlogins N
Specifies the number of logins.
passwdFormat password
Specifies the password string. If the string contains %ld, it will be replaced by the login number. For example: test%ld
portNum N
Specifies a non-default port number for a service. Defaults to the standard port for POP3.
sequentialDomains N
If 1, domain numbers should proceed in order. By default, use random domain numbers.
sequentialLogins N
If 1, logins should proceed in order. By default, use random login numbers.
server name
The name of the mail server being tested. For example: mailhost.example.com.

CONNECT by TCP to portNum on server.

Read the BANNER.

LOGIN with loginFormat and passwordFormat. Get response.

COMMAND: stat how many message are available. Get response.

IDLE for idleTime

Repeat this part numLoops times (or until out of messages or test ends)
{

   RETRIEVE message. Get data.

   Unless leaveMailOnServer is set, COMMAND: delete message. 
Get response.

   IDLE for loopDelay

}

LOGOUT

IDLE for blockTime

SMTP Protocol Steps

The SMTP protocol provides reliable message transfer. The mail server looks at the message addresses and either delivers the mail locally or sends it to another SMTP server. Each SMTP connection can deliver multiple messages, each of which can have multiple addresses.

Table 0.12 describes the various attributes you can use in the SMTP section.

Table 0.12 SMTP section attributes

Attribute
Description
addressFormat format
Specifies the address string. The first %ld is replaced by the address number, the second %ld (if present) is replaced by the domain number. The domain number is independent of the one used for login. For example: test%ld@server.com or server%ld@dom%ld.com.
file path
File to use as the message text. For example:
en-17k.msg.

firstAddress N
The first (lowest) address number. The address number will be inserted into the first %ld in the adressFormat. Defaults to 0.
firstDomain N
Specify the first domain number to use. The domain number will be inserted into the second %ld in the loginFormat. Defaults to 0.
firstLogin N
The first (lowest) login number. The login number will be inserted into the first %ld in the loginFormat. Defaults to 0.
loginFormat format
Specifies the login string. The first %ld is replaced by the login number, the second %ld (if present) is replaced by the domain number. For example: test%ld or server%ld@dom%ld.com.
numAddresses N
Specifies the number of addresses.
numDomains N
Specifies the number of domains. Defaults to 0.
numlogins N
Specifies the number of logins.
numRecips N
Specifies the number of addresses used for each message. Defaults to 1.
passwdFormat password
Specifies the password string. If the string contains %ld, it will be replaced by the login number. For example: passwdFormat test%ld
portNum N
Specifies a non-default port number for a service. Defaults to the standard port for SMTP.
sequentialDomains N
If 1, domain numbers should proceed in order. By default, use random domain numbers.
sequentialLogins N
If 1, logins should proceed in order. By default, use random login numbers.
server name
The name of the mail server being tested. For example: mailhost.example.com.
smtpMailFrom address
Specifies the from address for each message. For example: test0@server.example.com.
useAuthLogin N
If 1, authenticate to the SMTP server using loginFormat and passwordFormat. Requires that useEHLO is set. Defaults to 0, which uses no authentication.
useEHLO N
If 1, use the SMTP enhanced capabilities greeting. Must be enabled to use useAuthLogin. Defaults to 0.

CONNECT by TCP to portNum on server.

Read the BANNER.

If useEHLO is set 
{

   COMMAND: send EHLO. Get response and capabilities.

}

Else 
{

   COMMAND: send HELO. Get response.

}

If useAUTHLOGIN is set and AUTH=LOGIN capability is available on 
the server
{

   LOGIN with loginFormat and passwordFormat. Get response.

}

IDLE for idleTime

Repeat this part numLoops times (or until test ends)
{

   COMMAND: send mail from smptMailFrom. Get response.

   Repeat numRecips times.

   {

       COMMAND: send mail to addressFormat. Get response.

   }

   COMMAND: initiate message body. Get response.

   SUBMIT message. Get response.

   IDLE for loopDelay

}

LOGOUT

IDLE for blockTime

WMAP Protocol Steps

The WMAP protocol is the heart of Netscape Messenger Express web mail. It shares many similarities with IMAP but works on top of HTTP. Clients are usually connected for long periods of time and periodically check for new messages.

The underlying HTTP connection may be dropped and re-established as needed. The WMAP session is encoded in the URL used to access the data pages and is independent of the HTTP connection.

Many parameters for WMAP can occur multiple times. Each occurrence is added to a list of things to do. These parameters end in "cmds" to indicate that they can occur more than once.

The WMAP protocol requires special handling of connection termination. Since WMAP is based on HTTP, the connection may be dropped and re-established arbitrarily. This will show up as a connection error and then an additional connection. Repeat connection failures or any other kind of error will cause the block to abort.

Table 0.13 describes the various attributes you can use in the WMAP section.

Table 0.13 WMAP section attributes

Attribute
Description
checkMailInterval
time

Specifies how often to check for new mail. The default is every 10 minutes (10m). For example, `5s' means check for mail every five seconds.
firstDomain N
Specify the first domain number to use. The domain number will be inserted into the second %ld in the loginFormat. Defaults to 0.
firstLogin N
The first (lowest) login number. The login number will be inserted into the first %ld in the loginFormat. Defaults to 0.
leaveMailOnServer N
If 1, don't delete messages after reading them.
loginFormat format
Specifies the login string. The first %ld is replaced by the login number, the second %ld (if present) is replaced by the domain number. For example:
test%ld or server%ld@dom%ld.com.

numDomains N
Specifies the number of domains. Defaults to 0.
numlogins N
Specifies the number of logins.
passwdFormat password
Specifies the password string. If the string contains %ld, it will be replaced by the login number. For example:
passwdFormat test%ld.

portNum N
Specifies a non-default port number for a service. Defaults to the standard port for HTTP.
sequentialDomains N
If 1, domain numbers should proceed in order. By default, use random domain numbers.
sequentialLogins N
If 1, logins should proceed in order. By default, use random login numbers.
server name
The name of the mail server being tested. For example:
mailhost.example.com.

wmapClientHeader
string

This string is attached to all information set to the server.
wmapBannerCmds
string

Each URL to get for the banner (login) screen. Can be used more than once.
wmapLoginData string
Data to post to login. The first %s is replaced with the loginFormat. The second %s is replaced with the passwordFormat.
wmapInboxCmds string
Each URL to get for the main screen. Can be used more than once.
wmapCheckCmds string
Each URL to get to update the message list. Can be used more than once.
wmapMsgReadCmds
string

Each URL to get to read a message. Can be used more than once.
wmapMsgWriteCmds|
string

Each URL to get to write a message. Can be used more than once.
wmapLogoutCmds
string

Each URL to get to terminate the session. Can be used more than once.

CONNECT by TCP to portNum on server.

Get the BANNER screen URLs using wmapBannerCmds.

LOGIN with loginFormat and passwordFormat using wmapLoginCmd and
wmapLoginData. Get redirect URL response.

CMD: get the redirect URL.

CMD: get the INBOX screens URLs using wmapInboxCmds

IDLE for idleTime.

Repeat this part numLoops times (or until test ends)
{

   HEADERS: get the message list using wmapCheckCmd.

   For each new message
{

      RETRIEVE message using wmapMsgReadCmds.

	}

   IDLE for loopDelay

}

LOGOUT using wmapLogoutCmds.

IDLE for blockTime

Interpreting Test Results

Consider these points when reviewing the results of a Mailstone test:

Testing is often an iterative process where server configuration parameters are adjusted to get the best performance for your requirements. You can record much of this configuration using PRETEST sections to list configuration state in the results report. For example, both Netscape Messaging Server and Directory Server are shipped with access logging enabled. This may reduce performance, but may be required for other reasons.

Modern servers use many caching techniques to improve performance. You should repeat any tests until you get repeatable results.

Mailstone reports everything that it measures, but not all measurements will be meaningful. For maximum rate tests, the server is under extreme stress and only the throughput or transaction rate have real meaning. For specific workload rate tests, then the throughput and transaction rate are fixed and parameters like response time and CPU usage are meaningful.

Every mail server has a series of components that can limit either the maximum throughput or the maximum transactions rate. These are the typical critical components: client network, mail server CPU/memory, mail server disk, and directory server. For example, a message download test with very large messages will most likely be limited by the client network throughput whereas the same test with very small messages will most likely be limited by the disk seek latency. It is often useful to determine these maximums for the system and then compare them to your performance requirements.

You can see from the above example that you get two very different tests depending on what was in the mail server message store when the test was run. This kind of information will not show up in the results report unless you explicitly specify it using the -b banner and/or the -n notes arguments. You will often have to perform multiple Mailstone runs to get the message store in the right state for your tests. For example, you might run this sequence to test the maximum throughput.

mstone pop-every -b 'Drain messages from store'

mstone smtp32k-every -b '32K messages for each mailbox'

mstone pop -t 5m -b 'POP download rate of 32K messages'

The first two tests would be custom to your installation. They would access every mailbox so that each is in the same state. You would configure this using sequentialLogins, sequentialAccounts, and maxBlocks. Then you have a known state for the download throughput test.


Removing Mailstone
To remove Mailstone files from testclient machines, run the cleanup utility. The cleanup utility uses the following syntax:

cleanup

Save your test configuration and results by moving the conf and results directories to a safe place.

To uninstall Mailstone, remove the mailstone directory and all of its subdirectories.


Appendix
Adding Test Users to Directory Server 3.x

You can use the Database Manager feature to add the test user account information in the conf/mailhost.ldif file to the LDAP Directory on your instance of Directory Server.

Note: This process will go faster if you first disable access logging on Directory Server.

Directory Server 3.x. To add test users with Directory Server version 3.x, follow these steps:

  1. Log in to the Directory Server.
  2. You must have read and execute permission to use Database Manager and read and write permissions for the targeted entries in the LDAP directory. (You can run this remotely using -h host.)

  3. Go to the Database Manager screen as described in your Directory Server documentation.
  4. Select the `Add Entries' form.
  5. In the Full Path to LDIF File field, enter the full absolute path name to the accounts.ldif file that contains the test user entries you want to add.
  6. For example: /usr/netscape/mailstone/conf/mailhost.ldif

  7. Change the `Bind to Server' field if needed.
  8. Enter the Password.
  9. Click Okay. The entries are added to your Directory Server manager.
Caution: If you are using the Database Manager to update a Netscape Directory Server, be sure to use the Directory Server's Add Entries user interface so as not to overwrite any other entries.

See the Directory Server Administrator's Guide and online help for more information on adding entries to Netscape Directory Servers.

Adding Test Users with ldapmodify

You can use the ldapmodify command line utility to add the test user account information in the conf/mailhost.ldif file to the LDAP Directory instance on your Directory Server.

Note: This process will go faster if you first disable access logging on the Directory Server.

The ldapmodify utility is provided with Messaging Server in server-root/shared/bin.

To update the LDAP Directory with ldapmodify, follow these steps:

  1. In Unix environments, make sure your LD_LIBRARY_PATH environment variable is correctly set.
  2. For example:

    setenv LD_LIBRARY_PATH server-root/lib

  3. Go to the directory containing the LDIF files.
  4. For example: cd ../mailstone/conf/mailhost.ldif

  5. Run ldapmodify to update the LDAP Directory.
  6. For example:

    server-root/shared/bin/ldapmodify -h mailhost -a \
    -D 'cn=directory manager' -w
    d_m_password \
    < accounts.ldif

Caution: If you are using ldapmodify to update a Netscape Directory Server be sure to use the -a option so as not to overwrite any existing entries.

See the Directory Server Administrator's Guide, and online Help for more information on adding entries to Netscape Directory Servers and the ldapmodify utility.

 

© Copyright 1999 Netscape Communications Corp., a subsidiary of America Online, Inc. All rights reserved.