This document describes how to use Mailstone 4.1 for capacity planning and testing on Netscape Messaging Server and other mail servers.

• About the Mailstone Utility
• Installing Mailstone


• Setting Up Mailstone Tests


• Synchronizing Client Machines


• Running Mailstone


• Reviewing Test Results


• Detailed Protocol Test Steps


• Removing Mailstone

• In Unix environments, the test can be configured to include additional client machines (referred to as testclients in these instructions). These additional clients can be any supported Unix machine. The testmaster machine can also function as a testclient machine.
• In NT environments, Mailstone runs only on the single testmaster machine which acts as both testmaster and testclient.

1. Command line arguments are combined with configuration files to define the test.
2. The test processes are executed across the testbed. (In Unix environments rsh is used to run the processes on the other client machines.)
3. Results are collected and displayed on the HTML results page.

1. Locate the Mailstone archive file for your platform.

(The contents of both files are identical. Different file types are provided for convenience.)

For Unix the file is mailstone-4.15.tar.gz.

For Windows NT the file is mailstone-4.15.zip.

2. Choose the testmaster machine on which you will run Mailstone and copy the archive file to a temporary installation directory, such as /tmp, on that machine.

The testmaster machine can be the same machine as the server being tested, or a different machine.

3. Go to the testmaster directory immediately above where you want the Mailstone software to be installed.

For example, if you go to /usr/netscape Mailstone will be installed in
/usr/netscape/mailstone.

4. Extract the Mailstone files.

In Unix environments:

gunzip -c /tmp/mailstone-4.15.tar.gz | tar xf -

In Windows NT environments:

unzip c:\temp\mailstone-4.15.zip

The extraction process creates the required Mailstone files and places them in a mailstone directory and associated subdirectories.

See Table 0.2 for a description of the main Mailstone files.

If you initially install Mailstone as part of Messaging Server installation and later want to install the all-platforms version of Mailstone, or you want to reinstall Mailstone for any other reason, you must first perform the following steps before using the installation procedure described above.

1. Go to the mailstone directory.

For example: cd /usr/netscape/mailstone.

2. Run the cleanup program to remove installed files.

Type: ./cleanup.

3. Rename the mailstone/conf directory.

Type: mv conf conf.save.

1. Go to the mailstone directory (e.g. cd /usr/netscape/mailstone).
2. Run setup config. This will display the LICENSE and ask if you agree to it.
3. Answer yes to set up an initial configuration.
4. Enter the name of the mail server.
5. Enter the test user account name and password. The user account names should be separate from any actual users and should be unique to the server being tested (e.g. mailhost-test).

By default, the user number will be appended to the end of the user account name. If the name contains %ld, then this will be replaced with the user number. If the password contains %ld, then it will be replaced with the user number (otherwise, no numbering will be used in the password).

6. Enter the test client machine names. For NT, only localhost is allowed.
7. Answer yes to create an LDIF file for users and a broadcast account.
8. Answer yes to view the new configuration file. Make any additional changes to conf/general.html with a text editor. See Configuring Test Execution for details about adding clients and controlling test execution. To re-run the simple configuration editor, run setup config.
9. Edit the contents of your System Data file to specify the system configuration data you want appended to test results.

By default, this file is named conf/general.html. See System Data File (general.html).

10. On UNIX systems, make sure you can rsh to each client machine. You may have to edit the ~/.rhosts files to enable permission.
11. Run setup to copy messages and executables to test clients machines. If you encounter problems with configuration or access permissions, correct them and re-run setup.

(Even though there are no additional testclient machines in Windows environments, you must still run setup because the testmaster machine acts as a testclient machine.)

12. Important. If you are using more than one testclient machine, run checktime as described in Synchronizing Client Machines to verify that the system clocks are within 2 seconds of each other.

In order to test a messaging server, you need to create a set of test users. During set up, you created an LDIF file that will create these users in the Directory Server used by Netscape Messaging Server. If you are testing another kind of mail server, you will have to use other means to create these accounts.

• Database Manager. The Netscape Directory Server provides a Database Manager that can be used to update the LDAP Directory as described in Adding Test Users with Database Manager.
• ldapmodify. You can also use the ldapmodify command-line utility to update the LDAP Directory as described in Adding Test Users with ldapmodify.
• Edit the database. (Experts only) You can export the current database, append the new entries, and then import the new version. This is the fastest method and makes adding millions of users practical.

You can use the Database Manager feature to add the test user account information in the conf/mailhost.ldif file to the LDAP Directory on your instance of Directory Server.

1. Start up Netscape Console and log in.
2. Open the Server Group containing the instance of Directory Server you want to use.
3. Double-click the name of the Directory Server.
4. Click the `Configuration' tab.
5. Right-click Database and select the `Import' option.
6. Enter the name of the file containing your LDIF modifications in the dialog box.

For example, /usr/netscape/mailstone/conf/mailhost.ldif.

7. Select `Append Data to Database'.
8. Select `Add Only'.
9. Select `Continue on Error'.

Caution: If you are using the Database Manager to update a Netscape Directory Server be sure to use Directory Server's Append Data and Add Only options so as not to overwrite any other entries.

10. Click `OK'.

For accurate test results it is important that the testmaster and all of the testclient machines show exactly the same time on their system clocks. If at all possible, the times should be identical to less than a second.

checktime [-w machine.tbd]

• The best way to synchronize system clocks is with a utility that uses the Network Time Protocol (NTP) or an equivalent protocol. Two such utilities are rdate or timeslave.
• If you do not have an NTP utility, Mailstone provides the timesync utility. Note that the timesync utility will only work with operating systems that support setting seconds with the date command.

timesync [-w machine.tbd]

1. Go to the mailstone directory (e.g. cd /usr/netscape/mailstone).
2. If you have additional testclient machines in a Unix environment, synchronize them as described in Synchronizing Client Machines.
3. Run the mstone command using the syntax:

mstone testname [options]

where testname is the test workload file. For example:

mstone popsmtp

4. View the results by copying the display URL into a browser.

mstone popsmtp -t 10m

Even though you have created test users, the Messaging Server does not create each mailbox until it is accessed. One way to create mailboxes for the test users in your LDAP directory is by simply running a Mailstone test.

1. Run the pop test as follows:

./mstone pop -t 1m

This will access each login and force the mailbox to be created.

2. Look at how many logins were achieved in one minute and re-run with enough time to get to all mailboxes.

During a long test run, you can display the current results by running the process command. For example:

process

You can also use the process command to reprocess and display older test runs by specifying the data directory on the command line. For example, to reprocess the results of a test that was run on July 14, 1999 at 14:31 (24 hour clock) which are stored in the results/19990714.1431 directory, you would enter this:

process 19990714.1431

To stop a test run before it has completed, enter Control-C or use any other standard method of halting an ongoing process. Then run process to generate a report up to the point of termination.

Mailstone displays the results of a test run as an HTML page named results.html. There is one results file in each test run subdirectory.

• The timestamp in the title "Netscape Mailstone Results 19991215.2009" gives the time of the test run. In this case, December 15, 1999, at 20:09 hours.
• The headline "Wmap/Imap/Pop/Http/Smtp torture test" is the text that was specified for the title variable in the primary configuration file (testname.wld).
• The headline "Netscape Messaging Server 4.15" is the text that was specified for the comments variable in the primary configuration file (testname.wld).
• Under the line are general information about the test and links to additional supporting files.
• Then a series of tables report the test results for each protocol.

Mailstone breaks each client session into timers. Each timer tracks these variables:

• Tries. A try is an attempt to perform an operation. The exact meaning of a try depends on the protocol and the timer.
• Errors. An error indicates that a try did not succeed and the connection will be dropped. A line in the stderr log will indicate the cause of the error.
• Bytes written. See explanation below.
• Bytes read. See explanation below.
• Time. This is the average transaction time for each try.
• TMin. This is the minimum time for a transaction. A TMin of exactly 0 means that no transactions were performed.
• TMax. This is the maximum time for a transaction. A TMax of exactly 0 means that no transactions were performed.
• TStd. This is the standard deviation of the transaction time.

Each row in the report is a timer for a specific aspect of the protocol test. The one exception is "Total", which is a sum of the other timers. Details of how each timer is used in each protocol are found in Detailed Protocol Test Steps. Here is the current list of possible timers:

• Total. The total timer contains a sum of all the other timers.
• Connect. The connect timer is the establishment of the TCP connection to the server.
• Banner. The banner timer includes the message sent from the server when a connection is established.
• Login. The login timer includes sending the user ID and password and having it verified.
• Command. The command timer records all other protocol exchanges between the client and server.
• Submit. The submit timer records message data sent to the server.
• Retrieve. The retrieve timer records messages received from the server.
• Headers. The headers timer records non-message HTML pages received from the server.
• Logout. The logout timer records normally closed connections.
• Idle. The idle timer records time that the client spent sleeping between requests to the server.

• The first section reports all the tries, errors, bytes read, bytes written, and the average time value per operation.
• The second section lists all the operation rates. These are the values reported in the first table divided by the number of seconds that the test was run.

You may want to extract additional information from the data that Mailstone collects. Summaries of all the information are available in two additional formats for further processing.

At the end of the results.html file under the heading "Details" is a description of the test environment. This is the text you specified in the system data file (conf/general.html). This allows you to capture all the configuration that is beyond the scope of Mailstone, but critical to what you are testing. Typical information to document includes: software version, operating system version, hardware capacities, network configuration, disk organization, caching configuration, and so on. A simple description is shown in Figure 0.3.

Mailstone files are stored in a directory named mailstone with several subdirectories. All programs must be run from the mailstone directory. These directories are created when the Mailstone files are extracted with tar (Unix) or are unzip (NT). File names are the same for both Unix and NT platforms, except that in some cases NT files have filename extensions such as .exe or .bat.

Configuration Files

Configuration files describe both how the test will be run (e.g. what are the client machines and client counts) and the test itself. A number of test configurations are supplied and you can add more test configurations as needed. All configuration files are stored in the conf directory.

<CONFIG>
   title   \        # title of this test
POP reads

# blank lines and comment lines don't matter
   #telemetry 1     # uncomment this to enable telemetry logs
</CONFIG>

• General. The general configuration file specifies parameters that apply to all your Mailstone tests. Each test includes conf/general.wld to define common parameters. The mailstone utilities setup, cleanup, checktime, and timesync also read conf/general.wld for test client machines and test account information. See General Configuration File (general.wld).
• Workload. The workload configuration file specifies the protocols to be tested, the attributes for each protocol, and the test messages to be used. By default, workload configuration files use the filename extension .wld. Mailstone provides sample workload files for different kinds of tests: imap.wld, pop.wld, smtp.wld, popimapsmtp.wld, and popsmtp.wld. See Workload Configuration File (testname.wld).
• System Data. The system data file specifies any information that you wish to have appended to test results. By default, this file is named general.html. See System Data File (general.html).

Times are specified in configuration commands (or on the command line) by a number followed by one of these time unit designations:

• s for seconds
• m for minutes


• h for hours


• d for days

The general configuration file specifies general configuration parameters that apply to all Mailstone tests across this testbed. The general configuration file is named conf/general.wld.

<Default>
    server         servername
    smtpMailFrom   from-address
    addressFormat  addressformat
    loginFormat    loginformat
    passwdFormat   password
    numAddresses   N
    numLogins      N
    firstAddress   N
    firstLogin     N
</Default>

<CLIENT HOSTS=localhost>
</CLIENT>

The system data file specifies additional data to be added to the test report. That is, at the end of each test run the information in the system data file is appended to the HTML-formatted test results report. By default, this file is named general.html. Typically, the system data file describes the Messaging Server being tested. However, you can use this file to add any information you wish to the test report.

<PRE>
<B> Mailhost.domain.com </B>
  Netscape Messaging Server 4.1 (2/18/99)
  E6000
  26x250MHz UltraSPARC-II
  6.5GB RAM
  A3500 w/ 60x9GB 7200 RPM
  store: 5 x (10x9 RAID-0)
  queue: 1 x (10x9 RAID-0)
</PRE>

The workload configuration file specifies the services and commands to be tested, their weights, and the test message files to be used. There are sample workload files for different kinds of tests: imap.wld, pop.wld, smtp.wld, popimapsmtp.wld, popsmtp.wld, and wmap.wld. You can create additional workload files as necessary.

<include conf/general.wld> 

<CONFIG>                         # test specific config
     title              POP, IMAP, SMTP combined load
     clientCount        100      # set an appropriate client count
</CONFIG>

<SMTP HOSTS=client1>
     file               en-1k.msg
     weight             100
     blockTime          1s
</SMTP>

<POP3 HOSTS=client1>
     weight             50
     blockTime          1s
     #leaveMailOnServer 1
</POP3>

# Since IMAP connections are much longer lived than POP or SMTP,
# run them on a separate host for clearer results.
<IMAP4 HOSTS=client2,client3>
     weight             100
     loopTime           60s
     numLoops           10
</IMAP4> 

The CONFIG section is used to control the test startup and report generation. Nothing in CONFIG directly affects a particular protocol test.

The CLIENT section defines one or more machines to be used to run the test. The machines are specified using the HOSTS=client1,client2 section attribute. There are often multiple CLIENT sections.

<CLIENT HOSTS=client1,client2>
    arch          SunOS5.6
    maxThreads   500
    maxClients   2000
</CLIENT>

The PRETEST section defines commands to be run before the test starts. This is often used to check system state before the test (e.g. disk usage, server configuration, etc). The machine(s) are specified using the HOSTS=client1 section attribute.

The POSTTEST section defines commands to be run after the test ends. This is often used to check system state after the test (e.g. disk usage, error logs, etc.). The machine(s) are specified using the HOSTS=client1 section attribute.

The MONITOR section defines commands to be run during the test. This is often used to monitor system load. The machine(s) are specified using the HOSTS=client1 section attribute.

The GRAPH section defines each graph to be generated for the final report. If no GRAPH sections are defined, then Mailstone will use a pre-defined set of graphs. Additional graphs can be added after the test is complete, by running process timestamp -a moregraphs.wld.

The DEFAULT section defines attributes that may be used by all the protocol sections. Note that attributes in DEFAULT that are never used are silently ignored. If there is more than one DEFAULT section, they will be merged into one when the test is run.

<Default>
    server         mailserver.example.com
    smtpMailFrom   test0@mailserver.example.com
    addressFormat  test%ld@mailserver.example.com
    loginFormat    test%ld
    passwdFormat   mozilla
    numAddresses   1000
    numLogins      1000
    firstLogin     0
    firstAddress   0
</Default>

The standard test messages used by Mailstone during a test run are stored in the data subdirectory. Five message files of varying sizes are provided for you: en-1k.msg, en-5k.msg, en-17k.msg, en-20k.msg, and en-32k.msg. The number indicates the size of the message in kilobytes. For example, the en-17k.msg file is about 17K long.

Select a section using a random number and the weight of each section.

Run the protocol initialization.

IDLE for idleTime.

Repeat this part numLoops times (or until test ends)
{

   Run the protocol loop body.

   IDLE for loopDelay

}

Run the protocol conclusion.

IDLE for blockTime

Commands in workload files are assigned a weight factor that determines the relative frequency of the command's occurrence. For example, if the numeric weights you have assigned to all commands called by the test add up to 25, and a particular command has a weight of 15, then that command has a 60 percent chance of being chosen every time a command is executed, while a command with a weight of 5 has only a 20 percent chance of being chosen. By adjusting the weight factors for the different commands, you can balance them according to your needs.

weight nn

weight 9

Wherever loginFormat is used, the first %ld will be replaced by a user number and the second %ld will be replaced by a domain number. The user number is controlled by firstLogin, numLogins, and sequentialLogins. The domain number is controlled by numDomains, firstDomain, and sequentialDomains. Wherever passwordFormat is used, the first %ld will be replaced by the user number.

The IMAP protocol stores mail on the mail server and allows message management without downloading the whole message. Clients are usually connected for long periods of time and periodically check for new messages.

CONNECT by TCP to portNum on server.

Read the BANNER.

LOGIN with loginFormat and passwordFormat. Get response.

IDLE for idleTime.

Repeat this part numLoops times (or until test ends)
{

   COMMAND: select INBOX folder. Get response.

   For each new messages
   {

      COMMAND: Fetch the message size. Get response.

      RETRIEVE message. Get data.

      COMMAND: Send noop. Get response.

      COMMAND: Mark message as deleted and seen. Get response. 

   }

   COMMAND: Expunge deleted messages. Get response. 

   IDLE for loopDelay

}

LOGOUT

IDLE for blockTime

The POP protocol transfers messages to the client for viewing and storage. The client connects long enough to download (and usually delete) messages and then disconnects.

CONNECT by TCP to portNum on server.

Read the BANNER.

LOGIN with loginFormat and passwordFormat. Get response.

COMMAND: stat how many message are available. Get response.

IDLE for idleTime

Repeat this part numLoops times (or until out of messages or test ends)
{

   RETRIEVE message. Get data.

   Unless leaveMailOnServer is set, COMMAND: delete message. 
   Get response.

   IDLE for loopDelay

}

LOGOUT

IDLE for blockTime

The SMTP protocol provides reliable message transfer. The mail server looks at the message addresses and either delivers the mail locally or sends it to another SMTP server. Each SMTP connection can deliver multiple messages, each of which can have multiple addresses.

CONNECT by TCP to portNum on server.

Read the BANNER.

If useEHLO is set 
{

   COMMAND: send EHLO. Get response and capabilities.

}

Else 
{

   COMMAND: send HELO. Get response.

}

If useAUTHLOGIN is set and AUTH=LOGIN capability is available on 
the server 
{

   LOGIN with loginFormat and passwordFormat. Get response.

}

IDLE for idleTime

Repeat this part numLoops times (or until test ends)
{

   COMMAND: send mail from smptMailFrom. Get response.

   Repeat numRecips times.

   {

       COMMAND: send mail to addressFormat. Get response.

   }

   COMMAND: initiate message body. Get response.

   SUBMIT message. Get response.

   IDLE for loopDelay

}

LOGOUT

IDLE for blockTime

The WMAP protocol is the heart of Netscape Messenger Express web mail. It shares many similarities with IMAP but works on top of HTTP. Clients are usually connected for long periods of time and periodically check for new messages.

CONNECT by TCP to portNum on server.

Get the BANNER screen URLs using wmapBannerCmds.

LOGIN with loginFormat and passwordFormat using wmapLoginCmd and
wmapLoginData. Get redirect URL response.

CMD: get the redirect URL.

CMD: get the INBOX screens URLs using wmapInboxCmds

IDLE for idleTime.

Repeat this part numLoops times (or until test ends)
{

   HEADERS: get the message list using wmapCheckCmd.

   For each new message
   {

      RETRIEVE message using wmapMsgReadCmds.

	}

   IDLE for loopDelay

}

LOGOUT using wmapLogoutCmds.

IDLE for blockTime

Consider these points when reviewing the results of a Mailstone test:

• Was the system properly configured?
• Are the results repeatable?
• Did the test measure what you wanted?
• Was there a performance bottleneck?
• Was the mail server message store in the proper state for the test?

mstone pop-every -b 'Drain messages from store'

mstone smtp32k-every -b '32K messages for each mailbox'

mstone pop -t 5m -b 'POP download rate of 32K messages'

cleanup

You can use the Database Manager feature to add the test user account information in the conf/mailhost.ldif file to the LDAP Directory on your instance of Directory Server.

1. Log in to the Directory Server.

You must have read and execute permission to use Database Manager and read and write permissions for the targeted entries in the LDAP directory. (You can run this remotely using -h host.)

2. Go to the Database Manager screen as described in your Directory Server documentation.
3. Select the `Add Entries' form.
4. In the Full Path to LDIF File field, enter the full absolute path name to the accounts.ldif file that contains the test user entries you want to add.

For example: /usr/netscape/mailstone/conf/mailhost.ldif

5. Change the `Bind to Server' field if needed.
6. Enter the Password.
7. Click Okay. The entries are added to your Directory Server manager.

You can use the ldapmodify command line utility to add the test user account information in the conf/mailhost.ldif file to the LDAP Directory instance on your Directory Server.

1. In Unix environments, make sure your LD_LIBRARY_PATH environment variable is correctly set.

For example:

setenv LD_LIBRARY_PATH server-root/lib

2. Go to the directory containing the LDIF files.

For example: cd ../mailstone/conf/mailhost.ldif

3. Run ldapmodify to update the LDAP Directory.

For example:

server-root/shared/bin/ldapmodify -h mailhost -a \
-D 'cn=directory manager' -w d_m_password \
< accounts.ldif