BENCHMARK(1)

NAME

benchmark - runs the BRL-CAD benchmark

On this page

SYNOPSIS
DESCRIPTION
COMMANDS
OPTIONS
REPORTING RESULTS
EXAMPLES
SEE ALSO
AUTHOR
COPYRIGHT
BUG REPORTS

SYNOPSIS

[OPTION=value] benchmark [command] [OPTION=value] [RT_OPTIONS]

The benchmark suite will test the performance of a given system by iteratively rendering several well-known datasets into 512x512 images where performance metrics are documented and fairly well understood. The local machine performance is compared to the base system (called VGR) and a numeric "VGR" multiplier of performance is computed. This number is a simplified metric from which one may qualitatively compare CPU and cache performance, versions of BRL-CAD, and different compiler characteristics.

Run without any arguments or variables set, the benchmark suite will run a series of tests where it renders several image frames. By default, the benchmark suite will attempt to calibrate the intensity of the test frames with the performance characteristics of the machine running the tests.

When the benchmark completes, output is saved to several log files including a summary file containing tabulated results, a benchmark.log file containing the output from a given run, and multiple log files for each test frame. The generated tabular summary file will not be removed automatically regardless of invocation command, and must always be manually deleted.

COMMANDS

There are various commands that may be given to the BRL-CAD benchmark to perform actions such as invoking computation tests or filesystem clean-up:

clean: Remove the test-specific .pix and .log files.
clobber: Same as clean, but also removes benchmark.log files and prompts the user before proceeding.
help (or h): Display a brief usage statement.
instructions (or instruct): Display more detailed usage instructions.
quiet: Quell printing output (output is still logged).
verbose: Turn on extra debug output.
run: Initiate the benchmark analysis.
start: Legacy compatibility alias (no operation).

OPTIONS

There are several environment variables (passed as OPTION=value) that modify how the benchmark behaves, allowing it to be run in a stand-alone environment. Unrecognized options not containing = are appended to the raytracer invocation as RT_OPTIONS (e.g., -P1).

RT: The ray-trace binary (e.g., ../src/rt/rt or /usr/brlcad/bin/rt).
DB: The directory containing the reference geometry (e.g., ../db).
PIX: The directory containing the reference images (e.g., ../pix).
LOG: The directory containing the reference logs (e.g., ../pix).
CMP: The name of a pixel comparison tool (e.g., ./pixcmp or cmp). The benchmark interprets status 0 as an exact match, 1 as an acceptable difference, and 2 as an unacceptable difference. Other statuses indicate a comparison failure. A generic cmp(1) reports every content difference as status 1, so it cannot distinguish acceptable pixel variation from an incorrect image; when used as CMP, the benchmark consequently accepts any content difference.
ELP: The name of an elapsed time tool (e.g., ../sh/elapsed.sh).
TIMEFRAME: Minimum number of seconds each ray-trace frame should take.
MAXTIME: Maximum number of seconds to spend on any test.
DEVIATION: Minimum sufficient percentage deviation from the average.
AVERAGE: How many frames to average together.
VERBOSE: Turn on extra debug output (set to 1 or yes).
QUIET: Turn off all output (set to 1 or yes).

The TIMEFRAME, MAXTIME, DEVIATION, and AVERAGE options control how the benchmark will proceed and how long it should take. Each individual benchmark run will consume at least a minimum TIMEFRAME of wallclock time so that the results can stabilize. After consuming at least the minimum TIMEFRAME, additional frames may be computed until the standard deviation from the last AVERAGE count of frames is below the specified DEVIATION. When a test is run and it completes in less than TIMEFRAME, the raytrace is restarted using double the number of rays from the previous run. If the machine is fast enough, the benchmark may accelerate the number of rays being fired. These additional rays are hypersampled but without any jitter, so it is effectively performing a multiplier amount of work over the initial frame.

REPORTING RESULTS

Please send your BRL-CAD Benchmark results to the developers along with detailed system information to devs@brlcad.org. Include at least:

Compiler name and version (e.g. gcc --version)
CPU configuration (e.g. cat /proc/cpuinfo, hinv, or sysctl -a)
Cache (data and/or instruction) details for L1/L2/L3 and system
All generated log files (i.e. .log after benchmark completes)
Anything else you think might be relevant to performance

EXAMPLES

benchmark run: Default run of the suite, taking approximately 10 minutes.
benchmark run TIMEFRAME=1: Quick test run for testing functionality and performance ballpark.
benchmark run DEVIATION=1 TIMEFRAME=60 MAXTIME=600: Excessive analysis, attempt to stabilize within 1 percent deviation with each frame taking at least 60 seconds but no more than 10 minutes per test (the entire analysis will probably take 30 to 60 minutes).
benchmark run QUIET=1 -P1: Perform a benchmark analysis only using one CPU and only logging results to a file.
benchmark clean: Delete all of the log and pix image files generated during a benchmark analysis, leaving only the summary file and any *benchmark.log files.