LogonTest_BenchmarkingSuite

Overview

The LOGON Benchmarking Suite is a collection of scripts to execute and time various tasks that are representative of the LOGON MT system (and the larger collection of [http://www.delph-in.net/ DELPH-IN] NLP tools). The suite is intended as a cross-platform performance meter.

Benchmarking Tasks

• tadm Maximum Entropy Estimation using the [http://tadm.sourceforge.net TADM] learner. Given a training sample of 98109 events in 1317 contexts (using 323665 features), estimate parameters of a discriminative model. Without a prior and convergence thresholds of 1e-12 (relative) and 1e-24 (absolute), we expect to run 493 and 446 iterations, respectively, in 32- and 64-bit mode. This task is heavy on floating point arithmetic, manipulating large, sparse matrices. TADM is implemented in C++ and centrally builds on the [http://www-unix.mcs.anl.gov/petsc/ PETSc] and [http://www-unix.mcs.anl.gov/tao/ TAO] numerical optimization libraries.

• cheap Unification-Based Parsing using the [http://www.delph-in.net/pet PET] parser and [http://www.delph-in.net/erg LinGO English Resource Grammar]. Using 1154 inputs from the first section of the [http://www.emmtee.net LOGON] Jotunheimen corpus, the parser is run in exhaustive mode with pro- and retro-active packing (imposing a maximum chart size of 100,000 edges). Results are enumerated from the parse forest and scored, though not recorded. The specific cheap binary used for this benchmark (i.e. the one in the LOGON tree) is compiled off the PET main branch with mmap() disabled. This task has virtually no floating point activity but is almost exclusively occupied with unifying and copying relatively large feature structures: blocks of memory are allocated, initialized, traversed, compared to other feature structures (where values are exclusively ints), and eventually either copied or released. PET is implemented in C++ with the feature structure core in pure C.

• ape

• generate

Running the Suite

Benchmarkin Results

• node	tadm	tadm (32-bit)	generate	generate (32-bit)	ape	ape (32-bit)	cheap (32-bit)
  mt (2 single Xeon, 2.8ghz, 3gbyte, 32-bit)
  cc (2 single Xeon, 3.4ghz, 6gbyte, 32-bit)
  nm (2 single Xeon, 3.4ghz, 6gbyte, 64-bit)	292	406
  tiger (4 single Opteron 850, 2.4ghz, 32gbyte, 64-bit)
  teflon (2 single Xeon, 3.4ghz, 4gbyte, 64-bit)
  c0-4 (2 dual Opteron 275, 2.2ghz, 16gbyte, 64-bit)
  dalco (16 dual Opteron 880, 2.4ghz, 128gbyte, 64-bit)

  node	tadm (2-way)	tadm (4-way)	generate (2-way)	generate (4-way)	ape (2-way)	ape (4-way)	cheap (2-way)	cheap (4-way)
  mt
  cc
  nm
  tiger
  teflon
  c0-4
  dalco