LogonTest_BenchmarkingSuite

Overview

The LOGON Benchmarking Suite is a collection of scripts to execute and time various tasks that are representative of the LOGON MT system (and the larger collection of [http://www.delph-in.net/ DELPH-IN] NLP tools). The suite is intended as a cross-platform performance meter.

Benchmarking Tasks

• tadm Maximum Entropy Estimation using the [http://tadm.sourceforge.net TADM] learner. Given a training sample of 98109 events in 1317 contexts (using 323665 features), estimate parameters of a discriminative model. Without a prior and convergence thresholds of 1e-12 (relative) and 1e-24 (absolute), we expect to run 493 and 446 iterations, respectively, in 32- and 64-bit mode. This task is heavy on floating point arithmetic, manipulating large, sparse matrices. TADM is implemented in C++ and centrally builds on the [http://www-unix.mcs.anl.gov/petsc/ PETSc] and [http://www-unix.mcs.anl.gov/tao/ TAO] numerical optimization libraries.

• cheap Unification-Based Parsing using the [http://www.delph-in.net/pet PET] parser and [http://www.delph-in.net/erg LinGO English Resource Grammar]. Using the 1154 inputs from the first section of the [http://www.emmtee.net LOGON] Jotunheimen corpus, the parser is run in exhaustive mode with pro- and retro-active packing (imposing a maximum chart size of 100,000 edges).

• ape

• generate

Running the Suite

Benchmarkin Results

• node	tadm	tadm (32-bit)	generate	generate (32-bit)	ape	ape (32-bit)	cheap (32-bit)
  mt (2 single Xeon, 2.8ghz, 3gbyte, 32-bit)
  cc (2 single Xeon, 3.4ghz, 6gbyte, 32-bit)
  nm (2 single Xeon, 3.4ghz, 6gbyte, 64-bit)	292	406
  tiger (4 single Opteron 850, 2.4ghz, 32gbyte, 64-bit)
  teflon (2 single Xeon, 3.4ghz, 4gbyte, 64-bit)
  c0-4 (2 dual Opteron 275, 2.2ghz, 16gbyte, 64-bit)
  dalco (16 dual Opteron 880, 2.4ghz, 128gbyte, 64-bit)

  node	tadm (2-way)	tadm (4-way)	generate (2-way)	generate (4-way)	ape (2-way)	ape (4-way)	cheap (2-way)	cheap (4-way)
  mt
  cc
  nm
  tiger
  teflon
  c0-4
  dalco