WeSearch_LexicalFiltering

Background

Given a lattice of lexical hypotheses, _aL_bⁱ, and a tagger t^j(_aL_bⁱ), develop a filter function, filter(_aL_bⁱ, t^j(_aLbⁱ)), -> (0,1) that discards unlikely lexical hypotheses.

TNT output for filtering of LE types

One such filter function is to intuitively map PTB tags output from the TNT tagger onto LE Types:

• t^j(_aL_bⁱ) : map_t(t^j(_aL_b)) = map_l(_aL_bⁱ) => 1

  • map_t() ∩ map_l() ≠ ∬

	aj	av	c	cm	d	n	p	pp	pt	v	x
__unmapped	631		76			757	2		2
#		2				2
$						16
:	1	20				3	189	278	74	166
CC	2	13	4032	3	15	7	21	1		1
CD	663	1	1		11	88	1			1
DT	38	52	56	17	12422	396	8	8
EX						220
FW	10	19	22		3	268	45	3		4
IN	10	269	90	355	9	273	14135	132		2	2
JJ	9736	167	2		106	1794	303	34	80	736	1
JJR	236	28	1		13	2	2
JJS	110	31			103	8	2	1		1
LS	1	1				2
MD						3	1			1189
NN	204	168	5		4	19185	86	31	29	977	3
NNP	779	17	10			8804	6	5		105	2
NNPS	2					231	1
NNS	29	1				10246	4			128
PDT			7		33	42
POS	1					314				1
PRP	2	10				1322			4
PRP$					550	2
RB	106	3717	201	1	8	121	193	96		19	3
RBR	24	181				3	1
RBS	2	116			29
RP	1	2				2	69	7
SYM	2	1				26
TO	14	4		1171		2	1419	2
UH		3				19				1
VB	16	3				105				3331
VBD	10	2				16	2			1520	1
VBG	13	1				84	34			1823
VBN	78	3				33	3			4234
VBP	2	1				37				2206
VBZ	2	18				104	3	5		3965	1
WDT		8		6	28	709	5
WP					12	118					2
WP$					24
WRB	2	7				92	115	115			3

Related Work

• Rebecca Driden (2009), [http://www.dridan.com/research/papers/dridan-phdthesis.pdf Using Lexical Statistics to Improve HPSG Parsing], PhD Thesis, Saarland University