Name indexer

Name indexing happens during the "processing" phase of ingestion. If a customised/national checklist is required, a new name index can be created. Details of the required structure are here.

Using the playbook nameindexer-standalone.yml

Assuming you have an instance of ALA portal running, when you need to use your own name index, or update the existing name index, you want to run it separately so it doesn't potentially break the running service.

So as the tutorial of building a customised name index, we start from a vanilla vagrant ubuntu box.

$ cd ala-install/vagrant/ubuntu
$ vagrant status

At this point, if you have a vagrant box running, consider destroy it and start a fresh one.

$ vagrant destroy
$ vagrant up

Ensure the virtual box is up-and-running and you can login:

$ vagrant ssh

Now, run the playbook for creating the instance that builds nameindexes:

$ cd ../../ansible
$ ansible-playbook -i inventories/vagrant nameindexer-standalone.yml --private-key ~/.vagrant.d/insecure_private_key -u root

Where the name sources are stored:

$ cd /data/lucene/sources

Observe the target folder before we do name indexing:

$ ls /data/lucene

It should have only 'sources' as a directory.

scp your zipped custom DwCA in to the directory /data/lucene/sources, and expand the zip.

Now, to create a name index, for instance:

$ sudo nameindexer -dwca /data/lucene/sources/dwca-col-mammals

(As 23 Jul 14, the default owner of /data is root so you need sudo to run nameindexer.)

Notice that here the 'dwca-col-mammals' is a directory extracted from dwca-col-mammals.zip. If you are working on your own checklist in DwC-A format, make sure you extract it before fire the nameindexer. Do $ nameindexer -help for more information.

Test search to see if the name index has been built successfully:

$ sudo nameindexer -testSearch "Macropus rufus"

And you should see:

Search for name
ID: 6863103
GUID: urn:lsid:catalogueoflife.org:taxon:d9f7aefa-29c1-102b-9a4a-00304854f820:col20120124
Classification: "(Desmarest, 1822)",Animalia,Chordata,Mammalia,Diprotodontia,Macropodidae,Macropus
Scientific name: Macropus rufus
Authorship: (Desmarest, 1822)
Rank: SPECIES
Synonym: null
Match type: exactMatch

That means you've got a working index.

Using your new nameindexer

After regenerating the name index, in your /data/lucene directory, you'll see two new directories:

namematching
nmload-tmp

To use the new name index, you can zip these two directories and unzip and replace the same one on your production site in the VMs that use nameindex (that is, bie-index, biocache-cli, specieslists, sandbox, biocache-service).

Reindexing your data with the new name index

The next step would be to update the occurrence solr index to use this new name index.

You can test the biocache by processing a single record using biocache process-single <Record UUID>.

Later you can re-process your data in the biocache with biocache process -dr [druid] and re-index your data to see the changes biocache index -dr [druid]. But probably you will need a full reindex of your data.

Supported checklist formats

@Todo : Text to add (maybe images as well)

About homonyms

The name indexing by default looks for /data/lucene/sources/IRMNG_DWC_HOMONYMS to analyse homonyms. If you have alternative homonyms to detect against, run nameindexer with -irmng flag and point to your own extracted homonym DwC-A.

To avoid obvious homonym indexing error, you can provide taxonomy hints in the Collectory-hub when you edit the metadata of data resources by providing a taxonomy hint. The URL would be /collectory/dataResource/edit/[druid]?page=%2Fshared%2FeditTaxonomyHints. (Replace [druid] with your own.)

About using the GBIF Backbone

In you want to use the GBIF Backbone Taxonomy, you have to take into account that right now ALA nameindex uses a mandatory scientificName without scientificNameAuthorship and an optional nameComplete, not like the GBIF backbone that uses a full scientificName following Darwin Core. Right now (Nov-2019) if you want to use the GBIF Backbone Taxonomy and avoid the duplication of authors in BIE (and other related issues), you can, for instance adapt the backbone using this gist to remove the author from the scientificName.

More information

For more name indexing information read and follow the Guide to Getting Names into ALA.

This bie-index wiki page about the full reindex tasks is outdated but quite informative about the whole re index process.