SOLR Admin Tasks

Creating a new SOLR core

Do some command line steps:

# First create the directories for the new core
sudo mkdir -p /data/solr/data/biocache-reindex-2/data
# Copy the previous solrconfig, schema there
sudo cp -a /data/solr/data/biocache/conf/ /data/solr/data/biocache-reindex-2/conf
# Set the right perms
sudo chown -R solr:solr /data/solr/data/biocache-reindex-2/

Later you can add the core via the SOLR UI interface:

Full biocache-store indexing

Now we'll do a copy of the index on the filesystem.

Remove previous reindex:

rm -rf /data/solr/solr-create/biocache/data0*
rm -rf /data/solr/solr-create/biocache/data1*
rm -rf /data/solr/merged_*

# biocache index-local-node options:
# -t threads
# -ms, mergesegments, The number of output segments. No merge output is produced when 0. 
# -wc, writercount, The number of index writers. 
# -wt, writerthreads, The number of threads for each indexing writer. There is 1 writer for each -t. 
# -wb, writerbuffer, Size of indexing write buffer. The default is  + writerBufferSize, {
# -pt, processthreads, The number of threads for each indexing process. There is 1 process for each -t. 
# -pb, processbuffer, Size of the indexing process buffer. 
# -r, writerram, Ram allocation for each writer 
# -ws, writersegmentsize, Maximum number of occurrences in a writer segment. There is 1 writer for each -t. 
# -ps, pagesize, The page size for the records. 
# -max, maxrecords, Maximum number of records to index. This is mainly for testing new indexing. 

# basic text index
biocache index-local-node -t 4 -max 1000

# With more options, sample from: https://github.com/AtlasOfLivingAustralia/biocache-store/issues/329
biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -max 50000

# or ull reindex
biocache index-local-node -t 4 -max -1

Manual copy

The previous generated index need to manually copied into SOLR.

rsync --delete -aH --rsync-path "sudo rsync" /data/solr/merged_0/ 172.16.16.200:/data/solr/data/biocache-reindex-2/data/index/
ssh 172.16.16.200 "sudo chown -R solr:solr /data/solr/data/biocache-reindex-2/data/index/"

Swapping cores

When your data is ready (for instance after a BIE or with the previous biocache-cli local indexing), you can swap the cores to put the core in production: