How to run the silver annotation pipeline

[foolfun]

I ran the DrKIT code which includes 'sling/local/data/distant/facts-0000%d-of-00010.json', I have no idea how to get it?

[CSQianDong]

+1

[ringgaard]

The "silver annotation pipeline" is not yet properly documented as it is still under development, but you should be able to run it. First you run the wiki pipeline as described here. Then you need to build an IDF table using this command:

sling build_idf

Then you can run silver annotation on all the Wikipedia articles:

sling silver_annotation

It takes quite a while to run the silver annotation pipeline (10 hours on my machine). Please let me know if this works for you.

[CSQianDong]

Hi, thanks for your reply. When I run sling silver_annotation, I got the error massage:
[2020-11-27 17:39:19.446544: F sling/nlp/kb/calendar.cc:41] Check failed: num >= 0
do you have any idea about this error?

[ringgaard]

I remember having seen this error before. Let me check if there are some changes from the dev branch that I haven't submitted to the master branch.

[foolfun]

The "silver annotation pipeline" is not yet properly documented as it is still under development, but you should be able to run it. First you run the wiki pipeline as described here. Then you need to build an IDF table using this command:

sling build_idf

Then you can run silver annotation on all the Wikipedia articles:

sling silver_annotation

It takes quite a while to run the silver annotation pipeline (10 hours on my machine). Please let me know if this works for you.

Thanks for your reply! When I ran ‘ sling fuse_items’ met #4. I have no idea why it happened, can you help me?

[ringgaard]

It seems like I will have to do a complete test run of the wiki and silver annotation pipelines. I run these in a slightly different mode using wiki snapshots to get a wikidata dump and the reconciler for fusing items. It seems like there is some bug in the old pipeline.

You should check if you have enough disk space. You will need something like 500 GB free space on you hard drive including your temp directory (usually /tmp). There has been reports about out-of-disk-space is not always reported correctly. You should also check that you don't have a bunch of temp files from runs that crashed. You can remove old temp files using this command:

rm -r /tmp/local.*

It is going to take a while to rerunning the pipelines, so please be patient. I will try to do this over the weekend. I have a server upgrade Sunday which will also delay this.

[foolfun]

It seems like I will have to do a complete test run of the wiki and silver annotation pipelines. I run these in a slightly different mode using wiki snapshots to get a wikidata dump and the reconciler for fusing items. It seems like there is some bug in the old pipeline.

You should check if you have enough disk space. You will need something like 500 GB free space on you hard drive including your temp directory (usually /tmp). There has been reports about out-of-disk-space is not always reported correctly. You should also check that you don't have a bunch of temp files from runs that crashed. You can remove old temp files using this command:

rm -r /tmp/local.*

It is going to take a while to rerunning the pipelines, so please be patient. I will try to do this over the weekend. I have a server upgrade Sunday which will also delay this.

Thank you so much. I will take your advice to try it again.

[foolfun]

I have enough disk space, remove the old temp files and run the command follow. However, it seems that I met the same problem again. Looking forward to your reply

export TMPDIR=/mnt/hdd1/tmp

sling build_wiki --lbzip2 --languages en

[CSQianDong]

@foolfun try sling build_wiki. Withou lbzip2 and languages. It woks for me.

[ringgaard]

I think I managed to fix the error that caused fuse_items to crash, so if you sync to HEAD you should be able to run the wiki pipeline. See this commit.

You can just resume from the fuse_items stage, so you don't need to re-run the whole wiki pipeline again:

sling fuse_items build_kb extract_names build_nametab build_phrasetab

Next, I will try to see if I can reproduce the CHECK fault in the silver annotation pipeline:
[2020-11-27 17:39:19.446544: F sling/nlp/kb/calendar.cc:41] Check failed: num >= 0

[foolfun]

I think I managed to fix the error that caused fuse_items to crash, so if you sync to HEAD you should be able to run the wiki pipeline. See this commit.

You can just resume from the fuse_items stage, so you don't need to re-run the whole wiki pipeline again:

sling fuse_items build_kb extract_names build_nametab build_phrasetab

Next, I will try to see if I can reproduce the CHECK fault in the silver annotation pipeline:
[2020-11-27 17:39:19.446544: F sling/nlp/kb/calendar.cc:41] Check failed: num >= 0

it works! I have been troubled by the issue for nearly two weeks, thank you very much!

[foolfun]

The "silver annotation pipeline" is not yet properly documented as it is still under development, but you should be able to run it. First you run the wiki pipeline as described here. Then you need to build an IDF table using this command:

sling build_idf

Then you can run silver annotation on all the Wikipedia articles:

sling silver_annotation

It takes quite a while to run the silver annotation pipeline (10 hours on my machine). Please let me know if this works for you.

I have run the silver annotation pipeline and the result shows in the following picture. However, I still can not find 'local/data/e/silver/en/silver-00000-of-00010.rec'. I don`t know whether I miss some important steps. Can you help me?

By the way, the files I can find are:

[ringgaard]

The output looks correct. The silver-annotated Wikipedia documents are in train-*.rec and eval.rec. Together these contain all the Wikipedia articles. They are split into train and eval because I use this data as noisy training data for the semantic parser.

NB: I did a complete run of the silver annotation pipeline and it did not get the Check failed: num >= 0 error. This error could be due to Wikidata errors in the date items. My version of Wikidata is from Nov 25.