plain text processing

[Tortoise17]

Feature description

I have plain text, without any punctuation in English and German. I want to ask that is there any possibility to

1. make capital upper case where necessary? like some nouns are in capital.
2. punctuation? or at least full stop?

Please guide

[svlandeg]

Do I understand your question correctly that you have a non-capitalized, non-punctuated text, and you would like to automatically add punctuation and capitalization to it?

Because sentences are determined with the dependency parser, you can use a pretrained model to (at least partly) achieve what you want:

python -m spacy download en_core_web_lg

import spacy
nlp = spacy.load("en_core_web_lg")
text = "this is a sentence without punctuation this is another sentence"
doc = nlp(text)
output_text = ""
for sent in doc.sents:
    output_text += sent.text.capitalize() + ". "
print(output_text)

which will print

This is a sentence without punctuation. This is another sentence.

Obviously you'll need to do some more work if you'd also like to determine whether a sentence needs a full stop . or a question mark ?. Depending on the language, you could implement some heuristics by looking at the order of the part of speech tags (e.g. is the verb before the noun, etc).

For capitalization of nouns in German, you want to check the POS tags on the token level:

python -m spacy download de_core_news_md

import spacy
from spacy.parts_of_speech import NOUN, PROPN

nlp = spacy.load("de_core_news_md")
text = "san francisco erwägt verbot von lieferrobotern autonome fahrzeuge verlagern haftpflicht auf hersteller"
doc = nlp(text)
output_text = ""
for sent in doc.sents:
    for index, token in enumerate(sent):
        token_text = token.text
        if index == 0 or token.pos in (NOUN, PROPN):
            token_text = token_text.capitalize()
        output_text += token_text + token.whitespace_
    output_text = output_text.strip() + ". "
print(output_text)

Which gives:

San Francisco erwägt Verbot von Lieferrobotern. Autonome Fahrzeuge verlagern Haftpflicht auf Hersteller.

Also note that the pretrained models are trained on properly punctuated text for the most part, so they may not obtain the same accuracy on your type of input. But at least it's a starting point.

FYI - for future reference, these kind of usage questions are probably better asked at StackOverflow, as we try to keep this issue tracker focused on bug reports and feature requests specifically. Thanks!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.