SkimLit feature engineering

[tims457]

In model_5 we encoded both the sentence position and number of sentences in the abstract as one-hot features. If it's the relative position of the sentence in the abstract that is important, why not just create one new feature which is the normalized sentence position from 0 to 1? For example sentence 0 of 10 would have the value 0.0, sentence 2 would be 0.1, and so on. This should encode the same information but with fewer parameters.

The new features would be:

train_sent_position = train_df["line_number"] / train_df["total_lines"]
val_sent_position = val_df["line_number"] / val_df["total_lines"]

Is there a specific reason behind the one-hot choice?

Thanks!

[mrdbourke]

Hey Tim,

No specific reason for one-hot other than its ease of use.

Your suggested option is valid as well.

Did you see how it affected the results?

[tims457]

Dan,

I got similar results

{'accuracy': 82.01707930623593,
 'f1': 0.8198785512015585,
 'precision': 0.8201347091205505,
 'recall': 0.8201707930623593}

And this is how I initially set up the model.

Cheers!

-Tim