Understanding the schema

[ainamdar-ag]

Hello I've started to look at this data. I understand the high level structure of the XML files but I still have questions which I can't find answers to.

Focusing on debates XML:

1. What is scraperversion? How do I know which version to read from if there are multiple XML files for each day?
2. How do I distinguish different debates? The IDs are only for individual statements within the debates but how does that correlate to each of the debates listed on the Hansard website?
3. Where do I find detailed data dictionary for this data and schema?

Thank you
Atharva

[dracos]

Hi,

1. If revised data is provided for a particular day, we create a new version of the XML, a, b, c, etc. You should always take the latest; if you look at an example, you'll see the earlier versions only contain links to the later version, along with details of anything that has been changed.
2. IDs are not only for individual statements - all major-heading and minor-heading have IDs as well, and that is the structure there is, as mapped onto display at theyworkforyou.com. The source data does not really provide nicely nested information, see eg Better parsing of nested sections theyworkforyou#1750 and related.
3. The documentation is what's available at https://parser.theyworkforyou.com/

[ajparsons]

In python there's also a model for interacting with transcripts in https://github.com/mysociety/mysoc-validator/