ArgMining22-Shared-Task

Shared task predicting validity and novelty

View the Project on GitHub phhei/ArgsValidNovel

Data Description

All data is manually annotated by three annotators. All three annotators annotated individually without any consultation with each other during the annotation process.

Subtask A

Each line consists following fields:

  1. topic: title of the debate
  2. Premise
  3. Conclusion
  4. Validity
    1. 1 means: conclusion is valid
    2. 0 means: conclusion is defeasibly valid (this is not an own class itself - you are free to exclude such samples)
    3. -1 means: conclusion is not valid
  5. Validity-Confidence
    1. “very confident”: all three annotators agree in the validity-judgement
    2. “confident”: two out of three annotators agree in the validity-judgment while one annotator abstains
    3. “majority”: two out of three annotators agree in the validity-judgement, one disagrees
    4. “defeasible”: there is no majority in votes - either all three annotators can’t decide a validity rating or one annotator can’t decide and the two others disagree each other (probably a very subjective sample)
  6. Novelty
    1. 1 means: conclusion is novel
    2. 0 means: conclusion is somewhat novel/ borderline case (this is not an own class itself - you are free to exclude such samples)
    3. -1 means: conclusion is not novel
  7. Novelty-Confidence
    1. same as validity-Confidence, only for novelty

Additional remarks for Subtask A

Subtask B

Each line consists following fields:

  1. topic: title of the debate
  2. Premise
  3. Conclusion 1
  4. Conclusion 2
  5. Validity
    1. 1 means: Conclusion 2 is more valid
    2. 0 means: both conclusions are equally valid
    3. -1 means: Conclusion 1 is more valid
  6. Votes_Concl1IsMoreValid: number of votes voting for Conclusion 1 in terms of validity
  7. Votes_ValidTie: numbers of votes voting for a tie in terms of validity
  8. Votes_Concl2IsMoreValid: number of votes voting for Conclusion 2 in terms of validity
  9. Novelity
    1. 1 means: Conclusion 2 is more novel
    2. 0 means: both conclusions are equally novel
    3. -1 means: Conclusion 1 is more novel
  10. Votes_Concl1IsMoreNovel: number of votes voting for Conclusion 1 in terms of novelty
  11. Votes_NovelTie: numbers of votes voting for a tie in terms of novelty
  12. Votes_Concl2IsMoreNovel: number of votes voting for Conclusion 2 in terms of novelty

Additional remarks for Subtask B