The evaluation will be done at CodaLab. SocialDISNER submissions will be ranked by Precision, Recall, and F1-score for each ENFERMEDAD [disease] mention extracted, where the spans overlap entirely (F1-score is the primary metric). A correct prediction must have the same beginning and ending offsets as the Gold Standard annotation

Evaluation dates

July 11, 2022, midnight – ┬áJuly 15, 2022, UTC

Submission format

Participating teams will have to generate a *.tsv file with the annotations detected for each of the test documents according to the following column structure:

  • tweets_id: Name of the file from which the disease mention has been extracted.
  • begin: Character number where the detected mentions start.
  • end: Character number where the detected mention ends.
  • type: In this case it will always be ENFERMEDAD
  • extraction: Mention extracted from text
tweets_id	begin	end	type	extraction
1357198223706894339	12	19	ENFERMEDAD	alergia
1357198223706894339	21	26	ENFERMEDAD	covid

CodaLab

Predictions for each subtask should be contained in a single .tsv (tab-separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as a submission. For the evaluation phase which will start on the 11th of July, you are allowed to add the validation set to the training set for training purposes.

Codalab: https://codalab.lisn.upsaclay.fr/competitions/3531

  1. Register and wait for approval
  2. To make submissions : Participate -> Submit/View Results -> Click on Task -> Click Submit -> Select File
    Refresh your submission. It goes from Submitted -> Running -> Finished. Scores should be available in the files. You can choose to submit your best scores to the Leaderboard.
  3. To view results : Results -> Click on Task -> View results in table

You will be allowed to make unlimited submissions during the validation stage. During the evaluation stage only 2 submissions will be allowed.

A step-by-step tutorial of the previous SMM4H 2021 ProfNER tack (check sub-track B) format is found here!