The evaluation will be done at CodaLab. SocialDISNER submissions will be ranked by Precision, Recall, and F1-score for each ENFERMEDAD [disease] mention extracted, where the spans overlap entirely (F1-score is the primary metric). A correct prediction must have the same beginning and ending offsets as the Gold Standard annotation

Evaluation dates

July 11, 2022, midnight – ┬áJuly 15, 2022, UTC

Submission format

Participating teams will have to generate a *.tsv file with the annotations detected for each of the test documents according to the following column structure:

  • tweets_id: Name of the file from which the disease mention has been extracted.
  • begin: Character number where the detected mentions start.
  • end: Character number where the detected mention ends.
  • type: In this case it will always be ENFERMEDAD
  • extraction: Mention extracted from text
tweets_id	begin	end	type	extraction
1357198223706894339	12	19	ENFERMEDAD	alergia
1357198223706894339	21	26	ENFERMEDAD	covid


Predictions for each subtask should be contained in a single .tsv (tab-separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as a submission. For the evaluation phase which will start on the 11th of July, you are allowed to add the validation set to the training set for training purposes.


  1. Register and wait for approval
  2. To make submissions : Participate -> Submit/View Results -> Click on Task -> Click Submit -> Select File
    Refresh your submission. It goes from Submitted -> Running -> Finished. Scores should be available in the files. You can choose to submit your best scores to the Leaderboard.
  3. To view results : Results -> Click on Task -> View results in table

You will be allowed to make unlimited submissions during the validation stage. During the evaluation stage only 2 submissions will be allowed.

A step-by-step tutorial of the previous SMM4H 2021 ProfNER tack (check sub-track B) format is found here!