The evaluation will be done at CodaLab. SocialDISNER submissions will be ranked by Precision, Recall, and F1-score for each ENFERMEDAD [disease] mention extracted, where the spans overlap entirely (F1-score is the primary metric). A correct prediction must have the same beginning and ending offsets as the Gold Standard annotation
Evaluation dates
July 11, 2022, midnight – July 15, 2022, UTC
Submission format
Participating teams will have to generate a *.tsv file with the annotations detected for each of the test documents according to the following column structure:
- tweets_id: Name of the file from which the disease mention has been extracted.
- begin: Character number where the detected mentions start.
- end: Character number where the detected mention ends.
- type: In this case it will always be ENFERMEDAD
- extraction: Mention extracted from text
tweets_id begin end type extraction
1357198223706894339 12 19 ENFERMEDAD alergia
1357198223706894339 21 26 ENFERMEDAD covid
CodaLab
Predictions for each subtask should be contained in a single .tsv (tab-separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as a submission. For the evaluation phase which will start on the 11th of July, you are allowed to add the validation set to the training set for training purposes.
Codalab: https://codalab.lisn.upsaclay.fr/competitions/3531
- Register and wait for approval
- To make submissions : Participate -> Submit/View Results -> Click on Task -> Click Submit -> Select File
Refresh your submission. It goes from Submitted -> Running -> Finished. Scores should be available in the files. You can choose to submit your best scores to the Leaderboard. - To view results : Results -> Click on Task -> View results in table
You will be allowed to make unlimited submissions during the validation stage. During the evaluation stage only 2 submissions will be allowed.