The evaluation of the competition will be done against a manually annotated data set purposely created for this task.

Submission format

Sub-track 1 DisTEMIST-Entities

Participating teams will have to generate a tsv file with the annotations detected for each of the test documents according to the following column structure:

  • filename: Name of the file from which the disease mention has been extracted.
  • mention label: In this case it will always be ENFERMEDAD
  • starting character offset: Character number where the detected mention starts.
  • ending character offset: Character number where the detected mention ends.
  • mention span: Mention extracted from text
filename	label	off0	off1	span
es-S0004-06142007000600016-2	ENFERMEDAD	330	374	lesión nodular sólida en cabeza de epidídimo
es-S0004-06142007000600016-2	ENFERMEDAD	436	467	tumor adenomatoide de epidídimo
es-S0004-06142007000600016-5	ENFERMEDAD	1136	1167	tumor adenomatoide de epidídimo

Sub-track 2 Distemist-Linking

Participating teams will have to generate a tsv file with the annotations detected for each of the test documents according to the following column structure:

  • filename: Name of the file from which the disease mention has been extracted.
  • mention label: It will always be ENFERMEDAD
  • starting character offset: Character number where the detected mention starts.
  • ending character offset: Character number where the detected mention ends.
  • mention span: Mention extracted from text
  • code: List of Snomed-CT concept codes linked to the mention. If there is more than one code associated with a mention, they will be concatenated by the symbol “+”
filename	label	off0	off1	span	code
es-S0004-06142007000600016-2	ENFERMEDAD	330	374	lagunaas coriorretinianas	302893000
es-S0004-06142007000600016-2	ENFERMEDAD	436	467	anomalías de la migración neuroral	253146009
es-S0004-06142007000600016-5	ENFERMEDAD	1136	1167	afectación macular	312999006

To evaluate your systems AFTER June 2022, do it in Codalab: https://codalab.lisn.upsaclay.fr/competitions/7162

The following sections are kept here for historical reasons. They are not relevant AFTER June 2022

Evaluation dates (OLD: shared task)

  • 10 May 09:00 GMT – 15 May 09:00 GMT

Submission instructions (OLD: shared task)

Download here the submission tutorial

Download here a submission ZIP example.

To submit your results, (1) register at BioASQ website and (2), follow the following instructions:

A. 5 submissions per sub-track will be allowed.

B. You must submit ONE SINGLE ZIP file with the following structure:

  • One subdirectory per subtask in which you are participating. 
  • In addition, in the parent directory, you must add a README.txt file with your contact details and a really short explanation of your system.
  • If you have more than one system, you can include their predictions and we will evaluate them (up to 5 prediction runs).
  • For each prediction run:
    • You must include the tab-separated file with your predictions. 
    • One single file with all the predictions.
    • With a .tsv file extension.  
    • If you have more than one system, include one tab-separated file for each system.
    • If you have more than one system, name the tab-separated files with numbers and a more or less recognizable name. For example, 1-systemDL.tsv and 2-systemlookup.tsv

C. With this ZIP file, go to the participants area of BioASQ (http://participants-area.bioasq.org/Tasks/distemist/) and submit there your file.