Email your doubts to Salvador Lima (salvador [dot] limalopez [at] gmail [dot] com).
- Q: What is the goal of the shared task?
The goal is to predict the annotations (or codes) of the documents in the test and background sets. - Q: How do I register?
Instructions here: https://temu.bsc.es/distemist/registration/
(BioASQ registration is essential) - Q: How to submit the results?
Instructions here: https://temu.bsc.es/distemist/submission/
Download the example ZIP file. - Q: Can I use additional training data to improve model performance?
Yes, participants may use any additional training data they have available, as long as they describe it in the working notes. We will ask you to summarize such resources in your participant paper. - Q: The task consists of two sub-tasks. Do I need to complete all sub-tasks? In other words, If I only complete a sub-task or two sub-tasks, is it allowed?
Sub-tasks are independent and participants may participate in one or the two of them. - Q: How can I submit my results? Can I submit several prediction files for each sub-task?
You will have to create a ZIP file with your predictions file and submit it to BioASQ (instructions here: https://temu.bsc.es/distemist/submission/).
Yes, you can submit up to 5 prediction files, all in the same ZIP.
Download the example ZIP file. - Q: Should prediction files have headings?
Yes - Q: Are all codes and mentions equally weighted?
Yes - Q: DISTEMIST-linking. What version of the Snomed-CT is in use?
We are using the 2021-10-21 version.
There is a complete list of the valid codes on Zenodo. Codes not present in this list will not be used for the evaluation. - Q: DISTEMIST-linking. What is meant by the NARROW/EXACT column in the task 2 (DISTEMIST-linking) files?
Some disease mentions are more specific than the most granular term existing in Snomed-CT. Then, we assign the most granular possible Snomed-CT term to those mentions and mark them as NARROW.
If the Snomed-CT term corresponds exactly to the mention, we mark it as EXACT. - Q: DISTEMIST-linking. Do I need to add a NARROW/EXACT column in task 2 (DISTEMIST-linking) predictions?
No. Just the codes. - Q: DISTEMIST-linking. If a predicted mention has several codes, do I need to provide them in some particular order?
No. The evaluation library accepts combined codes in any order.
For instance, there is no Snomed-CT term for “orquiepididimitis crónica”. However, there is one for “epididimitis crónica” (236768005) and another for “orquitis crónica” (10194007). Then, the final assignment is 10194007+236768005. - Q: When will the test set be released?
The test set is openly available on Zenodo.