Email your doubts to Antonio Miranda (antoniomiresc@gmail.com)
- Q: What is the goal of the shared task?
The goal is to predict the annotations (or codes) of the documents in the test and background sets. The goal of the subtask 2 (CLINICAL IMPACT) is to predict the document category and the evidence for those categories - Q: How do I register?
Here: https://temu.bsc.es/livingner/2022/01/28/registration/ - Q: How to submit the results?
We will provide further information in the following days.
Download the example ZIP file.
See Submission page for more info. - Q: Can I use additional training data to improve model performance?
Yes, participants may use any additional training data they have available, as long as participants describe it in the working notes. We will ask you to summarize such resources in your participant paper. - Q: The task consists of three sub-tasks. Do I need to complete all sub-tasks? In other words, If I only complete a sub-task or two sub-tasks, is it allowed?
Sub-tasks are independent and participants may participate in one, two, or the three of them. - Q: How can I submit my results? Can I submit several prediction files for each sub-task?
You will have to create a ZIP file with your predictions file and submit it to EasyChair (further details will be soon released).
Yes, you can submit up to 5 prediction files, all in the same ZIP.
Download the example ZIP file.
See Submission page for more info. - Q: Should prediction files have headings?
No, prediction files should have no headings. - Q: Are all codes and mentions equally weighted?
Yes. - Q: LivingNER-NORM and CLINICAL Impact. What version of the NCBI Taxonomy is in use?
We are using the latest version available in January 2021.
There is a complete list of the valid codes on Zenodo. Codes not present in this list will not be used for the evaluation. - Q: LivingNER-NORM and CLINICAL Impact. What is meant by the /H appended to various codes?
Some SPECIES mentions are more specific than the most granular term available in the NCBI Taxonomy. Then, we append /H to the code.
For example, “K pneumoniae BLEE” is not specified in the NCBI Taxonomy. But “Klebsiella pneumonia” is (code 573). Then, we assign 573|H. - Q: LivingNER-NORM and CLINICAL Impact. What do the codes separated by a “|” mean?
Some SPECIES mentions are only correctly described by a combination of NCBI Taxonomy codes. For instance, “virus B y C de la hepatitis” does not exist as such in NCBI Taxonomy. However, we may express it as a combination of the NCBI Taxonomy terms “Hepatitis B virus” (10407) and “Hepacivirus C” (11103). Then, we assign 10407|11103. - Q: LivingNER-NORM and CLINICAL Impact. If a predicted mention has several codes, do I need to provide them in some particular order?
No. The evaluation library accepts combined codes in any order. - Q. In LivingNER-App. When one of the entities results in a NOCODE (the placeholder for when the NCBI taxonomy does not contain any suitable code), this NOCODE marker should appear in the final output for subtrack3? Or should we totally omit them?
As an example, totally made up (see the bold part):
(1) caso_clinico_neurologia78 Yes NOCODE No NA Yes 9031+NOCODE No NA <– adding the NOCODE just like it was any other code
or
(2) caso_clinico_neurologia78 Yes NA No NA Yes 9031 No NA <— totally ignoring the NOCODES putting an NA when no other evidence is available despite the flag being “Yes”
The correct answer is the first one. As a general rule, every time you try to add an NCBI code but your mention is not in the scope of the terminology, you should add NOCODE. That is what we did in the manual annotation process (https://doi.org/10.5281/zenodo.6385162)