Evaluation process: (1) you submit your results, (2) we perform the evaluation off-line and (3) return the final scores.

Submission format

LivingNER – Species NER track. A TSV file with one row per mention, headers and the following columns:

  • filename
  • mark
  • label (mention label)
  • off0 (starting character offset)
  • off1 (ending character offset)
  • span (mention span)
Figure 1. Example of submission file for LivingNER-Species NER track

LivingNER – Species Norm track. A TSV file with one row per mention, headers and the following columns:

  • filename
  • mark
  • label (mention label)
  • off0 (starting character offset)
  • off1 (ending character offset)
  • span (mention span)
  • NCBITax (mention code in NCBITaxonomy)
Figure 2. Example of submission file for LivingNER-Species Norm track

LivingNER – Clinical IMPACT track. A TSV file with one row per document, headers and the following columns:

  • filename
  • isPet (Yes/No)
  • PetIDs (NCBITaxonomy codes of pet & farm animals present in document)
  • isAnimalInjury (Yes/No)
  • AnimalInjuryIDs (NCBITaxonomy codes of animals causing injuries present in document)
  • isFood (Yes/No)
  • FoodIDs (NCBITaxonomy codes of food mentions present in document)
  • isNosocomial (Yes/No)
  • NosocomialIDs (NCBITaxonomy codes of nosocomial species mentions present in document)
Figure 2. Example of submission file for LivingNER-Clinical Impact track

To evaluate your systems AFTER June 2022, do it in Codalab: 

The following sections are kept here for historical reasons. They are not relevant AFTER June 2022

Submission method

Submissions will be made via SFTP.

Submission tutorial

Submission instructions

5 submissions per sub-track will be allowed.

You must submit ONE SINGLE ZIP file with the following structure:

  • One subdirectory per subtask in which you are participating. 
  • In each subdirectory, you must include the results in a TSV file with the format defined in the “Submission format” section.
  • The TSV file must have the .tsv file extension and include ALL your predictions.
  • If you have more than one system, you can include their predictions and we will evaluate them (up to 5 prediction runs). Use separate files for each run, with numbers and a recognizable name. For example, 1-systemDL and 2-systemlookup.
  • In addition, in the parent directory, you must add a README.txt file with your contact details (team name, affiliation, and authors) and a really short explanation of your system.

Example submission ZIP file