{"id":17,"date":"2025-11-07T11:07:23","date_gmt":"2025-11-07T10:07:23","guid":{"rendered":"https:\/\/temu.bsc.es\/MultiClinAI\/?page_id=17"},"modified":"2026-02-26T16:33:49","modified_gmt":"2026-02-26T15:33:49","slug":"evaluation-submission","status":"publish","type":"page","link":"https:\/\/temu.bsc.es\/MultiClinAI\/evaluation-submission\/","title":{"rendered":"Evaluation & Submission"},"content":{"rendered":"\n

Evaluation and Submission<\/strong><\/h1>\n\n\n\n

Evaluation<\/h2>\n\n\n\n

Evaluation of automatic predictions for this task will be conducted under two different scenarios or sub-tracks. In all cases, the primary evaluation metrics will be micro-averaged precision, recall, and F1-scores<\/strong>. The evaluation scripts, along with proper documentation, will be freely available on GitHub to allow participating teams to test the evaluation tools locally.<\/p>\n\n\n\n

As for the baseline systems, for the MultiClinNER<\/strong> subtask the prediction baseline will be based on vocabulary transfer results derived from the training set entities, combined with gazetteer lookup on the test set corpus. For the MultiClinCorpus<\/strong> task, a simple lexical lookup of translated entity mentions will serve as the baseline.<\/p>\n\n\n\n

More information coming soon!<\/p>\n\n\n\n

Metrics Definition<\/h2>\n\n\n\n

The following metrics are reported:<\/p>\n\n\n\n

    \n
  • Micro-F1<\/strong>: Micro-averaged F1-score computed over all sentence segmentation decisions.<\/li>\n\n\n\n
  • Micro-Precision<\/strong>: Micro-averaged precision over all sentence segmentation decisions.<\/li>\n\n\n\n
  • Micro-Recall<\/strong>: Micro-averaged recall over all sentence segmentation decisions.<\/li>\n\n\n\n
  • Macro-F1<\/strong>: Macro-averaged F1-score computed at the document (note) level.<\/li>\n\n\n\n
  • Macro-Precision<\/strong>: Macro-averaged precision at the document level.<\/li>\n\n\n\n
  • Macro-Recall<\/strong>: Macro-averaged recall at the document level.<\/li>\n<\/ul>\n\n\n\n

    Ranking Criteria<\/h3>\n\n\n\n

    The official ranking of submissions will be determined according to the following priority:<\/p>\n\n\n\n

      \n
    1. Micro F1-score<\/strong><\/li>\n\n\n\n
    2. Macro F1-score<\/strong><\/li>\n<\/ol>\n\n\n\n

      Additional metrics are reported for completeness and analysis purposes.<\/p>\n\n\n\n

      For full implementation details, participants are encouraged to consult the official evaluation scripts<\/strong>.<\/p>\n\n\n\n

      \n\n\n\n

      Submission<\/h2>\n\n\n\n

      The official submission process for MultiClinAI will be conducted through the CodaBench evaluation platform<\/strong>.<\/p>\n\n\n\n

      Step 1 \u2013 Prepare your submission files<\/h3>\n\n\n\n

      Teams must prepare a single ZIP file<\/strong> containing all the run.tsv<\/code> files corresponding to the submitted runs.<\/p>\n\n\n\n

      Each run.tsv<\/code> file must be a tab-separated (.tsv) file<\/strong> and must include a header with exactly the following columns<\/strong>:<\/p>\n\n\n\n

        \n
      • filename<\/code><\/li>\n\n\n\n
      • label<\/code><\/li>\n\n\n\n
      • start_span<\/code><\/li>\n\n\n\n
      • end_span<\/code><\/li>\n\n\n\n
      • text<\/code><\/li>\n<\/ul>\n\n\n\n

        All columns are mandatory<\/strong>. The start_span<\/code> and end_span<\/code> values must correspond to character offsets<\/strong> in the original document text.<\/p>\n\n\n\n

        Files that do not strictly follow this format may be rejected by the evaluation system.<\/p>\n\n\n\n

        Step 2 \u2013 Upload to evaluation platform<\/h3>\n\n\n\n

        The ZIP file must be uploaded to the official MultiClinAI task page on our evaluation platform<\/strong> during the evaluation phase.<\/p>\n\n\n\n

        The evaluation platform will automatically validate and score the submission. Only submissions successfully uploaded to the evaluation platform will be considered official and included in the ranking.<\/p>\n\n\n\n

        Step 3 \u2013 Send backup email (mandatory)<\/h3>\n\n\n\n

        Immediately after uploading to the evaluation platform, teams must send an email to the task organizers including:<\/p>\n\n\n\n

          \n
        • The exact same ZIP file<\/strong> uploaded to CodaBench (as a backup copy).<\/li>\n\n\n\n
        • A mandatory README file<\/strong> describing in detail the methodology used for each submitted run.<\/li>\n<\/ul>\n\n\n\n

          The README must clearly explain the approach followed for each run (e.g., model architecture, training procedure, external resources, prompting strategy, projection method, hyperparameters, and any additional data used).<\/p>\n\n\n\n

          Submissions without a README will be considered incomplete<\/strong> and may not be included in the final evaluation or official results.<\/p>\n\n\n\n

          \n\n\n\n

          Detailed deadlines, submission limits, and the official evaluation platform link will be announced before the evaluation phase opens.<\/p>\n","protected":false},"excerpt":{"rendered":"

          Evaluation and Submission Evaluation Evaluation of automatic predictions for this task will be conducted under two different scenarios or sub-tracks. In all cases, the primary evaluation metrics will be micro-averaged precision, recall, and F1-scores. The evaluation scripts, along with proper documentation, will be freely available on GitHub to allow participating teams to test the evaluation […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-17","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/pages\/17","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/comments?post=17"}],"version-history":[{"count":9,"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/pages\/17\/revisions"}],"predecessor-version":[{"id":221,"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/pages\/17\/revisions\/221"}],"wp:attachment":[{"href":"https:\/\/temu.bsc.es\/MultiClinAI\/wp-json\/wp\/v2\/media?parent=17"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}