{"id":3975,"date":"2019-09-19T12:34:05","date_gmt":"2019-09-19T12:34:05","guid":{"rendered":"http:\/\/temu.bsc.es\/meddocan\/?p=3975"},"modified":"2021-02-09T10:45:31","modified_gmt":"2021-02-09T10:45:31","slug":"evaluation","status":"publish","type":"post","link":"https:\/\/temu.bsc.es\/smm4h-spanish\/?p=3975","title":{"rendered":"Evaluation"},"content":{"rendered":"\n

The evaluation will be done at CodaLab<\/a><\/p><\/blockquote>\n\n\n\n

Precision (P) = true positives\/(true positives + false positives)<\/em><\/strong><\/p>

Recall (R) = true positives\/(true positives + false negatives)<\/em><\/strong><\/p>

F-score (F1) = 2*((P*R)\/(P+R))<\/em><\/strong><\/p><\/blockquote>\n\n\n\n

Track A \u2013 Tweet binary classification<\/strong><\/h4>\n\n\n\n

Submissions will be ranked by Precision, Recall and F1-score for the positive class (F-score is the primary metric).<\/p>\n\n\n\n

Prediction format:<\/strong> tab-separated file with headers:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Track B \u2013<\/strong> NER offset detection and classification<\/strong><\/h4>\n\n\n\n

Submissions will be ranked by Precision, Recall and F1-score for each PROFESION [profession] or SITUACION_LABORAL [working status] mention extracted, where the spans overlap entirely (F-score is the primary metric).<\/p>\n\n\n\n

A correct prediction must have the same beginning and ending offsets as the Gold Standard annotation, as well as the same label (PROFESION or SITUACION_LABORAL)<\/p>\n\n\n\n

Prediction format: <\/strong>tab-separated file with headers. Same as the tab-separated format of the Gold Standard.<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n
\n\n\n\n

CodaLab<\/h4>\n\n\n\n

Predictions for each subtask should be contained in a single .tsv (tab-separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as submission. For the evaluation phase which will start on the 1st of March, you are allowed to add the validation set to the training set for training purposes. <\/p>\n\n\n\n

Codalab: https:\/\/competitions.codalab.org\/competitions\/28766<\/a><\/p>\n\n\n\n

  1. Register and wait for approval<\/li>
  2. To make submissions : Participate -> Submit\/View Results -> Click on Task -> Click Submit -> Select File
    Refresh your submission. It goes from Submitted -> Running -> Finished. Scores should be available in the files. You can choose to submit your best scores to the Leaderboard.<\/li>
  3. To view results : Results -> Click on Task -> View results in table<\/li><\/ol>\n\n\n\n

    You will be allowed to make unlimited submissions during the validation stage. During the evaluation stage only 2 submissions will be allowed.<\/p>\n","protected":false},"excerpt":{"rendered":"

    The evaluation will be done at CodaLab Precision (P) = true positives\/(true positives + false positives) Recall (R) = true positives\/(true positives + false negatives) F-score (F1) = 2*((P*R)\/(P+R)) Track A \u2013 Tweet binary classification Submissions will be ranked by Precision, Recall and F1-score for the positive class (F-score is the primary metric). Prediction format: tab-separated […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3975","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=\/wp\/v2\/posts\/3975","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3975"}],"version-history":[{"count":12,"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=\/wp\/v2\/posts\/3975\/revisions"}],"predecessor-version":[{"id":4526,"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=\/wp\/v2\/posts\/3975\/revisions\/4526"}],"wp:attachment":[{"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3975"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3975"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/temu.bsc.es\/smm4h-spanish\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3975"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}