{"id":24,"date":"2020-12-23T15:27:44","date_gmt":"2020-12-23T15:27:44","guid":{"rendered":"https:\/\/temu.bsc.es\/meddoprof\/?p=24"},"modified":"2021-04-21T16:15:05","modified_gmt":"2021-04-21T14:15:05","slug":"evaluation-method","status":"publish","type":"post","link":"https:\/\/temu.bsc.es\/meddoprof\/evaluation-method\/","title":{"rendered":"Evaluation Method"},"content":{"rendered":"\n<p>MEDDOPROF Shared Task&#8217;s sub-tracks will be evaluated in the following way:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Track A \u2013<\/strong>&nbsp;<strong><strong>MEDDOPROF-NER<\/strong><\/strong><\/h4>\n\n\n\n<p>Submissions will be ranked by Precision, Recall and F1-score for each PROFESION [profession] or SITUACION_LABORAL [working status] mention extracted, where the spans overlap entirely (F-score is the primary metric).<\/p>\n\n\n\n<p>A correct prediction must have the same beginning and ending offsets as the Gold Standard annotation, as well as the same label (PROFESION or SITUACION_LABORAL)<\/p>\n\n\n\n<p><strong>Prediction format: <\/strong><a href=\"https:\/\/brat.nlplab.org\/standoff.html\">brat annotation files<\/a> (.ANN) with your predictions.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Track B \u2013<\/strong>&nbsp;<strong><strong>MEDDOPROF-CLASS<\/strong><\/strong><\/h4>\n\n\n\n<p>Submissions will be ranked by Precision, Recall and F1-score for each PACIENTE [patient], FAMILIAR [family member], SANITARIO [health professional] or OTROS [others] mention extracted, where the spans overlap entirely (F-score is the primary metric).<\/p>\n\n\n\n<p>A correct prediction must have the same beginning and ending offsets as the Gold Standard annotation, as well as the same label.<\/p>\n\n\n\n<p><strong>Prediction format: <\/strong><a href=\"https:\/\/brat.nlplab.org\/standoff.html\">brat annotation files<\/a> (.ANN) with your predictions.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Track C \u2013<\/strong>&nbsp;<strong>MEDDOPROF-NORM<\/strong><\/h4>\n\n\n\n<p>For this track, participants will be provided with a list of unique concept identifiers from the European Skills, Competences, Qualifications and Occupations (ESCO) classification and relevant SNOMED-CT terms. Participants will have to detect PROFESION and SITUACION_LABORAL mentions and map each of them to one of the terms in the list. Then, their mappings will be compared to the manually annotated concept ids and evaluated using F1-score.<\/p>\n\n\n\n<p>Precision, Recall and F1-score will be calculated using the following formula:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote has-text-align-center is-layout-flow wp-block-quote-is-layout-flow\"><p><strong><em>Precision (P) = <\/em><\/strong><em>true positives\/(true positives + false positives)<\/em><\/p><p><strong><em>Recall (R) = <\/em><\/strong><em>true positives\/(true positives + false negatives)<\/em><\/p><p><strong><em>F-score (F1) = <\/em><\/strong><em>2*((P*R)\/(P+R))<\/em><\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>MEDDOPROF Shared Task&#8217;s sub-tracks will be evaluated in the following way: Track A \u2013&nbsp;MEDDOPROF-NER Submissions will be ranked by Precision, Recall and F1-score for each PROFESION [profession] or SITUACION_LABORAL [working status] mention extracted, where the spans overlap entirely (F-score is the primary metric). A correct prediction must have the same beginning and ending offsets as [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-24","post","type-post","status-publish","format-standard","hentry","category-evaluation"],"_links":{"self":[{"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/posts\/24","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/comments?post=24"}],"version-history":[{"count":10,"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/posts\/24\/revisions"}],"predecessor-version":[{"id":247,"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/posts\/24\/revisions\/247"}],"wp:attachment":[{"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/media?parent=24"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/categories?post=24"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/temu.bsc.es\/meddoprof\/wp-json\/wp\/v2\/tags?post=24"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}