{"id":3966,"date":"2019-09-19T12:27:20","date_gmt":"2019-09-19T12:27:20","guid":{"rendered":"http:\/\/temu.bsc.es\/meddocan\/?p=3966"},"modified":"2019-09-19T12:27:20","modified_gmt":"2019-09-19T12:27:20","slug":"examples","status":"publish","type":"post","link":"https:\/\/temu.bsc.es\/meddocan\/index.php\/examples\/","title":{"rendered":"Examples"},"content":{"rendered":"\n

In this section, we show examples of how the evaluation will be carried out to make it clearer. For the following examples, we will consider that this set of MEDDOCAN tags is our Gold Standard (GS):<\/p>\n\n\n\n

\"\"
Example of tags in a Gold Standard file
<\/figcaption><\/figure>\n\n\n\n

This GS file is in i2b2<\/em> format. We have a DATE<\/em> entity, a CONTACT<\/em> entity, four LOCATION<\/em> entities, and a NAME<\/em> entity. Each entity tag is composed of the ID, the START<\/em> and END<\/em> offsets, the text snippet between the previous offsets, the entity TYPE<\/em> and a COMMENT<\/em>.<\/p>\n\n\n\n

NOTE:<\/strong> The ID<\/em>, TEXT<\/em>, and COMMENT<\/em> fields are not used for any of the evaluation metrics. The number in the ID<\/em> field and the COMMENT<\/em> field are arbitrary, and the evaluation of the TEXT<\/em> field is implicit in the offset evaluation, as the text is the same for the GS and the systems.<\/p>\n\n\n\n

Sub-track 1: NER offset<\/strong><\/h2>\n\n\n\n

In this sub-track we want to match exactly the beginning<\/strong> and end<\/strong> locations of each PHI entity tag, as well as detecting correctly the annotation type<\/strong>. The following system annotations will be accepted by the evaluation script even if the numbers on the ID fields are different, and additional comments are included.<\/p>\n\n\n\n

\"\"
Example of tags in a system submission file<\/figcaption><\/figure>\n\n\n\n

For this example the scores obtained by this system are the following:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Precisi\u00f3n<\/em><\/strong> has been computed dividing true positives (4) by the sum of true positives and false positives (0), scoring 4\/(4+0) = 1.0<\/strong><\/em>. Recall<\/em><\/strong> has been computed dividing true positives by the sum of true positives and false negative (3: Tags with IDs T1, T5, and T7 in the GS), scoring 4\/(4+3) = 0.5714<\/strong><\/em>. Finally, F1<\/em><\/strong> is computed using precision and recall, scoring 2*((1*0.5714)\/(1+0.5714))= 2*(0.5714\/1.5714) = 0.7273<\/strong><\/em>.<\/p>\n\n\n\n

NOTE:<\/strong> This is just an example. We are aware that achieving a precision score of 1.0 is quite a difficult task.<\/p>\n\n\n\n

Sub-track 2: Sensitive span detection<\/strong><\/h2>\n\n\n\n

For this second sub-track the goal is only to identify sensitive data. Thus, we will consider a span-based evaluation, regardless of the actual type of entity or the correct offset identification of multi-token sensitive phrase mentions.<\/p>\n\n\n\n

Strict span evaluation<\/strong><\/h4>\n\n\n\n

In the strict evaluation metric, the evaluation script will accept as correct annotations in the submissions that match exactly the start and end offset<\/strong> of the annotations in the GS file. For instance, it will accept as correct these tags:<\/p>\n\n\n\n

\"\"
Example of tags in a system submission file<\/figcaption><\/figure>\n\n\n\n

But also the following ones, even if the entity tag and type are not specified<\/strong> (the system has submitted PHI<\/em> and OTHER<\/em> as ENTITY<\/em> and TYPE<\/em>, respectively):<\/p>\n\n\n\n

\"\"
Example of a system submission file with unspecified tags<\/figcaption><\/figure>\n\n\n\n

Merged spans evaluation<\/strong><\/h4>\n\n\n\n

For this sub-track we will also additionally compute another evaluation <\/strong>where we will merge the spans of PHI connected by non-alphanumerical characters<\/strong>. For instance, consider this tag in a GS file<\/strong>:<\/p>\n\n\n\n

\"\"
Example of merged span evaluation: GS tag<\/figcaption><\/figure>\n\n\n\n

For the previous tag, in the merged spans evaluation, the script will accept as correct the following annotation (this example is also accepted by the strict span evaluation metric):<\/p>\n\n\n\n

\"\"
Example of merged span evaluation: system annotation with an unspecified tag

<\/figcaption><\/figure>\n\n\n\n

But also the following one, because the scripts merges the spans of the annotation if there are ony non-alphanumerical characters between them:<\/p>\n\n\n\n

\"\"
Example of merged span evaluation: system annotations with unspecified tags matching the GS annotation using merging of spans
<\/figcaption><\/figure>\n\n\n\n

This merging process is carried out both in the GS file and in the system submission file. Therefore, the result is symmetric even if the larger span is given by the system (“Navarro Cu\u00e9llar” and “Ignacio” spans in the GS file and “Navarro Cu\u00e9llar, Ignacio” in the system file).<\/p>\n","protected":false},"excerpt":{"rendered":"

In this section, we show examples of how the evaluation will be carried out to make it clearer. For the following examples, we will consider that this set of MEDDOCAN tags is our Gold Standard (GS): This GS file is in i2b2 format. We have a DATE entity, a CONTACT entity, four LOCATION entities, and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[5],"tags":[],"_links":{"self":[{"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/posts\/3966"}],"collection":[{"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/comments?post=3966"}],"version-history":[{"count":1,"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/posts\/3966\/revisions"}],"predecessor-version":[{"id":3967,"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/posts\/3966\/revisions\/3967"}],"wp:attachment":[{"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/media?parent=3966"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/categories?post=3966"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/temu.bsc.es\/meddocan\/index.php\/wp-json\/wp\/v2\/tags?post=3966"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}