BioNLP'09 Shared Task

Bio Event Server

U-Compare supported the NAACL BioNLP'09 Shared Task on Event Extraction. Evolved from the shared task, we provide servers of the event extraction tools, both/either locally and/or remotely deployed.
We have just started this work, no service available yet. Please wait for our announcement of the upcoming services.

For users, just to drag and drop the server component to call services. You can also call services from command-line. See U-Compare User Guide and Developer Guide for details of such U-Compare functions.

U-Compare provides an easy wrapper package for the service providers. If you are interested in providing your own service, please go to our google group website and request invitation for your participation.

Advantages

U-Compare provides a set of interoperable (compatible) componenets based on UIMA, such as protein mention taggers, syntactic parsers, part-of-speech taggers, etc. The bio event servers are also one of such interoperable components, easily combined to build your workflow regardless of local or web services.

By using U-Compare, you can easily

  • compare and evaluate generated annotations together with the gold standard data if any exist
  • visualize the annotations of the shared task corpus and your own tool, together with the exisiting tools if any specified

The bio event servers takes raw text and protein mention annotations as inputs, outputing extracted events. You can input

  • any text with our protein mention taggers, or manually annotated protein mentions
  • shared task format gold standard data (a1 format)
  • etc.

The output of the servers can be converted into many formats

  • UIMA annotations
  • shared task format (a2 format)
  • U-Compare simple stand-off format
  • etc.

How to Use U-Compare Shared Task Format Reader

Visualizing your annotations together with the corpora annotations

What users required is just to specify the directory where your annotation files are stored. However if you just wish to see the gold standard, no need to specify. Follow these steps:

  • Once you created your annotations for the test/training dataset of the shared task in the shared task format, put them to a single directory somewhere in your local file system.
  • Start U-Compare, select "BioNLP'09 Shared Task Visualization" predefined workflow from the "Workflow" menu.
  • (Optional) Click edit button or double click on the "BioNLP'09 Shared Task Annotation Reader", specify the directory where you stored your annotation files for the "Annotation Directory" configuration parameter field.
  • Click the play button in the bottom of the window.
  • When the session statistics window is shown, click "Show" button and get the visualization.
  • Because the event relations are very dense, increasing font size and narrowing down the windows width (and changing font) will improve the impression.

Obtaining annotations from U-Compare ready-to-use components for your tool

You can obtain annotations generated by the U-Compare ready-to-use components. Obtained annotations are all in a simple same format in the stand-off style, regardless of which tool you specified. In addition to the current components listed above, we have prepared the Stanford Dependency Parser and SecondString based user-dictionary matching tool. [While implementing the U-Compare version of Stanford Parser, we referred and reused the codes from the University of Colorado Health Science Center. Thank you very much for your cooperation!]

For example,

  • open the "BioNLP'09 Shared Task Visualization" predefined workflow as above
  • remove the the "BioNLP'09 Shared Task Annotation Reader" component by clicking the red X icon
  • drag and drop a sentence detector, e.g. the "UIMA Sentence Splitter" component into the component list where the removed component was shown before, from the the "Sentence Detectors" category in the component library on the right
  • drag and drop the "Stanford Parser" component into the bottom of the component list after the sentence detector (because it requires sentence detections preprocessed), from the the "Parsers" category in the component library on the right
  • drag and drop the "U-Compare Stand-off File Writer" component into the bottom of the component list (after Stanford Parser in this case), from the the "Components for Develepers" category in the component library on the right
  • Click edit button or double click on the "U-Compare Stand-off File Writer" component, and specify a directory where the generated annotation files will be stored. Optionally you can turn "generateRawText" off not to attach the original rawtext part as a header.
  • (Optional) Click edit button or double click on the "BioNLP'09 Shared Task Annotation Reader", specify the directory where you stored your annotation files for the "Annotation Directory" configuration parameter field.
  • Click play button
Obtained files are formatted like:
186
The document length in byte is output in the first line (end with new line), 
then the raw text follows as is (attaching a new line in the end), 
finally annotations follow line by line.
0 186 Document id="u1"
0 3 POSToken id="u2" pos="DT"
....
id is a special feature to make cross-annotation references. Please refer to the developer guide in this web site for details.

Embed your tool into U-Compare, make automatic combination expansion and comparison

Advanced topic, information will be provided later.