Sections:
- Input Text Reader
- Parallel Aggregate
- Auto-Wrapping
- Configuring the Parallel Aggeragate
- Interpreting Results
- Saving Results
- Saving Components
- Combinatorial Comparison
- Fixed Flow Aggregates
- Section Summary
Comparison Workflows
The original aim of the U-Compare project was to develop a tool that made it easy for developers to compare different components and different combinations of components against each other. In this section we see how this can be done by stepping through the creation of a workflow to compare three different tokenizer components.
Input Text Reader
To begin create a blank workflow. Into this blank workflow we are going add a different collection reader than we have been using up until now, the ``Input Text Reader''. Drag and Drop this component from the Collection Readers directory of the Component Library into the Collection Reader box of the workflow viewer.
As you can see, this component appears quite different from the ``Aimed Collection Reader'' (and in fact from all other included components). It is a special collection reader which allows the user to directly input text to be used as the ``Collection'' when the workflow is processed. To use it simply cut and paste the desired text into the box. For this tutorial any piece of English text should work fine, so please find something and paste it into the box.
Parallel Aggregate
The core of any U-Compare comparison workflow is a Parallel Aggregate Component. As its name suggests this component is comprised of several sub-components, which we wish to compare. These sub-components will be run in parallel when the workflow they are included in is processed. U-Compare collects the results of each of these parallel runs and uses them to generate the component comparison statistics.
The best way to see how this works is probably to look at an example, but before we do this we need to know a couple of things about Parallel Aggregates. Firstly, the way they are able to run multiple components on the same incoming document is by implementing a custom UIMA flow controller. One restriction this places on using them is that all other components in the workflow must also be similarly controlled by this custom flow manager.
The other thing that is different with Parallel Aggregate components is that the ``Outputs'' component parameter field is replaced by an ``Outputs to Compare'' field. And unlike the ``Outputs'' field of other components, the setting of this new field has a direct affect on the components behaviour.
The types added here are the ones for which comparison statistics will be generated, so it is important that this field is set correctly when using Parallel Aggregate components. Keeping these differences from other components in mind, lets start building the rest of the workflow.
Auto-Wrapping
Locate the ``Parallel Aggregate Template'' in Aggregate Component Templates directory of the component library and drag and drop it into the ``Analysis Engines and Cas Consumers'' box. When you do this the following confirmation dialog should appear.
This dialog concerns the need to wrap all other components in a comparison workflow by a U-Compare flow manager that was noted above. Manually wrapping components would be quite frustrating so U-Compare provides automatic component wrapping and unwrapping. Automatic wrapping effects components already existing in the workflow before we add the Parallel Aggregate, as well as any others that we might add later.
The current workflow doesn't include any components in the ``Analysis Engines and Cas Consumers'' so the first of these two cases does not apply. Click ``Ok'' in the dialog box, nothing unexpected should happen and the Parallel Aggregate should appear be added to the workflow.
The tokenizers we are going to add to the aggregate in a moment require Sentence annotations as their inputs which currently aren't generated by our workflow. Adding a sentence splitter just before the Parallel Aggregate will produces these annotations, and give us a chance to see how auto-wrapping works. Drag and drop the ``UIMA Sentence Spilter'' from the Sentence Detectors directory of the component library to a position just before the Parallel Aggregate.
Looking at the workflow viewer you can see that the sentence splitter hasn't simply been added to the workflow, but has first been wrapped by a U-Compare Parallel Aggregate.
The only situation in which users will likely want to unwrap components is if all parallel aggregates have been removed from the workflow. If this situation occurs, U-Compare will prompt users with a dialog asking if they want any remaining components to be unwrapped saving them from having to do it manually.
Configuring the Parallel Aggeragate
We have now seen how auto-wrapping works and have set up all the prerequisites needed for the tokenizer comparison we want to create. All that remains is to setup the Parallel Aggregate itself. To do this begin by opening the edit view of the aggregate by clicking the orange spanner icon. The edit view looks as follows.
As before the edit view begins with a title and description field for the component; try changing the title of the component to ``Tokenizer Comparison''. This is followed by a field for ``Inputs'' and as was mentioned before ``Outputs to Compare'', which we will return to in a minute. What we are interested in at the moment is the space below this that includes the message ``Drag and Drop an Analysis Engine or Cas Consumer here from the Component Library''. It is into this space that we are going to add the components we want to compare.
The three components we are going to add are all located in the Tokenizers directory of the component library, they are UIMA Tokenizer, Open NLP Tokenizer and Genia Tagger with Tokenization. Drag and drop all three of these components into the space at the bottom of the edit view, the order does not matter.
The final thing we need to do to configure the parallel aggregate is to set the ``Outputs to Compare'' field. Click the ``Add Output Type'' button to open the type system viewer which will let us add the required type.
The type system viewer may take some time to load. It has to compute which parts of the type system to display; this calculation is based on the inputs and outputs given for each of the sub-components, so displayed types will vary depending on which components have been included. Even with this calculation to determine which parts of the type system are relevant, the eventually displayed type system can be quite complex.
The type we want to add in this case is Token which is a direct subtype of the SyntacticAnnotation type. The easiest way to find it is to use the search function at the bottom of the the type system viewer window, but be careful as there are several types named Token and we need to find the particular one that is a direct subtype of SyntacticAnnotation.
Once you find this specific Token type, click its name or the green ``+'' symbol next to it to set is as an ``Output to Compare''.
Generally the output to compare that we will want to add is the lowest common super-type of all of the output types of the components we are trying to compare. This is quite easy to get wrong, so if the results generated by a comparison workflow you have created are not correct please check this setting. For more details on the U-Compare type system see the appropriate section of the U-Compare Developers Guide.
Interpreting Results
Once you have added the Token type to the ``Outputs to Compare'', confirm the changes and return to the workflow level view. Once there, run the workflow. This workflow may take slightly longer to complete than previous workflows as it has to process some sections of the workflow multiple times.
Once processing completes the Session Results window should open with the ``Token Comparison'' tab displayed. We have looked at the statistics provided by the other tabs before, we will now look at this tab, which is fairly representative of the statistics produced by comparison workflows.
In the results table each row represents the results of one set of components compared against another set of components which are assumed to be the gold standard. The first set of columns specifies the assumed gold standard and the next set of columns specifies what set of components to compare against it. So for example if we take the combinations of Sentence annotations generated by AIMED reader and Token annotations generated by the OpenNLP as being our gold standard data, then we would be interested in the second and third rows of the table.
The third row shows statistics generated by comparing this assumed gold standard against UIMA Sentence annotations and UIMA tokenizer Token annotations and the fourth row against Token annotations generated by ``Genia with Tokenization'' tokenizer.
Each row gives statistics for each document processed as well as the totals for all documents. The statistics in a given tab only refer to a single type of output annotation as specified by the tabs title. In cases where multiple output types are being compared, these will be spread across multiple tabs. The following statistics are generated for each document:
| G |  ; number of annotations generated by the Gold Standard components. |
| T |  ; number of annotations generated by the comparison components. |
| M |  ; number of annotations that match position exactly between G and T. |
| F1 | F-value score;
 |
| PR | Precision score;  |
| RC | Recall score;  |
Saving Results
Processing workflows can be quite time consuming so for results that you may wish to come back to in future U-Compare provides lets you them. To save the results of the currently open session select ``Save Current Session'' from the ``Sessions'' menu.
To restore saved sessions select the ``Session History..'' item from the same menu, which will open the following dialog.
From this dialog you can open any saved session results, and if necessary can then restore the workflow that produced them from the performance statistics tab.
Saving Components
The creation and configuration of complex aggregate components can be time consuming so if there is a chance that we may want to use a created aggregate component again it can be useful to add it to the component library. We will demonstrate this by adding the Tokenizer Comparison that we just created to the library. To do this begin by returning to the edit view of the Tokenizer Comparison.
From the configuration drop down, select ``set as default'' which will set the current configuration as the default for this Parallel Aggregate.
Now from the configuration drop down, select ``Add to Library''. U-Compare will prompt asking if you want to add this component to the library, click ``OK''. If you look in the component library down the bottom under the Customs Components directory you should now see the Tokenizer Comparison component.
This component, complete with its settings, can now be used simply by dragging and dropping it into any future workflows you create.
Combinatorial Comparison
The example of a Tokenizer comparison workflow given in this section shows how several different components of the same type can be compared against each other, but there are also situations in which users might want to compare different combinations of several different types of components to see which ones work best together.
To do this with U-Compare simply make a workflow with two (or more) parallel aggregate components, each containing a single type of component. If inputs and outputs for these aggeragates have been properly defined, when the workflow is run, statistics for all possible combinations of the components will be generated automatically. U-Compare comes with a predefined example of such a combinatorial comparison. From the workflow menu load and run the U-Compare POS Combinatorial Comparison workflow.
The comparison results may seem quite daunting, but in fact they follow exactly the same pattern as for normal comparison workflows. The only difference is that combinatorial comparison workflow results include more than just a single comparison statistics tab; they include one for each comparison component included in the workflow.
Fixed Flow Aggregates
In this section we made use of a Parallel Aggregate Component which is one of two types of Aggregate Components available in U-Compare. The other is the Fixed-Flow Aggregate which simply runs any sub components in sequence using the standard UIMA Fixed-Flow flow controller. It can be added to workflows in the same way as the Parallel Aggregate by dragging and dropping a ``Fixed-Flow Aggregate Template'' from the Component Library.
The Fixed flow aggregate does not provide comparison statistics or require other components to be wrapped. It is simply included as a way of structuring complex workflows and in combination with the ability to save components, gives users a way of saving commonly used sequences of components for use again in future.
Section Summary
In this section we have seen how to build and configure comparison workflows and interpret the statistics that they produce. Such workflows take a bit more effort to construct than normal UIMA workflows, but we believe it is still the easiest way to produce the sort of comparison statistics that U-Compare makes available, while it is extremely difficult to create comparisons just by what the UIMA framework offers.
We have now covered all of the core functionality of U-Compare, and in the next section will just tidy up a few issues regarding interoperability.
Previous: Editing a workflow Index: User Guide Next: Importing and Exporting
