Store experimental factor values as part experiments

[Nicklas Nordborg]

The current implementation of experimental factor values rely on annotations (primary or inherited) on the raw bioassays. There are some drawbacks with this:

• Bioassays that are part of multiple experiments need to inherit annotations for experimental factors for both experiments. This can impractical since it require WRITE permission on the raw bioassays and it becomes difficult to share data with users that you don't want to give that permission to.

• Changing the annotation values are immediately reflected in the experimental factors. This might be undesirable if the old values have been used in the analysis already.

So, we need a solution that can store experimental factor values as part of the experiment instead of as part of the raw bioassays.

• The values should be copies of the annotations that was used to create them.
• A link to the originating annotation should be stored, but it should not block changing or removing the original annotation.
• It should be possible to manually change the experimental factor values.
• It should be possible to display the current "synchronization" status and re-sync with the original annotation.
• More...?

Downstream tools that currently find experimental factors by loading annotations from the raw bioassays need to be updated.

[Nicklas Nordborg]

The current idea is to replace the Experiment<->Raw bioassay link which is currently a simple many-to-many relation with a new item (RootRawBioAssay?). The item should be Annotatable and the annotation set should be used for storing the experimental factor values. This solves the first problem with inheriting annotations to the raw bioassay since they are now inherited to the root raw bioassay instead.

Cloning annotations need additional support. We need to add rules so that the annotations targeted for other item types can be stored on the root raw bioassay. There is already similar functionality for plug-ins.

Additional functionality is also needed for the link back to the originating annotation. In the suggested design this link need to go from Annotation to the originating Annotation. The drawback with this is that the link is lost if the original annotation is lost and then re-created. It might be better to store this as a "soft" link using type+id to the originating item instead.

This change will introduce incompatibilities in the API and underlying database that affect down-stream analysis tools that use annotations or other information from the raw bioassay level.

[Nicklas Nordborg]

(In [6908]) References #1941: Store experimental factor values as part experiments

Added RootRawBioAssayData class. A database schema update is moving over existing raw bioassays to the new table and then drops the old table.

Some old mappings have been kept and this change doesn't introduce any incompatibility in the public API, but the new class is still only used for the link and not for annotations/experimental factors.

[Nicklas Nordborg]

(In [6909]) References #1941: Store experimental factor values as part experiments

Introduced RootRawBioAssay and use this item for inheriting experimental factors. There is still only inheritance by linking to the original value. Cloning of values remains to be implemented.

Downstream tools such as Experiment explorer, the plot tool, etc. have been updated to use the new items when searching for experimental factor values.

[Nicklas Nordborg]

(In [6910]) References #1941: Store experimental factor values as part experiments

Removing support for inheriting annotation sets. Annotations must now be inherited one by one but utility methods exists that work on all current annotations in an annotation set. But it will no longer be possible to automatically inherit new annotations from an item by inheriting the full annotation set.

The update will transform all current inherited annotation sets to inherited annotations.

The snapshot cache version number was increased to 4 to make sure no old snaphots remains.

[Nicklas Nordborg]

(In [6911]) References #1941: Store experimental factor values as part experiments

Replace test code related to inherting annotation sets with variants that work on with updated API.

[Nicklas Nordborg]

(In [6912]) References #1941: Store experimental factor values as part experiments

Updated code in the web client that worked with inheriting annotation sets to work with individual annotations instead.

[Nicklas Nordborg]

(In [6914]) References #1941: Store experimental factor values as part experiments

Experimental factor annotations for an experiment which are found as either primary or inherited annotations on the raw bioassays in the experiment are inherited to the root raw bioassay items. This means that existing experiments will continue to have access to the experimental factors as before.

[Nicklas Nordborg]

(In [6915]) References #1941: Store experimental factor values as part experiments

Making RootRawBioAssay items implement the Nameable interface. The name is initially set to the same name as the raw bioassay. The intention is that the items should be fully editable as part of the experiment.

[Nicklas Nordborg]

(In [6916]) References #1941: Store experimental factor values as part experiments

Added jsp files for editing and viewing root raw bioassays. They are still prelimiary and changes are expected when support for cloning experimental factor values are introduced.

[Nicklas Nordborg]

(In [6917]) References #1941: Store experimental factor values as part experiments

Started to redesign the AnnotationData class so that it can hold both primary/inherited/cloned annotations. The source column is an enum indicating the source of the annotation 0=primary, 1=inherited, 2=cloned. All annotations should be associated with an annotation set and an annotation type. The primary and cloned annotation have values and possible a unit and last updated date. Inherited annotations and cloned annotations have a link to the original annotation.

If the original annotation is removed, the link is should be nullified (but values are kept) for cloned annotations, but removed by cascade for inherited annotations. This is not yet implemented.

The update script should move existing inherited annotations to the new schema, but much of the existing code in the core is currently not functional and has been commented out while waiting to be fixed.

[Nicklas Nordborg]

(In [6918]) References #1941: Store experimental factor values as part experiments

Started to fix the core API so that the current functionality remains intact as much as possible. Annotating and inheriting annotations should work, except for some utility functions.

[Nicklas Nordborg]

(In [6919]) References #1941: Store experimental factor values as part experiments

Temporary fix for annotation snapshots to make them work as before. Bigger changes are expected to support cloned annotations.

[Nicklas Nordborg]

(In [6920]) References #1941: Store experimental factor values as part experiments

Fixes some more places in the core API that used the old "InheritedAnnotations" table.

[Nicklas Nordborg]

(In [6921]) References #1941: Store experimental factor values as part experiments

Started to implement support for cloning by adding a second boolean parameter to AnnotationSet.inheritAnnotation().

Existing code seems to not be affected much but behaves as if the annotation was inherited by linking.

The TestAnnotation test code has some new test cases that are preliminary so far. Some things may need to change when implementing more realistic scenarios in the web client.

[Nicklas Nordborg]

(In [6922]) References #1941: Store experimental factor values as part experiments

Lots of changes in AnnotationSnapshot to make it possible to use with cloned annotations. Almost all existing methods in this class has been deprecated and replaced with new variants. The deprecate method should behave as before for primary and inherited annotations. Cloned annotations will get some information from the inherited annotation and some from the cloned information. This might confuse existing code that is not prepared to handle cloned annotations.

The "Annotations" tab in the web gui has been updated to display the correct information but it is not yet possible to edit or otherwise manage cloned annotations.

[Nicklas Nordborg]

(In [6923]) References #1941: Store experimental factor values as part experiments

Fixes the query API so that it treats cloned annoations as inherited annotations and not as primary annotations.

[Nicklas Nordborg]

(In [6924]) References #1941: Store experimental factor values as part experiments

Some places that use inherited annotations have been updated to use the new API methods:

• Table exporter
• Batch inherit annotations
• BioAssaySetUtil (used a lot in the analysis)

[Nicklas Nordborg]

(In [6925]) References #1941: Store experimental factor values as part experiments

Updated the batch inherit annotations feature to support cloning.

[Nicklas Nordborg]

(In [6926]) References #1941: Store experimental factor values as part experiments

Updated the batch inherit annotations feature with support for re-syncing cloned annotations with their parent annotations.

[Nicklas Nordborg]

(In [6937]) References #1941: Store experimental factor values as part experiments

Removing the annotation on a parent item will nullify the link from cloned annotations instead of removing the clones.

Other changes are related to fixing various NullPointerExceptions and other problems resulting from having "inherited" annotations without a parent item.

But more work is needed on this since the cloned annotations now end up in a state that is almost impossible to manage via the gui. The "batch inherit annotations" can be used to remove or update cloned annotation.

[Nicklas Nordborg]

(In [6938]) References #1941: Store experimental factor values as part experiments

Started with redesigning the annotations edit dialog. The idea is to include both primary and inherited/cloned annotations in the "Annotations" tab and manage all types of annotations there. The "Inherited annotations" tab will be removed.

So far, the new implementation will list all types of annotations. PRIMARY and CLONED annotations are editable, INHERITED annotations only display some information. Support for multi-valued annotations is not implemeted.

[Nicklas Nordborg]

(In [6939]) References #1941: Store experimental factor values as part experiments

Updated "Item overview" functionality to become compatible with cloned annotations. There may still be some issues with suggested fixes related to inherited annotations since it is not yet clear how those will be managed.

[Nicklas Nordborg]

(In [6940]) References #1941: Store experimental factor values as part experiments

The AnnotationSet.copyFrom() method should now be working again.

[Nicklas Nordborg]

(In [6942]) References #1941: Store experimental factor values as part experiments

Updated lists that display experimental factors so that they are able to handle inherited/cloned annotations correctly.

[Nicklas Nordborg]

(In [6943]) References #1941: Store experimental factor values as part experiments

Adding support for synchronizing a cloned annotation that has been modified, and for converting between cloned/inherited annotation.

[Nicklas Nordborg]

(In [6944]) References #1941: Store experimental factor values as part experiments

Fixes an issue with the annotations tab being reloaded in some cases when moving back and forth between tabs in an edit dialog. This could cause modified annotation values be reset to their original values.

[Nicklas Nordborg]

(In [6945]) References #1941: Store experimental factor values as part experiments

Implemented support for syncing/cloning/deleting annotations in batch by selecting multiple annotations in one go.

New icons for inherited annotations depending on inherited/cloned/sync status.

[Nicklas Nordborg]

(In [6946]) References #1941: Store experimental factor values as part experiments

Adding 'check all' icons to the annotations list and put some operations in a submenu to save space.

[Nicklas Nordborg]

(In [6947]) References #1941: Store experimental factor values as part experiments

Implemented support for inheriting new annotations via the "Inherit" button. The 'Inherited annotations' tab has been removed from all dialogs.

[Nicklas Nordborg]

(In [6955]) References #1941: Store experimental factor values as part experiments

Implemented support for multi-valued annotations.

[Nicklas Nordborg]

(In [6958]) References #1941: Store experimental factor values as part experiments

Updated test program to set experimental factor values on root raw bioassays instead of raw bioassays.

Noted that the "Item overview validation" now complains about missing experimental factors, probably because it has not been updated and still check the raw bioassays and not the root raw bioassays.

[Nicklas Nordborg]

(In [6959]) References #1941: Store experimental factor values as part experiments

The "Item overview" functionality now load "root raw bioassays" instead of "raw bioassays" when going up from the experiment level. This change is needed so that experimental factor values can be found. The loading then skips the actual raw bioassay item. This change causes lots of other loaders/validators to fail since the are expecting or looking for a raw bioassay. Those have been updated so that they can work with either a root raw bioassay or a raw bioassay.

[Nicklas Nordborg]

(In [6960]) References #1941: Store experimental factor values as part experiments

Added validation rule to "Item overview" that check if a cloned annotation is in sync or not.

[Nicklas Nordborg]

(In [6961]) References #1941: Store experimental factor values as part experiments

Updated documentation and screen shots related to the new functionality.

[Nicklas Nordborg]

(In [6962]) References #1941: Store experimental factor values as part experiments

Using a different "Edit" icon for cloned annotations that are out-of-sync.

[Nicklas Nordborg]

(In [6964]) References #1941: Store experimental factor values as part experiments

Adding notes about incompatible API changes.

[Nicklas Nordborg]

(In [6968]) References #1941: Store experimental factor values as part experiments

The annotation logger implementation did not handle inherited annotations correctly when logging of old property valus had been enabled (NullPointerException when trying to get the values).

[Nicklas Nordborg]

(In [6969]) References #1941: Store experimental factor values as part experiments

Fixes a NullPointerException in array design validator when the item overview was rooted on an array design (parentNode==null).

[Nicklas Nordborg]

(In [6974]) References #1941: Store experimental factor values as part experiments

The current unit was not selected in IE, since the selected status is set by 4th parameter instead of 3rd (both works in Firefox).

[Nicklas Nordborg]

(In [6975]) References #1941: Store experimental factor values as part experiments

Added 'Clone' button in the 'Inherit annotations' dialog to make it possible to clone annotations in a single step without having to re-select and clone in a second step.

[Nicklas Nordborg]

(In [6976]) References #1941: Store experimental factor values as part experiments

When updating a MySQL installation it is not possible to drop indexes without also dropping foreign keys that use the same columns. After dropping the index, the foreign keys need to be re-created. In schema version 124 we need to drop the unique index on annotationset_id+annotationtype_id in the Annotations table since cloning annotations means that there can be multiple annotations for the same annotation type.

When updating a MySQL installation it is not possible to drop a NOT NULL constraint on a column. Instead the column definition must be modified as if a new column was created without a NOT NULL constraint. This requires that we know the data type of the column. Since the SQL to do this is so different from how PostgreSQL does this, we put this in the mysql-queries.xml file.