Subtypes of items

[Nicklas Nordborg]

It could be useful to be able to assign subtypes to biomaterials. The use case is that the lab procedure involves several steps at the same biomaterial major type. Eg. sample --> sample --> extract --> extract --> extract --> labeled extract.

For filtering, batch importing, and other gui purposes it would be useful to assign subtypes for each step.

[Nicklas Nordborg]

Maybe we should extend this to more than biomaterial... I am thinking that the current plan for #1153 could use the same mechanism on PhysicalBioAssay to discriminate between hybridizations and sequence-type bioassays. It could also make the biomaterial type Library not needed since we can use the subtype to discriminate between LabeledExtract and Library.

To provide a kind of semi backwards compatibility the subtype can also be added to DerivedBioAssaySet (so that we can know what was a Scan in BASE 2).

[Nicklas Nordborg]

The idea is to introduce a new interface in the data layer: SubtypableData. This creates an optional link from an item to an ItemSubtypeData instance. The subtype information is more or less only a name and description (N), but we also make this a system item (T) since we need to define subtypes for Hybridization and Scan for backwards compatibility with BASE 2. We may add some more system subtypes in the future.

Each subtype must specify which main item type it can be used on. We also define a unique constraint on the name+itemType combination so that subtypes can be properly identified.

To begin with we add the SubtypeableData interface to MeasuredBioMaterialData and some of the new classes for #1153.

[Nicklas Nordborg]

Added links to ProtocolType, HardwareType and SoftwareType. This should be used to sub-classify protocols/hardware/software that can be used to create the item. Fallback is to the default protocol/hardware/software for the given item type. Example:

The 'Labeling' and 'Library preparation' protocol types are linked from the 'Labeled extract' and 'Library' subtypes so that the protocol selection list when doing action 'Create labeled extract'/'Create library' can be pre-filtered on the proper protocol type. If a subtype doesn't specify a protocol type the default ('Extraction' in this case) filter is used.

The validators in the item overview functionality must be updated to handle this.

[Nicklas Nordborg]

Hmmm... I have been working a bit with the implementation of this and realized that we can maybe do much more...

The first idea is to replace ProtocolType, HardwareType, and some other *Type classes with ItemSubtype. This should make Protocol, Hardware, etc. Subtypable implementations.

But now we have some loose ends... consider, for example, EXTRACT and two subtypes: Library and Labeled extract. A labeled extract is required to have a Label (which is a subtype to TAG), but a library may or may not have a Barcode (which is another subtype to TAG). We do not want to hardcode this into the gui client... And... other subtypes of EXTRACT should not have any TAG at all... So how do know when a related item is needed and when it's not?

My idea is that we need associations between subtypes. For example, the Labeled extract subtype is related to the Label subtype (required) and the Library subtype is related to the Barcode subtype (optional). Repeating the same thing with protocol types: Labeled extract is related to Labeling (a PROTOCOL subtype) and Library is related to the Library preparation protocol subtype.

Basically the 3 associations in the attachment:item-subtypes-2.png​ image is replaced with a many-to-many between ItemSubtype and itself. The association should include a 'required' flag and must be restricted to one association for each main item type. Some combinations doesn't make sense. For example, EXTRACT doesn't link with SOFTWARE, and PROTOCOL, HARDWARE, etc. doesn't link with other subtypes at all.

I'll try to make an updated diagram.

[Nicklas Nordborg]

(In [5630]) References #1597: Subtypes of items

Added Subtypable interface and ItemSubtype item (and corresponding data-layer classes). File, Protocol, Software and Hardware are implementing the new interface and the FileType, ProtocolType, SoftwareType and HardwareType + data-layer classes have been deleted.

The rest of the changes are dependencies that have been updated to use the new subtype functionality.

There is no updgrade script yet. This will need to move existing *Type items into ItemSubtype and update all links from the other items. Then it can remove the old columns and tables. We also need to fix context-related information, but I think we have most functionality for this.

[Nicklas Nordborg]

Note! The implementation so far is like the attachment:item-subtypes.png​ diagram. Relations between subtypes have not yet been implemented.

[Nicklas Nordborg]

(In [5631]) References #1597: Subtypes of items

Added support for subtypes with related subtypes.

[Nicklas Nordborg]

Upgrading this to "critical" since a lot of the new functionality needed for #1153 depend on it.

[Nicklas Nordborg]

(In [5643]) References #1597: Subtypes of items

Cleanup code and JSP pages. Added some utility functions to make it easier to load and use subtypes. Added subtype functionality to BioSource.

[Nicklas Nordborg]

(In [5644]) References #1597: Subtypes of items

Implemented a function that automatically selects the "Annotation category" in the "Annotations" tab in edit dialogs that has a name matching the name of the current subtype for a Subtypable item.

So, by naming annotation categories with the same name as item subtypes a sort of "soft" links are created that may make data input easier when there are many annotation types for the main item type, but only a few are actually used for each subtype.

Note that this doesn't prevent annotations with other annotation types.

[Nicklas Nordborg]

(In [5645]) References #1597: Subtypes of items

Added subtype functionality for sample list, view and edit dialog. Show the inversely related subtypes on the subtype view page.

[Nicklas Nordborg]

(In [5646]) References #1597: Subtypes of items

Added subtype functionality to the biosource importer.

• The "Identification method" section has a new parameter were it is possible to select one or more subtypes to use when searching for existing items. If exactly one subtype is selected in this list new items will automatically be assigned to that subtype. If no subtype or multiple subtypes are selected, the subtypes must be assigned using a column in the imported file.
• Added a 'subtype' column mapping that can be used to assign subtypes to items. If a mapping is given, it always override the subtype used in the identification section.

[Nicklas Nordborg]

(In [5647]) References #1597: Subtypes of items

Added subtype functionality to the sample importer. The importer uses the "related subtypes" feature when looking for protocols, biosources and parent samples. So, if the imported sample has subtype S1 that is related to the protocol subtype P1 the importer will first try to find a protocol with that subtype.

Re-organized code in the abstract item importer to avoid code duplication.

Discovered a flaw in how the cachec system works when used with subtypes. The cache would not consider the subtype setting and could possible return an item with a different subtype.

[Nicklas Nordborg]

(In [5648]) References #1597: Subtypes of items

Added subtype functionality for extracts. Including the web gui and batch importer.

Fixed some minor problems with samples.

[Nicklas Nordborg]

(In [5649]) References #1597: Subtypes of items

Added subtype functionality for physical bioassays. Including the web gui and batch importer.

Replaced the HybridizationImporter with PhysicalBioAssayImporter. The hybridization importer is kept for supporting older servers only and so is the LabeledExtractImporter. They are not installed on new installations.

[Nicklas Nordborg]

(In [5650]) References #1597: Subtypes of items

Allow selection of more default items for a project. Previously only a single item for each subtype could be selected (eg. a single sampling protocol). Now that it is possible to create more subtypes there is a need to be able to select multiple default items. Since the subtype can be changed at any time, there is also no need to restrict the selection to a single item for each subtype.

The project edit dialog has changed so that it is now possible to select multiple platforms, array designs, protocols, software and hardware. The underlying storage has been changed from AnyToAny links to a list of ItemParameterValue (eg. the same mechanism that is used for job parameters).

Some use cases related to the default items have only been fixed to the level that the code compiles. There are more things to fix before everything can be said to work:

• Edit dialogs for subtypable items: The list of default items (eg. protocol) should be dynamically updated when the subtype selection changes. We need to allow multiple default items in the lists.
• Overview validators: subtypes are not validated
• Automatically set default properties for new items: I think this is implemented, but needs to be tested.

[Nicklas Nordborg]

(In [5651]) References #1597: Subtypes of items

Implemented more functionality for the item overview that is related to item subtypes.

• Validation of project default subtypes.
• Validation of related subtypes between items. For example, if we have a sample with subtype S that has a related protocol subtype P then the protocol attached to the sample should use the P subtype (if not an INCORRECT_PROTOCOLTYPE warning is generated). On the other hand, if the sample subtype doesn't have a related protocol subtype, warnings about missing items are not generated.
• Display subtype information in the overview.

There are still a few labeled extract/hybridization validation rules that have not been fixed.

[Nicklas Nordborg]

(In [5666]) References #1597: Subtypes of items

Fixes an issue with the table exporter plug-in and item subtypes. The exporter now supports setting a data type specifically for the exportproperty attribute of a <columndef> tag, when the property to export is different than what is used for filtering.

Eg.

<tbl:columndef
  ...
  property="itemSubtype"
  exportproperty="itemSubtype.name:string"
/>

In this case, filtering is done on the id (INT) column but the name is exported.

[Nicklas Nordborg]

(In [5669]) References #1597: Subtypes of items

Update UML diagrams.

[Nicklas Nordborg]

(In [5686]) References #1597: Subtypes of items

Smarter selection of default subtypes and related items (protocols, etc) when new items are created. The 'recently used items' are remembered per subtype so the list should contain more relevant entries when creating items of different subtypes.

Implemented by the sample edit dialog so far. The current implementation more or less prevents the use of project default items (which also depend on the subtype). This need to be fixed and integrate into the current solution.

[Nicklas Nordborg]

(In [5687]) References #1597: Subtypes of items

Smarter selection of default subtypes and related items (protocols, etc) have now been implemented in the other important edit dialogs.

Added support for also loading and displaying project default items dynamically based on the selected subtype.

[Nicklas Nordborg]

(In [5698]) References #1597: Subtypes of items

Implemented linking of item subtypes and data file types so that we can show/hide the suitable file types when editing an item.

This should more or less complete the item subtype feature. I'll keep the ticket open a little longer in case something pops up.

[Nicklas Nordborg]

(In [5701]) References #1597: Subtypes of items

Better implementation of 'recently used' and 'project default' options in selection lists. Hopefully it is more bug-free than before which had problems causing multiple 'project default' sections to be created and not auto-selecting options in some cases.

[Nicklas Nordborg]

(In [5709]) References #1597: Subtypes of items

Added link to item subtype from bioplate type which makes it possible to restrict/filter on the biomaterial subtype when selecting biomaterial for a plate. The "restriction" is in the gui only and is not checked by the core since it is possible to change the subtype of a biomaterial after it has been placed on a plate.

As a result of this the "Labeled extract reaction plate" is brought back to life again.

Also discovered that the bioplate event wizards should probably be able to utilize subtype information for some improvements. So there are still some things to fix before this ticket can be closed.

[Nicklas Nordborg]

(In [5710]) References #1597: Subtypes of items

The place-on-plate wizard now uses item subtypes when selecting plate, protocol and hardware if all selected items are of the same subtype.

[Nicklas Nordborg]

(In [5712]) References #1597: Subtypes of items

Fixes a NPE when there is no active project.

[Nicklas Nordborg]

(In [5722]) References #1597: Subtypes of items

Update the "Move biomaterial" plate event wizard to use item subtypes when possible.

Display item subtype for biomaterials in tooltip on the single-item page for plates.

[Nicklas Nordborg]

(In [5723]) References #1597: Subtypes of items

Update the "Create child plate" event wizard to use item subtypes when possible.

[Nicklas Nordborg]

The last UML diagram adds the ItemSubtypeFileTypeData which provides a link between item subtypes and data file types. This means that any FileStoreEnabled item that is also Subtypable can select files based on the linking information. The item overview validation also use this information to check that all required files are linked, etc.

I'll close this ticket now since I don't think any major issues remains. Minor issues will be fixed, other enhancements might be scheduled for BASE 3.1 or later.

[Nicklas Nordborg]

(In [5728]) References #1597: Subtypes of items

Added unique constraint on the subtype+datafiletype combination. Seems like the updatedb script is creating duplicate links... needs to be fixed...

[Nicklas Nordborg]

(In [5785]) References #1597: Subtypes of items

Parts of the subtype functionality was missing from the derived bioassay importer.

[Nicklas Nordborg]

(In [5786]) References #1597: Subtypes of items

Only check for WRITE permission if a new item is actually created.

[Nicklas Nordborg]

(In [5814]) References #1597: Subtypes of items

Validation rules for "project default items" should only be used if a default item with a matching type/subtype has been selected. For example, if no default array design has been selected, no warnings should be created for array designs.