Lists of items (similar funcion as biomaterial lists)

[Jari Häkkinen]

Starting to work with biomaterials and biomaterial lists makes us realize that it would be nice to extend the list creation functionality down to raw bioassays. We would like to be able to filter say extracts and the get a list of all raw bioassays associated to this list. (Of course adding this functionality could include hyb and scan lists but it is most urgent for raw bioassays)

[Nicklas Nordborg]

If we are going to do this it should be a generic 'list-of-items' functionality that also mergers with 'list-of-biomaterials' functionality. The current 'list-of-biomaterials' functionality can't be used except for copy-and-paste and we don't want to have lots of code that is similar but slightly different.

[Johan Vallon-Christersson]

Replying to nicklas:

If we are going to do this it should be a generic 'list-of-items' functionality that also mergers with 'list-of-biomaterials' functionality. The current 'list-of-biomaterials' functionality can't be used except for copy-and-paste and we don't want to have lots of code that is similar but slightly different.

If this can be made generic so that lists can handle different items it would be most useful. For a generic 'list-of-items' functionality the important core of the functionality would be to create item lists and to be able to create “child” and “parent” item lists.

One user case (basically the ticket title):

• 1. View a table listing of biomaterials, e.g., displaying five samples.
• 2. Create an item list containing Rawbioassays derived from the listed samples, i.e., those Rawbioassays that are associated with the 5 listed samples.

To be able to handle items from Biosource to Rawbioassay (Biosource, Sample, Extract, Labeled extract, Hybridization, Scan, Rawbioassay) would last a long way.

If the 'list-of-items' functionality could be extended to handle additional items this would be useful as well.

Conceivable user cases involving array designs:

• 1a. From array design list view, list array design X
• 1b. Create item list of hybridizations, i.e., hybridizations that are associated to array slides of array design X

• 2a. From hybridization list view, list a number of hybridizations
• 2b. Create item list of array designs, i.e., array designs for array slides associated with the listed hybridizations.

[Nicklas Nordborg]

(In [6738]) References #1325: Lists of items (similar funcion as biomaterial lists)

Added ItemList as a new item type for holding lists of any type of items. In principle, the underlying implementation can store any type of items, but since we need to support this in the GUI as well, the Listable interface was introduced as a tagging interface for items that we want to put in lists. The initial items type supporting this:

• Biomaterial (Biosource, Sample, Extract)
• Bioassays (Physical, Derived and Raw)
• Files

Adding support for more item types is not difficult, but requires some work in the GUI.

I don't know yet how to handle the "old" BioMaterialList. There is a lot of existing functionality when it comes to working with biomaterial lists and creating lists with parent or child items.

The new funcationality can replace this and it would be relatively easy to transfer existing biomaterial lists to item lists, but API-wise it would introduce a lot of binary incompatibilities. But keeping the biomaterial lists as a separate functionality is not a very good idea either, so maybe it's worth to break the binary compatibility in this case...

[Nicklas Nordborg]

(In [6739]) References #1325: Lists of items (similar funcion as biomaterial lists)

Added test program for item lists.

[Nicklas Nordborg]

(In [6740]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding Namable as a superinterface to Listable. It will make life easier if we can assume that all items in a list has a name and description.

[Nicklas Nordborg]

(In [6741]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding functionality to keep item lists up to date when deleting items from the database. Since this should be possible we must check/update two things:

• The size of item lists must be decreased if an item in the list is deleted
• The entry in the ItemListMembers table must be deleted

This is done in BasicItem.onBeforeCommit() for all Listable items.

[Nicklas Nordborg]

(In [6742]) References #1325: Lists of items (similar funcion as biomaterial lists)

Started with GUI for working with item lists. Current functionality is limited and only includes listing, creating/editing and viewing the "ItemList" item. It is not possible to add/view members to the list, merge lists or use the lists for filtering.

[Nicklas Nordborg]

(In [6743]) References #1325: Lists of items (similar funcion as biomaterial lists)

Add list page for members of the item list. It can currently only display a limited set of columns (eg. id, name and description).

[Nicklas Nordborg]

(In [6744]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding functionality for manually adding/removing members to a list.

[Nicklas Nordborg]

(In [6745]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding columns and other functionality for removable, ownable and shareable member items.

[Nicklas Nordborg]

(In [6746]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding annotations columns if the member items are annotatable.

[Nicklas Nordborg]

(In [6747]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding columns for subtypable items and registered items (=items that have an automatic registration date recorded). Note that most items use the same underlying database proprety execept samples and extracts which use the date from the linked creation event item.

[Nicklas Nordborg]

(In [6748]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding extension points for gui (toolbar, etc.) related to item lists.

[Nicklas Nordborg]

(In [6749]) References #1325: Lists of items (similar funcion as biomaterial lists)

Added support for filtering on membership in item lists. Used samples list view for testing. Property prefix to use is '§' (actual property name is not currently be used but recommended to be 'itemLists').

[Nicklas Nordborg]

(In [6750]) References #1325: Lists of items (similar funcion as biomaterial lists)

Added functionality for creating a new item list from the regular list view. The samples list view has been used for testing.

Note that there is no support for walking up the parent/child chain here any longer since it felt like a confusing way to do things.

The new idea is that lists should be created at various levels and then use the "union/intersection" functionality to create a final list by checking for members using the entire parent/child chain.

[Nicklas Nordborg]

(In [6751]) References #1325: Lists of items (similar funcion as biomaterial lists)

Added item list -> itemsubtype/rawdatatype link which is intended to be used as a recommendation for the types of items to be members of the list. The raw data type is used for raw bioassay lists and the item subtype for other lists.

[Nicklas Nordborg]

(In [6752]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding check for item lists using a subtype to keep trashcan functionality is up to date.

[Nicklas Nordborg]

(In [6753]) References #1325: Lists of items (similar funcion as biomaterial lists)

Adding subtype and raw data type to list page, view page and edit dialog. If this is set the information is used to filter the "Add members" list.

[Nicklas Nordborg]

(In [6754]) References #1325: Lists of items (similar funcion as biomaterial lists)

Options in selection lists with html tags are now handled betted. The html is removed when displaying the filter summary and click event handler now use the correct target element.

[Nicklas Nordborg]

(In [6755]) References #1325: Lists of items (similar funcion as biomaterial lists)

Deprecated several classes related to biomaterial lists to prepare for it's removal.

Implemented support for item lists in the 'Item overview' feature.

[Nicklas Nordborg]

(In [6756]) References #1325: Lists of items (similar funcion as biomaterial lists)

Removed some functionality related to biomaterial lists and replaced that with item lists:

• Filter on list pages for biosources, samples, extracts and biowells
• Create biomaterial list button on the same list pages
• Removed biomaterial lists from "Item overview" completely.

Installation now create Item.ITEMLIST using the same code as Item.BIOMATERIALLIST which should make it a lot easier when upgrading since we don't have to remap permissions, annotation types, any-to-any-links, stored filters etc. that use the code in order to work.

[Nicklas Nordborg]

(In [6757]) References #1325: Lists of items (similar funcion as biomaterial lists)

Added update script for moving existing biomaterial lists to item lists. ID values are re-used. The old database tables are removed and property filters on biomaterial lists are updated to filter on item lists instead.

[Nicklas Nordborg]

(In [6758]) References #1325: Lists of items (similar funcion as biomaterial lists)

Removing BioMaterialList class. Deprecating classes in package 'net.sf.basedb.util.biomaterial' since I think they need to be replaced with a better solution for walking up/down parent/child relationships.

[Nicklas Nordborg]

(In [6759]) References #1325: Lists of items (similar funcion as biomaterial lists)

Remove Biomaterial lists from the gui.

[Nicklas Nordborg]

(In [6760]) References #1325: Lists of items (similar funcion as biomaterial lists)

Make sure that old *hbm.xml files are not included when building.

Installation should update name/description of role keys (so that the key for item lists is renamed from biomaterial lists).

[Nicklas Nordborg]

(In [6768]) References #1325: Lists of items

Added functionality for storing the active filter used to create a list. The functionality for re-syncing the list with the filter at a later time is still rudimentary but will be improved...

[Nicklas Nordborg]

(In [6769]) References #1325: Lists of items

Improved code for printing out the synchronization filter on the information page. Instead of jus using the PropertyFilter.toString() method a PropertyFilterFormatter is used. This makes some cosmetic changes as well as loading names of annotation types, etc. which are used instead of the ID values.

[Nicklas Nordborg]

(In [6770]) References #1325: Lists of items

Re-factored the syn with filter implementation. Each item list can now have multiple "Synchronization filters". This is now it's own item (Item.SYNCFILTER) which is a child item to item an item list.

Each filter have settings for the relation to the type of items it filters on (eg. parent or child items).

The ListableChildItems and ListableParentItems annotations was added to make it possible to annotate a class which parent and child item types it can navigate to.

The GUI has been updated to make it possible to create and specify filters for a list, but all synchonrization code has been temporarily removed since it no longer works.

[Nicklas Nordborg]

(In [6771]) References #1325: Lists of items

Added Re-sync functionality to the web client and started to implement this in the core. Only items on the same level are current synced. Parent/child syncing has not been implemented, except that a count is made to find the number of matching items.

[Nicklas Nordborg]

(In [6772]) References #1325: Lists of items

Added functionality for keeping track of the sync status of a single SyncFilter. Each time a synchronization is made, an MD5 value is calculated from the ID:s of items matching a filter. If the number of ID and their MD5 value doesn't change the filter is assumed to be in sync. Note that parent/child relationships are not taken into account so the even if a filter appears to be in sync it may result in a different child/parent list.

It could also be a performance hog to load this information every time on the view page, so it may have to be changed to lazy-loading in the future.

[Nicklas Nordborg]

(In [6773]) References #1325: Lists of items

Added "Sync date" column to list view.

[Nicklas Nordborg]

(In [6774]) References #1325: Lists of items

Started to implement parent/child loading of synchronization filters.

The foundation is two new interfaces SourceItemTransformerFactory and SourceItemTransformer.

The factory implementations should target a specific end item type (eg. the member item type of an item list). Each factory implementation has a list of supported source items and should be able to create transformers that takes a list of source items and transforms them to the target item type.

For example, the transformer factory for SAMPLE should be able to create transformers for transforming from other items types to the SAMPLE items.

The new interfaces also replaces the ListableParentItems and ListableChildItems annotations that was used to specify supported parent/child transformations. They will be removed as soon as a few more factories are in place.

To begin with only the transformations between BIOSOURCE->SAMPLE and SAMPLE->SAMPLE has been implemented.

[Nicklas Nordborg]

(In [6775]) References #1325: Lists of items

Added transformer factories for biosource and extract levels.

Added transformer implementations for Extrac<->Extract, Extract<->Sample, Sample->Sample (parent) Sample->BioSource

This should make it possible to implement all functionality that existed with th old biomaterial lists. The test cases has been updated to use the new code but the old implementation is still used by the annotation inheritance loading and the item overview.

[Nicklas Nordborg]

(In [6776]) References #1325: Lists of items

Remove action when deleting biomaterial that tried to update size of "BioMaterialLists" which failed since those lists no longer exists. The corresponding functionality for item lists is already implemented in BasicItem.

[Nicklas Nordborg]

(In [6777]) References #1325: Lists of items

Added stub source item transformer factories for the remaining item types (Physical bioassay, derived bioassay and raw bioassay). Currently they only return information about support parent/child item types.

This means that the ListableParentItems and ListableChildItems could be removed.

[Nicklas Nordborg]

(In [6785]) References #1325: Lists of items

Handling lack of write permission better. Added link to modify a filter directly on the view page without having to go through the filter edit dialog.

[Nicklas Nordborg]

(In [6786]) References #1325: Lists of items

Added one option that disable manual modification of members and one option that disable sync filter modification of the list. Setting both to disabled will effectively lock the list for editing.

[Nicklas Nordborg]

(In [6787]) References #1325: Lists of items

Added option to a sync filter that allows matching parent/child items that either must exist or must not exist.

[Nicklas Nordborg]

(In [6788]) References #1325: Lists of items

Initializing syncfilters to the current context settings. This avoids having a starting filter list that doesn't match any items due to all 'include' options in the 'view/preset' dropdown being unchecked.

[Nicklas Nordborg]

(In [6789]) References #1325: Lists of items

Schema update must set 'disable_syncfilters' and 'disable_manual' to false when transferring existing biomaterial lists.

[Nicklas Nordborg]

(In [6790]) References #1325: Lists of items

Adding transformers to/from the physical bioassay and derived bioassay level. Currently implemented as "full" child/parent only. Eg. there is no limitation on parent extracts or child derived bioassays when passing through the physical bioassay level.

[Nicklas Nordborg]

(In [6791]) References #1325: Lists of items

Adding support for keeping track of extracts that we should treat as parents when we pass through the physical bioassay level down and up.

Basically, the CollectExtracts class is loading and storing all extracts that we see on the derived and raw bioassay level. Then, when we move upwards from the physical bioassay it will match the loaded parent extracts with those seen earlier.

All transformers are now expected to work in the child->parent direction.

[Nicklas Nordborg]

(In [6792]) References #1325: Lists of items

Adding final transformation steps in the parent->child direction on the derived bioassay and raw bioassay level. The physical bioassay->derived bioassay transformation should now be able to keep track of parent extracts if the start level was from extract or higher.

[Nicklas Nordborg]

(In [6793]) References #1325: Lists of items

Improve sorting of filters.

[Nicklas Nordborg]

(In [6794]) References #1325: Lists of items

Added feature for disabling filters. This should make it easier to test different filter settings.

[Nicklas Nordborg]

(In [6795]) References #1325: Lists of items

Displaying a progress bar when syncing list with filters.

[Nicklas Nordborg]

(In [6796]) References #1325: Lists of items

Add support for filters that result in lots of matching items (eg. larger than the number of parameters allowed in a query).

[Nicklas Nordborg]

(In [6797]) References #1325: Lists of items

Adding column for storing the number of matched transformed items during the last sync. Updated the filter list to also display this information.

[Nicklas Nordborg]

(In [6798]) References #1325: Lists of items

Enabling "edit" extension point for sync filters. Removing merge functionality that was never implemented.

[Nicklas Nordborg]

(In [6799]) References #1325: Lists of items

Replaced all uses of the old biomaterial list transformers with the new implementations. Added some more information to the javadoc about replacements to use instead of deprecated classes/methods.

[Nicklas Nordborg]

(In [6800]) References #1325: Lists of items

Disabled filters should not affect the 'in sync' status.

[Nicklas Nordborg]

(In [6801]) References #1325: Lists of items

Adding support for loading parent/child relations that goes only between raw bioassay/derived bioassay <-> extract without going through a physical bioassay.

This is basically a union between items loaded from the two paths.

[Nicklas Nordborg]

(In [6802]) References #1325: Lists of items

Send notification for reloading to view page after syncing is complete.

[Nicklas Nordborg]

(In [6803]) References #1325: Lists of items

Adding functionality for cloning an existing item list. The clone will contain the same member items and all sync filters from the original list.

[Nicklas Nordborg]

(In [6804]) References #1325: Lists of items

Build failed on fresh checkout due to trying to delete from non-existing directory (see also [6760]).

[Nicklas Nordborg]

(In [6807]) References #1924, #1927 and #1325. Added information about incompatible changes and things to consider when updgrading to BASE 3.5.

[Nicklas Nordborg]

(In [6808]) References #1325: Lists of items

Typo.

[Nicklas Nordborg]

(In [6811]) References #1325: Lists of items

Added documentation about item lists.

[Nicklas Nordborg]

(In [6813]) References #1325: Lists of items

Added "Create item list" buttons on all bioassays lists pages.

[Nicklas Nordborg]

(In [6814]) References #1325: Lists of items

Sort order of sync filters updated to sort by strict parent->child order before sorting by name.

[Nicklas Nordborg]

(In [6815]) References #1325: Lists of items

Adding "Item list" column to all bioassay list pages and the members list page for item lists.

[Nicklas Nordborg]

(In [6816]) References #1325: Lists of items

Move "Re-sync" button to first place in toolbar. Renaming "In sync" column to "In sync/enabled".

[Nicklas Nordborg]

(In [6817]) References #1325: Lists of items

Adding a link to set a sync filter as the current filter for a list page and then go to that list page.

[Nicklas Nordborg]

(In [6819]) References #1325: Lists of items

Updating test code since FILE is no longer possible to use in item lists.

[Nicklas Nordborg]

(In [6840]) References #1325: Lists of items

Added 'select filter' mode for popups. The major difference is that the title says 'Select filter' instead of 'Select one item'.

[Nicklas Nordborg]

(In [6841]) References #1325 and #1933. Synchronization filter that filters on data file types are now formatted with the name of the file type instead of just the id.

[Nicklas Nordborg]

(In [6845]) References #1325: Lists of items

Move subtype filtering to be the first filter and use this to limit the transformed transformed parent/child items before reporting the matching numbers. The end result is identical, but the numbers makes more sense in from a user perspective. For example, when creating a "specimen" item list it seems strange that a filter on the "RNAQC" child level that matches a few hundred items suddenly matches several thousands of parent items. This happened beacuse the subtype filter was not applied until after all other filters and all samples in the parent/child chain that links to the "RNAQC" level was selected by the transformation.

[Nicklas Nordborg]

(In [6846]) References #1325: Lists of items

Moving 100% progress report outside the resync method since the transaction has not yet been committed and this may take some time. The problem is that the progress dialog will close automatically when 100% is reached but the worker thread may still be working with the commit and actual database update.

The synchronization now replaces the 'members' set with a new set which is considerably faster (Hibernate is able to make better optimizations) if there are big changes in the set.

[Nicklas Nordborg]

(In [6847]) References #1325: Lists of items

Need to create copy of matching subtypes since otherwise it will be modified as a side-effect when merging other filters.

[Nicklas Nordborg]

(In [6848]) References #1325: Lists of items

Use the configured Include options when finding child/parent items instead of Include.ALL.

[Nicklas Nordborg]

(In [6855]) References #1325: Lists of items

Use 'mode' parameter when selecting items.

[Nicklas Nordborg]

(In [6871]) References #1325: Lists of items

Adding @Deprecated annotation to methods only marked as deprecated with javadoc.

[Nicklas Nordborg]

(In [6876]) References #1325. Fixed compiler warning.