Project-specific annotations

[Nicklas Nordborg]

This is a new feature that allows an annotation type to be configured as "project-specific". The main idea here is that annotations on items can be entered on a per-project basis. Switching to another project would make it possible to enter a different value that is only visible in that project.

At the core level this must be implemented in a way that doesn't upset existing client code (web client/extensions/plugins).

1. This feature must be enabled per annotation type. Existing annotation types will have it disabled by default. Once this feature has been enabled it will not be possible to disabled it unless all existing project-specific annotations are first deleted.
2. The current core API should be usable and behave more or less in a backwards compatible way for annotation types that have the option enabled. Exception: The existing API is not enough to be able to create a project-specific annotation when a default value already exists.
3. Existing annotation values doesn't belong to any project and should be treated as default values if the project-specific setting is enabled for an annotation type.
4. When a users is working without an active project only the default values are visible/editable for annotations that are project-specific.
5. If an annotation has a default value but not any project-specific value when a project is active, the default value is visible and should be treated as if it was a project-specific value.
6. Default values are not editable (unless maybe via new API methods) when a project is active. Trying to change a default value results in a project-specific value being created instead. The default value is then no longer visible and should not match queries either.

The core API needs to be updated for this behavior. The existing API will modified the default annotation values even when a project is active.

7. Deleting a project-specific value makes the default value visible again.
8. Deleting a default value can only be done when no project is active (unless maybe via new API methods).
9. It is important that queries are working in a sensible way. When no project is active only default values can be searched. When a project is active both project-specific and default values are searched but default values that have been replaced with a project specific value should be ignored.
10. If a default value exists it is not possible to "unset" the annotation in a project, only to change it to a different value.
11. Inherited annotations always belong to the same project as the parent annotation. No exceptions! This means that inheriting a default value always create a default inherited value even if a project is active. If a project-specific value is later created on the parent item it will not automatically be inherited. This may cause some confusion since the child item may display one value (the default) but navigating to the parent item will display the project-specific value.
12. Inheriting a project-specific value should override an earlier inherited default value.

The caching functionality used for speeding up annotation retrieval must be updated to follow the same rules as when using the regular API.

The batch functionality for speeding up annotation changes must also be updated.

Change history logging might be affected as well. Should changes be visible across projects?

Some implementation details The approach taken to implement this feature is to add two columns to the Annotations table.

• project_id: The id of the project the annotation belongs to or 0 for default values
• override_id: The id of the annotation that contains the default value. 0 if this annotation is a default value or if there is no default value.

When searching for annotations we can introduce filters in HQL/SQL statements:

• where ... project_id = <pid> to find only project-specific annotations (or 0 to only find default values).
• where ... project_id = <pid> OR (project_id = 0 and id not in (select override_id from "Annotations" where ....)) to find project-specific annotations and default values that are not overidden. The subquery may have a different where clause depending on what we are looking for (for example, all annotation in an annotation set or all annotations of a specific annotation type).

The project_id column is easy to handle since we always know if a project is active or not. The override_id column is more complicated since annotations may be created/updated/removed in any order. For example, creating a default value on an item that already has project-specific values must update the override_id column on all project-specific annotations to point to the newly created default value.

Inherited annotations are even more complex since we must go up and check the parent annotations in order to know what to set for the override_id column.

More....

[Nicklas Nordborg]

(In [7244]) References #2049: Project-specific annotations

Implemented a flag option for annotation types that enabled project-specific annotations.

[Nicklas Nordborg]

(In [7245]) References #2049: Project-specific annotations

Implemented a test case designed for testing project-specific annotations. The test will create one project-specific annotation type and two projects.

• An array design is created and a default annotation value is set
• A project-specific annotation is set for one of the projects

Tests

• Check that the expected value is returned by the AnnotationSet API.
• Check that the expected value is returned by the SnapshotManager API.
• Check that a query looking for the expected value is returning the correct item.

Of course, the tests currently fail.

[Nicklas Nordborg]

(In [7246]) References #2049: Project-specific annotations

The test must also check that searching for a value that has been changed by a project-specific annotation doesn't match default values.

[Nicklas Nordborg]

(In [7247]) References #2049: Project-specific annotations

Changes the order of the tests so that the query tests are made before the snapshot manager tests.

[Nicklas Nordborg]

(In [7248]) References #2049: Project-specific annotations

Added a column override_id to keep a reference back to the default value for project-specific annotations that override a default value. Both this and the project_id column has been changed to NOT NULL since it will be easier to handle 0 instead of null when comparing values. Both columns are also update=false. The project id will never change. The override id may change, but that is going to be handled by batch SQL updates.

[Nicklas Nordborg]

(In [7249]) References #2049: Project-specific annotations

Started to implement this in the core. It is now possible to create a project-specific annotation. Currently it only works if there is no default annotation already. However, it is possible to create a default annotation if it is done after the project-specific annotation.

The snapshot manager and queries do not understand this yet so they may experience unexpected behaviour.

[Nicklas Nordborg]

(In [7250]) References #2049: Project-specific annotations

The snapshot manager should now be able to find the correct annotation values. It is still a bit difficult to test since the web interface also depend on queries which are currently not returning correct results.

[Nicklas Nordborg]

(In [7251]) References #2049: Project-specific annotations

The AnnotationSet.getAnnotations() method should now work with project-specific annotations. It is mainly used in some places when editing annotations for a single item.

[Nicklas Nordborg]

(In [7252]) References #2049: Project-specific annotations

AnnotationRestriction and AnnotationJoin has now been updated to work with project-specific annotations. Both are used in queries generated by list pages when searching.

The join support to handle default vs. project-specific overrides was a bit complicated since regular HQL doesn't support subquires inside the WITH statement. The workaround was to invent a way to inject native SQL into HQL queries by defining the native() function that is implemented by NativeSQLFunction class.

[Nicklas Nordborg]

(In [7253]) References #2049: Project-specific annotations

Removing deprecated functionality that is too difficult to fix.

[Nicklas Nordborg]

(In [7254]) References #2049: Project-specific annotations

Started to work on inherited annotations that are project-specific. It seems to work so far as long as project-specific annotations are inherited before default annotations (the same as for primary annotations).

[Nicklas Nordborg]

(In [7255]) References #2049: Project-specific annotations

The code for keeping the "override_id" column i sync with everything was getting out of hand. A new approach collects all of this into ProjectSpecificAnnotationsManager which will do all of it's work just before the transaction is committed. All other annotationd data should already have sent to the database by Hibernate.

[Nicklas Nordborg]

(In [7256]) References #2049: Project-specific annotations

If a project is deleted all project-specific annotations for that project are also deleted. This currently happens in the main transaction but in a big project with lots of annotations this might not be a good idea since it can take a long time. It might be better to just ignore that the project-specific annotations are there and then make regular cleanups instead.

[Nicklas Nordborg]

(In [7257]) References #2049: Project-specific annotations

Added a check to block setting projecAnnotations=false if there are existing project-specific annotations for an annotation type. The only other option is to delete the annotations but that feels a bit too complicated.

[Nicklas Nordborg]

(In [7258]) References #2049: Project-specific annotations

Updated the API so that it is possible to create a project-specific annotation when a default value already exists. Unfortunately it was not possible to do this in a fully backwards compatible way. This means that existing code (clients, extensions, etc.) that uses the core API must be modified to be able to create project-specific annotations when a default value already exists. If not change is made, the old code will simply modify the existing default value instead.

If not default value exists or if a project-specific annotation already exists, the existing code will work as expected.