Type.getHibernateType() and VirtualColumn.getType()

Due to internal changes in Hibernate 3.6 the two methods have been removed from the BASE API. Existing code should be use Type.getTypeWrapper() and VirtualColumn.getTypeWrapper() instead. Since the new methods may not provide the same information as the old methods developers are encouraged to discuss any problems and workarounds with the BASE team on the developer mailing list.

Location.EXTERNAL, File.getSize() and ProgressReporter.display()

We have added support for file items that are stored externally by adding a File.getUrl() method. This is mostly invisible and methods such as File.download() will work as expected. The Location enumeration has a new option, Location.EXTERNAL. This may cause problems with code that doesn't know about it assumes that Location.PRIMARY is the only location when a file can be used. Another issue is that the size for external files is not always known. The File.getSize() method is now allowed to return -1 to indicate that the file size is not known. This may be a problem for code that is unprepared to handle it. For example, many plug-ins rely on the file size for progress reporting. For the same reason ProgressReporter implementations should also accept -1 as a valid value indicating unknown progress.

Type.getHibernateType() and VirtualColumn.getType()

Due to internal changes in Hibernate 3.5.2 the two methods have been deprecated. We expect more changes in the future (Hibernate 3.6) that may force us to remove the two methods from the BASE API. Existing code should be updated to use Type.getTypeWrapper() and VirtualColumn.getTypeWrapper() instead. Since the new methods may not provide the same information as the old methods developers are encouraged to discuss any problems and workarounds with the BASE team on the developer mailing list. Under all circumstances it is important to stop using the deprecated methods.

Date annotations and plug-in parameters

The underlying table used to store date annotation values was actually using a timestamp column type. With default settings this was not an issue since the time part of a date was removed before being saved. However, it was possible to write code or configure BASE so that the time part also was saved. While this worked fine in some cases, it also caused trouble in others (eg. in the Query API and in the table exporter). The BASE 2.16 update changes the underlying database column to a date-only column. All time information that has been stored in it will be lost. The update also introduces the Type.TIMESTAMP data type that can be used with annotations and uses a timestamp column in the database. Most of BASE has been updated to work with the new data type with some exceptions:

For more information see ticket 1512 (Add support for datetime annotation types).

Base1PluginExecuter and enumerated parameters

Enumerated parameters as handled by the Base1PluginExecuter consists of key/value pairs, for example 0=No, 1=Yes. The 'value' is what the user sees in the GUI while running jobs and the 'key' is what is sent to the plug-in in the parameter file. Starting with BASE 2.15 the 'value' is saved in the database instead of the 'key'. At runtime, a reverse lookup is used to convert the 'value' into the 'key' that is written to the parameters file. The reverse lookup only works if all values are unique. BASE will check for this at configuration time and throw an exception if this is not the case. To solve this problem the plug-in should be reconfigured.

ReporterFlatFileImporter store file parser settings as job parameters

This change was introduced since it should be possible to run the plug-in without a configuration. Existing code that uses this plug-in outside the web client, may fail due to not setting required parameter values as job parameters.

Reporters, Raw data, Features and Array design blocks are now proxied

This was previously not possible due to a Hibernate problem with stateless sessions. See http://opensource.atlassian.com/projects/hibernate/browse/HHH-3528 for more information about this problem.

This change means that items linking to reporter, raw data, feature or array design blocks will no longer load the linked items automatically. This is usually not a problem since the proxies will be initialised if needed. The exception is when a stateless session was used to create the proxy since the stateless can't initialise proxies. In BASE, stateless sessions are only used by DataQuery instances, eg. queries that returns reporter, raw data or features. When this type of query is used and when linked items are used in a way that causes proxy initialization the linked item must be explicitely FETCH JOIN-ed by the query. Here is an example:

 
RawBioAssay rba = ...
DataQuery<RawData> rawQuery = rba.getRawData();
rawQuery.join(
  Hql.leftJoin(null, "reporter", Item.REPORTER.getAlias(), null, true));
// NOTE! Last parameter is 'true' to FETCH JOIN the reporter!!
...
DbControl dc = ...
DataResultIterator<RawData> rawData = rawQuery.iterate(dc);
while (rawData.hasNext())
{
  RawData rd = rawData.next();
  ReporterData reporter = rd.getReporter();
  int reporterId = reporter.getId();
  // Always safe since getId() doesn't cause proxy initialization

  reporter.getName();
  // The above statement will fail in BASE 2.13 if 
  // the FETCH JOIN is no included
  ....
}

The error message to look out for is: org.hibernate.SessionException: proxies cannot be fetched by a stateless session

Re-attaching a detached item no longer assumes that it has been modified

The DbControl.reattachItem(BasicItem) method has been deprected and replaced with DbControl.reattachItem(BasicItem, boolean). The boolean parameter is used to tell BASE if the item has been modified or not while it was detached from the session. The previous behaviour of DbControl.reattachItem(BasicItem) was equivalent to DbControl.reattachItem(BasicItem, true), but this is now changed to DbControl.reattachItem(BasicItem, false) since in most cases there are really no changes to the item. The problem with the old behaviour was seen in the new (in BASE 2.13) change history functionality which would create UPDATE events even when there was no change.

Log-2 and log-10 transformed spot intensity data is now allowed

Prior versions of BASE only allowed unlogged spot intensity values. Analysis plug-ins that operate on spot data should be updated to check the kind of values that are present in the source bioassay set and either:

Plug-ins that are not aware of the type of data may produce unexpected results if they are applied on logged data. The core plug-ins that are shipped with BASE has been fixed and they should work with any kind of data. The Base1PluginExecuter that is used for executing BASE 1 plug-ins can be configured to work with only a specific kind of data. After upgrading to BASE 2.12 a server admin should manually update the configuration of all registered BASE 1 plug-ins with information about what kind of source data that is required and what kind of result data the plug-in produces. The default setting is that a plug-in works with any kind of data and produces the same kind of data used as source.

This change also affects formulas, which now has two additional properties: source and result intensity transform. The source intensity transform property tells BASE what kind of source data that the formula can be used with. If this property is not specified the formula can be used with any kind of data. The result intensity transform property tells BASE what kind of result the forumla generates. If this property is not specified the formula is expected to generate the same kind of data as the source data. All existing user-defined forumlas will not have any of the properties set. After upgrading to BASE 2.12 user should check their formulas and set appropriate values for the source and result intensity transform attributes.

The ch() function automatically converts logged intensities to unlogged

In order to maintain as much backwards compatibility as possible the ch() function will automatically convert logged data back to unlogged. This means that many formulas will continue to work unmodified, but some may create unneccesary complex formulas. Consider, for example, the log-ratio formula: log2(ch(1) / ch(2)), which will be converted to: log2(2 ^ rawCh(1) / 2 ^ rawCh(2)) if it is applied on logged values. A better re-write is: rawCh(1) - rawCh(2).

Biomaterial batch importers uses a different coordinate system to target biowells

The batch importers previously used the same coordinate system for locating biowells on a plate that BASE uses internally. A 0-based numerical coordinate pair. This has now been changed to use the more logical alphanumeric 1-based coordinate system typically found on plates. As an example files should now specify A1, B2, C3 instead of [0,0], [1,1] or [2,2]. Files that use the "old" coordinate system must be updated to the new coordinate system, or the imported data will end up in incorrect wells or in no well at all. The change affects three batch importers:

	Note
	It is still possible to use purely numerical coordinates, but they must be 1-based and not 0-based as before!

	Warning
	The new coordinate system only affects the batch importers. The BASE web client will still display the internal 0-based coordinate values. BASE users should be aware of this, particularly if they use the table exporter to generate template files intended for import at a later time. In this case the coordinates in the template file needs to be edited before importing.

Added 'entryDate' property to a lot of items

Including reporters and users. Since those two items are extendable by the server admin this addition may break a custom property with the same name.

Must use a database that supports UTF-8

If you have been running BASE on a database that is not using UTF-8 as the character set you need to convert the database as part of the update. Read Section L.1, “BASE 2.9 must use a database that supports UTF-8” for more information.

Type.BOOLEAN.parseString()

This method now converts the empty string, "no", "0", and "false" (ignoring case) to boolean false. A null value as input still returns null as output. All other values are converted to true. Previously, all values except null and the string "true", was converted to false. The change was made to make this method behave the same as other string conversion methods.

Tomcat 6 or higher is required

If you have been running BASE with Tomcat 5.5 you need to upgrade your Tomcat installation before installing BASE 2.7.

Feature identification methods

Array design features can now be identified by three different methods: COORDINATES, POSITION and FEATURE_ID. The coordinates method was the only one supported earlier.

Data file types

The DataFileType.setValidatorClass() and DataFileType.setMetadataReaderClass() no longer verifies that the specified class names exists and implements the required interfaces. This validation will instead happen at commit time. The reason for this change is the added support for having the validator/meta data reader in external JAR files. This means that the validation can't happen until both the class names and JAR paths has been set. If a client application need validation before commit time it should use DataFileType.getValidator() and DataFileType.getMetadataReader() after the class names and JAR paths has been set.

Job.Status.ABORTING

Added the ABORTING option to the Job.Status enumeration. This option is used when the ABORT signal has been sent to an executing job, but the job has not yet responded to it. Code that uses the job's status to decide what action to take may fail since this is a new option. In most cases, it should be handled in the same way as if the job is EXECUTING.

Hybridization to Labeled extracts link

This link can now hold information about which sub-array a labeled extract belongs to on a multi-array hybridization. Code that is unaware of the concept of sub-arrays may find hybridizations where the number of labeled extracts doesn't match the number channels in the platform used, and that more than one labeled extract has the same label. This was previously considered as an error condition by the experiment validator. With the new scheme the validation has to be done on a sub-array basis instead of on the complete hybridization.

A similar issue arises when inheriting annotations to a raw bioassay which stems from a specific sub-array on a multi-array hybridization. This raw bioassay should only inherit annotations from the labeled extracts that are part of the same sub-array. For API compatibility reasons the Hybridization.getAnnotatableParents() will still return all labeled extracts. Code that is calling this method in order to find the parents to a raw bioassay should instead call the new method, Hybridizations.getAnnotatableParents(int), where the int is the sub-array index value for the raw bioassay.

A related issue arise when querying labeled extracts and joining the source events collection to use the linked hybridizations in the query. Here is an example:

// Find all labeled extracts on a given hybridization
Hybridization current = ...
ItemQuery<LabeledExtract> labeledExtractQuery = LabeledExtract.getQuery();

// This no longer works
// labeledExtractQuery.join(Hql.innerJoin("sourceEvents", "evt"));

// Replace the above line with these two line
labeledExtractQuery.join(Hql.innerJoin("sourceEvents", "srcevt"));
labeledExtractQuery.join(Hql.innerJoin("srcevt", "event", "evt"));

labeledExtractQuery.restrict(
  Restrictions.eq(
    Hql.property("evt", "hybridization"), 
    Expressions.integer(current.getId())
  )
);

The good new is that the modifications makes it possible to filter the query on used quantity and sub-array. See the Javadoc for Hybridization to get more information.

Raw data types

The storage attribute of the <raw-data-type> has been deprecated for the raw-data-types.xml file. BASE will refuse to start if it finds this attribute. Raw data types which doesn't use the database for storing data should be registered as Platform:s instead.

Applications or plug-ins that filters on the rawDataType property for RawBioAssay or Experiment may need to change. The ID given to raw data types that doesn't use the database for storing data are prefixed with "platform.". The ID for the Affymetrix platform has changed from affymetrix to platform.affymetrix.

Raw bioassays

The property spots which used to hold the number of spots in the database OR in the file(s) has been split into two properties:

Applications or plug-ins that filters on the spots property may no longer work as expected for file-only platforms, since this value is now always 0. They should use the numFileSpots property instead.

Array designs

The ArrayDesign.isAffyChip() method has been deprecated. Applications or plug-ins that filter on the affyChip property may no longer work as expected. The applications should instead filter on the platform.externalId and look for the value given by the constant Platform.AFFYMETRIX.

// This may no longer work
query.restrict(
   Restrictions.eq(
      Hql.property("affyChip"), 
      Expressions.parameter("affyChip", true, Type.BOOLEAN)
   )
);

// Use this instead
query.restrict(
   Restrictions.eq(
      Hql.property("platform.externalId"), 
      Expressions.string(Platform.AFFYMETRIX)
   )
);

Plug-in API

The InteractivePlugin.isInContext() method may now throw exceptions to indicate error-level messages. Messages that are returned by the method are considered as a warning message and are by default no longer shown to the users. See the section called “The net.sf.basedb.core.plugin.InteractivePlugin interface” and the section called “The Plugins tab”.

Custom JSP pages for plug-ins

Plug-ins using custom JSP pages for parameter input are recommended to pass along the requestId parameter. The parameter is optional, but if used it will prevent data from two different plug-ins configurations to get mixed up. See Section 26.2.3, “Using custom JSP pages for parameter input”.

JEP parser

The Jep.nodeToExpression() and Jep.nodeToString() methods return NULL if they find an unqouted NULL string in the expression. This allows JEP to convert expressions like ch(1) == NULL into a Query API expression testing for null. To get the old behaviour use ch(1) == 'NULL'.

Parsing strings into numeric values

The Type.parseString(String) method for Type.FLOAT and Type.DOUBLE has changed it's behaviour for NaN and Infinity values. The methods used to return Float.NaN, Float.NEGATIVE_INFINITY, Float.POSITIVE_INFINITY or the corresponding Double values. Since databases doesn't like these special values and eventually most values will go into the database, the parseString method now returns null instead.

Extended properties and raw data types

We have added validation code to check for invalid values. If you have modified the extended-properties.xml or the raw-data-types.xml file and they contain invalid values, you may not be able to start BASE until they are fixed. The validation is rather strict and things that may have worked before (because you were lucky or the because the database has been forgiving) may no longer work. Here is an overview of the most important validation rules:

FlatFileParser

The hasMoreData() method has changed the order of checks made to determine the line type. The checks are now made in the following order:

The data line check has been moved to the last since it was difficult to create settings that made sure section and data footer lines where matched correctly.

BASE 1 Plug-in executer

Changed to store information about plug-in parameters as XML in the database instead of in the original BASE version 1 plug-in definition file. Existing BASE version 1 plug-ins must be re-configured before they can be used. To do this:

BASE 1 Plug-in executer

No longer provides a default mapping between BASE version 1 and BASE version 2 raw data columns. To solve this add a Formula item with the same name as the BASE version 1 column name and an expression that picks the BASE version 2 raw data property. For example: