Lowess normalization fails on trunk but works on latest 2.7-stable

[Jari Häkkinen]

Running trunk version lowess on my development machine creates the below stack trace whereas 2.7-stable lowess on the same machine works.

Version info: BASE 2.8.0pre (build #4311; schema #61) and BASE 2.7.2pre (build #4354; schema #55)

22:27:16,593 ERROR jobqueue:519 - Exception while executing job: Job[id=128; name=Run plugin: Normalization: Lowess]
net.sf.basedb.core.BaseException: Unknown column 'raw.block' in 'field list'
        at net.sf.basedb.core.InternalJobQueue$JobRunner.run(InternalJobQueue.java:513)
        at java.lang.Thread.run(Thread.java:637)
Caused by: net.sf.basedb.core.BaseException: Unknown column 'raw.block' in 'field list'
        at net.sf.basedb.core.QueryExecutor.getFutureResult(QueryExecutor.java:196)
        at net.sf.basedb.core.QueryExecutor.executeQuery(QueryExecutor.java:93)
        at net.sf.basedb.core.AbstractSqlQuery.iterate(AbstractSqlQuery.java:193)
        at net.sf.basedb.core.DynamicQuery.iterate(DynamicQuery.java:48)
        at net.sf.basedb.plugins.LowessNormalization.normalize(LowessNormalization.java:455)
        at net.sf.basedb.plugins.LowessNormalization.run(LowessNormalization.java:231)
        at net.sf.basedb.core.PluginExecutionRequest.invoke(PluginExecutionRequest.java:108)
        at net.sf.basedb.core.InternalJobQueue$JobRunner.run(InternalJobQueue.java:505)
        ... 1 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'raw.block' in 'field list'
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.Util.getInstance(Util.java:381)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
        at net.sf.basedb.core.QueryExecutor$FutureResultSet.call(QueryExecutor.java:226)
        at net.sf.basedb.core.QueryExecutor$FutureResultSet.call(QueryExecutor.java:214)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        ... 1 more

[Nicklas Nordborg]

I get the same error. Looking at the SQL, it seems like the table containing the raw data is never joined into the query:

SELECT 
  `spt`.`position` AS `position`, 
  `spt`.`ch1` AS `ch1`, 
  `spt`.`ch2` AS `ch2`, 
  `raw`.`block` AS `block`, 
  0 AS `present` 
FROM `base2dynamic`.`D1Spot` `spt` 
WHERE 
  `spt`.`cube` = 2 
  AND `spt`.`layer` = 1 
  AND ((`spt`.`ch1` > 0.0) AND (`spt`.`ch2` > 0.0)) 
  AND (`spt`.`column` = ?) 
ORDER BY 
  `raw`.`block` ASC

This is probably caused by the fix to #1016, which suffered from joining the same table multiple times.

There are other bugs that leads to this one:

1. The DynamicQuery.isReadOnly() method is not implemented correctly. Once the query has been built it is not allowed to modify it by adding more restrictions, joins, etc. unless reset() is called.
2. The Lowess plug-in should not work as it is now. Just after it has created the query, it calls count() on it. After that, more restrictions and other query elements are added. This should never have been allowed, but works because of (1).

[Nicklas Nordborg]

The same problem exists in the Median-ratio normalization plug-in.

[Nicklas Nordborg]

(In [4358]) Fixes #1075: Lowess normaliztion fails on trunk but works on latest 2.7-stable