BASE 1 lowess plugin description too long for BASE 2 migrate tool

[base]

I've patched the migrate tools to handle a customized Reporter table, but when I run it, it falls over before it gets to the Reporters. It's something to do with the description of the Lowess plugin being too long. I've tried to delve into the code but I'm not getting very far, sorry! It seems that the description is plenty small enough to fit in PluginConfigurations' description column of type 'text'.

Here's the stack trace:

15:44:46,454 INFO migration:77 - Migrater: startTransfer: Plugins 15:44:46 Plugins 15:44:46,622 INFO migration:68 - Free memory : 383 MByte 15:44:46 Left Done Items/s ETE 15:44:46 5 : 0 0% 15:44:46,678 INFO migration:77 - PluginTransfer: createItem: Base1 name=Normalization: Global median ratio; id=1 15:44:46,984 INFO migration:77 - PluginTransfer: createItem: Ok 15:44:47,067 INFO migration:77 - PluginTransfer: createItem: Base1 name=Normalization: Lowess; id=2 15:44:47,211 INFO migration:77 - PluginTransfer: createItem: Ok 15:44:47,245 INFO migration:77 - PluginTransfer: commit(): could not insert: [net.sf.basedb.core.data.PluginConfigurationData] commit() net.sf.basedb.core.BaseException: could not insert: [net.sf.basedb.core.data.PluginConfigurationData]

at net.sf.basedb.core.HibernateUtil.saveData(HibernateUtil.java:967) at net.sf.basedb.core.DbControl.commit(DbControl.java:362) at net.sf.basedb.clients.migrate.Transfer.commit(Transfer.java:492) at net.sf.basedb.clients.migrate.PluginTransfer.createItem(PluginTransfer.java:167) at net.sf.basedb.clients.migrate.Transfer.runUnBatched(Transfer.java:398) at net.sf.basedb.clients.migrate.PluginTransfer.start(PluginTransfer.java:93) at net.sf.basedb.clients.migrate.Migrater.startTransfer(Migrater.java:218) at net.sf.basedb.clients.migrate.Migrater.run(Migrater.java:92) at net.sf.basedb.clients.migrate.Migrater.main(Migrater.java:477)

Caused by: org.hibernate.exception.GenericJDBCException: could not insert: [net.sf.basedb.core.data.PluginConfigurationData]

at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103) at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:40) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2050) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2515) at org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:47) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248) at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:290) at org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:180) at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:108) at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:186) at org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:33) at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:175) at org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:27) at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:70) at org.hibernate.impl.SessionImpl.fireSave(SessionImpl.java:535) at org.hibernate.impl.SessionImpl.save(SessionImpl.java:523) at org.hibernate.impl.SessionImpl.save(SessionImpl.java:519) at net.sf.basedb.core.HibernateUtil.saveData(HibernateUtil.java:958) ... 8 more

Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'description' at row 1

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2973) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1600) at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1125) at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:677) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1357) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1274) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1259) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) at org.hibernate.id.IdentityGenerator$GetGeneratedKeysDelegate.executeAndExtract(IdentityGenerator.java:73) at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:33) ... 24 more

Users, groups etc have transfered OK it seems. So has the Normalization: Global median ratio plugin (which has a shorter description).

Has anyone else found this? I'm 99.9% sure I didn't alter the BASE 1 plugins.

Other than my patches to ReporterTransfer.java and the BASE1/2 schemas, this is a fresh BASE 2.1 install from source.

Please reply to Bob MacCallum via basedb-devel list or dig up my email. Happy to give more info if you tell me where to look. Thanks!

[Nicklas Nordborg]

I googled around for the error message: MysqlDataTruncation: Data truncation: Data too long.

It could be caused by a bug in MySQL: ​http://bugs.mysql.com/bug.php?id=18908 I think it is very unlikely that the description is longer than 64K characters, which a text column should be able to handle, and if the description was longer, it would be caught in the code.

In fact, when I look at the description the lowess plugin contains the character 'ö', but the median ratio plugin doesn't.

I am running MySQL 5.0.15 and it seems not to be affected by the bug. The latest release is 5.0.27

[base]

Hi Nicklas,

Thanks very much for looking - I'll also do some more work on it today and keep you informed. We're on MySQL 5.0.21 by the way.

keep you posted. Bob

[base]

fixed - thanks to your help - will post more details tomorrow - have some suggestions for improving documentation!

[base]

OK. So I read the info in doc/configure_unicode_support.txt and added characterEncoding=utf8 to the db.url in base.config, however that didn't help.

I then read some JDBC docs and decided to try characterEncoding=UTF-8 and it worked.

Note that the database is created with default charset = utf8, however the MySQL server's default charset is latin1 (it's also running BASE 1). I suspect that at BASE HQ you have a dedicated BASE 2 MySQL server which has utf8 as default, so you wouldn't encounter these problems.

Thanks again for pointing me in the right direction, Nicklas.

[Nicklas Nordborg]

I'll change it in the docs. We have used characterEncoding=utf8 without problem so far but the MySQL docs says that UTF-8 should be used. The use of UTF-8 is not something we have spent much time on exploring. All our servers are running with default settings which probably is ISO-8859-1.

[Nicklas Nordborg]

(In [2940]) References #438