Preparations for Java 11 support

[Nicklas Nordborg]

Java 11 was released in september 2018 and with it comes a lot of big changes.

• The Oracle JDK is no longer free to use in a production environment (only for development). However, OpenJDK has been technically improved and will be aligned to the Oracle JDK.
• A new major version will be released every 6 months. Security updates will only be provided (by Oracle) for the last major version. Other vendors (eg. Red Hat) may provide support also for some of the older versions.
• Java 8 is supported until 2019.

Sooner or later we will have to move from Java 8 to Java 11 (or some later version). I think it would be a good idea to start this work now. At this stage:

• BASE must still work with Java 8.
• We should identify problems that need to be fixed.
• Changes that can be made without breaking Java 8 support.

For more information about migrating to Java 11:

• ​https://blog.codefx.org/java/java-11-migration-guide/
• ​https://medium.com/codefx-weekly/no-free-java-lts-version-b850192745fb
• ​https://openjdk.java.net/
• ​https://www.oracle.com/technetwork/java/javase/downloads/index.html

[Nicklas Nordborg]

First step is trying to run initdb with Java 11 (the code is compiled with Java 8). It fails almost immediately with an exception:

java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
        at org.hibernate.boot.cfgxml.internal.ConfigLoader$1.initialize(ConfigLoader.java:41)
        at org.hibernate.boot.cfgxml.internal.ConfigLoader$1.initialize(ConfigLoader.java:38)
        at org.hibernate.internal.util.ValueHolder.getValue(ValueHolder.java:55)
        at org.hibernate.boot.cfgxml.internal.ConfigLoader.loadConfigXmlResource(ConfigLoader.java:57)
        at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:163)
        at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:152)
        at net.sf.basedb.core.HibernateUtil.init1(HibernateUtil.java:206)
        at net.sf.basedb.core.Application.start(Application.java:514)
        at net.sf.basedb.core.Install.createTables(Install.java:131)
        at net.sf.basedb.install.InitDB.main(InitDB.java:75)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 10 more

We can search Maven (​https://search.maven.org/) for this class/package (eg. javax.xml.bind) and find the jaxb-api: ​https://search.maven.org/search?q=g:javax.xml.bind%20AND%20a:jaxb-api&core=gav We download the 2.3.1 version since it seems to be most recent. The exception is replaced with a new one:

net.sf.basedb.core.BaseException: Unable to perform unmarshalling at line number 0 and column 0 in RESOURCE hibernate.cfg.xml. Message: null
        at net.sf.basedb.core.HibernateUtil.init1(HibernateUtil.java:217)
        at net.sf.basedb.core.Application.start(Application.java:514)
        at net.sf.basedb.core.Install.createTables(Install.java:131)
        at net.sf.basedb.install.InitDB.main(InitDB.java:75)
Caused by: org.hibernate.internal.util.config.ConfigurationException: Unable to perform unmarshalling at line number 0 and column 0 in RESOURCE hibernate.cfg.xml. Message: null
        at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:133)
        at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:65)
        at org.hibernate.boot.cfgxml.internal.ConfigLoader.loadConfigXmlResource(ConfigLoader.java:57)
        at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:163)
        at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:152)
        at net.sf.basedb.core.HibernateUtil.init1(HibernateUtil.java:206)
        ... 3 more
Caused by: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
        at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:278)
        at javax.xml.bind.ContextFinder.find(ContextFinder.java:421)
        at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:721)
        at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:662)
        at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:122)
        ... 8 more
Caused by: java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at javax.xml.bind.ServiceLoaderUtil.nullSafeLoadClass(ServiceLoaderUtil.java:122)
        at javax.xml.bind.ServiceLoaderUtil.safeLoadClass(ServiceLoaderUtil.java:155)
        at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:276)
        ... 12 more

We need to search for com.sun.xml.bind and find the jaxb-impl (​https://search.maven.org/search?q=g:com.sun.xml.bind%20AND%20a:jaxb-impl&core=gav) and jaxb-core (​https://search.maven.org/search?q=g:com.sun.xml.bind%20AND%20a:jaxb-core&core=gav). Once again, we download version 2.3.1 and 2.3.0.1 resepectively. Next exception:

java.lang.NoClassDefFoundError: javax/activation/DataSource
        at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.<clinit>(RuntimeBuiltinLeafInfoImpl.java:474)
        at com.sun.xml.bind.v2.model.impl.RuntimeTypeInfoSetImpl.<init>(RuntimeTypeInfoSetImpl.java:63)
        at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:128)
        at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:84)
        at com.sun.xml.bind.v2.model.impl.ModelBuilder.<init>(ModelBuilder.java:162)
        at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.<init>(RuntimeModelBuilder.java:92)
        at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:455)
        at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:303)
        at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:139)
        at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1156)
        at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:165)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:297)
        at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:286)
        at javax.xml.bind.ContextFinder.find(ContextFinder.java:409)
        at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:721)
        at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:662)
        at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:122)
        at org.hibernate.boot.cfgxml.internal.JaxbCfgProcessor.unmarshal(JaxbCfgProcessor.java:65)
        at org.hibernate.boot.cfgxml.internal.ConfigLoader.loadConfigXmlResource(ConfigLoader.java:57)
        at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:163)
        at org.hibernate.boot.registry.StandardServiceRegistryBuilder.configure(StandardServiceRegistryBuilder.java:152)
        at net.sf.basedb.core.HibernateUtil.init1(HibernateUtil.java:206)
        at net.sf.basedb.core.Application.start(Application.java:514)
        at net.sf.basedb.core.Install.createTables(Install.java:131)
        at net.sf.basedb.install.InitDB.main(InitDB.java:75)
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 29 more

Here we search for javax.actication and find ​https://search.maven.org/search?q=g:javax.activation Version 1.2.0 is downloaded.

And finally, the installation is running and seems to be working! There are a few warnings, but more about them later...

[Nicklas Nordborg]

To summarize the above. It seems like we need four additional JAR files to get back things that has been removed from core java:

• javax.activation-api-1.2.0.jar
• jaxb-api-2.3.1.jar
• jaxb-core-2.3.0.1.jar
• jaxb-impl-2.3.1.jar

[Nicklas Nordborg]

The initdb now works but causes warnings. To get all warnings except the first one --illegal-access=warn must be added to the command line.

WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils 
(file:/..../www/WEB-INF/lib/postgresql-42.0.0.jar) 
to field java.util.TimeZone.defaultTimeZone
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions 
(file:/..../www/WEB-INF/lib/javassist-3.20.0-GA.jar) 
to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

Updating to the latest JDBC driver for PostgreSQL (42.2.25, ​https://jdbc.postgresql.org/) and the latest Javassist (3.23.1-GA, ​https://github.com/jboss-javassist/javassist/releases) seems to fix the problems without breaking the installation.

[Nicklas Nordborg]

The other command-line programs also seems to work. I have tested:

• updatedb
• dbinfo
• migrate export
• migrate import

[Nicklas Nordborg]

TestAll fails in 3 tests:

• TestNumberFormat:

  --Parse string '1.23456E2' --> 123.456f FAILED
  java.lang.NumberFormatException: For input string: "1.23456E2"
  	at net.sf.basedb.core.Type.parseString(Type.java:716)
  	at TestNumberFormat.test_parse_number(TestNumberFormat.java:103)
  	at TestNumberFormat.test_all(TestNumberFormat.java:50)
  	at TestAll.main(TestAll.java:139)
  ... and a lot of more similar errors
  

• TestSpotImages:

  Error: One factory fails for the operation "encode"
  Occurs in: javax.media.jai.ThreadSafeOperationRegistry
  java.lang.reflect.InvocationTargetException
  	at java.base/jdk.internal.reflect.GeneratedMethodAccessor1114.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  	at javax.media.jai.FactoryCache.invoke(FactoryCache.java:122)
  	at javax.media.jai.OperationRegistry.invokeFactory(OperationRegistry.java:1674)
  	at javax.media.jai.ThreadSafeOperationRegistry.invokeFactory(ThreadSafeOperationRegistry.java:473)
  	at javax.media.jai.registry.RIFRegistry.create(RIFRegistry.java:332)
  	at javax.media.jai.RenderedOp.createInstance(RenderedOp.java:819)
  	at javax.media.jai.RenderedOp.createRendering(RenderedOp.java:867)
  	at javax.media.jai.RenderedOp.getRendering(RenderedOp.java:888)
  	at javax.media.jai.JAI.createNS(JAI.java:1099)
  	at javax.media.jai.JAI.create(JAI.java:973)
  	at javax.media.jai.JAI.create(JAI.java:1395)
  	at net.sf.basedb.core.SpotImages.saveImage(SpotImages.java:965)
  	at net.sf.basedb.core.SpotImages.createSpotImages(SpotImages.java:496)
  	at TestSpotImages.test_create_spotimages(TestSpotImages.java:102)
  	at TestSpotImages.test_all(TestSpotImages.java:64)
  	at TestAll.main(TestAll.java:119)
  Caused by: java.lang.NoClassDefFoundError: com/sun/image/codec/jpeg/JPEGCodec
  	at com.sun.media.jai.codecimpl.JPEGImageEncoder.encode(JPEGImageEncoder.java:261)
  	at com.sun.media.jai.opimage.EncodeRIF.create(EncodeRIF.java:70)
  	... 18 more
  Caused by: java.lang.ClassNotFoundException: com.sun.image.codec.jpeg.JPEGCodec
  	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
  	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
  	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
  	... 20 more
  ... and then several other due to spot images not existing in the first place
  

• TestDirty:

  java.lang.Exception: Unexpected sum for 'x': 8010.0; expected 9920.0
  	at TestDirty.test_sum_count(TestDirty.java:126)
  	at TestDirty.test_all(TestDirty.java:85)
  	at TestAll.main(TestAll.java:155)
  ... and some more that are very similar...
  

[Nicklas Nordborg]

(In [7509]) References #2129: Preparations for Java 11 support

Updated Javassist to version 3.23.1 and PostgreSQL JDBC driver to version 42.2.5.

[Nicklas Nordborg]

(In [7510]) References #2129: Preparations for Java 11 support

Fixes the errors in TestNumberFormat and TestDirty.

The problem was caused by a change in the default number formatting.

• Scientific notation had changed from using E to ×10^. We need to explicitely change it back to E
• Minus sign had changed from the regular minus at ascii(45) to a unicode minus at U+2212 (​https://www.fileformat.info/info/unicode/char/2212/index.htm). We need to explicityely change it back to a regular minus sign.

[Nicklas Nordborg]

(In [7511]) References #2129: Preparations for Java 11 support

Fixes the errors in TestSpotImages.

The problem was caused by the JAI toolkit that tries to use an old and internal (com.sun.image.codec.jpeg.JPEGCodec) implementation of JPEG encoding/decoding. The old implementation was replaced with a new one (in javax.imageio) a long time ago, but JAI has not been updated.

The code in BASE has been updated to use javax.imageio for handling JPEG images.

We have also deprecated all functionality that is related to spot images, because we don't know how much longer JAI will be working and we don't have the resources to fully replace it with something new.

[Nicklas Nordborg]

The "roles" test program worked without errors or warnings. Starting up a Tomcat instance (version 8.5.15) also worked and it was possible to login and navigate around in BASE without any problems. The CompileAll servlet seems to be able to compile all JSP pages.

So, basically it seems like BASE is working with Java 11 if we just add some extra JAR files (see comment:3 above) to the WEB-INF/lib directory.

[Nicklas Nordborg]

(In [7512]) References #2129: Preparations for Java 11 support

Adding extra JAR files that are required for running BASE with Java 11.

[Nicklas Nordborg]

First try to compile with Java 11

[Nicklas Nordborg]

Compiling with Java 11 causes 1 error and 25 warnings before the compilation stops. See ant-core.compile-1.out​

[Nicklas Nordborg]

(In [7513]) References #2129: Preparations for Java 11 support

Get rid of warnings related to Class.newInstance(). Basically we need to replace it with Class.getDeclaredConstructor().newInstance() but this can throw some extra exceptions that also need to be handled or declared.

[Nicklas Nordborg]

(In [7514]) References #2129: Preparations for Java 11 support

Get rid of warnings related to new Long(String) by replacing it with Long.parseLong(String).

[Nicklas Nordborg]

(In [7515]) References #2129: Preparations for Java 11 support

Get rid of the error related to Thread.stop() by replacing it with Thread.interrupt().

The remaining warnings that are related to Object.finalize() and ClassLoader.getPackage() seems a bit more complex to get rid of, so this will have to wait until later.

[Nicklas Nordborg]

Second try to compile with Java 11

[Nicklas Nordborg]

After fixing the error, the full build completes with only warnings. See ant-dev-1.out​

[Nicklas Nordborg]

(In [7516]) References #2129: Preparations for Java 11 support

Get rid of warnings related to new Integer(), new Float(), etc.

[Nicklas Nordborg]

(In [7517]) References #2129: Preparations for Java 11 support

Get rid of some more warnings warnings related to Class.newInstance().

[Nicklas Nordborg]

(In [7547]) References #2129: Preparations for Java 11 support

More information in the installation documentation about Java 11.

[Nicklas Nordborg]

(In [7551]) References #2129: Preparations for Java 11 support

Added the AutoCloseable interface to several classes and interfaces that already has a close() method. This should make it easier to write code that makes sure that resources are closed even after an exception (try-with-resources pattern).

[Nicklas Nordborg]

(In [7555]) References #2129: Preparations for Java 11 support

Found another class related to spot images that should be deprecated.