External Plugin Executor

[Nicklas Nordborg]

This is a new plug-in that is intended to be a more powerful complement to the existing !Base1PluginExecuter. The main problem with the old plug-in is that it needs to follow the BASE 1 rules and it can never take full advantage of new BASE 2 features.

The general protocol of the new plug-in is:

1. Export data from BASE to files.
2. Run some external program.
3. Import data from files that the external program produced.

All steps should be higly configurable. It should be possible to select which exporter to use, which external program to use and which importer to use. To accomplish this we need to define:

• A set of interfaces that compatible exporters/importers need to implement. Some exporters/importers may need configuration-time settings and/or run-time settings.
• A way to define additional run-time parameters that are needed by the external program. The selected values must go through the exporter which is responsible for creating ALL files.

We will use the BaseFileSet (#1440) format as the first use case. When this plug-in is in place it should also be possible to easily fix #741. With the proper wrapper classes executing BASE 1 plug-ins is just another use case. In theory we could deprecate the !Base1PluginExecuter, but in practice it may not work since existing configurations for the !Base1PluginExecuter needs to be transfered to the new plug-in. At this time it is hard to tell if this is going to be easy or if it requires a lot of work.

[Martin Svensson]

(In [5200]) References #1443 Started to build up the configuration part of the plugin.

[Martin Svensson]

(In [5217]) References #1443 Created a dtd-file to describe how the external parameters should be defined in XML. Only a few parametertypes are still supported. More will come when most of the plug-in is done. A jsp-page is created to manage the parameters needed by the external plug-in. The functionality is far from finished and only part of gui is done.

[Martin Svensson]

Changeset [5222] should address to this ticket.

[Martin Svensson]

(In [5226]) References #1443 Continuing develop the jsp-page specilized to let the user create parameter for the external plug-in.

[Martin Svensson]

(In [5234]) References #1443 Extended the external-parameter wizard to also support larger text-, boolean- and float- parameters.

[Martin Svensson]

(In [5247]) References #1443 The job-configuration part of the ExternalPluginExecutor is more or less done now. Things left to do in this ticket is the run-part of the plugin and the plugin's interaction with the exporter-plugin and the importer-plugin.

[Martin Svensson]

(In [5250]) References #1443 Moved the import/export configurations to an own step in the plugin-configuration part. Have done some documentation of the code.

[Nicklas Nordborg]

(In [5265]) References #1443: External Plugin Executor

• Moved around and renamed some things.
• Added possibilty for the external executor to communicate with exporter and importer plug-ins.
• Implemented support plug-ins for BFS export and import (has very limited functionality)

It should now be possible to complete both a configuaration sequence and a job execution sequence without errors. But there are lot of holes that needs to be fixed. For example:

• The external program is never executed
• The data to export is hard-coded into the exporter
• No context-checking that experiment has proper raw data type, number of channels, etc.
• No parameters are included in the export
• Not possible to define parameters that selects experimental factors, bioassays, etc.
• And probably other things as well...

[Nicklas Nordborg]

(In [5266]) References #1443: External Plugin Executor

• Implemented isInContext() method
• Restructured the run() method to make it more usable with subclassing
• Started to implement method that executes the external program but this is not yet finished

[Nicklas Nordborg]

(In [5270]) References #1443: External Plugin Executor

• Exection of external program implemented including error handling
• Implemented sending/receiving of standard input/output between the executor and the program
• Added test program for the external executor (using BFS format for a simple copy of the source bioassay set)

[Nicklas Nordborg]

(In [5271]) References #1443: External Plugin Executor

• Implement support for runtime export/import parameters
• Added parameters for child name, extra files directory to BFS importer (nned to add test case for extra files)

[Nicklas Nordborg]

(In [5272]) References #1443: External Plugin Executor

Fixes a classpath issue with the test program on windows.

[Nicklas Nordborg]

(In [5274]) References #1443: External Plugin Executor

Reworked parameter definition part. It is possible to define basic parameter types and they will show up in job configuration, but...

• Almost no error handling. Some errors may cause the configuration to be unusable and forced to be deleted.
• Doesn't work with IE (XML handling with javascript).
• Exporters/importers are not aware of the parameters.

[Nicklas Nordborg]

(In [5275]) References #1443: External Plugin Executor

• The BFS export now includes the program parameters in the exported metadata file
• Added support for parameters that can have multiple values.

[Nicklas Nordborg]

(In [5277]) References #1443: External Plugin Executor

• Added support for specifying which reporter/spot/assay fields to export
• Reworked string to field conversion since the current implementation was too Base1-specific.

[Nicklas Nordborg]

(In [5278]) References #1443: External Plugin Executor

• Added support for program parameters for selecting experimental factors and formulas.
• If the selected experimental factor/formula is not already included in the export it is added as a new field.

[Nicklas Nordborg]

(In [5280]) References #1443: External Plugin Executor

• Added support for program parameter for selecting bioassays.

[Nicklas Nordborg]

(In [5281]) References #1443: External Plugin Executor

Internet explorer can now be used to create program parameter definitions.

[Nicklas Nordborg]

(In [5282]) References #1443: External Plugin Executor

If executed on a job agent, also load the exporter/importer plug-ins using their configured JAR path.

[Nicklas Nordborg]

The last changeset [5282] also adds proper support for aborting the plug-in.

[Nicklas Nordborg]

(In [5283]) References #1443: External Plugin Executor

Properly tokenize command line arguments for the external program instead of submitting the entire command line as one argument.

[Nicklas Nordborg]

(In [5285]) References #1443: External Plugin Executor

Added test case for importing extra files.

[Nicklas Nordborg]

(In [5286]) References #1443: External Plugin Executor

Added TestExternalProgramExecutor to TestAll. Removed leftover debug output.

[Nicklas Nordborg]

(In [5287]) References #1443: External Plugin Executor

Added error checking of parameter definitions.

[Nicklas Nordborg]

(In [5289]) References #1443: External Plugin Executor

Check that defined fields in the export are unique.

[Nicklas Nordborg]

(In [5290]) References #1443: External Plugin Executor

Implemented toString() in exportable field implementations.

[Nicklas Nordborg]

(In [5291]) References #1443: External Plugin Executor

Completed implementation and documented ParameterDefinition.java

[Nicklas Nordborg]

(In [5292]) References #1443: External Plugin Executor

Replaced the field converter used by Base1PluginExecutor with an implementation that extends ExportableFieldConverter

[Jari Häkkinen]

I am seeing a strange thing. I am creating a new configuration. If I select the wrong name for the export plug-in and click next I properly get an Invalid data dialogue. If I now click back, I get back to the first dialogue, correcting the export plug-in name to the class name, and clicking next again ... I will get back to the Inavlid data dialogue displaying the first entered name (not the corrected string) and complaining about the string. That is, the correction made is not registered in the GUI. The work around is to close and start over.

[Nicklas Nordborg]

It is not possible to go backwards in the plug-in configuration wizard, particularly not if a generic error dialog is displayed. Unfortunately this kind of dialog always include the "Back" button. This doesn't happen if the error is catched by the wizard, which then displays the old configuration page with together with the error message.

In any case I suspect that everything is not installed properly, since it should not be possible to enter a name. It should be a selection list with the installed plug-ins that can be used.

[Nicklas Nordborg]

(In [5300]) References #1443: External Plugin Executor

Field conversion now supports using metadata to find existing property fields.

[Nicklas Nordborg]

(In [5302]) References #1443: External Plugin Executor

Field conversion for assay fields now checks annotations on the assays and not just experimental factors.

This more or less completes the feature set for the external executor and it's BFS export/import support classes. Now it only remains to weed out bugs and maybe polish a few corners.

[Jari Häkkinen]

Replying to nicklas:

In any case I suspect that everything is not installed properly, since it should not be possible to enter a name. It should be a selection list with the installed plug-ins that can be used.

It maybe that something was not installed properly but I follow the instructions to upgrade the database. The database I currently test on has been around for almost two years now with incremental upgrades.

The problem with not getting the list is that the external program plug-in (EPE) does not find any matching plug-ins. Digging somewhat deeper I remove the ​restriction on import type in the plug-in and print out the result of the query while running EPE configuration. The list of plug-ins returned is

Import: Illumina BGX reporter importer
Import: Illumina BGX feature importer
Import: Illumina SNP raw data importer
Import: Illumina SNP reporter importer
Import: Batch Data Importer
Import: Illumina BGX reporter importer
Import: Illumina BGX feature importer
Import: Illumina SNP reporter importer
Import: Illumina BGX reporter importer
Import: Illumina BGX feature importer

which interesting since I have many more importers installed. (I run the test code as an administrator so there should be no sharing issues.)

Why is some plug-ins appearing more than once? What is common denominator? Why aren't there any exporters (or other types)? Looking into the list of import and export plug-ins in my test server I notice that all five unique plug-ins are owned by the user running EPE whereas all other plug-in definitions are owned by 'root'. Actually there is another importer plug-in owned by my admin user that does not show up in the list! So, I run the configuration as 'root' and find many plug-ins owned by root. Plug-ins owned by my admin userid is not reported.

I suppose this explains why I get no selection list during EPE configuration. If no plug-in is found, apparently I get presented a string box. So, the query should really look for all plug-ins accessible to me not only the ones owned by me.

I cannot explain why the query could not find all plug-ins owned by me and why some plug-ins appear more than one time in the list. I looked through the list of all plug-in definitions (including removed) and there are no duplicate names.

[Jari Häkkinen]

Replying to nicklas:

This more or less completes the feature set for the external executor and it's BFS export/import support classes. Now it only remains to weed out bugs and maybe polish a few corners.

I have an issue defining parameters for my external binary (the 3rd dialogue in configuration). I tried to select a Name like "Select reference assays" but it fails with message "Error invoking plugin". The problem is probably the spaces, changing the Name to "Select_reference_assays" will successfully end the configuration dialogues.

While I am writing this, how do I define defaults for bioassay selection (multiple values). I would like to have default all. There are basically two options, none or all.

[Nicklas Nordborg]

While I am writing this, how do I define defaults for bioassay selection (multiple values). I would like to have default all. There are basically two options, none or all.

The default value that can be entered in the dialog is matched against the name of the bioassays and the first one that is found is used as the default. It is not possible to have more than one default value.

[Nicklas Nordborg]

(In [5303]) References #1443: External Plugin Executor

Fixes an issue with spaces in parameter names and making sure that names are unique.

[Nicklas Nordborg]

I cannot explain why the query could not find all plug-ins owned by me and why some plug-ins appear more than one time in the list.

Some plug-ins appear more than one time or not at all because the restriction was removed but not the join to the pluginTypes table. Some plug-ins implement more than one type and some plug-ins doesn't implement any type at all. The query is also missing the option to include plug-ins not owned by the current user. Eg. ItemQuery.include(Include.SHARED, ....)

[Nicklas Nordborg]

(In [5304]) References #1443: External Plugin Executor

Include import/export plug-ins not owned by the current user.

[Nicklas Nordborg]

(In [5310]) References #1443: External Plugin Executor

Added debug-mode parameter which leaves the temporary working directory after the plug-in has finished. The option must be enabled on the configuration to be visible at runtime.