Opened 14 years ago
Closed 14 years ago
#1582 closed enhancement (fixed)
Extended support for external files
Reported by: | Nicklas Nordborg | Owned by: | Nicklas Nordborg |
---|---|---|---|
Priority: | critical | Milestone: | BASE 3.0 |
Component: | web | Version: | |
Keywords: | Cc: |
Description ¶
The current support for external files introduced in BASE 2.16 by #1485 is limited to http and https. There is also support for Basic and Digest authentication and client-side certificates. We would like to be able to make this support more generalized. Here are a few examples:
- Files on the Amazon S3 cloud storage. The files can be accessed by regular http or https, but they use a different authentication mechanism (http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RESTAuthentication.html).
- Files on a Hadoop distributed file system (HDFS). The files need to be accessed thorugh a special API. http://hadoop.apache.org/hdfs/
This ticket is about adding support in the BASE core api for various kinds of external file access. We hope to be able to do this in a somewhat pluggable fashion, although it may not work for all cases (eg. if the case needs more configuration options). The actual support for the S3 and Hadoop will be added as separate tickets.
Change History (7)
comment:1 by , 14 years ago
comment:2 by , 14 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:3 by , 14 years ago
(In [5582]) References #1582: Extended support for external files
Created framework for external file support. A registry (ConnectionManagerRegistry
) is keeping track of registered factories (ConnectionManagerFactory
). Typcially, a specific protocol is handled by a single factory.
An implementation for http
and https
have been created. I also have code for HDFS but I think that should be made as an extension. To do this we first need #1593 and parts of #1592.
The current implementation only supports automatic selection of connection factory (based on the protocol). This will probably not work with, for example, Amazon S3, since this also uses http (+custom authentication procedure). We need a way to manually select a factory to use. This is probably best done at the FileServer
level.
comment:4 by , 14 years ago
comment:5 by , 14 years ago
comment:6 by , 14 years ago
(In [5614]) References #1582: Extended support for external files
Fixed a NullPointerException found while testing. Store latest robots.txt.
comment:7 by , 14 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
For HDFS there already exists a sub-project, 'HDFS proxy'. This is a https proxy server that can be run inside Tomcat that enables regular https access to the files in the HDFS file system. It seems like it comes with complete support for client-side certificates which means that everything should work with the current version of BASE.
Pros
Cons
More information about HDFS proxy at: http://hadoop.apache.org/hdfs/docs/current/hdfsproxy.html