File upload times out

[Nicklas Nordborg]

The file upload may randomly get a timeout exception:

java.net.SocketTimeoutException on page /base21/filemanager/upload/progress.jsp
    
Version     BASE 2.1.1pre (build #0; schema #20)
Web server     Apache Tomcat/5.5.20
Database Dialect     org.hibernate.dialect.MySQLInnoDBDialect
JDBC Driver     com.mysql.jdbc.Driver (version 3.1)
Java runtime     Java(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc.
Operating system     Linux i386 2.6.5-7.276-default
Memory     Total: 60.3 MB
Free: 11.6 MB
Max: 494.5 MB
Browser     Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0
Error message     Read timed out
Stack trace    

...at java.net.SocketInputStream.socketRead0(Native Method)
...at java.net.SocketInputStream.read(SocketInputStream.java:129)
...at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:747)
...at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:777)
...at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:115)
...at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:712)
...at org.apache.coyote.Request.doRead(Request.java:418)
...at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:284)

...at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:371)
...at org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:293)
...at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:104)
...at javax.servlet.ServletInputStream.readLine(ServletInputStream.java:93)
...at org.apache.catalina.connector.CoyoteInputStream.readLine(CoyoteInputStream.java:198)
...at net.sf.basedb.clients.web.fileupload.FileUpload$UploadStream.readMoreData(FileUpload.java:796)
...at net.sf.basedb.clients.web.fileupload.FileUpload$UploadStream.read(FileUpload.java:750)
...at net.sf.basedb.util.FileUtil.copy(FileUtil.java:81)
...at org.apache.jsp.filemanager.upload.upload_jsp._jspService(upload_jsp.java:205)
...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
...at net.sf.basedb.clients.web.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:71)
...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
...at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
...at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
...at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
...at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
...at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
...at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
...at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
...at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
...at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
...at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
...at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
...at java.lang.Thread.run(Thread.java:595)

• It appears to happen at random times. Sometimes it works, sometimes not.
• It seems to be related to slower network connections. We have never been able to reproduce the error locally.
• We have only seen this error when the client machine is a windows box.
• It has happened both with IE6, IE7 and Firefox
• We have only seen it on the 2.1.x versions, not in the trunk so far

What we can try:

• First, we must be able to reproduce this error locally
• Check if it has something to do with the character encoding filter servlet. If it does we should replace the file upload with a real servlet instead of a JSP page
• Check if it has something to do with the session=false attribute that is present in the trunk but not in the 2.1.x branch
• Check if has something to do with the reload of the progress bar

[Nicklas Nordborg]

I have testet this from my home which has a very slow connection. None of the ideas mentioned above made any difference. Searching on the net finds some other user with similiar problems but almost none with any solutions. One advice was found on ​http://www.tek-tips.com/viewthread.cfm?qid=627123. After setting disableUploadTimeout="false" in my server.xml in TOMCAT_HOME/conf I have not been able to reproduce the error. I could upload at least twice as many files as before without getting any error.

[Nicklas Nordborg]

I have now investigated this a bit more by delving in the Tomcat code and I think I have a possible solution. This is what I found:

• Tomcat has two different timeouts. One timeout for parsing the headers of the request and another timeout for the rest of the POST:ed data

• The header-timeout is initially the timeout specified in server.xml as the connectionTimeout attribute for the connector tag. This timeout is normally 20000ms.

• Depending on the number of threads that Tomcat thinks are active the header-timeout can be decreased by as much as a factor of 20 or more. This may decrease the header-timeout to less than a second.

• When disableUploadTimeout="true" is specified no timeout is set for the data part. The timeout is left to the same used for parsing the headers. This seems to me like a bug in Tomcat.

• When disableUploadTimeout="false" is specified the timeout is set to the unmodified connectionTimeout value.

I think this explains most of the problems we have had:

• It doesn't happen locally since there is very little probablity that one has to wait 1 second for the next data-package to arrive.

• It doesn't happen if the "Max upload" setting is decrease in BASE since then there will always be an incoming packet on queue.

• The problem only happens randomly and works the first couple of times because Tomcat doesn't start modifying the timeout until it thinks it is busy.

• Setting disableUploadTimeout="false" beacuse even with a slow connection we never have to wait 20 seconds.

I now also have a possible solution. The SocketTimeoutException only tells us that no data was read during the time we waited. The connection is still open and usable and we can simply try reading data one more time (or as many times as we think is needed).

[Nicklas Nordborg]

(In [3003]) References #454: File upload times out

Possible fix but have to be tested with slow connection.

[Nicklas Nordborg]

(In [3010]) References #454: File upload times out

Use Ajax for updating the progress bar if the browser supports it.

[Jari Häkkinen]

The fix is now working on my slow connection test machines.