A couple of weeks back, I spent time setting up a training environment for our webapp. We also needed to integrate with another application which forwards users to our site. When coming from the other site, our webapp was non functional (returned 500 or 503 errors). In our apache logs, I noticed the errors:
ajp_msg_append_cvt_string(): BufferOverflowException 4 7969
Googling this error, I found a possible solution was to use ProxyIOBufferSize in our proxy config. I added this, but then the apache errors were replaced by:
ajp_marshal_into_msgb: Error appending the header value
ajp_send_header: ajp_marshal_into_msgb failed
[error] (120001)APR does not understand this error code: proxy: AJP: request failed to ....
I eventually traced this to cookies, and after I cleared some of the larger cookies, things started working. So, I left the working configuration as-is and went away.
That is until today, when we were trying to upload files using apache commons-fileupload. Two of us spent the better part of the day trying to figure out what changes we had to our webapp, comparing the checksums of the jar files, trying to figure out if we had the classes were being loaded from a different jar file, comparing our training environment to production (where it worked).. We even debugged it to a filesize where the upload was successful and one where it was not. Files lesser than 24k worked, but larger than that threw an exception:
ajp_msg_append_cvt_string(): BufferOverflowException 4 7969
Googling this error, I found a possible solution was to use ProxyIOBufferSize in our proxy config. I added this, but then the apache errors were replaced by:
ajp_marshal_into_msgb: Error appending the header value
ajp_send_header: ajp_marshal_into_msgb failed
[error] (120001)APR does not understand this error code: proxy: AJP: request failed to ....
I eventually traced this to cookies, and after I cleared some of the larger cookies, things started working. So, I left the working configuration as-is and went away.
That is until today, when we were trying to upload files using apache commons-fileupload. Two of us spent the better part of the day trying to figure out what changes we had to our webapp, comparing the checksums of the jar files, trying to figure out if we had the classes were being loaded from a different jar file, comparing our training environment to production (where it worked).. We even debugged it to a filesize where the upload was successful and one where it was not. Files lesser than 24k worked, but larger than that threw an exception:
java.lang.IndexOutOfBoundsException
java.io.BufferedInputStream.read(BufferedInputStream.java:306)
org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:627)
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:584)
org.apache.jk.common.JkInputStream.receive(JkInputStream.java:200)
org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:179)
org.apache.coyote.Request.doRead(Request.java:419)
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
org.apache.commons.fileupload.MultipartStream.discardBodyData(MultipartStream.java:651)
org.apache.commons.fileupload.MultipartStream.skipPreamble(MultipartStream.java:690)
org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:339)
We had finally given up trying to debug the application, and in desperation had a look at the apache logs. We noticed that there was an error message that was written everytime our upload failed:
[error] ajp_read_header: ajp_ilink_receive failed
We revisited the proxy configuration and found that ProxyIOBufferSize statement was the only difference between our training and production (and development) environment. Commenting it out and restarting apache made our upload problems go away. Whew.. this was one difficult day.
No comments:
Post a Comment