Visualizing daily data trends.

I had to revisit some R code that I had written a while back. Since I had additional data, I decided to rework it a little bit. The code generates 2 plots. The first one is just a line plot of a daily count of something ( e.g. number of files generated). The second one shows the trend.

Gist

I ran it on 2 1/2 years of data, and came up with the following plots.

The scatter plot just shows the daily values. The lower values are on the weekends. The trend graph shows a clearer pattern. It shows a sharp dip during the year end (vacation time).

certificates and expiry.

Today we needed to find a way to monitor our ssl certificate expiry dates. A quick search on google pointed me to this document. It was a simple enough matter to get the days remaining before the certificate expires.

Gist

This does not work for us because openssl cannot use a proxy. See openssl-s-client-using-a-proxy.

The option of using another package (proxytunnel) was not possible on our restricted access servers.

So, back to the drawing board. This time, I went for a java solution. A quick and dirty solution can be found  at this gist.

To use this java program, you would call it like so:

java -Dhttps.proxyHost=.. -Dhttps.proxyPort=... -cp SSLCertExpiry www.google.com

ProxyIOBuffer and file uploads

A couple of weeks back, I spent time setting up a training environment for our webapp. We also needed to integrate with another application which forwards users to our site. When coming from the other site, our webapp was non functional (returned 500 or 503 errors). In our apache logs, I noticed the errors:

ajp_msg_append_cvt_string(): BufferOverflowException 4 7969


Googling this error, I found a possible solution was to use ProxyIOBufferSize in our proxy config. I added this, but then the apache errors were replaced by:

ajp_marshal_into_msgb: Error appending the header value
ajp_send_header: ajp_marshal_into_msgb failed
 [error] (120001)APR does not understand this error code: proxy: AJP: request failed to ....

I eventually traced this to cookies, and after I cleared some of the larger cookies, things started working. So, I left the working configuration as-is and went away.

That is until today, when we were trying to upload files using apache commons-fileupload. Two of us spent the better part of the day trying to figure out what changes we had to our webapp, comparing the checksums of the jar files, trying to figure out if we had the classes were being loaded from a different jar file, comparing our training environment to production (where it worked).. We even debugged it to a filesize where the upload was successful and one where it was not. Files lesser than 24k worked, but larger than that threw an exception:

java.lang.IndexOutOfBoundsException

        java.io.BufferedInputStream.read(BufferedInputStream.java:306)

        org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:627)

        org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:584)

        org.apache.jk.common.JkInputStream.receive(JkInputStream.java:200)

        org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:179)

        org.apache.coyote.Request.doRead(Request.java:419)

        org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)

        org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)

        org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)

        org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)

        org.apache.commons.fileupload.MultipartStream.discardBodyData(MultipartStream.java:651)

        org.apache.commons.fileupload.MultipartStream.skipPreamble(MultipartStream.java:690)

        org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:339)

We had finally given up trying to debug the application, and in desperation  had a look at the apache logs. We noticed that there was an error message that was written everytime our upload failed:

[error] ajp_read_header: ajp_ilink_receive failed

We revisited the proxy configuration and found that ProxyIOBufferSize statement was the only difference between our training and production (and development) environment. Commenting it out and restarting apache made our upload problems go away. Whew.. this was one difficult day.

buttons buttons and counters.

I needed to keep track of some different actions during a basketball game, and needed a simple interface to count the number of times that each action took place. Since I wanted to play with bootstrap, I decided to write a simple html "application" that I could use on my phone.

To begin with, I needed to keep track of 8 actions (blocks, cross-overs, ...). A 2x4 grid seems apt. The buttons needed to be big enough for clicking. One possibility is to use rows with 2 buttons of  col-xs-6 sizes so that it will appear to be 2x4 in every form factor. The buttons are styled as btn-block so that they are big.

Badges are used to count the number of clicks. The initial count is set to zero in the html, and are incremented every time the button is clicked. It uses javascript to increment the counter.

This has been hosted at: http://209.141.44.113/counter1.html. It doesn't save the results, nor does it provide edit capabilities to write your own categories. The source can be viewed using view-source on your browser.

saved by junit.

I have been helping migrate a java application originally written in 2001 to a new cloud based environment. The application uses a mysql database and needs to be migrated to SQLServer. A contractor was assigned to this effort, and I was to help her out, because I had written code for it in 2001. The code used Torque as an orm, so the initial conversion of the db schema was done using the torque tools.

Once we had installed everything, we asked for some testing. We got a whole lot of issues reported back. Progress was slow, because everything that the user found incorrect stopped the rest of his testing. We were able to isolate the following differences between SQLServer and mysql.

• LONGVARCHAR. In most cases, we could get away with using a regular VARCHAR(4000). In a single case, we needed to make it a VARCHAR(max)
• TINYINT. this is signed on mysql and unsigned on SQLServer.

We also needed to change some of the sql code that we had written to adapt it to SQLServer.

When reviewing the code, I discovered that we also had junit tests, and these tests are comprehensive. So kudos to younger me (and colleague) for having taken time then to do things correctly. Now, i am running the junit tests to figure out if we have other things to worry about. I already found issues with making two fields nullable and fixed them.. Keeping my fingers crossed.

jar versions.

Ran into the dreaded "UnsupportedClassVersionError: Bad version number in .class" exception today when building a cxf based webservice client on java 1.5. Went searching through the jars to find out the jar that had classes for 1.6 and above. Since there were a number of jar files, I decided to write a shell script.

find_version_of_jars.ksh

Strangely, all the jars had a major version of <=49 (java 1.5). 

Tried to figure out which class was being loaded using the "java verbose:class" option, but this just gives you the classes after they have been loaded. The UnsupportedClassVersionError really needs to print out the class name (and file) to make things easier..

I was able to replace cxf with java ws reference implementation. I used their 2.2.6_4 version which said it required java 1.5, but there were two jars in there which had been targetted to 1.6. Using their 2.2.5 version seemed to work (atleast it ran the wsimport without issues..)..