So this is twice that I have been upstaged by my lack of Google skills. I have some small background in compilers, languages close to the processor, and optimization. So when I see something obviously lacking in even rudimentary speed-ups, I think about taking a look and doing a better job myself. Usually I restrain myself otherwise I’d spend all my time re-writing Matlab to support NOT ridiculously slow for loops (please please please Mathworks, can’t you hire a decent compilers person or two and at least get your for loops to run faster than QBasic?). But sometimes, when running batch processes takes days, I start getting involved.<\/p>\n
My first attempt was to make some Chi-Squared distance metric Matlab scripts run faster – a friend needed to run this over gigabytes of data, which was going to take months using his simple Matlab script. Months is bad so I sat down with him and we optimized the Matlab scripts, then moved to C with a mex interface, then delved into assembly language, and then into the SSE2 optimizations. In the end we had something that was an order of a magnitude faster, so it would only take several days to run. Of course, a few days later I found a guy at Google who had done the exact same thing<\/a>. And oh look now that I’m trying to find a link to his website, I’ve found somebody else with the similar code with Matlab wrappers<\/a>. Sigh…that was a wasted night and a half. Coulda gone to sleep at a decent hour instead of going home at 7 am after working all night.<\/p>\n And the story doesn’t end there! When running some of my algorithms, I was seeing rather slow JPEG decoding speeds so I went out searching for faster implementations. I found the libjpeg-SIMD version<\/a>, but it was only 32-bit and since it was coded in assembly language, porting it to 64-bit would be non-trivial (I only use 64-bit for my research work so I can access more than 3 gb of RAM). I figured somebody must have done it somewhere on the Internet, but nope, the I couldn’t find anything. I even spent an extra day just researching this to prevent the same thing that happened with the Chi-Squared code from biting me once again. The closest I could find was some discussion on tightvnc-devel about attempting to port it to 64-bit,<\/a> but it didn’t seem like anything came out of it (as Internet discussions have a tendency to do, the threads degenerated into open-source vs. Microsoft<\/a> and cygwin vs. Mingw). So I spent a good 4-5 nights porting the key bits of libjpeg SIMD to 64-bit one frightful file at a time, as detailed in a previous blog post<\/a>. And of course it turns out I again duplicated somebody else’s work. One of the developers of TightVNC posted a comment<\/a> to my blog entry announcing the completion of the port<\/a> informing me that they had indeed successfully ported libjeg SIMD to 64-bit a while back and it was in their SVN. Sigh…on one hand that’s really discouraging because well that is a good 25 or so hours I could have spent doing better things (like sleeping!), but on the other hand I did get a brush up on assembly language and especially some of the new 64-bit assembly language extensions.<\/p>\n So moral of the story is search hard and long so you don’t have to re-invent the wheel.<\/p>\n","protected":false},"excerpt":{"rendered":" So this is twice that I have been upstaged by my lack of Google skills. I have some small background…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,9,11],"tags":[],"class_list":["post-485","post","type-post","status-publish","format-standard","hentry","category-cc","category-code","category-research"],"_links":{"self":[{"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/posts\/485","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/comments?post=485"}],"version-history":[{"count":3,"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/posts\/485\/revisions"}],"predecessor-version":[{"id":488,"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/posts\/485\/revisions\/488"}],"wp:attachment":[{"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/media?parent=485"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/categories?post=485"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.briancbecker.com\/blog\/wp-json\/wp\/v2\/tags?post=485"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}