Defining the problem actually takes up a lot of time

I've been flying between Tokyo and Kagoshima for the last month and a half after doing my first 2 month stint in west Japan. I've already manage to take a 1 week process down to 3 days and while allowing 2x more samples to be processed simultaneously, meaning 4x throughput.

By the time I got to Kagoshima, some of  the engineers have been working on figuring out how to get the process to work and determine an algorithm for sample inspection. Armed with the basic information from these guys, I started writing software through the night shift and my work spawned into a project of it's own.

I've already finished writing an application and trained a technician to take over the job. I've started working on the next version of the program to try and have the entire process automated since my program doesn't work in some special cases. Classic example of the 20-80 rule, where 20% of the effort gets 80% of the output.

I am treading into territory where other people haven't gone into with the data analysis and I've noticed that I am spending less and less time writing code and more time trying to better define the problem and the exceptions that my code needs to deal with before writing any code. It's somewhat frustrating to not be writing code to get things done but I guess it's part of the process.


It's odd how the motion of "doing" something is so closely related to progress, though planning and preparation is also a very important part of solving any sort of problem. Being busy for the sake of being busy is detrimental. I generally like disappearing from the office when working on a tough problem that I haven't managed to wrap my head around and go for a coffee. I find that I understand a problem well, I work infinitely faster because everything is in my head and laid out.

There is something called the "drunken sailor's walk" which is a classic problem that nearly all probability courses covers at some point. After n random steps, the furthest you've traveled away from the origin is related to the root on n. If you walk with direction, the distance traveled is n.

In other words, if you plan things out properly and minimize mistakes, you'd be better off by the factor of a square. If you wonder why there can be such a gap in skill or level in the world sometimes, I think that it is the expression of this phenomenon.

For those that have the time, I suggest reading Richard Hamming's book The Art of Doing Science and Engineering: Learning to Learn, which is a fairly ok read. There are some interesting notes from a well known researcher, from which I gleaned this idea from.