We spent 10 times as we might have estimated!

Today Rock and I continued the work that should have been done on last Friday. There is a task on CIY requiring that when a teacher creates an assignment, the system should send email to every student of that course. Nate had created a skeleton of sending email for us. We were expected to spend no more 2 pairing hours on this. However, it did not seem so easy when we got down to work.

On last Friday afternoon we spent two hours on researching this part of the system, and how to send a email using Ruby on Rails, only to find that there exited some problem with database fields. What we could get is an assignment_id, with which we had to look into the database and find the email address of the students who chose that course. Unfortunately, the database was not set up as was design on ERD. The assignments table stored a field teacher_id which should have been course_id, while the students table stored a field class_id which should be course_id. Then we realized that we should use migration to modify the database before we can start to work on the mailing job.

So we started today by creating the migration to modify the database. However, when we update the local project from SVN, there were failed tests. We tried very hard to fix that failure until we found it out of our capabilities and responsibilities half an hour later. Then we turned Nate, finally finding that it was a test that had been committed before passed. After Nate fixed that failure, we came back to our machine and update again, ran all tests, it had been 9:30! We thus wasted more than an hour today just because a failed test not caused by us.

Modifying the schema of a database is a radical change; it may shake the foundation of the system. Rock and I had experienced such a nightmare on last Thursday evening when we were asked to change a field ideal_hour to ideal_day and divide the original data by 8. In that evening, we started to work in 7:30. It seemed simple initially. After spent some time checking out and passing all test, we start to change the database. Of course a lot of tests could not pass. Then we began to spend frustrating hours in fixing codes in order to pass those tests, until it was nearly 12 that night when we realized it impossible to fix those failure in a couple of hours. So we decided to roll back what we had done that evening. When we continued to work for more than half an hour, we thought that we had rolled back. Since run all the tests again may take a long time, Rock said I could leave for bed first, he could handle the rest. Not do I know the next morning that he worked until 2:30 a.m. just to make all test pass again!

So this time we were both taught that it would never be too cautious in making change to database schema, and the estimation time of that work would never too long. Before we did the migration, we decided to make a new branch in the SVN, because we did not think that we could fix in a single day all the failures of tests that were caused by changing the schema. Before we made a new branch, we had to pass all the tests and commit what we had done to the trunk. Then we could made a branch from the trunk. Then we could work on that branch.

We proceeded so smoothly this time. Rock even doubted whether there were still incorrect logics that could not figure out by passing all the tests. It is possible, but it is a test coverage issue, we were not able to do anything to it then. So we could not but trust the tests. Then it came to merge the branch to the trunk. To did this, first we had to commit what we had done to the branch, then we checked out from the trunk to another project, say admin_trunk. And then we merged the branch to admin_trunk, after resolving all the conflicts and fixing all failures of tests, we finally committed it to the trunk. It had been 3:30 then. If we started from over eight this morning, apart from an hour of meeting this morning, we had spent nearly 5 hours on it. Imaging we just modify two database fields, if we had been asked to estimate how long it would take, we might say at most 30 minutes pairing. It turned out to be 10 times we might estimate. In retrospect of how we spent such a long time, writing that two migration files only took us less than 30 minutes; we spent the rest of the time fixing the unpassed tests that were not written by us, waiting for SVN check and out, waiting when the tests were running. How to reduce such overhead time, or how to estimate such time during planning meeting, it is a question.