Archive

Archive for June, 2010

Functional Continuous Integration

Driving agile practices over the last 4 years in 3 SAAS companies, it is quite apparent to me that continuous integration (CI) requires both unit test build failure verification and regular deployed functional regression test verification to be really agile. Yes, you need all the SDL (software development lifecycle) practices to manage building the right product and completing work items, but quality of work cannot be compromised in the name of speed. Agile is all about completing small amounts of working (deliverable) software and iterating on continuous feedback. It also includes confidence that you are delivering tested software without regression defects (not breaking what already worked) and confidence future work will not break what was just delivered. Any remaining tests not completed in the scope of work items is considered technical debt. This technical debt is postponed work that results in missed defects.

Quality confidence is achieved by routinely running automated tests at both the code and system levels. Regardless of the agile practices used, design, development and test are interwoven and requires collabration of development and test resources in the delivery team. I believe this is the secret sause that differentiates a waterfall-ish team and an agile-ish team.

  • The waterfall-ish team has the mind-set of develop application code first and develop automation test code later, frequently not including testing in the work item scope.
  • The agile-ish team has the mind-set of developing both unit test and functional test code along with application code, either prior to application code (Test Driven Development) or just after application code, but within the scope of the work item. This includes meeting work item (e.g. user story) acceptance criteria.

Functional Continous Integration (FCI) is continuously creating and updating automated regression tests, and must be the expectation for PO’s when planning work commitments, Executives when assessing progress reporting, Developers including collaboration with QE in their estimating, and Quality Engineers in planning and completing test work. Infrastructure for FCI needs to include integration of automated tests with a test management and reporting database, and needs to be capable of running unattended. I’ve used SeleniumRC with both CruiseControl with Rails test scripts and Hudson with Java test scripts to run build-time deploy and unattended test runs. These CI applications can run with multiple client machines as slaves. This allows CI jobs to run each test suite on a different client machine simultaneaously to speed up the test duration.

The result is that test failures due to problems that break existing code, introduced with changes or new application code, are caught very early and corrected. Further, if these tests are run in the Developer’s sandbox and corrected prior to check-in, there is no defect created, which results in significantly reducing defect counts for the agile team.

Advertisements
%d bloggers like this: