So you have your escaping defects under control and your team is looking to optimize further to become an elite agile team…what more can you do towards defect prevention? Just as WIP Defects (‘Work In Process’ Defects) are an antidote to Defects, TDD (Test Driven Development) is an antidote to WIP Defects. WIP Defects, although a great way to contain faults from getting into the customer hands, is still an anti-pattern to elite agile teams. These teams are test-infected and write more tests than application code, and catch most problems while authoring the application code.
In an elite agile team, everyone writes and executes tests. Product Owners carefully craft use cases with well thought through acceptance criteria, and regularly validates both that the working application behaves as intended and is a delightful experience. Software Engineers write unit tests to assure solid code craftsmanship, performance tests in the sandbox to assure efficient basic application behavior, and manually inspect their user experience like a fine cabinetmaker inspecting the glide action of the drawers in the cabinet he is building. Quality Engineers work closely with the Product Owner to understand the intent of the features, and with the Software Engineer to understand the design and share test approaches, authoring and running tests along the way. Quality Engineers peer inspect tests with Software Engineers so that both are intimately familiar with the tests. Both Software and Quality Engineers regularly run unit and regression tests, and validate both performance and user experience in each context the application runs in.
Consider writing failing test cases instead of WIP Defects as a start down this path. I have found that Quality Engineers who are not use to developing tests early and conducting peer inspections of tests with Software Engineers, are initially uncomfortable by the idea of not writing defects. You need to document the defect somehow. The best documentation is the regression test and not a defect report. This will foster correcting the problem promptly as a failing test instead of a WIP Defect hand-off or scheduling an escaping Defect fix. Get into the habit of running regression tests for the feature being worked on often throughout the day in the sandbox environment, with visibility by the whole team, as part of the CI (Continuous Integration) cycle.
Measure cycle time and efficiency in defect prevention over defect counts. You may still need to track defect counts, fix/find rates, etc. (especially in a larger organization), however in an elite agile team these defect counts are so small that everyone in the team is aware of them. Defect prevention and correction are part of the cycle time to deliver the changes. Take the time to be clear on acceptance criteria, compatibility with standards and architecture, and stability of the code changes. Collaborate within the team such that progress on code design and development, and the tests to validate them, is well known. Elite agile teams write and execute the tests early and often that assure working software remains working.
Even in elite agile teams there are Defects that a found late in the iteration or are a larger problem than can be solved at the time of discovery to fix prior to declaring work done. This warrants creating an escaping Defect. The elite agile team carefully scrutinizes each escaping Defect for defect tolerance of the business and impact to the customer, before opting to let the escaping defect into the field. The intent of the agile life cycle is to add business value often, and in small enough iterations to foster continuous feedback, so it might be more important to deliver the change with the defect than delay delivery. An escaping Defect may survive past an iteration and still be held back to release into the field until fixed in another iteration.
Elite agile teams continually test with constant feedback of pass/fail results. There are no unknown failures left without attention to understanding the problem as it is introduced. Problems are corrected promptly and not let into the field for exposure to customers. The whole team is aware of test status through continuous integration practices. There are very few, if any, known defects in the field.
In this video, you’ll learn how Appfire worked with Constant Contact to help them map-out and execute a large-scale migration from competitive tools (including Rally and IBM’s ClearQuest) to Atlassian’s JIRA and GreenHopper. You’ll hear about Contant Contact’s organization-wide adoption of Agile, from their early days of Scrum, to trials of Kanban, to their now current deployment of Scrum-ban. Learn why Constant Contact selected Appfire and Atlassian, and why they recommend other enterprise organizations do as well.
As a Computer Science major (undergraduate or graduate degree programs), why should I choose Quality Engineering as part of my internship or co-operative work experience? I am studying in a computer science degree program to be a software developer and expect to have a developer position upon graduation from college. This post is a pitch to College Students to consider at least one Quality Engineer internship/CoOp job so that you become a more well rounded developer. You should find the job to be career enriching and offers you another career choice.
Quality Engineering (QE) is the practice of defect prevention (finding defects as the code is developed) as opposed to quality assurance (verification and validation after the code is developed), striving towards zero defects in the field. Quality Engineering is early acceptance test planning in agile product delivery teams, collaborative with developers for both exploratory and automated testing, and centers around continuous integration. Quality Engineering embraces traditional testing techniques of defect discovery and test coverage in parallel with developing features.
Quality Engineering as has come a long way over the years. Testing was once considered a downstream activity in a separate test group that pushed defects into a large backlog for developers to fix in subsequent phases. This was usually made up of minimally technical testers with experience and domain knowledge of the customer’s needs. Test automation in the past was a small group of highly skilled test developers who automated tests written by others, requiring stable code to be worthwhile automating. In contrast, the modern software tester has a technical degree (usually computer science), is embedded in a software development project team, is highly collaborative with the developer counterparts in the team, is skilled in traditional test approaches, is skilled in exploratory and automated tests (creating regression tests along the way), and is the voice of the customer throughout the feature development life cycle. Today both the developer and the tester are engineers working together to deliver working software in small iterations, where software is delivered in small testable chunks.
Quality Engineering is a role in Agile product delivery organizations that:
- develop and maintain test automation infrastructure and processes
- develop automated tests (functional, integration, and load/performance) for GUI (graphical user interface) and API (application programming interface) automated tests
- contribute to continuous integration practices
- exercise continuous deployment and both regression and exhaustive testing
- apply customer domain expertise for comprehensive test coverage, and exploratory and fuzz testing
- validate acceptance criteria that developed and re-factored code meets customer and stakeholder expectations
- develop and work with test fixtures (software containers to test components in isolation) and mocking (simulated responses from third party and internal integration interfaces)
Quality Engineering role has become increasingly more developer oriented, by participating in:
- code design and test planning reviews
- early tests of newly developed code in developer sandboxes
- develop, review, and use deployment automation scripting (including up-time deployment)
- develop, review, and use monitoring and up-time reporting tools
- assess load distribution for high availability, redundancy, and fault tolerance, in hosted systems
- test integration with third party services, and their mocks
Quality Engineering is a an integral part of scaling development. Agile development includes designing for re-use, routine re-factoring, unit tests (including mocking and stubbing for negative tests), and building integrated code frequently. Work is broken down into the smallest testable work items, such that development and test is a continuous practice. A combination of unit tests (authored by the Developer) and regression tests (authored by the Quality Engineer) provide a library of automated tests that run often and identify problems that break running software. This library of automated regression tests allows the planned iteration time (to visually test and develop additional automated tests) to remain constant and predictable.
Without well written regression tests, the amount of time it takes to test increases exponentially with the addition of new working code (Test Accumulation graph). While adding features to a product with each development cycle iteration, the time to test manually increases (usually linearly). Automated regression tests absorb the time to test, enabling the available test time to be focused on testing changes and writing new automated regression tests. The library of automated can be run over and over again frequently to promptly catch problem regressions (code or configuration changes that result in breaking another part of the system). Therefore, agile product delivery teams rely on the accumulation of tests as part of delivering working software to the customer.
Having experience working as a Quality Engineer will provide you with a deep understanding of core test techniques, a test perspective on continuous integration and continuous delivery approaches, a sense of what acceptance criteria means to consider development tasks done, and designing software applications for test-ability. CI and CD drive defect prevention, assuring that working software remains working. Quality Engineering is a different kind of developer and knowing this role well will help you work collaboratively with quality engineers while working as a development engineer in the future. You may also find that Quality Engineering is a career path that interests you more than Developer Engineering.
Constant Contact Quality Engineering is organized into roles of QE Architecture, a specialized test team as a center of excellence in test automation, and embedded quality engineers on each agile product delivery team. There is typically a 3:1 ration of developers to quality engineers whom operate collaboratively with developers to deliver new and re-factored infrastructure and customer facing features. Most of our Quality Engineers have a degree in computer science (or related major), many have an MSCS, and we even have a PhD in the team. Consider joining our Constant Contact QE organization for one or more of your internship or co-op work sessions. This experience will contribute to building a well-rounded developer education and a chance to learn the other side of delivery software.
I recall a day in the late ’90′s when assessing readiness for deployment of an application at Kodak after a several month long release cycle having 3,000 deferred defects. WOW! I can’t believe that was acceptable at that time, but in long waterfall release cycles that was the norm at the time. How can you manage defects like this? Today, this is unacceptable. The idea of “deferred defects” has always bothered me in software development. So, what can we do about this?
Along comes agile software development cycles where a defect backlog is an anti-pattern (No Bugs). The idea is that through continuous integration, unit tests, early inspection, and regression tests, your team finds problems as they are introduced. This is great in theory, but how do we manage the inevitable defects that we can’t get to and is an acceptable risk to meeting business requirements, and those defects that will come in from the field as support requests to fix? My approach to managing this is to focus on “defect prevention” as opposed to “defect tracking.”
Defect prevention, really? …is that possible? Imagine counting defects on your two hands. How can we accomplish this? Efficient agile organizations focus on defect prevention rather than downstream defect discovery. A culture of defect prevention includes separating “work in process” defects (WIP Defect) from “escaping” defects (Defect) to minimize defect management that escape beyond the sprint that features are developed in. This results in a much smaller defect backlog to manage and dramatically increased customer satisfaction. Agile is not just about releasing more often, but also with complete and tested features. So, we need to treat defects found in development as actionable sub-tasks of the feature work item. If we treat these WIP Defets as sub-tasks and acceptance criteria of completing the development tasks, then we are not introducing them to the field and not adding to the project team backlog as technical debt.
Escaping defects should then be treated as ranked backlog work items, along with other project work items. They should be prioritized high enough to resolve them within the next sprint or two and not accumulate a growing backlog. Watch the defect backlog as part of the project metrics. A growing defect backlog is a key indicator that the team is taking on more new work than it can handle. It may also be a key indicator that the team is operating as a “mini-waterfall” project, rather than a agile project, requiring more collaboration between Dev and Quality Engineers and early testing. Drop the number of new items the team works on until the escaping defects are well managed or eliminated.
When a WIP Defect must exist past the completion of the parent development task, then promote it to a Defect and place it in the backlog in rank order with other work. However, whenever possible the team should heavily scrutinize this practice and opt to hold delivery of the feature until the WIP Defects are complete. Also, a Defect in the backlog could be demoted and attached to an active development task to include it in the acceptance criteria for that task.
At Constant Contact, we now have our defects in the same tool (Jira/GreenHopper) that we manage new feature work, such that defects are in the same project and iteration backlogs. This provides greater visibility to the product owner ranking the work and the team implementing the work.
Scrum works well to well when there is a distinct backlog of prioritized work and distinct time-boxed release cycles. Scrum scales well with a Scrum-of-Scrums group where scrum masters scrum across teams, provided you introduce program management to oversee the progress across teams. As an organization grows in agile product teams that align with a single delivery cycle, agile service teams emerge (e.g. Engineering Services, Database, Application Operations, automation infrastructure, etc.). Many of these teams have both planned backlog work and on-demand unplanned work (work request tickets).
Scrum is a push model where work is pushed into the team in prioritized order. Many times work on an item may be blocked by a dependency on another team, so the team picks another work item to work on, and so on. This leads to many things started without being completed and accepted in an consistent flow throughout the length of the sprint. The result is a lot of task switching and inefficiencies near the end of the sprint to get the work done in the sprint, and technical debt (deferred defects, refactoring for reuse, etc.). This is especially difficult when you have many interdependencies between scrum teams.
There are three key challenges that emerge as you scale these needs:
- minimizing work in process (WIP) with smaller work items
- balance the planned and unplanned work and their priorities
- minimize bottlenecks in the flow of work
Kanban is another agile concept borrowed from the lean principles of the manufacturing industry in Japan. Kanban (or kamban in Hepburn romanization–kanji 看板, katakana カンバン, meaning “signboard” or “billboard”) is a concept related to lean and just-in-time (JIT) production. According to Taiichi Ohno, the man credited with developing JIT, kanban is a means through which JIT is achieved (en.wikipedia.org/wiki/Kanban). Kanban is a pull model, rather than the push model of Scrum. It uses the same idea of a user story broken into work items with tasks attached. It uses the card (sticky note) concept as in Scrum, but it uses the idea of clarifying distinct process steps (columns) and swim lanes (rows), where each step has a distinct WIP limit. You still have a prioritized backlog to pull work from. Each swim lane has a purpose for processing work, and each step can only pull work from the previous step once a work item or task is pulled from the downstream step. If the downstream step does not have capacity you can’t push the work to them. This leads to tuning the efficiencies and capacities in each step to accommodate a cadence of pulling work through the swim lanes.
Constant Contact (CTCT) is moving from Scrum to Kanban to address these needs. Mike Fitterman (Development Manager) and Rick Simmons (Agile Coach) illustrate the learning of the pilot team (website) at Constant Contact, as presented at the Agile2010 conference: AgileConference2010-Upstream_Kanban_at_CTCT. At Constant Contact, we have nearly all (both product delivery and service) agile teams using this approach at the time of this writing. You can see Kanban boards in many of our conference room and work area walls as you walk through the engineering space. The Scrum-of-Scrums still occurs daily, however it is more focused on larger project progress, interdependencies, and organizational impediments. We are still working out our inefficiencies with this new process, but it seems to be working well for us so far.
Kanban addresses the need to minimize the WIP by constraining each step in each swim lane. It allows for an express lane for high priority on-demand unplanned work (defects and footprint tickets). It makes inefficiencies and bottlenecks apparent to the team to self-correct and tune how the work is done at each step (colored sticky notes and visible policies). Overall it smooths out the cadence of work to keep work flowing through the team with visible status of all work items to the whole team.
Visit agilemanagement.net to explore more about Kanban agile practices, references, training with David Anderson, founder of Kanban as a software development practice.
What if we could apply the zero-defects vision of highly efficient Continuous Integration to our highways? We could then travel our highways at the full speed limit, at a sustained pace during rush hour. We would not have to expand our capacity of the highways nor extend our travel time with travel debt that eats into our private lives. Well, it appears Google is taking a crack at it (http://bit.ly/90RF3Q).
Driving agile practices over the last 4 years in 3 SAAS companies, it is quite apparent to me that continuous integration (CI) requires both unit test build failure verification and regular deployed functional regression test verification to be really agile. Yes, you need all the SDL (software development lifecycle) practices to manage building the right product and completing work items, but quality of work cannot be compromised in the name of speed. Agile is all about completing small amounts of working (deliverable) software and iterating on continuous feedback. It also includes confidence that you are delivering tested software without regression defects (not breaking what already worked) and confidence future work will not break what was just delivered. Any remaining tests not completed in the scope of work items is considered technical debt. This technical debt is postponed work that results in missed defects.
Quality confidence is achieved by routinely running automated tests at both the code and system levels. Regardless of the agile practices used, design, development and test are interwoven and requires collabration of development and test resources in the delivery team. I believe this is the secret sause that differentiates a waterfall-ish team and an agile-ish team.
- The waterfall-ish team has the mind-set of develop application code first and develop automation test code later, frequently not including testing in the work item scope.
- The agile-ish team has the mind-set of developing both unit test and functional test code along with application code, either prior to application code (Test Driven Development) or just after application code, but within the scope of the work item. This includes meeting work item (e.g. user story) acceptance criteria.
Functional Continous Integration (FCI) is continuously creating and updating automated regression tests, and must be the expectation for PO’s when planning work commitments, Executives when assessing progress reporting, Developers including collaboration with QE in their estimating, and Quality Engineers in planning and completing test work. Infrastructure for FCI needs to include integration of automated tests with a test management and reporting database, and needs to be capable of running unattended. I’ve used SeleniumRC with both CruiseControl with Rails test scripts and Hudson with Java test scripts to run build-time deploy and unattended test runs. These CI applications can run with multiple client machines as slaves. This allows CI jobs to run each test suite on a different client machine simultaneaously to speed up the test duration.
The result is that test failures due to problems that break existing code, introduced with changes or new application code, are caught very early and corrected. Further, if these tests are run in the Developer’s sandbox and corrected prior to check-in, there is no defect created, which results in significantly reducing defect counts for the agile team.