Archive for the ‘Quality Engineering’ Category

Quality Engineering in a Scaled Agile Organization

February 4, 2017 Leave a comment

Test Pyramid – test at the earliest and lowest level feasible

Organizational structure is an important part of creating the culture and capability for Quality Engineers to be successful. In this post I share my insights in the journey of evolving the Quality Engineering role of a scaled Agile organization in a SaaS company. The Quality Engineering function is to advocate defect prevention, assure continuous delivery confidence, and advocate for customer satisfaction. Defect prevention is efficiently testing the right thing at the earliest and lowest level feasible (test pyramid). Continuous delivery confidence begins with the Story acceptance criteria, and ends with doneness criteria and automated pipeline success. Customer satisfaction is both meeting all the expectations as core value, and adding more value with delighters that differentiate you from your competition, usually reflected in the business metric NPS (Net Promoter Score).


Follow Conway’s Law


model team and app/test architecture to mimic each other to reduce friction

Model your team organization and your application/test architecture in similar ways to maximize capability and productivity in each…commonly known as Conway’s Law. Shaping engineering culture and software design are two sides of the same coin. They are synergistic patterns that must flow together or friction will emerge in each. As a change agent, realizing this natural tendency is eye opening and makes driving change much more scientific, shaping change as an engineer. Whether you adopt a formal process like SAFe to scale your agile teams or simply looking for ways to continuously improve specific issues, these insights apply.


Organizing Teams, Architecture, and Interfaces

In a modern SaaS (Software as a Service) Scrum Agile organization with more than a couple teams, the teams and the architecture should be based on small cohesive units that operate with common interface standards, and are loosely coupled to the organization they interface with. In monolithic applications, the teams, architecture, and interfaces tend to be more tightly coupled with their consumers, and standards managed within each team. In micro-services applications, the teams, architecture, and interfaces tend to be more loosely coupled with their consumers, and centralized standards managed across teams such that the interfaces are consistent. Most organizations operate somewhere in-between these two polar extremes.

In the past I have inadvertently created friction in one of these areas while attempting to solve problems in another area. I have since realized that both have to change to achieve the aspirational results I was seeking. Making sense of this tendency makes strategic planning much clearer and achievable, and easier to communicate.

Over the last 30 years I’ve worked in many software engineering organizational shapes and sizes, from basement start-up to giants including Eastman Kodak and Agfa Typographic Systems (in their heyday). This notion applies in all cases, but the approach will differ in the size of the organization. As organizations grow, so does the approach needed to move the organization along in the direction you want to take it. In the past 7 years I’ve experienced maturity in moving from the monolith pole towards the distributed pole.

Quality Assurance to Quality Engineering

In 2010 I re-branded Quality Assurance to Quality Engineering at the advice of our VP of Engineering at the time, with the interest of moving the bar on our technical expectations up the Engineering scale significantly, towards defect prevention. This proved to be a very effective idea as we continued to drive test automation out the each of the Scrum teams and improve the technical capability within each of these teams. Looking back over the last 7 years of improvement, we went from ~800 tests per week to over 10,000 tests in about 3 to 4 hours. Through growth and attrition we transformed the make-up of the organization be mostly software engineers with a quality engineering specialty. The regression test automation push swung the pendulum significantly in the direction of automating everything we wanted to assure confidence that working software stays working. So much so, that we then had to reinforce balancing spending time manually working through exploring the customer experience against building/maintaining an adequate regression suite.

During this time we evolved our application architecture to be much more distributed (pulling business logic back to services with RESTful interfaces), expanding application stacks (Java, RoR, JS), and provide shared components. Our test approaches also continue to evolve to align with our application code designs, requiring common continuous integration test infrastructure and reporting services with RESTful interfaces, and test repositories co-existing in application code repositories. We continues to invest in test infrastructure, evolving the functional test automation team (dubbed the “Autobots”) to a “Center of Excellence” (CoE) team, and further spinning off a Performance CoE team.

As we grew in size so did we grow the Quality Engineering leadership into a matrix-ed organization. By the Summer of 2015, my Quality Engineering organization at Constant Contact grew to the size of 50 (in a 200-person engineering and operations organization). This includes 5 Quality Engineering Managers, 2 QE Architects (Functional and Performance engineering), with test automators embedded in 18 product delivery scrum teams, distributed across 5 geographical locations. During this time, we were growing the role of the Scrum Master, limiting the backlog control of Development Managers to enabling the team. There were a lot of efficiencies in Quality Engineering, including a closely nit community of quality engineers. However, there was a growing animosity amongst the Development Managers about silos between Dev and Test roles under different line managers, and for concern of losing control of the work towards self-organizing teams. We knew we needed to change again to improve our efficiencies in product delivery teams.

Continuous Delivery

The notion of DevOps was introduced to us in 2013. Spawned by the novel “The Phoenix Project” authored by Gene Kim, and the book “Continuous Delivery” authored by Jen Humble and David Farley, we charted a direction to improve our efficiencies in how we deliver changes and value to the customer. We were locked into monolithic release cycles for most applications and needed to get to independent release capability. This spawned the creation of a CD team to stitch together a continuous delivery pipeline, including both commercial and internally developed services. Using a pipeline like this requires re-plumbing our applications toward micro-services and static UI application design, changing the way we manage our backlogs and delivery commitments, and how we operate as teams. All changes should be independently testable and deliverable. The vision is to release value to the customer an Agile Story at a time, mid-Sprint with confidence.



automate the product delivery pipeline with validation and test feedback loops as gates


As we looked to reduce to cultural leadership friction in these teams as well, we re-organized the Quality Engineering Organization again. This time re-aligning Quality Engineers embedded in the scrum product delivery teams to report to the Engineering Managers (formerly Development Managers), re-assigning QE Managers to other roles. There was also a conscious movement to reduce number of quality engineers on some teams, in favor of more test development during application development. There was an expectation for software engineers to take more ownership for testing, sharing the test burden across the team. The whole teams is responsible to quality craftsmanship and the quality engineer brings the quality consciousness, test leadership, and customer advocacy.

Preserve the Quality Engineering Community – Guild and CoE

These changes led to the destruction of the QE Community as we knew it, and the Quality Engineers remaining had a sense of abandonment by the Quality Engineering leadership. The Engineering Managers felt the rising animosity about this need for resurrecting the QE Community. To address this, I introduced formation of the QE Guild community of practice, inspired by the Guild approach pioneered by Spotify and encouraged by Janet Gregory and Lisa Crispin in their book “More Agile Testing” under Organizational Culture.  This introduces a loosely grouped community of test-minded engineers focused on career and capability development in the craft of quality engineering, led by a guild coordinator (me as Dir QE for now). We also preserved the CoE (Center of Excellence) “capability radiator” team approach as centralized teams that provide evolving capability to the product delivery teams for the craft of quality engineering.


Quality Engineering Guild is an organic and wide-reaching “community of interest” group of people across the organization that want to share knowledge, tools, code, and practices –  (search Spotify Guilds for more about Guilds)


Continuous Improvement

Reflecting on where we are now and where we go from here, I can tie this back to the notion of Conway’s Law. Our software architecture is driven more tightly around RESTFul Standards, micro-services, single-page GUI’s (CDN edge-cached), and a managed public authentication layer.The Product Manager (owner) organization also now tightly manage an organizational ranked backlog or Epics that direct the priorities and alignment of the agile teams. Scrum teams organize around delivering features with these architectural constructs in mind. Continuous test is now part of delivering each Story and we have a zero-defect policy (issues found during the Story that gate delivery). These teams operate with a cohesive “developer”-“tester”-“product owner” partners, where the whole team owns quality. It’s not all that way yet, but we continue to strive for this culture.

Constant Contact was purchased by Endurance International Group in Feb 2016 with the same mission to improve online marketing success of small businesses. The mission is the same, but the brand organizations are very different. At the time of this writing we are in the midst of aligning the many brands, including the Constant Contact premium email marketing brand. Merging different organizations like this, although at a grander scale, require similar patterns of re-organizing we experienced within the Constant Contact brand and Conway’s Law continues to apply. The product offerings and feature teams may change, but the idea of architecture and team organization beget each other is a truth that leads to greater efficiency in Scaling Agile as the organization grows...tightly integrated teams, loosely coupled architectural layers with strict interface and performance standards, and continuous delivery pipelines with fast feedback loops.

In the end, the customer wins with a stronger company delivering better results faster.


My Biggest Test Mistake

October 13, 2013 Leave a comment

Lesson learned: A one character change can have a very significant impact on quality.

I love to hear and tell stories in illustrating lessons learned. They really convey the emotional impact of the lesson at the time it was learned. This is a story about a very costly mistake when attempting to make a small one character change in application resources that I would never want to repeat again.

In 1993, I was working at Custom Applications on the FreedomOfPress product line. This is a print driver application that provided a software PostScript RIP (raster image processor) for rendering pages to over 150 different printers.  We were releasing a new Mac version for mass distribution to the major software distributors (Ingram, MicroD, etc.). At the time, software was distributed on CD’s from a glass master and silk-screened. This included sending a CD image to the disc duplicator, who created a glass master image, and then stamped out tens of thousands of resin discs. The discs were silk-screened with images and print, and assembled into a printed box along with its printed manual and other material.

Macintosh-centris-660avA blockbuster release of the new Macintosh Centris 660 AV was was the hottest thing in desktop publishing and the CEO had it in his office. We were able to get Apple hardware ahead of distribution to develop and test with the hardware, so we only had one. We had a chance to test with it at times and didn’t have it available to us at all times. As we were preparing for the trade show that Apple was unveiling this new hardware we were received registered trademark ® certificate. Of course we had to have this symbol on our box and on the software! The marketing department was all over this.

We received a request from marketing to add one last change and test, on the day we had to send out the disc image for duplication. This was only a one character change, right? How hard could it be? Our application artwork was created on a Solaris work station and saved to the Mac as a resource. We were asked to add the ® symbol was the splash screen when you start up the application. The text was updated on the Solaris workstation and saved to the Mac (as we normally did, right?). We built and spot checked this new version with the ® symbol proudly displayed on the splash screen on machines we had in the lab. We then sent the new disc image to the duplicators by overnight courier.

The next morning the CEO comes into the office and installs the new version of FreedomOfPress onto the Centris 660AV. He launches the application and the splash screen appears first…the system restarts. What (?!) he says to himself…he thought he must have done something wrong. When the system comes back up he tries again with the same result. He calls me in and explains what he experienced and shows me what is happening. Wow! How did this happen and how did we miss this? Well that machine was nicely tucked into his office when we were testing permutation of machines and printers the day before.

The VP of Engineering was called in and we discussed what might be the issue. After all, we only made one character change from what was running properly on this machine only 24 hours earlier. This is bad…very bad. All the engineers involved in making this changed looked carefully at the code and resources and diff’ed the files. Ah, there is one difference other than the ® symbol. A control+D symbol appears at the end of the text file. This was a normal file termination character for the text file created by the editing application on the Solaris workstation. This control character means something very different on a Mac however, instructing the computer to reset. Ouch! We had to stop the disc duplication immediately and get a new image to the duplicators.

Marketing calls the disc duplicators and they find out that the duplicators had already created the glass masters and proceeded with high speed manufacturing of the disc copies. They had already created thousands of copies. Agh, they had to stop the line and these masters and copies had to be trashed. After removing this control+D character, we spent that day testing through this disc image again, including the Centris 660AV. The new image was sent out to the duplicators overnight again and they rushed through the order. We were able to get the new product boxes to the trade show just in the nick of time.

This was an expensive one character change. As it turns out, there was a corner cut when making this change and the text was not re-editing and saved on the Mac before adding the resource to the bundle. If this step was not skipped, the control-D would not have appeared in the final resource file and this problem would not have been introduced. Also, we should have included the new hardware in the list if things we had to verify.

The real lesson learned here is to pay very close attention to the proven process and not cut any corners when making even simple changes, especially when making a seemly benign change at the last moment.

‡ a.k.a. ColorAge, purchased by Splash in the late 1990’s

Agile Defect Prevention | Part 2

June 24, 2012 2 comments

Continuation of

So you have your escaping defects under control and your team is looking to optimize further to become an elite agile team…what more can you do towards defect prevention? Just as WIP Defects (‘Work In Process’ Defects) are an antidote to Defects, TDD (Test Driven Development) is an antidote to WIP Defects. WIP Defects, although a great way to contain faults from getting into the customer hands, is still an anti-pattern to elite agile teams. These teams are test-infected and write more tests than application code, and catch most problems while authoring the application code.

Fine Craftsmanship

In an elite agile team, everyone writes and executes tests. Product Owners carefully craft use cases with well thought through acceptance criteria, and regularly validates both that the working application behaves as intended and is a delightful experience. Software Engineers write unit tests to assure solid code craftsmanship, performance tests in the sandbox to assure efficient basic application behavior, and manually inspect their user experience like a fine cabinetmaker inspecting the glide action of the drawers in the cabinet he is building. Quality Engineers work closely with the Product Owner to understand the intent of the features, and with the Software Engineer to understand the design and share test approaches, authoring and running tests along the way. Quality Engineers peer inspect tests with Software Engineers so that both are intimately familiar with the tests. Both Software and Quality Engineers regularly run unit and regression tests, and validate both performance and user experience in each context the application runs in.

Consider writing failing test cases instead of WIP Defects as a start down this path. I have found that Quality Engineers who are not use to developing tests early and conducting peer inspections of tests with Software Engineers, are initially uncomfortable by the idea of not writing defects. You need to document the defect somehow. The best documentation is the regression test and not a defect report. This will foster correcting the problem promptly as a failing test instead of a WIP Defect hand-off or scheduling an escaping Defect fix. Get into the habit of running regression tests for the feature being worked on often throughout the day in the sandbox environment, with visibility by the whole team, as part of the CI (Continuous Integration) cycle.

Measure cycle time and efficiency in defect prevention over defect counts. You may still need to track defect counts, fix/find rates, etc. (especially in a larger organization), however in an elite agile team these defect counts are so small that everyone in the team is aware of them. Defect prevention and correction are part of the cycle time to deliver the changes. Take the time to be clear on acceptance criteria, compatibility with standards and architecture, and stability of the code changes. Collaborate within the team such that progress on code design and development, and the tests to validate them, is well known. Elite agile teams write and execute the tests early and often that assure working software remains working.

Even in elite agile teams there are Defects that a found late in the iteration or are a larger problem than can be solved at the time of discovery to fix prior to declaring work done. This warrants creating an escaping Defect. The elite agile team carefully scrutinizes each escaping Defect for defect tolerance of the business and impact to the customer, before opting to let the escaping defect into the field. The intent of the agile life cycle is to add business value often, and in small enough iterations to foster continuous feedback, so it might be more important to deliver the change with the defect than delay delivery. An escaping Defect may survive past an iteration and still be held back to release into the field until fixed in another iteration.

Elite agile teams continually test with constant feedback of pass/fail results. There are no unknown failures left without attention to understanding the problem as it is introduced. Problems are corrected promptly and not let into the field for exposure to customers. The whole team is aware of test status through continuous integration practices. There are very few, if any, known defects in the field.

Quality Engineering Intern/CoOp Position

October 3, 2011 Leave a comment

As a Computer Science major (undergraduate or graduate degree programs), why should I choose Quality Engineering as part of my internship or co-operative work experience? I am studying in a computer science degree program to be a software developer and expect to have a developer position upon graduation from college. This post is a pitch to College Students to consider at least one Quality Engineer internship/CoOp job so that you become a more well rounded developer. You should find the job to be career enriching and offers you another career choice.

Zero BugsQuality Engineering (QE) is the practice of defect prevention (finding defects as the code is developed) as opposed to quality assurance (verification and validation after the code is developed), striving towards zero defects in the field. Quality Engineering is early acceptance test planning in agile product delivery teams, collaborative with developers for both exploratory and automated testing, and centers around continuous integration. Quality Engineering embraces traditional testing techniques of defect discovery and test coverage in parallel with developing features.

Quality Engineering as has come a long way over the years. Testing was once considered a downstream activity in a separate test group that pushed defects into a large backlog for developers to fix in subsequent phases. This was usually made up of minimally technical testers with experience and domain knowledge of the customer’s needs. Test automation in the past was a small group of highly skilled test developers who automated tests written by others, requiring stable code to be worthwhile automating. In contrast, the modern software tester has a technical degree (usually computer science), is embedded in a software development project team, is highly collaborative with the developer counterparts in the team, is skilled in traditional test approaches, is skilled in exploratory and automated tests (creating regression tests along the way), and is the voice of the customer throughout the feature development life cycle. Today both the developer and the tester are engineers working together to deliver working software in small iterations, where software is delivered in small testable chunks.

Quality Engineering is a role in Agile product delivery organizations that:

  • develop and maintain test automation infrastructure and processes
  • develop automated tests (functional, integration, and load/performance) for GUI (graphical user interface) and API (application programming interface) automated tests
  • contribute to continuous integration practices
  • exercise continuous deployment and both regression and exhaustive testing
  • apply customer domain expertise for comprehensive test coverage, and exploratory and fuzz testing
  • validate acceptance criteria that developed and re-factored code meets customer and stakeholder expectations
  • develop and work with test fixtures (software containers to test components in isolation) and mocking (simulated responses from third party and internal integration interfaces)

Quality Engineering role has become increasingly more developer oriented, by participating in:

  • code design and test planning reviews
  • early tests of newly developed code in developer sandboxes
  • develop, review, and use deployment automation scripting (including up-time deployment)
  • develop, review, and use monitoring and up-time reporting tools
  • assess load distribution for high availability, redundancy, and fault tolerance,  in hosted systems
  • test integration with third party services, and their mocks

Quality Engineering is a an integral part of scaling development. Agile development includes designing for re-use, routine re-factoring, unit tests (including mocking and stubbing for negative tests), and building integrated code frequently. Work is broken down into the smallest testable work items, such that development and test is a continuous practice. A combination of unit tests (authored by the Developer) and regression tests (authored by the Quality Engineer) provide a library of automated tests that run often and identify problems that break running software. This library of automated regression tests allows the planned iteration time (to visually test and develop additional automated tests) to remain constant and predictable.

Test Accumulation Graph

Regression Time in this graph is the time it would take to manually exercise regression tests as the code base increases over time.

Without well written regression tests, the amount of time it takes to test increases exponentially with the addition of new working code (Test Accumulation graph). While adding features to a product with each development cycle iteration, the time to test manually increases (usually linearly). Automated regression tests absorb the time to test, enabling the available test time to be focused on testing changes and writing new automated regression tests. The library of automated can be run over and over again frequently to promptly catch problem regressions (code or configuration changes that result in breaking another part of the system). Therefore, agile product delivery teams rely on the accumulation of tests as part of delivering working software to the customer.

Having experience working as a Quality Engineer will provide you with a deep understanding of core test techniques, a test perspective on continuous integration and continuous delivery approaches, a sense of what acceptance criteria means to consider development tasks done, and designing software applications for test-ability. CI and CD drive defect prevention, assuring that working software remains working. Quality Engineering is a different kind of developer and knowing this role well will help you work collaboratively with quality engineers while working as a development engineer in the future. You may also find that Quality Engineering is a career path that interests you more than Developer Engineering.

Constant Contact Quality Engineering is organized into roles of QE Architecture, a specialized test team as a center of excellence in test automation, and embedded quality engineers on each agile product delivery team. There is typically a 3:1 ration of developers to quality engineers whom operate collaboratively with developers to deliver new and re-factored infrastructure and customer facing features. Most of our Quality Engineers have a degree in computer science (or related major), many have an MSCS, and we even have a PhD in the team. Consider joining our Constant Contact QE organization for one or more of your internship or co-op work sessions. This experience will contribute to building a well-rounded developer education and a chance to learn the other side of delivery software.

Agile Defect Prevention

September 23, 2011 1 comment

I recall a day in the late ’90’s when assessing readiness for deployment of an application at Kodak after a several month long release cycle having 3,000 deferred defects. WOW! I can’t believe that was acceptable at that time, but in long waterfall release cycles that was the norm at the time. How can you manage defects like this? Today, this is unacceptable. The idea of “deferred defects” has always bothered me in software development. So, what can we do about this?

No BugsAlong comes agile software development cycles where a defect backlog is an anti-pattern (No Bugs). The idea is that through continuous integration, unit tests, early inspection, and regression tests, your team finds problems as they are introduced. This is great in theory, but how do we manage the inevitable defects that we can’t get to and is an acceptable risk to meeting business requirements, and those defects that will come in from the field as support requests to fix? My approach to managing this is to focus on “defect prevention” as opposed to “defect tracking.”

Defect prevention, really? …is that possible? Imagine counting defects on your two hands. constraining escaping defects to what fits on your two handsHow can we accomplish this? Efficient agile organizations focus on defect prevention rather than downstream defect discovery. A culture of defect prevention includes separating “work in process”  defects (WIP Defect) from “escaping” defects (Defect) to minimize defect management that escape beyond the sprint that features are developed in. This results in a much smaller defect backlog to manage and dramatically increased customer satisfaction. Agile is not just about releasing more often, but also with complete and tested features. So, we need to treat defects found in development as actionable sub-tasks of the feature work item. If we treat these WIP Defets as sub-tasks and acceptance criteria of completing the development tasks, then we are not introducing them to the field and not adding to the project team backlog as technical debt.

Escaping defects should then be treated as ranked backlog work items, along with other project work items. They should be prioritized high enough to resolve them within the next sprint or two and not accumulate a growing backlog. Watch the defect backlog as part of the project metrics. A growing defect backlog is a key indicator that the team is taking on more new work than it can handle. It may also be a key indicator that the team is operating as a “mini-waterfall” project, rather than a agile project, requiring more collaboration between Dev and Quality Engineers and early testing. Drop the number of new items the team works on until the escaping defects are well managed or eliminated.

When a WIP Defect must exist past the completion of the parent development task, then promote it to a Defect and place it in the backlog in rank order with other work. However, whenever possible the team should heavily scrutinize this practice and opt to hold delivery of the feature until the WIP Defects are complete. Also, a Defect in the backlog could be demoted and attached to an active development task to include it in the acceptance criteria for that task.

At Constant Contact, we now have our defects in the same tool (Jira/GreenHopper) that we manage new feature work, such that defects are in the same project and iteration backlogs. This provides greater visibility to the product owner ranking the work and the team implementing the work.

Hiring Quality Engineers for Scrum Teams

December 14, 2007 Leave a comment

I have frequently been asked “how do you go about hiring good Quality Engineers for an Agile team?” I really feel strongly that there are several proficiencies that needed to round out a strong test group for agile teams. These proficiencies can be enbodied in more than one person, but rarely is any one Engineer proficient in all of these disciplines.

  • Domain Knowledge:
  • As with any Quality Assurance (Engineering) team, there needs to be enough representation of the customer for each actor (user) of the system. This requires insights into the goals and intended use of of the system, and most likely may be someone hired from the user community.

  • Development Prociency:
    • Unit Tests and representative Acceptance Tests in an XUnit Framework — This is a tennant capability for working with the developer on a story to write story acceptance tests first (or during the story development).
    • Separation of GUI/Presentation tier from Business Logic tier — This is key to being able to exercise the business logic (where most of the work is accomplished) in tests that reside with the source of that production code. There is no need to drive these test with the overhead of the client presentation tiers involved. The client and presentation tiers need their own representative acceptance test for regression, and exhaustive tests (e.g. different environments).
    • Exhaustive Tests through data driven tables — There are several approaches and tools for exercising acceptance tests with many combinations of data sets (e.g. FitNesse or Solenium tables, xml, csv, etc.)
    • Test Fixtures — This is key to exposing all of the system for comprehensive story acceptance tests. As a company matures with agile, I prefer to create a Test Automation Architect role who is responsible for oversight and creation of enough test fixture infrastructure to enable the team to develop comprehensive regression test coverage that can run following build and unit tests completion daily.
  • Traditional Tests
    • Performance Test Automation — This includes both local and load tests as found in traditional waterfall and other iterative SDL’s (software Development Lifecycles).
    • Manual Tests — There are some test that are purely asthetic or user experience oriented to assure that the application is pleasing to use and accomplishes all the user tasks and goals of each actor identified for the system.
  • Process Engineering:
  • This is discipline of best practices and well defined procedures that need to be in place for consistency across the team, and to maintain a project cadence. As with software, process can be over-engineered and requires a sensitivity to what is working efficiently, when to turn up the process to improve consistency and predictability, and when to turn down the process to eliminate unnecessary overhead.

The question that usually follows this list is “how do you find a Quality Engineer with a Development Proficiency?” There are talented Quality Engineers with CS and MSCS degrees with years of experience in both development and test engineering. If you can’t find them, create them. Find a developer who has an affinity for methodical test coverage in their own work, or who is good at and enjoys testing their own work to step in to a Quality Engineering role. I have been able to convert a few developers to this role and the results are fantastic.

%d bloggers like this: