Home > Agile, Lessons Learned, Quality Engineering > Quality Engineering in a Scaled Agile Organization

Quality Engineering in a Scaled Agile Organization


Test Pyramid – test at the earliest and lowest level feasible

Organizational structure is an important part of creating the culture and capability for Quality Engineers to be successful. In this post I share my insights in the journey of evolving the Quality Engineering role of a scaled Agile organization in a SaaS company. The Quality Engineering function is to advocate defect prevention, assure continuous delivery confidence, and advocate for customer satisfaction. Defect prevention is efficiently testing the right thing at the earliest and lowest level feasible (test pyramid). Continuous delivery confidence begins with the Story acceptance criteria, and ends with doneness criteria and automated pipeline success. Customer satisfaction is both meeting all the expectations as core value, and adding more value with delighters that differentiate you from your competition, usually reflected in the business metric NPS (Net Promoter Score).


Follow Conway’s Law


model team and app/test architecture to mimic each other to reduce friction

Model your team organization and your application/test architecture in similar ways to maximize capability and productivity in each…commonly known as Conway’s Law. Shaping engineering culture and software design are two sides of the same coin. They are synergistic patterns that must flow together or friction will emerge in each. As a change agent, realizing this natural tendency is eye opening and makes driving change much more scientific, shaping change as an engineer. Whether you adopt a formal process like SAFe to scale your agile teams or simply looking for ways to continuously improve specific issues, these insights apply.


Organizing Teams, Architecture, and Interfaces

In a modern SaaS (Software as a Service) Scrum Agile organization with more than a couple teams, the teams and the architecture should be based on small cohesive units that operate with common interface standards, and are loosely coupled to the organization they interface with. In monolithic applications, the teams, architecture, and interfaces tend to be more tightly coupled with their consumers, and standards managed within each team. In micro-services applications, the teams, architecture, and interfaces tend to be more loosely coupled with their consumers, and centralized standards managed across teams such that the interfaces are consistent. Most organizations operate somewhere in-between these two polar extremes.

In the past I have inadvertently created friction in one of these areas while attempting to solve problems in another area. I have since realized that both have to change to achieve the aspirational results I was seeking. Making sense of this tendency makes strategic planning much clearer and achievable, and easier to communicate.

Over the last 30 years I’ve worked in many software engineering organizational shapes and sizes, from basement start-up to giants including Eastman Kodak and Agfa Typographic Systems (in their heyday). This notion applies in all cases, but the approach will differ in the size of the organization. As organizations grow, so does the approach needed to move the organization along in the direction you want to take it. In the past 7 years I’ve experienced maturity in moving from the monolith pole towards the distributed pole.

Quality Assurance to Quality Engineering

In 2010 I re-branded Quality Assurance to Quality Engineering at the advice of our VP of Engineering at the time, with the interest of moving the bar on our technical expectations up the Engineering scale significantly, towards defect prevention. This proved to be a very effective idea as we continued to drive test automation out the each of the Scrum teams and improve the technical capability within each of these teams. Looking back over the last 7 years of improvement, we went from ~800 tests per week to over 10,000 tests in about 3 to 4 hours. Through growth and attrition we transformed the make-up of the organization be mostly software engineers with a quality engineering specialty. The regression test automation push swung the pendulum significantly in the direction of automating everything we wanted to assure confidence that working software stays working. So much so, that we then had to reinforce balancing spending time manually working through exploring the customer experience against building/maintaining an adequate regression suite.

During this time we evolved our application architecture to be much more distributed (pulling business logic back to services with RESTful interfaces), expanding application stacks (Java, RoR, JS), and provide shared components. Our test approaches also continue to evolve to align with our application code designs, requiring common continuous integration test infrastructure and reporting services with RESTful interfaces, and test repositories co-existing in application code repositories. We continues to invest in test infrastructure, evolving the functional test automation team (dubbed the “Autobots”) to a “Center of Excellence” (CoE) team, and further spinning off a Performance CoE team.

As we grew in size so did we grow the Quality Engineering leadership into a matrix-ed organization. By the Summer of 2015, my Quality Engineering organization at Constant Contact grew to the size of 50 (in a 200-person engineering and operations organization). This includes 5 Quality Engineering Managers, 2 QE Architects (Functional and Performance engineering), with test automators embedded in 18 product delivery scrum teams, distributed across 5 geographical locations. During this time, we were growing the role of the Scrum Master, limiting the backlog control of Development Managers to enabling the team. There were a lot of efficiencies in Quality Engineering, including a closely nit community of quality engineers. However, there was a growing animosity amongst the Development Managers about silos between Dev and Test roles under different line managers, and for concern of losing control of the work towards self-organizing teams. We knew we needed to change again to improve our efficiencies in product delivery teams.

Continuous Delivery

The notion of DevOps was introduced to us in 2013. Spawned by the novel “The Phoenix Project” authored by Gene Kim, and the book “Continuous Delivery” authored by Jen Humble and David Farley, we charted a direction to improve our efficiencies in how we deliver changes and value to the customer. We were locked into monolithic release cycles for most applications and needed to get to independent release capability. This spawned the creation of a CD team to stitch together a continuous delivery pipeline, including both commercial and internally developed services. Using a pipeline like this requires re-plumbing our applications toward micro-services and static UI application design, changing the way we manage our backlogs and delivery commitments, and how we operate as teams. All changes should be independently testable and deliverable. The vision is to release value to the customer an Agile Story at a time, mid-Sprint with confidence.



automate the product delivery pipeline with validation and test feedback loops as gates


As we looked to reduce to cultural leadership friction in these teams as well, we re-organized the Quality Engineering Organization again. This time re-aligning Quality Engineers embedded in the scrum product delivery teams to report to the Engineering Managers (formerly Development Managers), re-assigning QE Managers to other roles. There was also a conscious movement to reduce number of quality engineers on some teams, in favor of more test development during application development. There was an expectation for software engineers to take more ownership for testing, sharing the test burden across the team. The whole teams is responsible to quality craftsmanship and the quality engineer brings the quality consciousness, test leadership, and customer advocacy.

Preserve the Quality Engineering Community – Guild and CoE

These changes led to the destruction of the QE Community as we knew it, and the Quality Engineers remaining had a sense of abandonment by the Quality Engineering leadership. The Engineering Managers felt the rising animosity about this need for resurrecting the QE Community. To address this, I introduced formation of the QE Guild community of practice, inspired by the Guild approach pioneered by Spotify and encouraged by Janet Gregory and Lisa Crispin in their book “More Agile Testing” under Organizational Culture.  This introduces a loosely grouped community of test-minded engineers focused on career and capability development in the craft of quality engineering, led by a guild coordinator (me as Dir QE for now). We also preserved the CoE (Center of Excellence) “capability radiator” team approach as centralized teams that provide evolving capability to the product delivery teams for the craft of quality engineering.


Quality Engineering Guild is an organic and wide-reaching “community of interest” group of people across the organization that want to share knowledge, tools, code, and practices –  (search Spotify Guilds for more about Guilds)


Continuous Improvement

Reflecting on where we are now and where we go from here, I can tie this back to the notion of Conway’s Law. Our software architecture is driven more tightly around RESTFul Standards, micro-services, single-page GUI’s (CDN edge-cached), and a managed public authentication layer.The Product Manager (owner) organization also now tightly manage an organizational ranked backlog or Epics that direct the priorities and alignment of the agile teams. Scrum teams organize around delivering features with these architectural constructs in mind. Continuous test is now part of delivering each Story and we have a zero-defect policy (issues found during the Story that gate delivery). These teams operate with a cohesive “developer”-“tester”-“product owner” partners, where the whole team owns quality. It’s not all that way yet, but we continue to strive for this culture.

Constant Contact was purchased by Endurance International Group in Feb 2016 with the same mission to improve online marketing success of small businesses. The mission is the same, but the brand organizations are very different. At the time of this writing we are in the midst of aligning the many brands, including the Constant Contact premium email marketing brand. Merging different organizations like this, although at a grander scale, require similar patterns of re-organizing we experienced within the Constant Contact brand and Conway’s Law continues to apply. The product offerings and feature teams may change, but the idea of architecture and team organization beget each other is a truth that leads to greater efficiency in Scaling Agile as the organization grows...tightly integrated teams, loosely coupled architectural layers with strict interface and performance standards, and continuous delivery pipelines with fast feedback loops.

In the end, the customer wins with a stronger company delivering better results faster.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: