Archive

Posts Tagged ‘automation’

Continuous Integration on our Highways

November 14, 2010 Leave a comment

What if we could apply the zero-defects vision of highly efficient Continuous Integration to our highways? We could then travel our highways at the full speed limit, at a sustained pace during rush hour. We would not have to expand our capacity of the highways nor extend our travel time with travel debt that eats into our private lives. Well, it appears Google is taking a crack at it (http://bit.ly/90RF3Q).

Continuous Integration: Selenium RC v.s. XUnit tests

December 26, 2008 Leave a comment

Since April 2008 I’ve been a Consultant/QE Architect at Sermo in Cambridge, MA USA on Ruby on Rails agile teams. The first team started as an experiment to prove rapid development of rails applications, composited with the JBoss-based java core community, could work seamlessly. We have continued to successfully add several more rails applications with this approach. We are now undergoing a major rewrite of the core community and it’s applications entirely in ruby on rails. This new design includes formal SOA interfaces. We are continuously refining our scrum lifecycle as well as our test automation approaches.

We have been focusing on continuous integration with Cruise Control and comprehensive regression testing, including Test:Unit and Selenium tests, an automated test plan generator, and ci_reporter test report, that run with every SVN commit. I also added nightly and weekly batch runs for runtime and more timely tests. The big issue we have been wresting with is…at what level {unit, functional, integration, runtime, browser DOM, load} should acceptance/regression tests be created? This question led to some interesting and healthy debate between development and test staff.

To summarize the testing levels…

  • Rails Test::Unit test levels:
    • unit test coverage is important to verify the methods and their paths
    • functional tests validate controllers operate as intended, including environment and database configuration
    • integration tests validate systemic operations that cross controllers, in render pages properly, irrespective of the browser, including AJAX responses for page load
  • Runtime tests are run on a deployed fleet, either headless or in a simulated browser DOM
  • Selenium tests exercise interactive AJAX and JavaScript in the page, requiring a separate client machine running the selenium-server.jar (Selenium RC for rails) or webrat gem (includes selenium-server.jar and the webrat IDL)
  • Jmeter for load tests (including performance counters)

We found that the most important part of a story is the list of acceptance tests. This list shapes and clearly defines the expectations of the story and what makes it complete. Many times we stub out tests for stories in the sprint in test suites at the beginning a sprint and can be implemented by either a test or development engineer. The biggest problem we found was that adding too many selenium tests made the automated build validation time increase dramatically (4 to 10 times that of integration tests), difficult for developers to run regression tests prior to source code commit, and more fragile as GUI implementation changed. We had to re-factor many tests from selenium to Test::Unit functional or integration tests to improve test performance and reliability.

The key to successful continuous integration is to test continuously, either with TDD practices or TIA (test immediately afterward), as part of accepting stories. This includes all code implemented for the story and any additional tests to cover the acceptance criteria. Testing at the lowest level feasible for code coverage is important for test efficiency. This may require the creation or test fixtures, mocking response expectations, and data factories.

Testing should include both happy path and negative tests (exception handling). Development Engineers need to have a sense of ownership for regression tests. Quality Engineers need to have a sense of test coverage completeness. Together the scrum team needs to hold themselves and each other accountable for not leaving test coverage technical debt beyond the story acceptance. Plan this test engineering time into the story. It may mean that your velocity is a little less than not doing this, but the overall sustainable stride is greater and you really do catch problems prior to (or at the time of) committing changes.

There is a place for automating at the GUI with Selenium or a comparable HTML element or application control automation tool. By limiting the use of these tools to testing AJAX or behavior that requires interactive javascript to render the page, workflows between systems, use cases (data driven), and cross-browser testing.

What we found is that ideally…

1. Test Engineers embedded in a development scrum team should have the ability to:

  • read and exercise application code
  • author unit test cases
  • create and work with test fixtures, test mocks and test data factories
  • assess adequate test coverage for development stories

2. Test Engineers chartered with testing external to the development teams should be able to:

  • deploy to fleets (fully automated is preferred)
  • read and understand mocked interfaces (to exercise actual interfaces)
  • author and exercise run-time tests (cover GUI and API workflows across the system)
  • author and exercise performance/load tests

Continuous Integration assures a solid application code base with full test coverage. It engages all engineers in responsibility for application testing. It allows dedicated Test Engineers to focus on system-level functionality, deployment, load, and user experience.

Hiring Quality Engineers for Scrum Teams

December 14, 2007 Leave a comment

I have frequently been asked “how do you go about hiring good Quality Engineers for an Agile team?” I really feel strongly that there are several proficiencies that needed to round out a strong test group for agile teams. These proficiencies can be enbodied in more than one person, but rarely is any one Engineer proficient in all of these disciplines.

  • Domain Knowledge:
  • As with any Quality Assurance (Engineering) team, there needs to be enough representation of the customer for each actor (user) of the system. This requires insights into the goals and intended use of of the system, and most likely may be someone hired from the user community.

  • Development Prociency:
    • Unit Tests and representative Acceptance Tests in an XUnit Framework — This is a tennant capability for working with the developer on a story to write story acceptance tests first (or during the story development).
    • Separation of GUI/Presentation tier from Business Logic tier — This is key to being able to exercise the business logic (where most of the work is accomplished) in tests that reside with the source of that production code. There is no need to drive these test with the overhead of the client presentation tiers involved. The client and presentation tiers need their own representative acceptance test for regression, and exhaustive tests (e.g. different environments).
    • Exhaustive Tests through data driven tables — There are several approaches and tools for exercising acceptance tests with many combinations of data sets (e.g. FitNesse or Solenium tables, xml, csv, etc.)
    • Test Fixtures — This is key to exposing all of the system for comprehensive story acceptance tests. As a company matures with agile, I prefer to create a Test Automation Architect role who is responsible for oversight and creation of enough test fixture infrastructure to enable the team to develop comprehensive regression test coverage that can run following build and unit tests completion daily.
  • Traditional Tests
    • Performance Test Automation — This includes both local and load tests as found in traditional waterfall and other iterative SDL’s (software Development Lifecycles).
    • Manual Tests — There are some test that are purely asthetic or user experience oriented to assure that the application is pleasing to use and accomplishes all the user tasks and goals of each actor identified for the system.
  • Process Engineering:
  • This is discipline of best practices and well defined procedures that need to be in place for consistency across the team, and to maintain a project cadence. As with software, process can be over-engineered and requires a sensitivity to what is working efficiently, when to turn up the process to improve consistency and predictability, and when to turn down the process to eliminate unnecessary overhead.

The question that usually follows this list is “how do you find a Quality Engineer with a Development Proficiency?” There are talented Quality Engineers with CS and MSCS degrees with years of experience in both development and test engineering. If you can’t find them, create them. Find a developer who has an affinity for methodical test coverage in their own work, or who is good at and enjoys testing their own work to step in to a Quality Engineering role. I have been able to convert a few developers to this role and the results are fantastic.

%d bloggers like this: