The BBC is developing a shared platform for building its digital products to reduce development complexity and duplication as much as possible. The aim being to enable quicker and more efficient software development processes resulting in quicker delivery of digital content to our audiences. Read more about the technology changes.

A key aspect of this project has been having a shared repository for the Presentation layer code with different teams working on this platform. This blog will be sharing our experiences so far through the lens of quality engineering by answering three commonly occurring questions that pop up before, during, and after product development — who is going to use the product, how will we ensure quality, and what have we learned so far?

Who will be using the product?

Engineers across different teams working on the platform directly and our digital products consumers are the main product users. We want to keep making great digital products (quality, usability and design), even as we change technology platforms, while also minimising bugs in our software as much as possible.

In order to reduce bugs and issues raised, the testing is integrated into the development workflow, and team members across disciplines have ownership of the product quality. Having a consistent approach to testing features in the platform and having a quick feedback loop for spotting and fixing defects early, helps in minimising the risks and impact across different teams. This is an ongoing process with fine tuning based on feedback from the development teams.

Solution: The users’ needs, (in our case digital products users and engineering teams building on the shared platform) help to define the product requirements that influence the test process.

How do we do the testing?

One of the key things Test engineers and other project stakeholders consider are the risks. The impact of different code merges and changes cascading to different teams was one such risk in the shared repo. Having a shared platform meant sharing other infrastructure (besides a GitHub repo), such as deployment pipelines, communication channels in Slack, documentation, etc.

A consequence of this is that deployments are now visible to multiple teams or stakeholders, with the notifications in our Slack channels flagging failing builds. Bugs get flagged up quickly and when needed, different development team members are able to ‘swarm’ (even while working remotely) to collaboratively debug and resolve these issues. This has led to more frequent and better communication across teams and we think this has been a beneficial and worthwhile project just for getting more people talking and working together more often.

There has been consistency planned into the project as a whole from the start, for example with the Design system. Similarly, it was important to have consistency across teams when it came to testing the features developed in the platform, as we simultaneously worked on this shared code space, in order to minimise bugs, regressions and other product risks. Having an overarching Test strategy considering approaches to manual and automated testing (guided by the Test pyramid and Testing trophy principles) has informed our testing.

Automated tests form part of pull request checks and before deployment to Live. We have consistency in the automated test tools we use and engineers across teams are able to know what the expectations for testing are. This is by no means a finished endeavour, but a continuous work in progress so having forums like the Test Guild and team knowledge sharinghelp with communication, continuous learning, and further improvements.

Because of the scale of the project we rely on automated test tooling for regression testing. We also began to use fairly new test tools for visual regression testing like Storybook and Chromatic. Alternatives were Percy, Nightwatchjs and Browserstack. For other types of automated tests we use Puppeteer and formerly Cypress. We had communication channels with the test tool makers to feed back issues encountered and to request new features as we scaled and grappled with using the different test tools.

Solution: Have a test strategy and plan early to mitigate against identified project risks by including quick and early feedback during the development process.

What have we learned?

One of the benefits of a brand-new project is that there is no legacy code or technical debt at the start (this changes pretty quickly though!). Mature products have gone through the growth pains. There are known unknowns and workarounds for known problems or pain points which the development teams, (and Test engineers in particular) come to know and understand fairly well.

The challenge however, with new projects, is that there are lots of unknowns with the new technology stack. As the project has been growing we have also been dealing with and learning from the scaling challenges such as pipeline issues from multiple deployments taking place at the same time, improving monitoring of traffic and website status errors, as well as optimising our stack’s performance as more product features have been built. Being able to identify such issues early on has been important. Manual testing by different team members helps with identifying such issues that may not be covered by the automated processes initially.

Solution: Continuously learn and iterate as issues are identified and fixed.

Conclusion

Building quality engineering into a shared repository requires similar considerations to that of single-team projects but on a bigger scale and with a wider focus. These considerations are; who is the product being made for and by whom; what are the product risks, and what is the test approach or plan to reduce the impact from these risks. The aim is to provide quick feedback and monitoring for regressions during the software development process. Test automation and tooling are important for facilitating this. Continuously learning about our product, (including from our product users) by regular communication, exploring, and working collaboratively. This has helped with iterating on our quality processes based on our findings and has been important for our quality engineering.

Have you come across a project which heavily relies on a certain member of the team knowing everything about it ? During stand-ups you will see that person talking about what’s right and what is not and the whole team would nod in agreement.

Scenario 2: Change is partially understood by everyone

What about releases? Have you ever been scared to release something to production because you don’t know what else is included in the release apart from your own change? This becomes more complicated with time when the change starts bundling up to a point where you are releasing 50 commits in one release. How confident would you be with that?

How did we solve it?

For scenario 1, we propose XP practices which means constant pairing on writing code and cycling the pairs for tickets to spread the knowledge across the team.

For scenario 2, everyone in the team should have the ability to understand the change that is being made in the system clearly. This is only possible if the size of change is kept small. In an ideal world that would be one atomic commit that gets released to production after it goes through all the phases of Software Development Lifecycle.

If change is small, then it will be easy to understand and communicate, not only within the team but also outside the team to stakeholders. Everyone in the software delivery chain would be aware of what and why the change was made and would be able to support their customers better.

We understand that sometimes it is not possible to do atomic commits and the change could include multiple commits. Although we highly discourage this practice we have a temporary solution. Within a department where multiple teams are involved in the change you can use the method of “huddle” when the change is being released to production. It helps you share knowledge between teams about the changes committed by members of each team and ask questions about risks.

The problem with doing a bundled release is the rollback or revert process. For example you release five changes together to live and, out of the five, one of the features is completely broken. How would you deal with that? Would you just do a full rollback? Or do you prefer a roll forward approach or patch based approach? The roll forward approach takes longer as you would be putting a “proper” fix in the release and for that you will need developer effort. The patch approach since its a quick fix there are chances of side effects. Rollback would mean that all the features that you just demonstrated to your customers are taken away from them. This will damage your trust with the customers and could have a big impact on your organisations reputation.

A better approach is to keeping things simple, releasing everything you have from commit to production has several benefits:

• Team members are happy and onboard as they understand the change
• Customer happy as less disruption and confusion in the product
• Stakeholders will be happy because they can understand what is live and what did not
• The Test Team would not have to create complex test scenarios in order to approve the change which means testing would be faster
• Your overall pipeline process would be smoother and faster

In order to understand the impact of a change as a team, we need to create a shared understanding of the change. So how can teams go about building that shared understanding?

Help create a shared understanding between developers and testers by encouraging developer and tester pairing

When work in a team isn’t aligned and automation is put in place retrospectively rather than at the time of development, it can increase the burden on manual exploratory testing and ultimately increase risk of bug leakage. Which potentially requires rework on issues found late on in the development cycle. Collaboration between Dev and Test teams helps to ensure test automation is part of the definition of done. By not having to develop automation retrospectively you can maintain focus on the current task, smooth the process of delivery, and the team will have more confidence to deliver features faster.

In our experience by introducing pairing practices, dev and test can agree which tests are required and where to include them at UI level, integration or unit test level. This helps in a number of ways:

• Creating a shared understanding of test coverage
• Reducing duplication of tests and the effort to maintain them
• Exploratory testing focuses more ways to improve a product, rather than being a bug safety net.

Overall it contributes to a reduction of risk and can help teams go faster.

This is just one blog post out of the series that we are planning to write to share our experience with you. Please feel free to comment and share how you collaborate within your teams.

In the above example pyramid, TAP (Television Application Platform) is the primary launch mechanism for all our TV apps (iPlayer, News, Sport, RB+, Live Experience).

So how did it begin?

In August 2017, the PhantomJS repository was officially declared abandoned. This prompted us to investigate an alternative tool for running our integration tests. After several discussions and experiments we decided to build our new framework based on Puppeteer and JEST. This meant that we needed to migrate from Casper tests (which heavily relied on PhantomJS) to Puppeteer.
It gave us an “opportunity” to think about our current tests. We started asking ourselves questions like “Do we need to migrate this?”, “Is this still relevant?”, “What are we testing here?”. So instead of a lift-and-shift approach we decided to do cleanup and rewrite.

Cleanup and rewrite isn’t as easy as we thought, some of the tests were written by people who had already moved to different teams and some of the tests were testing multiple scenarios; others were doing unnecessary user journeys. With time, the framework had became a hoarder's palace. It was time for a change!

We started asking ourselves questions like how to create consistency on a single framework worked on by multiple teams. Every team has a different culture and method of implementation - how could we get everyone to agree on following one approach? We called a meeting between all teams and asked for test representation to provide suggestions. The most common suggestion was, “We need integration test guidelines”.

This blog post talks about the Integration Test Guidelines which we have written based on the experience and lessons learned from the previous framework in order to remind us not to fall into the same trap again.

So here they are:

Guidelines

• Avoid unnecessary user journeys
• Write atomic independent tests
• Don't overdo the DRY principle
• Abstract tests using Group by Intent not by page(s)
• Don't couple domain knowledge with reusable core functionality
• Don’t use artificial delays
• Don't use dependencies which are loaded externally
• Use anatomically correct test language
• Re-use rather than re-invent
• Shared code should be reusable and adoptable.

What does it mean ?

Avoid unnecessary user journeys

If you are testing the video player, please test the video player. Don’t open the homepage, click on an icon to play the video then go to video player then test it. Going to home page up until you reached the video player is an unnecessary user journey that you can and should avoid.

Write atomic independent tests

The tests should be simple and testing one aspect only. They should be independent of the other tests. Coupling another test because this test has already done half of your user journey is a bad practice and should be avoided at all times. The consequences of doing this is that you won’t be able to pin point the exact problem when the test fails and you will spend longer in debugging the issue then you saved time to write another test. It is a false economy. It will make it harder to understand what the test is actually trying to do because its so tightly coupled with the previous test.

Don’t overdo the DRY principle

DRY (Do Not Repeat Yourself) is a very well known principle used in the industry, but it is good to understand when to avoid the DRY principle. Duplication is an obvious problem for maintenance, but there's a secondary meaning to the DRY Principle... when adding new features to the framework or test, it should take the fewest steps possible with a minimum of repetition. Sometimes it is taken to an extreme where it would make the code difficult to read, in that case it is preferred to write WET (write everything twice) code.

Sandy Metz illustrates this really well in her talk titled ‘All the Little Things’

 - “Duplication is far cheaper than the wrong abstraction” - Sandy Metz, RailsConf 2014.

So, if we’re willing to tolerate some duplication, how do we avoid bugs caused by code duplication? One solution is to write a test that fails when the one piece of logic changes but the other does not.

Just remember, readability makes it easier to debug.

Abstract tests using Group by Intent not by page(s).

User flow is not equal to order of pages, therefore grouping should be done based on the responsibility of the code and not by order of the pages.

As mentioned by Soumya Swaroop in her blog post The Page Objects anti pattern

- Use the right abstractions! Group by intent not page(s).

and Robert C. Martin said in his book:

- "A class should have only one reason to change” - Agile Software Development, Principles, Patterns, and Practices. Prentice Hall. p. 95. ISBN 978-0135974445

When you don’t use the right kind of abstraction you increase the complexity of the code, hence making it difficult to read and understand your tests.

Don't couple domain knowledge with reusable core functionality

If you would like to verify some statistics event has been fired or not and your test also compares the objects of the result that has been fetched, don’t couple them together. Keep the reusable “comparison” bit part of your core and keep recording of the fired stats separate which is a part of your domain knowledge which understands what URLs will have what kind of parameters and in what pattern.

Don’t use artificial delays

Write tests which are eventful rather than time-based, don’t rely on waits or things to happen within X seconds otherwise, you will end up increasing the time until the test passes and that test will start failing in future when the “ideal” conditions change. This is not a healthy practice. It is actually an insult to your super fast resources which will always become better with time.

Don't use dependencies which are loaded externally

Have you thought about the dependency from which you are calling the function may or may not be loaded at the the time of invoking this function? This will result in a flakiness behaviour in your test. It may always pass on your local but not all environments are the same.

However, you can reference and load the external dependency internally and manage it as a part of your framework. It should not be loaded as a part of the application that you are testing.

Use anatomically correct test language

As an example, instead of coming up with an entirely new way of doing a NOT check:

Re-use rather than re-invent

If the code is already present don’t rewrite it to be slightly better. Re-use the existing functionality, more code means more maintenance. Make the core better so that every body can benefit from it.

If you see lots of repeated code in your framework, this is a red flag for “lack of communication” between the teams. This should be identified and addressed ASAP.

If you are unsure if the code already exists and you think it should have been there, ASK !

Shared code should be reusable and adoptable

With the right set of documentation around the functions its easy to understand the intent and adopt the library function. Therefore, ensure that your sharable code is wrapped with information that makes it easy to re-use and adopt.

Thats all folks!

So it would be quixotic(extremely idealistic) to say 100% of the tests will follow these guidelines. Our strategy is to influence the teams by sharing it as much as possible and getting more like-minded representation from each crew to help us implement it in their ownership.

Share your opinion with us to help us improve it further in the comments below.

I am keen on doing a follow-up post based on the interest so if you have any questions feel free to comment it and ill try to answer your question in the next one.

Cheers!

Team Structure

Testers are embedded within our agile development teams, as Test Engineers or Developers-in-Test. We work very closely with Software Developers, Product Owners, Project Managers and Business analysts to develop and test software to ensure we make a great end product.

Process

The team operates a pull-based workflow where we set a limit on our work in progress (WIP). This is set to four tasks, with each tester supporting two tasks each.

Developers pro-actively identify a tester to pair with who has the capacity to support the work moved to in-progress on the KANBAN board. This has helped us regulate work better, to avoid bottlenecks and overburdening. It has improved the engagement of testers and has resulted in more testing happening earlier in the workflow. It has also helped to breakdown the old divides of developer and tester ensuring we work collaboratively by developing and testing in parallel. Overall, the quality and the flow of work to "ready to deploy" has improved.

We use Behavior Driven Development (BDD) tools like cucumber to help us organise and automate our acceptance criteria. Testable Acceptance Criteria are created collaboratively for the feature to be developed; issues can be detected and even dealt with at this early stage. It also helps ensures that everyone has a clear view of what is required and importantly that the product developed and tested relates directly to what the product owner wanted in the first place!

We use feature toggling as a way of letting us ship releasable software any time we want. Even if we are in the middle of building a new feature that isn't ready for users to see. Whenever we develop a new feature it will be ‘feature toggled’ on or off according to its readiness and then a build is created for testing.

We initially test the feature running on a VM (Virtual Machine) and if we find any easily fixable bugs, they are fixed immediately and released in a new build on the VM. We monitor the health of builds using build verification tests running on a browser, as code is continually integrated into trunk and builds are created periodically

If a feature requires issues to be resolved then they will be triaged and added to the product backlog. When a viable VM tested build is ready it is released onto our TEST environment. Here we fully test the new feature that has been developed on a TV device and carry out a regression test of major existing features in the application using a combination of automated and manual testing. This is how we work now but it is constantly evolving as we try to continuously improve.

Automated Testing

Automated test are created as features are developed, using a Cucumber and Ruby framework which includes a message queue that sets up a communication channel between a browser or a device. These message queues are created on the fly whenever they are needed, and messages and responses are sent and received using simple HTTP requests. The test results obtained from the automation execution on devices are pushed automatically into our Test case management tool and from that we can produce a combined and dynamic view of product coverage and automated device execution in our test dashboard.

Our automation tests use a combination of live data and canned data from a static data provider. It is very useful during development of new features when the service layer code that supplies the data in normal operation is still under development.

Regression and Risk based testing

The regression testing of a product is a combination of automated and manual testing. Product coverage is set by prioritising areas of change in the code and key existing features that would cause the greatest impact to the product if broken.

Using a risk based test scope we can reduce our overall product regression effort whilst managing the risk that significant application issues could be introduced. The supported device list keeps increasing and we also take a risk-based approach to our selection of devices by selecting some representative models from each manufacturer to test.

Summary

Previously when we had only a small amount of automation a lot of time was spent manually testing builds on a device. Sometimes we needed to repeat this testing once a build was found to have bugs. With more automation both in validating developed builds and testing on devices, we can find bugs faster and spent less effort and time manual testing. Additionally process experiments across the whole development team have found ways to improve the flow of work, making it possible to increase the number of supported devices and still introduce new features into CRB quickly.

Krishnan Sambasivan is a Junior Developer-in-Test in Platform Test, BBC Future Media