July 05, 2020jeanlucc9 min read
Image from the Pact documentation
Modern software development organizations scale their development efforts by spreading the development of a system across different teams. Such projects can have team work issues because interconnected applications that change often, create a lot of errors breaking other teams' application. A typical problem due to a large organization is when the backend team deploys a change that is not compatible with the mobile app and breaks it.
Contract testing is a way to make sure that several applications developed by collaborating teams can understand each other. With no need to take extra manual care when modifying code, the teams avoid human errors, get rid of bugs, and on the way, develop faster.
With contract testing, a team can easily create a test checking that the application they are developing receives viable responses from the provider application that another team is developing. On their side, the provider application team only has to run a single automatically generated test. This allows both teams to know if something is broken before they deploy their applications: they can talk quickly about this precise problem, with little rework cost and no bug for the users.
Pact is the main framework to implement contract tests, and it works with many languages.
In order to avoid creating bugs coming from miscommunication between two parts of your applications you can set up different tactics, contract testing is one of them. Here are some alternatives with an evaluation of how they perform. You can find some comments on these alternatives in the appendix at the end of the article.
|Alternative||Maintenance cost||Setup cost||New interaction cost||Learning curve||Efficient|
|Versioned API without breaking change||$$$||$$||$||$$||+++|
|Massive documentation and conception||$$$||$||$$$||$$$||+++|
All these tactics can be used together where it is possible.
In fact contract testing needs some conception to create the contract or update it. But contract testing allows to have an accurate documentation when you want to do this conception. Once you discover a breaking change you still have to take a decision on how to handle it and versioned API are a good way to do it but the contracts will show you what you do not need to maintain because no one uses it anymore.
Conclusion: if you develop a distributed system communicating via HTTP APIs with heterogeneous technologies you should find someone to help your team set up contract testing.
Usually to achieve testing of several services interacting, we use some kind of end-to-end testing. This means you can test only after deploying your service to put it on the same environment as other services (e.g. staging, preprod, CI). What is called end-to-end testing here could be called broad stack tests, it matches the definition found here. If there are several teams, it becomes difficult to have someone responsible for end-to-end tests.
With contract testing you can test directly in your continuous integration (CI) workflow which means no one else than the developer invested time in checking the work done, this allows problems to be caught earlier. (cf. process just below)
doing > manual local test (few minutes) > push (few seconds) > CI (few minutes) > code review (some minutes) > merge (few seconds) > deploy (few minutes) > manual test on staging > validation by the PO
As you can see on the process above having the result in CI instead of after deployment creates a faster feedback loop for the developer and he can correct it before the merge which means corrections are done faster also.
Let me introduce the pyramid of tests briefly, there are several kinds of tests with different drawbacks, the idea is that even if useful, the more drawback you have, the less tests of this kind you do.
Image from the Martin Fowler article Test Pyramid
The main advantage is that contract tests are lower on the pyramid of tests (between unit for the cost and service for the speed) than end-to-end tests (UI). It means they are faster and cheaper, so we can test more cases which gives quickly more confidence for less money.
With the pyramid we saw that contract testing is cheap compare to the end-to-end alternative. Our experience is that once set up the cost is close to unit tests (only slower) which are widely used on most projects.
Each test, tests exactly one case, this means they do not change often (compared to end-to-end tests where any change can break the test) and that a change is easy to do in the test because the complexity is low (like a unit test).
Thanks to Pact that registers all your contract tests, you know precisely what are the interactions tested in your project. This is up to date even before going to production and you can easily visualize it from Pact.
Image from the Pact documentation
Details of an interaction:
Image from the Pact documentation
Tests are used to perform check that are hard to remember and take a lot of time to perform manually. For contract tests this means they are less useful when you only have one or two teams as you usually know exactly what the other team uses.
They remain relevant but the cost for an inexperienced team on how to use them could create more problems than it solves at first. For an experienced team the cost is not an issue making them a practical solution even to test the interactions between one frontend and its API developed by the same team.
To make contract tests work you need several days of work.
You can benefit of contract testing in many situations. Once you have overcome the initial cost to set up and understand how you should work with it for a big project it can even be a cost effective solution for smaller projects in your organization. So join the Pact users' community to improve your test stack and help your teams to ship faster. You are not the first, many companies like Atlassian use this tool already as they explain it in this video.
To know if you can get benefit from Pact you should read this page of the documentation, in case of doubt read this. Then to learn how to use it you should begin here and here. To know where these tests should be used you can read the excellent article The Practical Test Pyramid which describes most kinds of test you can use in a project.
If you want more details, here are some elements I used to create the evaluation table of the alternatives in the "For Whom?" part:
Use: When you want to check your most critic happy path and you have a way to have enough control on your system in an environment before prod (set up mocks for external services, set up well known data...)
Use: When you have a small enough system that you can deploy all at once and a technical way to share code between consumers and provider.
Versioned API without breaking change
Use: For public APIs this is a good solution as there is no way to contact each consumer directly.
Massive documentation and conception
this kind of massive
Use: When lives depends on your software working perfectly
Use: When you want to quickly develop a system with more than two interconnected applications (if you do not control the update of some applications like for mobile, each used versions count). And each application is developed by independent teams which communicate frequently. It can be better to have someone to train your team to reduce the setup cost.
Web Developer at Theodo