Theodo

Writing Quality Code Using the Chain of Responsibility Pattern

Fri, 29 Apr 2022 00:00:00 GMT

At the beginning of my software engineering internship, I had little experience with writing clean code. The features were functional but there was hardly any underlying organization. I was lacking guidelines on what classes to create and how they should interact with each other. That’s when my mentor introduced me to design patterns. These are solutions for creating flexible and maintainable modules.

In the upcoming sections, I will showcase the use of the Chain of Responsibility pattern and how it helped us write scalable and easily testable code on a production-level API.

A design problem

On a project for a renewable-energy provider, I was working on a consumption monitoring app. One goal was to retrieve the user’s activities over a time period. An activity is a carbon footprint calculated based on energy consumption. To compute the activities, we had to fetch and process the consumption in the following priority order:

The real consumption (meter reading), retrieved from an external API
An estimate of the household's annual consumption, retrieved from an external API
An estimate calculated from the user's behavioral data (thereafter labeled macro estimate).

The returned activities are the result of the first of these methods that returns a non-empty array.

A trivial solution

While designing a solution for the presented situation, we have to take into account the maintainability of the code, the simplicity of adding and removing data sources as well as testing our service. An initial solution may consist of cramming everything in one class.

namespace Component\Activity\Provider;

class ActivityProvider {

    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate): array
    {
        if (count(retrieveMeterActivities($startDate, $endDate)) > 0) 
            return retrieveMeterActivities($startDate, $endDate);

        if (count(retrieveEstimatedActivities($startDate, $endDate)) > 0) 
            return retrieveEstimatedActivities($startDate, $endDate);

        if (count(retrieveMacroActivities($startDate, $endDate)) > 0) 
            return retrieveMacroActivities($startDate, $endDate);

        return [];
    }
    
    private function retrieveMeterActivities(\DateTime $startDate, \DateTime $endDate)
    {
        $data = $this->fetchMeterData($startDate, $endDate);
        return $this->processMeterData($data);
    }

    private function fetchMeterData(\DateTime $startDate, \DateTime $endDate)
    {
        /* Fetch data */
    }

    private function processMeterData(array $data)
    {
        /* Process raw meter data */ 
    }

    private function retrieveEstimatedActivities(\DateTime $startDate, \DateTime $endDate)
    {
        $data = $this->fetchHouseholdEstimatedData($startDate, $endDate);
        return $this->processHouseholdEstimatedData($data);
    }

    private function fetchHouseholdEstimatedData(\DateTime $startDate, \DateTime $endDate) 
    {
        /* Fetch data */ 
    }

    private function processHouseholdEstimatedData(array $data)
    {
        /* Process raw estimated data */
    }

    private function retrieveMacroActivities(\DateTime $startDate, \DateTime $endDate)
    {
        $data = $this->fetchMacroEstimatedData($startDate, $endDate);
        return $this->processMacroEstimatedData($data);
    }

    private function fetchMacroEstimatedData(\DateTime $startDate, \DateTime $endDate) 
    {
        /* Fetch data */
    }

    private function processMacroEstimatedData(array $data)
    {
        /* Process raw macro estimated data */
    }
}

The outcome is a service that is hard to follow and maintain. It is also more complex to test the different ways of retrieving the activities.

The Chain of Responsibility approach

The Chain of Responsibility pattern consists of having multiple services (called handlers) and running through them in a determined order to handle a request. Each service decides in runtime to either handle the request or pass it to the next handler. The CoR pattern can be used for instance in multi-stage validation or in managing multiple data providers.

In our case, each data provider (meter reading, household consumption estimate, macro estimate) represents a "handler". A request is handled if the service returns an array of activities.

To promote loose coupling, the 3 data providers will implement an interface that exposes a retrieveActivities function.

namespace Component\Activity\Provider;

interface ChainedActivityProviderInterface {
    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate);
}

Each of the three services encapsulates the fetching and processing logic that match their respective data sources.

namespace Component\Activity\Provider;

class MeterReadingActivityProvider implements ChainedActivityProviderInterface {

    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate): array
    {
        $data = $this->fetchData($startDate, $endDate);
        return $this->processData($data);
    }

    private function fetchData(\DateTime $startDate, \DateTime $endDate)
    {
        /* Process Data */
    }
    
    private function processData(array $data)
    {
        /* Process Data */
    }
}

namespace Component\Activity\Provider;

class HouseholdEstimateActivityProvider implements ChainedActivityProviderInterface {
    // Same structure as MeterReadingActivityProvider
}

namespace Component\Activity\Provider;

class MacroEstimatedElectricityActivityProvider implements ChainedActivityProviderInterface {
    // Same structure as MeterReadingActivityProvider
}

Lastly, to link the services, we can use a “main” activity provider in which they are injected and ran through in the injected order. As soon as one of the handlers returns a valid response (a non-empty array in this case), we return its value. Note that the main activity provider handles the case where none of the chained providers returns a valid response.

namespace Component\Activity\Provider;

class ActivityProvider {
    /**
     * @var ChainedActivityProviderInterface[]
     */
    private array $chainedActivityProviders;

    /**
     * @param ChainedActivityProviderInterface[] $chainedActivityProviders
     */
    public function __construct(array $chainedActivityProviders)
    {
        $this->chainedActivityProviders = $chainedActivityProviders;
    }

    /**
     * @return Activity[]
     */
    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate): array
    {
        foreach ($this->chainedActivityProviders as $chainedActivityProvider) {
            $activities = $chainedActivityProvider->retrieveActivities($startDate, $endDate);
            if (!empty($activities)) {
                return $activities;
            }
        }

        return [];
    }
}

Finally, we need to inject the chained activity providers. In Symfony, one neat way of doing it is using the serices configuration file. This makes reordering as well as adding and removing the services an easy task.

# config/services.yaml
services:
    # ...

Component\Activity\Provider\ActivityProvider:
    arguments:
        $chainedActivityProviders: [
            '@Component\Activity\Provider\MeterReadingActivityProvider',
            '@Component\Activity\Provider\HouseholdEstimateActivityProvider',
            '@Component\Activity\Provider\MacroEstimatedElectricityActivityProvider'
        ]

With this setup in place, we call the retrieveActivities function of Component\Activity\Provider\ActivityProvider to retrieve a user's activities.

The trade-offs of the CoR pattern

Implementing the Chain of Responsibility pattern offered many advantages:

Testing the main activity provider: We only need to test the behavior of the chain. Given a mocked chain of activity providers, the function should return the first valid result.
Implementing the open/closed principle: the principle states that the behavior of classes should be changed through extension and not modification. This comes into play when we add, remove, or reorder the activity providers. We can easily change the behavior of the main activity provider by changing the configuration file.

However, the usage of the CoR pattern comes with a few drawbacks, namely:

Being unaware that the CoR pattern is implemented makes the code harder to understand and debug
Depending on the used implementation, it is possible to have no handler to fall back to (if none of the defined handlers can handle the request). Always check for the fallback strategy.

Conclusion

In the long run, the Chain of Responsibility pattern helps organize and write modular code. However, it also comes with an initial cost higher than the trivial solution. They become interesting when the implemented system is prone to expansion. Indeed, setting up the chain is the most expensive part, while adding new chained elements keeps a constant cost over time.

The implementation using injections is one of many ways to utilize this pattern. For implementations in other languages, refer to refactoring-guru.

Generally, design patterns should be perceived as blueprints that offer ready-to-use solutions to common problems. Since the human brain is built to work with patterns, understanding these solutions makes the code base more intelligible and easier to work with. However, one should be aware of the layer of complexity they can add due to their initial cost.

How to externalize large libraries from your application bundle?

Wed, 27 Apr 2022 00:00:00 GMT

During the past months, I spent a lot of time trying to improve the frontend developer experience of about 15 developers on a large e-commerce website. The objectives included the reduction of our bundle size and CI run times. During this time, the development team introduced 3D assets, using the BabylonJS library to provide users with a more immersive experience regarding the articles displayed on the website. Unsurprisingly, this new feature increased our bundle size by 8 MB, representing more than 40% of our application size of the time. This in turn significantly increased our build times locally but also on the CI runners.

To mitigate these consequences, the approved solution was to serve the Babylonjs library from a dedicated CDN excluding it from the bundle for each build.

I am writing this article to share my learnings on how we managed to keep this dependency and more generally any large app dependency while reducing the consequences on the build size and time to a minimum.

TL;DR

Adding a large library increases bundle size and building times locally and on the CI/CD environments.
Many solutions as lazy solution exist to separate large dependencies for client, but it does not solves the issues for building processes.
To shorten building process, we externalized a large library to remove it from my application bundle.
To do so, the library has been externalized in a script, and dynamically loaded the library logic from it, without modifying any import from my application
Finally I measured the performance gain through a minimal example that you can find at the end of the article.

What are the impact(s) of adding large libraries as dependencies?

As a general rule, the bigger the package is, the greater the impact will be on a project's developer experience.

Download time

First, big dependencies impact installation times. When a dependency is added to a project, the designated packages and all its sub-dependencies are added to the node_modules directory. The time to compute dependencies, download the packages and sometimes build them will increase proportionally to the size of the new package. Yet, fast Internet access and cache have made these delays short, and exporting a whole package as an external dependency would not worth it if it was only to shorten these download times.

Build size

Another concern could be the bundle size increase related to the added dependency. However, modern bundling techniques such as lazy loading, code splitting and tree shaking already solve this issue by bundling only necessary code and splitting it into chunks. Moreover, moving packages into CDN would hide the problem because the user browser would still require to fetch the library. It is just not counted in the bundle served by the server.

Build time

Finally, the biggest issue with large dependencies is the build time increase. Building the application happens very frequently in the development workflow: building locally in watch mode when adding or modifying code or remotely with CI/CD. Globally, all the library code will be parsed and built into the bundle, so the larger the package is, the longer it will take to be included in the bundle. Externalizing the dependency on a CDN allows developers to retain the package functionality without having to include it at every build step.

For instance, on my project, the frontend was managed by Next.JS and we regularly analyzed our bundle size with @next/bundle-analyzer. Just after installing BabylonJS and developing the 3D feature, a new chunk appeared, weighting more than 8 MB, which represented around 40% of total bundle size while increasing our CI build times by 30 seconds.

How to do it!

There are three main steps to externalize a large dependency into a CDN:

Configure the application bundler to load a global variable as an import
Find or build a script to globally scope the library logics
Dynamic loading of component once the external script is loaded

Bundler configuration

The purpose of a bundler is to take all the dependencies of a project and to split them all in a little number of files containing the module logic in order to improve the loading speed of the built application. This article is only going to focus on Webpack because it is the most used currently, but the method used here can be used on other bundlers as well.

In order to bundle modules, Webpack does multiple operations:

It recursively resolves all module imports from the application entrypoint
It packs the associated code in a JavaScript asset and divides it into multiple chunks

To externalize a dependency, Webpack needs to know which dependency we are talking about so that it may resolve the dependency imports differently. One way of doing this is to map the webpack dependency imports to a global window.myDependency variable. This is the goal of the next part of the article.

Webpack has to be configured to not pack the dependency to externalize and to link all its uses to window.myDependency. A single configuration option allows to do that: externals (link to Webpack documentation). This parameter takes an array of string or function returning a string. During the building, when Webpack reads an import, it tests if the dependency location is included in the external array (or function callback is called). If the matching element is a string externalElement, Webpack will replace the dependency location by reading directly into the variable global.externalElement. If the matching element is a function, it will follow the same process with the returned value.

Here we want to replace the Babylon import @babylonjs/core by BABYLON due to the way they will be exposed in the CDN's script (explained in the next part). The Webpack configuration looks like the following:

module.exports = (config, { isServer = false } = {}) => {
  config.externals = config.externals || [];
  config.externals.push(({ request }, callback) => {
    if (request.match("^@babylonjs/core")) {
      return callback(null, "BABYLON");
    }
    callback();
  });

  return config;
};

Find or create the new library script and export it into a CDN

A dependency can not be externalized directly: the associated script has to have some modification. Indeed, the classic NPM or Yarn packages contain modules which are not strictly speaking scripts. The main purpose of modules is to expose logic (function, constants, etc.) that will be imported from other files. As explained in the previous part, the variable specified by the Webpack external option has to be accessible in the global scope. The dependency script must therefore be set globally when running in the browser.

Most of the time, dependencies already provide usable scripts from an already existing CDN. This was the case for BabylonJS for example which simplifies the process a lot. If your dependency does not already provide a CDN solution, you have to build the library yourself. The simplest way to do this is to follow this Webpack documentation. That allows to assign to window or global variables the elements exposed by the library. The result script should be the equivalent of one used with CDN.

Once the necessary script is build, it can be easily hosted on a CDN like UNPKG or JS Delivr. Some libraries made this task even easier by providing these scripts through their own CDN. BabylonJS was one of these. For the ones who built their own files you can host them on a private CDN.

Conditionnal loading for components using the external script

If you followed the previous steps, you should have a fetchable script exposing one variable containing all the library logic and a Webpack configuration bounding the imports of the library to the global variable. The last part is to inject the script in the code. One approach would be to inject the script globally in the application. This would create performance issues for parts of the app that do not require the dependency. The other solution is to load the script dynamically. In this case, the library logic is not available once the application is running which means the application needs to be notified before actually being able to use the dependency.

There are multiple ways to load a script conditionally and attach a callback once loaded. The simplest is by manipulating directly the DOM: adding a script with the src pointing to the script URL and onload calling the callback function. In our case, we were working with Next.js, and since v12, this wonderful library provides a <Script /> component (documentation link) doing all this work! I used it this method to display my 3D asset (contained in the <Asset3d /> component):

import { useState } from "react";
import Script from "next/script";
import dynamic from "next/dynamic";

const Asset3d = dynamic(() => import("./Asset3d"));

const Asset3dContainer = () => {
  const [isBabylonScriptLoaded, setIsBabylonScriptLoaded] = useState(false);

  return (
    <div className="asset-3d">
      <Script
        id="babylonjs-core"
        onLoad={() => setIsBabylonScriptLoaded(true)}
        src="https://unpkg.com/babylonjs@4.2.0/babylon.js"
      />
      {isBabylonScriptLoaded && <Asset3d />}
    </div>
  );
};

export default Asset3dContainer;

Multiple elements have to be taken in account:

In my component, BabylonJS requires the external script to be fetch to have all the logic loaded. I used a state updated through <Script />'s onLoad callback
Once the script is loaded, BabylonJS logic is available in window.BABYLON so <Asset3d /> can be dynamically loaded

After these 3 steps, my 3D asset was finally displayed on my page! A check shows that BabylonJS chunk disappeared from my bundle!

With a wrapper as the one I showed above for your library, you should be able to display your component with all the library logic integrated while the latter is not considered as a dependency! Congratulations!

Limits

Yet, not every dependency has to be moved into a CDN. There are three limits on externalizing a dependency from an application:

Even if a CDN package is build, it has no other optimization as tree-shaking.
Moreover, CDN are maybe fast, but they require an Internet access during development time.
If your project is using TypeScript, then every import from the removed library in not typed. You may either ignore these issues locally but reduce type coverage, or you reinstall this dependency as a devDependency. In the second case, this would still mean downloading the dependency, but it won't be included in the final bundle or in the build process (except with Next.js, but type checking can be disabled during the build if already done previously).

Thus, it is a cost-benefit trade-off between development time and use frequency of the package on one hand and building time on the other hand. In my case, the exported package was BabylonJS because 3D was only used on a few pages, but it also could be React-pdf if it is rarely used in the application, or any other large package not often used.

Conclusion

On my project, based on Github Actions data, I reduced the CI building time processes from around 30 seconds. To have better metrics, I created a minimal Next.js project which externalize BabylonJS. In the commit history, the same 3D asset is displayed, first with BabylonJS as dependency, then without it. You can look at it to see a functional example.

To test the performance of this change, I ran eight times a Next.js build command and timed it with the time command. Median building time decreases by 30 seconds, from 64.4 to 33.295 seconds! This time reduction is almost identical to the one without BabylonJS at all.

Another benefit of shorter CI runs are lower costs. According to the current Github action pricing (0.008€ per second) and around 1000 builds per month, a reduction of 30 seconds on each build saves 240€ per month on the Github action bill.

To conclude, externalizing a dependency is a complex process to setup, but it reduce all building process, by just removing the time used to build the dependency. It is worthless for small libraries, but useful for large ones, even more if they are used only on a little pages of your web application. I hope you enjoyed this article and use it to speed your CI/CD!

Working with a Third-party Provider, Three Lessons I Learned to Reduce the Lead Time

Fri, 08 Apr 2022 00:00:00 GMT

For two years, I worked as a web developer with an investment bank. One of their key aims is to digitalize their finance system in order to speed up the procedure.

At this moment the company was developing a more stable third-party API to provide the core data and which could manage the required performance for all of its consumers.

Unfortunately, the new third-party service ran into a slew of issues. One of them was stability, and it's one of the most annoying, especially since the entire app depends on it.

The more reliant you are on a third-party service, the less control you have over your app. This suggests an increase in lead time when developing a feature or fixing an issue that is dependent on a third party.

This page is about what the team tried and what we were successful at. But also where we failed to save time with our provider in order to deliver a defined quality to the consumer.

From how we started :

From the start of the project, we had a third-party owner in our team. His objective was to prepare :

the migration of our core data,
the connection to the third-party provider APIs,
and ensure that everything worked flawlessly once completed.

However, he left the project for a new adventure before we obtain the agreement for the migration. I volunteered to replace him. Everything was pretty well-prepared and documented, and I only needed to follow the process.

But, the first weeks we plugged into it, we checked almost all the problems that you can ever have with a provider :

✔️ Availability

✔️ Data integrity

✔️ Breaking change in business rules

✔️ Endpoint performance

Every day, we accumulated more and more problems, and we didn’t know when we will see the end. The time spent asking for investigation or fix increased drastically.

From these issues, I learned how to shorten the time to make a request done and make issues disappear.

My First Lesson: Become aware of your problems before treating them

With experience, we reduce the investigation duration. However, switching context caused me to be less efficient in delivering the features.

So I approached my IT lead and asked him, "What can I do?" His first answer was, “Do you realize how much time you spent on the third party problems?”

Based on this question, I set a board where I record the request frequency, the time spent, the author, and the topic. The result was, in 20 days, I spent 255 min to treats 22 requests.

This board also sheds light on two typical causes :

Poor communication
Lack of autonomy

What I learned here is: that referencing all the out-of-context time, gives you a much clearer idea of how much time you spent on it, thinking about the issues, and will push to set up some actions.

So let’s tackle the causes now!

Second lesson: Making a request needs communication and negotiation

Knowing how to communicate the needs to the provider is a thing I learned and still learning in this project.

Making an efficient request is not only about how clear you ask it. It's also about the information you provide to your interlocutor to convince them to prioritize your request. For that the information should be relevant to help him evaluate the work needed and to commit faster.

Identifying the interlocutor

To save time, asking for a request from the right person avoid doing some back and forth to relay information.

So, identifying the positions of the provider team is the first thing I did.

My next action was to define with the provider an owner for each case/field. Since what I learned was, that putting ownership in only one person will «empower» and lead him to put more effort to solve your request.

An example was, I sent a request to the whole provider team to investigate why we couldn’t create any company. The answer I obtained was, “it’s not us it’s another team”.

I then asked a developer who treats most of the time with them, the owner of the authentication process.

The time to find the right owner took us 2 hours because he was not a part of the provider team. The fix, once we found the owner, took 10 minutes to update the API key.

Know each other :

It happened pretty often that when it comes to explaining problems after investigations from our side or the provider side, it’s hard to understand because :

we don’t know the architecture of each other systems,
each team uses their abbreviations,
we work differently.

A good idea from one of our lead tech was to clarify all these points:

Share the architecture of the application,
Ask each time the definition of an abbreviation that we don’t understand, but also we use the less abbreviation possible, and only the ones that everyone knows.
Ask how they do something if it takes longer than we expect.

Knowing their architecture, and how they work will make us understand the problematics, help us communicate more efficiently, and finally unlock situations faster.

The importance of building a clear request :

What we learnt then was: it’s crucial to prepare your request.

This mean that the request has :

a clear why
accessible materials
a measurable goal

With a clear request, you will make your provider understand better your needs, you’ll have less work to do, and it will be harder for them to refuse it facing facts.

A clear request is also a way to respect your provider. Advice from our sensei to make an even better request is to formally ask your provider to refuse the request if it’s not perfect.

This is the best way to learn how to create the perfect interface for your specific relationship.

It is essential that to respect the provider the same way you respect your customers.

The importance of the explaining the impact

What I often did before was in a hurry, ask them to investigate a problem, but without explaining to them the impact

What could happen is that they don’t answer. They are busy and let you request again, to be sure that it’s important, or they would take their time.

Here is a typical example I could do :

– We have a problem, the private individual creation endpoint is on timeout, could you investigate on it?

In this example, the provider doesn’t know the emergency level and will put the minimum effort to answer this.

After several failures, my lead tech told me to learn the standard of the company to do a good request.

The previous request became then :

– We have an emergency problem right now, 10000 customers cannot create their account on the production environment, because the private individual creation endpoint doesn’t work since 14h. Could you investigate, please?

Here you have a clear impact. It helps your provider to understand the emergency of your request and to prioritize it.

Make it faster : Bring the material

To reduce the time to convince our provider to help us, or to achieve a fix, I needed to prepare my talk and material. This help the provider to localize and take decision faster.

There are plenty of tools that help us to gather all important information to prepare our requests to the provider. Here are some that we use :

Rest API client to test your provider APIs (Postman / RestClient):
- What we do if their endpoint doesn’t work is to send them a screenshot of our Postman call, with the environment, URL, headers, body, response code, and response body visible. Your provider could extract more data from your screenshot than you imagine and can guide you.
Monitoring tools (Kibana, Datadog, ...):
- These tools allow us to visualize and analyze our application logs through different aspects like histograms, charts, ... We will see how essential it is below.
Sharing workspace for sharing documentation or joint action (Notion:
- What we do with this tool is a sharing board where we track the current or future problems that could impact our provider or us. We put significant information that helps us to track the progress and to define the priority to treat them.
Draw schema (Excalidraw, whimsical) :
- Explain problems, architecture or workflow with a drawing allow a better understanding. It makes easier to localize the problem, and to evaluate the time to fix it. Good advice that I learned was to draw with your customer/provider during a meeting help to synchronise your understanding of a problem.
Alerting tools (Sentry + Slack):
- To save time, and not be constantly checking our monitoring tools, we set a notification alert for each time something goes wrong from our provider server. This will help us to react faster, and to mention the problem to our provider. With this process, it happened that we detected the problem earlier than our provider.

Establish good quality criteria/KPI

One day, the provider, announce to us that they are preparing for critical data migration.

The risks if it is not done correctly can have impacts on :

data integrity ⇒ if you provide the wrong data, you lose the trust of the customer
performance of their SQL queries/ endpoint response time ⇒ a slow app (3 second to answer) can decrease 53% of your customer
and the availability ⇒ If you have international customers, your apps need to be available 7/7 and 24/24, and you must inform your customer when there is any maintenance

At this time, I was not confident to ask our provider for a guarantee that the process will work without any negative impacts.

So I asked for help from our “sensei”. He made me realize that I was not confident because we didn’t have any good criteria to which the third party is committed.

But what are good criteria?

Good criteria should be:

clear to make it easy to remember to constantly have it in mind
measurable to determine if it is reached, or if you will reach it soon
accessible with checkpoints to be able to check it at any time
realistic to be reachable

To define our criteria, our first step was to collect all problems that we encountered and sort them to have a map of the actual situation.

From this information, we could define a clear plan on what to measure.

Monitoring, monitoring, always monitoring ... and master your tools

In the beginning, we were monitoring the liveness of our provider endpoint by coding a scheduler that calls all the endpoints every 5 min and registers the endpoint state in our database. That was not a great idea.

The result was poorly shareable for everyone, and we needed to layout the data to make it understandable, which took way too much time. We even deactivated it, because it created too many logs.

Another bad idea was to monitor all the routes, by calling them several times to make an aggregate of the response time.

We got the results, but it was manual and painful to do it frequently.

We then asked our sensei for help again, and he made us realize, that we already have all the tools that we needed, but we didn’t know how to master them.

To avoid all these pains and to save some time, we spent 2 hours studying our monitoring tools.

We were then able to automatize the monitoring by defining the duration of a single event in our code (a span), and using our monitoring tools (Kibana) to display these events in a form of a dashboard.

After getting the performance monitoring data in an accessible form, we came to our sponsor with some checkable numbers to tell them that we were not confident about the migration if they could guarantee us that 95% of their endpoints response time are below 2sec.

This made them aware that they were not ready with some facts.

The impact was big. It results in our provider's PM coming to us to make an official communication that they had to delay the data migration, this in front of our sponsor.

Use monitoring to measure the KPI and to negotiate with your provider

Here is another example of how monitoring helps us to convince our provider, with minimum time.

Every time we encountered a data integrity problem we asked our provider to modify the data, but nothing will stop that if it happens again.

We needed to find the root cause, and some long-term solution to prevent it from happening again.

How? By gathering the material, we visually showed them:

how often does the problem occur
how much time both of us spent
how much time we could save.

With this information, we reconciled our interests (not having bugs anymore) with their interests (not having to waste time on hotfix frequently) and proposed a “win-win” situation.

We then convinced them to work together to find the root cause and fix it by adding a business rule in their code.

Using visual management helped us to have much more impact to negotiate with our provider to investigate the root cause instead of fixing the problem temporarily. We then never heard about the problem again until now.

Now commit!!

We saw how to do a great request, but still, something is missing.

The last point is to make your provider commit to it. A good commitment, has a confirmation message, with a check date and an identified owner.

So you know when to check and know who to talk with when the request is still not resolved.

Go further

Go drink a beer ... or a tea with the provider team :

A thing that we didn’t achieve was to invite the provider team to a bar.

This is not a joke. We could figure it out that they are probably not so different from you.

This could also make us aware of their working environment, and how they work, and so better interact together.

Third lesson : Share your tools and methodologies

Another important lesson, that I learned during these years was how to avoid overwhelm when I was in front of tons of requests from others teams or our providers.

What I learned was to delegate the request but not anyhow.

I aimed to make everyone a copy of myself when it comes to the method and knowledge of my topic.

Teaching to make everyone autonomous will release your stress, gain time, and last but not least, put your footprint on the methodology. So even if you leave, people will work as you do.

In this paragraph, I will tell you how I made things work as I was present, and not to be overwhelmed with my problems.

Make your team autonomous

Help your team to identify the problem

5 different teams, depending on us to get the provider data.

In the beginning, we got contacted multiple times a day for data reading or writing problems.

However, the provider data could be recomputed with many business rules set by others teams through different layers.

To overcome this situation, we helped everyone to identify the origin of the problem. For that, we set a status error (424) with a message containing the problem, when the problem comes from our provider.

Since that, the contact frequency reduced drastically and the teams became much more compliant.

Help your team pay attention to the provider problem

Previously I showed you that alerting tools can be great to gather material and be notified if something goes wrong.

But there is also a drawback that we experienced by not mastering it.

What we didn’t do well, was logging our error. The alerting tools caught all the error logs, which some of them were not, and sent too many false positive notifications.

The consequence was that we became immune from it, and we didn’t pay attention to the notifications.

So it is important to set up well your alerting tools and your log to make everyone use it.

Monitor your requests

In my first lesson, I learned to monitor all the requests, and the result was in 20 days I spent 255 min to threatening 22 requests.

So what did I do then?

I sorted all the requests and realized that most of them are easy to solve.

What they required is to know in which endpoint they could find the specific information they needed.

The problem was no one was familiar with these endpoints, and there is no easy way to get the information.

To solve this, I made a Q and A, postman collection, and documentation relative to the most frequent requests, to make everyone autonomous.

The 255 minutes dropped to 10, with 2 requests. Finally, I had some time to drink a coffee!

So teaching the team is time-consuming, but it’s a great investment in the long term.

If you work 6 months on the project, I let you count how much time you gain, and what you can do with that?

Make your provider autonomous in front of problems

A typical problem that we encountered was the increasing response time each time they did a deployment, for different reasons.

This leads us to always check the response time of all of their endpoints, after each deployment. It took us 30 min each time.

To avoid this, we shared our monitoring response time tools with them with the endpoints that we used the most.

It cost us 30 min to create an access login and to teach them how to use it.

With that, we are pretty confident, that they will be more aware of the impact of their deployments and fix it if necessary.

And this would make them also more reactive since they are committed to the response time that they need to deliver.

The result is that our provider, check frequently the dashboard, and contact us by themselves to tell us that they improved the response times of some endpoints which were already less than we expected.

Don’t miss any requests : track it with a co-board:

When it comes to new version deployment or data migration, to follow critical request resolution, we created an exclusive channel during important events.

But it would be counterproductive to create many channels for smaller requests, like data/logs analysis.

What we tried was to implement a light Kanban board (we used Notion for that) to track progress, and don’t forget any of them. This board could be filled by the problem owner team.

It is also useful to track the weekly/monthly meeting with the provider if you couldn’t attempt it.

To be honest, our first try doesn’t work. Our provider told us that they will have a hundred consumers and couldn’t maintain a board like that, if they have to do it for every one of them.

What we are planning to do is to maintain it at the beginning by ourselves, to track the requests that we needed to ask again, and the time spent. This will give us some material to determine if we should convince harder our provider to use it.

Invest into long term by teaching

There are two ways to put your footprint on a project:

you work a lot, it’s worked, hurra everyone will congrats you 5 min. However, you lost your hair and the problem should be fixed well, because the day you disappear, I let you imagine the rest of the story.
you are pretty smart (or have a good sensei), you teach everyone little by little a routine about how you do to deal with the provider. The day you disappear, everything is still working as you were here. You succeed to put your footprint on this ownership. People will work as you teach them. So you can enjoy your vacation.

Conclusion

What I can retain from all of these experiences was, that there is always a way to make things done by becoming proactive.

It’s not easy, and it’s time-consuming. But, you should not think about losing time, but investing time for the long term.

To sum up what I've learnt over this project:

Communicate clearly your request to the right interlocutor with a why, materials to only post it once with respect, and expect them a commitment.
Monitoring the most sensible “points” and making them accessible, would make you gain a bunch of time to prepare your request, and convince your provider to prioritize it.
Before looking for new tools or coding some new ones for monitoring, look at what you already have, and master it, it can be better than what you expect.
Make everyone autonomous to decentralize your task. This avoids you from switching contexts, and put your footprint on the methodology. The project will still work even if you disappear ... at the coffee shop.

... And last but not least don’t forget the drink with your provider, it’s important to drink!

Serverless, DynamoDB, Streams, Filter Patterns and You

Wed, 06 Apr 2022 00:00:00 GMT

This article will cover filter patterns in the serverless framework as they apply to DynamoDB streams, but the principles apply to filter patterns on other event types as well (kinesis, event bridge, etc...)

DynamoDB Streams provide near real time information on all data modifications that occur in your database at no extra cost (No paying for event messages or any in between computations). We can then use these streams to trigger Lambdas without having to setup any extra infrastructure! Read more about DynamoDB Streams here.

This sounds great but quickly the problem becomes that you have a Lambda triggered on every single data modification for your DynamoDB table wasting the saved costs by spinning up a Lambda whose only job is to recognize it has no job to do and then stop processing. The answer to this problem is to set up filter patterns. Filter patterns allow you to execute a listener to an event only under certain circumstances removing the data validation part of our Lambda and rely on our filter pattern to only invoke the Lambda when necessary. Plus stream filter patterns come with no extra costs making them the perfect solution.

Before we dive in

Filter patterns work on any event structure but our examples are specific to DynamoDB streams which have an event structure like the following:

{
  "eventName": "INSERT" | "MODIFY" | "REMOVE",
  "dynamodb": {
    "Keys": {...},
    "NewImage": {...},
    "OldImage": {...}
  }
}

The expamples below use the serverless-lift plugin to create our database instances. This is why the stream arn's look so nice they are a direct reference to a database created in the serverless file. Read more about serverless lift and DynamoDB here!

How to implement filter patterns in your project

Lets say you have a series of jobs making updates to a single record and you have a lambda that is going to send an email to your users when the jobs finish processing. To denote when a job is finished there is a jobState column on your record. You could have a serverless lambda function setup like this:

emailUsersWhenJobComplete:
  handler: src/emailUsersWhenJobComplete.handler
  events:
    - stream:
      type: dynamodb
      arn: ${construct:batchJobs.tableStreamArn}
      filterPatterns:
        - eventName: [INSERT, MODIFY]
          dynamodb:
            NewImage:
              jobState:
                S: [DONE]

Let's break the filter pattern down.

First the filter pattern says to only pay attention to INSERT or MODIFY events meaning any remove events are automatically ignored. The next few lines of the filter pattern drill down on the event structure to get to the attribute that we need. Then it says that we expect the value of jobState to be DONE. Meaning any inserts or updates to our table records that happen before the job is done are ignored by our pattern and our lambda doesn't even spin up.

This is good, but what about updates that happen to our record after it hits the done state. Those will trigger our lambda to run as well causing us to send emails we don't want to send. So we only want it to process when the jobState wasn't DONE before the modification. We can accomplish that with filter patterns as well. Our serverless lambda function will now look like this:

emailUsersWhenJobComplete:
  handler: src/emailUsersWhenJobComplete.handler
  events:
    - stream:
      type: dynamodb
      arn: ${construct:batchJobs.tableStreamArn}
      filterPatterns:
        - eventName: [INSERT]
          dynamodb:
            NewImage:
              jobState:
                S: [DONE]
        - eventName: [MODIFY]
          dynamodb:
            NewImage:
              jobState:
                S: [DONE]
            OldImage:
              jobState:
                S:
                  - anything-but: [DONE]

This updates the filter pattern we had before and splits out the insert and the modify into seperate patterns. AWS applies filter patterns in a way that ensures the lambda will be triggered if the stream event matches any of the provided patterns.

The MODIFY event has also been updated to reference OldImage. This is the state the database record was in before the modify event was emitted. By doing this, we have now specified that if an update happens and our record ends up with a jobState of DONE we still only want to trigger our lambda and send an email if the jobState before this update was not DONE, solving our earlier problem. Users will now only get notified when the job finishes and not if any updates are made after the job finishes.

What else can filter patterns do? Quite a lot. Lets say we have a shopping platform and we want to send our users a coupon if they make an order over $100.00. We could have a serverless lambda setup like this:

sendCoupon:
  handler: src/sendCoupon.handler
  events:
    - stream:
      type: dynamodb
      arn: ${construct:orders.tableStreamArn}
      filterPatterns:
        - eventName: [INSERT]
          dynamodb:
            NewImage:
              total_amount_cents:
                N:
                  - numeric: [ >, 10000]

Or if we need to run extra structural checks when a user adds a pipe with a radius greater than 5 cm, a length greater than 10 m, and any PVC based material type in a 3d modeling system:

structureCheck:
  handler: src/structureCheck.handler
    events:
      - stream:
        type: dynamodb
        arn: ${construct:structures.tableStreamArn}
        filterPatterns:
          - eventName: [INSERT]
            dynamodb:
              NewImage:
                type:
                  S: [piping]
                radius_cm:
                  N:
                    - numeric: [>, 5]
                length_m:
                  N:
                    - numeric: [>, 10]
                material:
                  S:
                    - prefix: [PVC]

As you can see pairing DynamoDB streams with serverless filter patterns lets you perform in-depth event logic without having to spin up any extra infrastructure or invoke any extra functions. Next time you need to publish an event after a database write in your serverless project why not try out a filter pattern. You just might save yourself some time and money.

Links

Images from unsplash

Information on DynamoDB stream event structures and testing events

Full list of filter pattern rules and raw json filter pattern examples

Information on the Serverless lift plugin

Understanding Codat Webhooks

Fri, 01 Apr 2022 00:00:00 GMT

What is a webhook?

As developers, we often find ourselves wanting to see exactly when an operation in an external API finishes or when a particular event happens within an external service. It would make sense to poll that API regularly to see if something has changed, checking in on all our operations until they’re complete. This polling can be taxing and really annoying to manage–which is the problem webhooks solve.

Let’s look at one simple example of using a webhook: Github and Slack. When we create a new pull request in Github, if you’ve integrated Slack, you’ll automatically get sent a Slack message if you’re assigned as a reviewer on that pull request. This is a perfect example of utilizing a webhook–instead of constantly polling Github to look for new pull requests, Slack simply listens for the Github webhook to call its API, then parses that request to send messages to the reviewers–no polling required.

Webhooks let us specify endpoints for our application that an external service will call when a particular event occurs. Whenever we are using a webhook, we need to specify to the external provider what type of event to notify us of and what API endpoint in our application we want this webhook to call. Once configured , we’ll get a request from the external service whenever that event occurs.

Why does Codat use webhooks?

Codat interfaces with a number of different accounting, banking, and eCommerce platforms. Each of these platforms have different rate limiting restrictions, and all these services have the potential to go down at any time. To deal with this, Codat’s API works asynchronously, so requests to Codat are acknowledged and given an operation key that can be used to get the status of that API call. So, when we start an operation in Codat, we don’t know immediately when that operation will complete. It might be a few seconds until the operation actually finishes–this is where we benefit from having a webhook to notify us that it’s finished.

One of Codat’s most important webhooks is the “Push Operation Status Changed” event. When we get a call of this type, it tells us that the operation we started to save data to Codat has completed. (Or, if it encountered a problem, it will give us a detailed description of what went wrong.) The webhook is what allows Codat to asynchronously carry out our requests without making us poll over and over to see the status of those requests.

What does this look like in practice? Let’s say our application has a table called “Expenses” and we want to sync this data as a directCost object in Codat. Additionally, we probably want to know when an expense has been successfully pushed. Using a webhook, our flow would be very simple: Create the directCost object based on a row in our Expenses table. Mark that expense as having a status of “Push in progress” and save the push operation key. When the push operation is complete, we’ll get a Codat webhook callback, and we’ll find our expense based on the push operation key we saved and update based on the push operation status: If it succeeded, we’ll mark our expense with “Push Complete”. If it failed, we’ll mark it with “Push Failed”. And that’s it–since we’re using a webhook, we never need to do any polling to get the status of our expense as it’s syncing.

Securing Codat Webhooks

If you’re using authentication for your backend, you’ll probably have to set up an exception for the Codat webhook endpoint. Codat offers two forms of authentication–basic auth with a set username and password and bearer authentication with a static token. Either way, you will have to check against these static credentials in your backend to make sure the webhook endpoint is available to Codat but not to the outside world.

Multiple push operations

Codat’s data types rely a lot on each other, so it’s not uncommon to be in a situation where you need to manage multiple sequential push operations.

For example, let’s say you again wanted to push a directCost, but as part of that directCost, you wanted a reference to a new Account object that did not exist yet in Codat. To do so, you would have to go through an exchange like this:

Managing the dependencies between different objects that you want to push to Codat will require special backend logic.

Managing connection status

The first step after creating a company in Codat is commonly to add a connection to an accounting platform–and webhooks can notify us when that connection is established.

The “Company data connection status changed” status is what notifies us when we successfully connect to an accounting platform like Quickbooks or Xero, and gives us the ID for that new connection. Many of the calls we will want to make to Codat will take in this connection ID in addition to the company ID, so storing this in the backend is a great idea–and getting it in a webhook makes that easy.

This can also be helpful when we disconnect an application.If the user disconnects Codat from their accounting platform, getting notified by the webhook will be crucial, otherwise our application won’t know that the connection has been invalidated and we’ll get errors when we try to push data.

It’s important to note that the Codat connection status changed webhook will not trigger automatically when Codat is disconnected on the accounting platform side. Only when Codat next attempts a sync will it realize the connection is severed and update the connection status for that platform.

Summary

It is crucial to understand webhooks when building applications with Codat. If you’re planning on using Codat to integrate with accounting, eCommerce, or banking platforms, you should be sure to include webhooks in your architecture’s design.

How to beautify java code reliably

Wed, 09 Mar 2022 00:00:00 GMT

I recently had to set up code formatting on a spring boot java application. Auto code formatting is important to avoiding useless diffs in source files, reducing noise in code review, allowing reviewers to focus on what matters.

Ideally we want:

Automatic formatting on file save in IDE's (vscode, IntelliJ)
Capacity to run in CI

I wanted to integrate well with vscode and after trying some solution like:

java autoformat, which I couldn't run in CI and that was not really portable between IDEs
maven auto formatting plugin (didn't run on save easily in IDEs)

I ended up, from a friend suggestion, looking into a java prettier plugin. Prettier allowed for an easy integration with most Web IDE, can be configured to format on save for vscode and is trivial to run in CI.

Installing prettier for java

Let's start with the setup in command line:

Install nodejs: (I highly recommend using a node version manager like fnm) and to install a recent node version (current long term support is 16+)
Add a package.json file at the root of your project and add prettier and the java plugin:

{
  "name": "api",
  "version": "0.0.1",
  "private": true,
  "devDependencies": {
    "prettier": "2.5.1",
    "prettier-plugin-java": "1.6.1"
  },
  "engines": {
    "node": "16.x"
  },
  "scripts": {
    "format": "prettier --write \"{,**/}*.{java,yml,yaml,md,json}\"",
    "check-format": "prettier --check \"{,**/}*.{java,yml,yaml,md,json}\"",
    "lint": "npm run check-format"
  }
}

Install the required node_module: npm install

💡 Don't forget to add node_modules to your .gitignore.

💡 I highly recommend adding a .prettierignore that list autogenerated files to avoid formatting those, eg:

node_modules
target

You can now format java code running npm run format 🚀

💡 Do not forget to add node as a dependency for your project installation and to run npm ci in your project installation script for new developers.

We now want to improve developer experience by integrating with IDEs, running the formatter in CLI being impractical.

Monorepo

If you are using a monorepo, I highly recommend setting up prettier in the root of your monorepo instead of setting up prettier in each sub directory.

Integrating with IDEs

vscode

To set up prettier in vscode: add the prettier for java extension to recommended vscode extensions for the project, to do so, you can simply add the id of the extension to: .vscode/extensions.json:

{
  "recommendations": [
	"pivotal.vscode-boot-dev-pack",
	"vscjava.vscode-java-pack",
	"redhat.vscode-xml",
	"sonarsource.sonarlint-vscode",
	"gabrielbb.vscode-lombok",
	"dbaeumer.vscode-eslint",
+	"esbenp.prettier-vscode", // the classic prettier extension
	"shengchen.vscode-checkstyle",
  ]
}

then set up this extension as the default linter for java file in .vscode/settings.json :

{
// rest of your configuration
// automatic format on save, I recommend the save on focus lost option as well
+	"editor.formatOnSave": true,
+	"[java]": {
+		"editor.defaultFormatter": "esbenp.prettier-vscode"
+	}
}

Already we can easily format java code on save:

I highly recommend checking in the .vscode folder in your versioning tool (most likely git) to share this configuration with all developers.

IntelliJ

To integrate with prettier java formatter:

First install the File Watchers IntelliJ plugin
Then configure the plugin Tools/File Watchers:

Name: Prettier-java
File type: Java
Program:  full path to `.bin/prettier` eg: `node_modules/.bin/prettier`
Arguments: --write $FilePathRelativeToProjectRoot$
Output paths to refresh: $FilePathRelativeToProjectRoot
Auto-save edited files to trigger the watcher: check it
Trigger the watcher on external changes: check it

prettier documentation can be found here for IntelliJ integration

Integrate code formatting check in CI

Now we only need to integrate this check in CI, I'm using gitlab for this example, but the principles can be transposed to any other CI.

First we need to add a new job:

stages:
  - build
  - test
  - deploy

# ... rest of the file

# beautify backend files with prettier
lint-back:
  image: node:lts-alpine # We use a node image to run prettier
  stage: test # must match an existing stage
  needs: [] # none required optimization that explain to gitlab ci that this test has no dependency and can start as soon as possible
  script:
    - npm ci
    - npm run lint
# ... rest of the file

now just push your change, and you are all set:

Choosing the best storage solution in GCP

Wed, 09 Mar 2022 00:00:00 GMT

Are you new to GCP, looking to make sense of all the storage offerings GCP provides or simply hoping to learn something new? You're in the right place!

In this blog we're going to compare the key differences between GCP's numerous storage solutions. We will start by taking a look at a handy decision tree which gives you an overview of the various options. We will then categorise the available storage solutions into blob, relational, NoSQL, file, block and other storage. For each option, we will delve a bit deeper into the specifics and we will highlight the best suited use cases as well as flag any potential downsides. Let's get started!

Decision tree

As promised, we're starting with a decision tree to help you choose the best storage tool for your use case. Don't worry if you're not sure what all this means - we'll take a look at each tool below.

Now let's look at each of the options listed above in a bit more detail.

Blob storage

Blob storage allows you to store unstructured object data. This means that we're looking at data which often consists of items (e.g. media files) which don't need to sit within a file structure.

Cloud Storage

We're starting with one of the most commonly used storage options - Cloud Storage. Cloud Storage is GCP's blob storage solution and it is often used alongside other tools such as Compute Engine (service for creating and running virtual machines), App Engine (platform for building highly scalable applications using serverless) and BigQuery (data warehousing solution - more info below). In terms of its structure, it uses buckets to group items.

Benefits

Buckets are really powerful organisation tools, which allow you to control lots of properties, including lifecycle policies (e.g. delete object 2 years after creation) and object versioning (by default, object versioning is switched off, so if updated, the object will be overwritten).
You can choose to either host your bucket in a single region, or it can be dual-regional or multi-regional. The dual-regional option in particular isn't seen across many storage options. It can be a great way of saving on costs (in comparison to multi-regional buckets) whilst bringing your data closer to your users and making sure it is available even in the event of a regional outage.
You can choose a suitable storage class based on how often you need to access your data. The four storage classes available are standard (access regularly), nearline (access once a month), coldline (access once a year) and archive (access less often than once a year), with the cost of storage decreasing with each subsequent class.

Downsides

Cloud Storage cannot mimic file structure of nested folders. Each item is accessed directly via its own url.
Coldline and archive classes carry an associated retrieval cost, which will be charged when you access your data. This means that the costs can sneak up on you, especially if you’re not keeping track of how often you’re accessing objects in your long-term storage buckets.

In terms of common use cases, Cloud Storage is an excellent option for archiving data for audits and for storing data for disaster recovery. It is also great for minimising latency for users who download large data objects since you can choose dual-regional or multi-regional storage to bring your data closer to your users.

Relational storage

Unlike with blob storage, relational databases organise data into tables which can be linked to each other. They are the industry standard and GCP provides a couple of options for storing relational data within the cloud.

Cloud SQL

Cloud SQL is the obvious choice when setting up a relational database in GCP. It provides a fully managed service which now allows you to create instances of MySQL, PostgreSQL as well as Microsoft SQL Server.

Benefits

Easy to migrate to from on-premise relational databases, either using MySQL dump or CSV file imports.
Variety of methods for connecting to your database, including access via private IP and Cloud SQL proxy.
Automatic data replication with failover.
Automated storage capacity management.

Downsides

Storage capacity is limited to 30 TB, which may catch some users out.
Cloud SQL is great for vertical scaling, but if you're interested in horizontal scalability, you'd best look elsewhere.

Cloud SQL is an excellent option if you're looking to replicate your on-premise database within the cloud, or if you'd simply like to stick with a database you know already. This is exactly what happened with AutoTrader, a digital automotive marketplace. They were looking to move their on-premise infrastructure to the cloud, and they chose to move their existing Oracle database into Cloud SQL.

Cloud Spanner

GCP’s alternative to Cloud SQL is Cloud Spanner. Cloud Spanner provides a relational database structure with the added advantage of horizontal scaling.

Benefits

If you're concerned that Cloud SQL doesn't provide enough capacity for your needs, Cloud Spanner is definitely worth a consideration since it scales to PB capacity.
Support for Google Standard SQL as well as PostgreSQL.
It provides strong consistency, meaning that once a user makes a change to the data, the update will be consistent for all users immediately.
It automatically replicates data across multiple zones.

Downsides

Whilst you can create backups on demand, there is no in-built option to generate them automatically.
Cloud Spanner isn't MySQL compliant.

Since Cloud Spanner offers strong consistency globally, it is well suited to financial applications. A fun real-life example is Pokemon Go, which uses Cloud Spanner due to the aforementioned feature.

NoSQL storage

By comparison to relational storage, NoSQL databases store data in non-tabular formats. This can include document storage, key-value pairs or wide-column databases. GCP offers two main NoSQL databases, which we will explore below.

Cloud Bigtable

Cloud Bigtable consists of sparsely populated columns, where rows are indexed by a single key. It is possible to group columns into column families to capture which columns are related. Each row-column intersection can contain multiple cells at different timestamps, and this allows Cloud Bigtable to record a history of data updates.

Benefits

It can comfortably handle PB capacity.
It offers very low latency, consistently hitting targets of sub-10ms.
Easy integration with big data tools, including Hadoop and BigQuery.
Clusters can be resized with no downtime.

Downsides

The smallest possible cluster consists of 3 nodes and provides 30,000 operations per second.
Cloud Bigtable doesn't scale to zero, so you pay for nodes regardless of whether they are being used.

Cloud Bigtable works well for operational and analytical use cases, and is often used with IoT applications. A real-life example is Dow Jones, a news content and business information provider, who uses Cloud Bigtable to store their data before processing it for analytics.

Firestore (previously Datastore)

Previously known as Datastore, Firestore is the new generation of this NoSQL document database. Unlike Cloud Bigtable, it uses JSON to store data.

Benefits

Integration with Firebase - GCP's app development offering.
Firestore offers strong consistency. This is a typical feature of relational databases.
Multi-regional replication is available.
It provides ACID transactions, allowing you to be confident that there are no transactional discrepancies within your data.

Downsides

It only scales up to TB capacity.

Firestore is an excellent choice for mobile and web applications at global scale. Thanks to its integration with Firebase, setting up Firestore can be a seamless experience. An example of Firestore being used in the real life is The New York Times, who use Firestore to support their real-time rich text editing tool.

File and block storage

We are now moving on from nonSQL storage to file storage, by which we mean storage which allows us to replicate the commonly used file structure of nested folders. The best way of thinking about it is that it mimics the file system of a computer. Block storage, by comparison, stores data as separate pieces, where each piece can be uniquely identified. Block storage is very performant since it allows for data to be accessed via multiple paths.

Filestore

Filestore (not to be confused with Firestore) is GCP's fully managed network attached storage (NAS) option. It is often used alongside instances of Compute Engine and Kubernetes Engine. Filestore also offers four tiers (basic HDD, basic SSD, high scale SSD and enterprise) to allow you to pick the best solution for your needs.

Benefits

Filestore offers high performance - throughput of 25 GB/s and 920,000 operations per second.
It scales up to 100 TB capacity.
Capacity automatically scales up and down to minimise cost.

Downsides

It is rarely used without an associated Compute Engine or Kubernetes Engine instance.

Filestore is often used for file sharing as well as high performance computing, e.g. genomics processing.

Persistent disk

We are now delving into the world of storage solutions for Compute Engine. GCP provides a variety of options within the persistent disk offering. As expected, at the the highest level of categorisation, you can choose between SSD and HDD.

Benefits

Automatic encryption using GCP-supplied keys or your own.
Durable storage - there is very low risk of loss of data, with the risk lower for regional persistent disks.
There is no charge for operations.
The disk types include standard, balanced, extreme or SSD, so you have a lot of freedom to choose the best suited alternative.

Downsides

Its usage is primarily restricted to Compute Engine.
A persistent disk can also be used with Kubernetes Engine as storage for persistent volumes.
Persistent disks can be zonal or regional, so they aren't well suited for global applications.

The most common use case for persistent disks is, unsurprisingly, attaching them to Compute Engine instances.

Local ephemeral disk

The final file storage type we'll cover is local ephemeral disk. The main difference between local ephemeral disks and persistent disks hides in the name - persistent disks retain data even if they are not attached to a virtual machine and the data will be kept until the disk is deleted. By comparison, the data stored on local disks is lost as soon as the virtual machine it is attached to is stopped or deleted.

Benefits

Local disks provide higher throughput and lower latency than persistent disks since they are physically attached to the server.
It is possible to attach up to 9 TB per instance using a maximum of 24 local disks.

Downsides

Local disks are only available as SSD, so HDD local disks don't exist.
You can't attach local disks to certain virtual machine types.
Whilst you can be confident that your disk is always encrypted using GCP-supplied keys, you can't supply your own encryption keys.

Local disks are best suited if you need fast scratch disk or cache.

Other storage types

In this final section we introduce the remaining two storage solutions which don't fit well into any of the categories mentioned above.

Memorystore

Memorystore is GCP's fully managed service for Redis and Memcached. If you are interested in using either Redis or Memcached but don't want the hassle of managing them, Memorystore is the storage option for you.

Benefits

Memorystore provides sub millisecond latency for lightning fast data access.
It is easy to migrate from an existing Redis or Memcached instance using the lift-and-shift functionality.
Memorystore natively integrates with GCP's IAM service for robust security.

Downsides

Memorystore's capacity is limited to 300 GB.
If cost is a significant factor, then you should consider hosting your own Redis or Memcached instance.

BigQuery

GCP's data warehouse solution, BigQuery can handle data storage as well as analysis. It isn't the most common storage solution, and is often used alongside other storage options for data analysis only.

Benefits

BigQuery is great for data analysis - all your data can be in one place so you don't have to worry about data ingestion.
It provides many specialised services, including BigQuery ML for machine learning and BigQuery BI Engine for interactive data analysis.
You can also stream data directly into BigQuery from other storage solutions.

Downsides

Cost is relatively high, so it may not be the most efficient long-term solution.
It's not recommended to use BigQuery for data storage unless you want to use its analysis capabilities as well.

One of GCP’s customers using BigQuery is The Home Depot, who use this tool to keep track of their stock across 50,000+ items and 2,000+ locations.

Conclusion

GCP provides a massive variety of data storage tools for all sorts of use cases. Selecting the best option can be difficult, since there are so many tools to choose from. My best advice is to start from your data type, this will often significantly narrow down the available possibilities. Afterwards, considering the location of your data (e.g. zonal or global), required consistency and capacity will help you decide. Make sure to consider cost as well - you can use GCP's pricing calculator to get a sense of how expensive different solutions would be.

I hope that you're this leaving this blog post with a clearer idea of the different storage options GCP provides. Remember that if none of the tools listed here fits your use case, you can always build your own solution on Compute Engine. Best of luck with your cloud journey!

Useful links

Intro to storage options

Cloud SQL for AutoTrader

Cloud Spanner for Pokemon Go

Cloud Bigtable for Dow Jones

Firestore for The New York Times

BigQuery for The Home Depot

Pricing calculator

Images from Unsplash

How AnotherDay and Theodo built a big data geospatial analytics platform

Wed, 02 Mar 2022 00:00:00 GMT

AnotherDay’s Cascade platform provides businesses with the ultimate risk and intelligence management platform. With risk assessment at the heart of so many international standards, many business suffer from competing standards for tracking and mitigating risk. Often, large spreadsheets, unwieldy slide decks and or folders with hundreds of sensitive intelligence files become impossible to manage with challenges such as version control and centralisation. Cascade provides both a single centralised platform to collate intelligence and assets and also a powerful analytics platform to make sense of this data and to extract effective risk mitigation strategies.

Today, an often used strategy for asset risk quantification is to run accumulations. This is where a radius is selected and for each of the businesses assets, the total value of all the other assets lying within that radius from the asset is calculated. A business can then look for where most value has accumulated, and therefore where they have incurred the most financial risk. This form of analysis is often used in areas such as terrorism modelling.

The Cascade platform needed the ability to run these sorts of accumulations across large datasets, in excess of 1 million assets. As these modelling runs were potentially long-running it was also desirable that these tasks could be run asynchronously from a queue.

AnotherDay worked with Theodo to build this functionality into the Cascade platform. As experts in React, Django and AWS the task was to make this end to end functionality production ready in 5 weeks.

The modelling flow begins by uploading the assets into the Cascade platform. The data was provided in CSV format which was uploaded to S3 via the frontend. This triggered a lambda function which performed validation and cleaning of the data before leveraging the aws_s3 extension for RDS PostgreSQL to transfer the data directly into RDS from S3.

Asset upload flow

To store the geospatial data in RDS, Theodo used the PostGIS extension for PostgreSQL. A second RDS instance was provisioned alongside the existing database for Cascade in order to protect the main database from the expensive modelling queries.

To trigger an accumulation modelling run, a user creates an accumulation entity in the frontend specifying parameters such as radius. Saving this entity triggers a JSON message sent to an SQS Queue with the model parameters and run details.

Celery (an asynchronous task runner) was provisioned in Fargate, providing a worker pool that subscribes to the SQS queue. The model task is picked up by a celery worker which then runs the model SQL against the PostGIS database. A naive implementation of the model scales as N² with an analysis of 20,000 assets taking 15 minutes. Leveraging the r-Tree index provided by PostGIS provides blazing fast performance, lowering the model runtime to 15 seconds.

Running an accumulation model

The top 50 results (in terms of accumulated value) are saved to the main database for display in the frontend. Theodo used the Mapbox API to build an interactive results map, displaying the numbered results and the accumulation radius. For the rest of the results the aws_s3 extension is used again to output a CSV to S3 which allows the entire result set to be downloaded.

Focusing on system design tailored to the business domain, and leveraging AWS & open source tools, sets AnotherDay up to create a new standard for risk management. Theodo ensured best practices were followed to create a state-of-the-art platform with many exciting avenues to explore around integrating live intelligence data into the modelling process.

Architecting a Modern Monorepo with NX and Turborepo

Tue, 01 Mar 2022 00:00:00 GMT

What is a Monorepo?

A monorepo is a consolidated repository containing the source code of multiple projects, which are commonly managed by independent teams and also often share common packages.

Not just Code Collocation

Recently a common software development practice is to have a full stack JavaScript application, this leads to many developer teams believing they have a Monorepo "Because all of my code is in the same repository".

A monorepo is far more than just code collocation, we'll address the benefits, drawbacks and showcase two industry leading Smart Build Systems.

Benefits of a Monorepo

There are a number of important benefits to using a Monorepo (with a Smart Build System).

Consistency - It's possible to share UI modules, documentation and other such shared packages
Visibility - Everyone in the organisation can see all the code, improving cross-team collaborations and communication
Build and Development Caching - Some tools like NX and Turborepo allow for efficient rebuilds to allow rebuilding only changed packages
Single source of truth - As all changes to the repo are atomic, there won't be a situation where one team is working with outdated legacy code
A single CI/CD pipeline - There is no need for multiple pipelines in each application as we can handle it in one centralized place
Deduplication of node modules - Node modules are the computational equivalent of a black hole already, so any deduplication or help here is always appreciated

Downsides of a Monorepo

There aren't many downsides to using a Monorepo, but there are some that are worth mentioning:

Slower IDE Performance - For large Monorepos it's possible to have a slower IDE performance due to the increased number of files, this can slow down imports, intellisense and other IDE features
Learning Curve - Some tools to manage a Monorepo add technical complexity, especially during the setup

How do I make a Monorepo?

Making a Monorepo for a fullstack JavaScript code base is easier than ever, the easiest solution with no additional dependencies is NPM Workspaces, and the other two solutions we will cover are NX and TurboRepo.

Workspaces - Keep It Simple Stupid

Workspace is a generic term that refers to a set of features in the NPM cli which adds support for managing multiple packages from the root of the repository, NPM workspaces allow you to deduplicate node modules and run commands against multiple projects at the same time.

Below is an example structure of a repository using NPM workspaces

├── package.json
└── packages
    ├── package-a
    │   └── package.json
    └── package-b
        └── package.json

# ./package.json
{
  ...
  "workspaces": ["./packages/*"]
}

Workspaces provides two basic features, as for the most part Workspaces are just code collocation with the ability to run projects from one terminal in conjunction.

Running commands against multiple packages from the top level repository, e.g. npm run test --workspaces
Deduplication of node modules

Ideal for 💡

Small projects
You don't need a smart build system
You don't want to add additional dependencies

NX - Advanced and Powerful

NX is one of the most advanced build systems available and offers some incredibly powerful features to take your codebase to the next level.

Some of the most acclaimed features of NX are as follows:

Integrations and plugins for many platforms (including a VSCode Plugin)
Rich plugin ecosystem 🚀
Ability to build and test only what you need to
Automatic project dependency management
Vastly faster builds with NX Cloud
Distributed task execution
Code generation tools

Create a Next.js NX project

Create a new NX project, with a Next.js application

npx create-nx-workspace@latest

The NX CLI will then ask you which template you would like to use to create this new application, for our case we will choose "Next.js".

Create a shared React library

One common usage of NX is to create a shared library which can be used by multiple apps. Shared libraries are created in the libs directory by default.

nx g @nrwl/react:lib ui-shared

Running this command generates a new ui-shared library, which can be used by multiple apps. The beauty of this pattern is that if you change ui-shared, it will rebuild other apps that use it, whereas, if you make a change in next-app NX won't rebuild ui-shared.

Here you can see the process of importing our new shared library, and using it in our Next.js app.

Affected Builds

When you run nx test next-app, you are telling Nx to run the next-app:test task plus all the tasks it depends on.

For a small project this is okay, however for a large project you really only want to run the tests for the files and changes you've made to the app. To handle this, NX has the affected command. nx affected --target=next-app will figure out which tasks to run, and only run those tasks.

To visualize these "affected libraries", use the nx affected:graph command, in this example below we've added a second-shared lib which is also used in next-app.

Now we're going to make a change to second-shared, as this is used by next-app, and next-app-e2e you should expect that they should both be re-built.

As epected you can see that the next-app and next-app-e2e projects are affected, but the ui-shared lib is not.

Incremental Builds

Once you've got an established NX Monorepo, you will likely have many applications and libraries. It is common that you will only be editing a small subset of these applications at once, Incremental builds allows NX to only rebuild a small subset of the libraries, which results in much better performance.

This then works closely with NX Cloud. With NX Cloud you and your team mates share the same cache, allowing you all to benefit from improved performance.

The above chart is from NX and highlights the performance differences between normal builds, and incremental builds (both cold and warm).

Running the Nx incremental build having already cached results from a previous run or from some other coworker that executed the build before. In a real world scenario, we expect always some kind of cached results either of the entire workspace or part of it. This is where the teams really get the value and speed improvements.

Ideal for 💡

Medium to massive projects
You want all of the fancy modern features you can get
You don't mind dealing with some complicated looking nx.json files and tuning them to your liking
You want a battle hardened, community supported platform with industry backing

Turborepo - The New Kid on the Block

Turborepo is a recently open sourced project acquired by Vercel, the company which brought us Next.js.

Turborepo's goal is to take what's great about other build systems such as Lerna, and NX, whilst shipping it in a small simple package, which works hard to stay out of your way.

Setup

Setting up Turborepo is as easy as it gets, just run npx create-turbo@latest.

This sets up an example project, with a web and docs apps, and a shared ui package. Unlike NX there is a lot less boilerplate and this resembles a typical project structure you'd see with a "normal" Monorepo, this makes Turborepo much more approachable to a monorepo newcomer.

Affected

Turborepo has its own flavour of affected from NX, both work very similarly, however Turborepo doesn't produce an interactive graph like NX does, Turborepo has a basic graphviz image export.

Other Features

Turborepo has a few other features:

Cloud caching
Parallel execution
Profile in the browser

Ideal for 💡

Vercel users, Remote Caching works great
Small to medium projects
You don't mind losing some features of NX

Feature Comparison

Below is a feature comparison for Workspaces, NX, and Turborepo.

Feature	Workspaces	NX	Turborepo
Node Module Deduplication	✅	✅	✅
Run Commands Across Projects	✅	✅	✅
Shared Libraries	✅	✅	✅
Profile in Browser	❌	❌	✅
Dependency Graph	❌	✅	🚧 (non interactive)
Incremental Builds	❌	✅	✅
Cloud Caching	❌	✅	✅
Code Generation	❌	✅	❌
Distributed Task Execution	❌	✅	❌

Conclusion

Hopefully you now appreciate the importance of a Smart Build System within a monorepo. As much as I'd recommend giving both NX and Turborepo a shot, if you don't have time to experiment with both, I recommend trying NX.

Turborepo has most of the features that NX has, but NX does that and more, in a more performant package. Given the current state of Turborepo due to it's recent inception, I'd rather have a NX monorepo that has been battle hardened which you know won't let you down.

However, with Turborepo's recent backing of Vercel, I expect NX to face some strong competition in 2022, and as you all know, competition brings innovation.

References and Reading Materials

NX Documentation
Turborepo Docs
NX and Turborepo comparison (Disclaimer, it's from NX)
monorepo.tools Fantastic Resource! 🔥

Bringing the client closer to their product using Contentful

Tue, 15 Feb 2022 00:00:00 GMT

You need to make sure that your website content keeps up-to-date with changing designs and client requirements. The trade-off? This often impacts developer time. Small changes, like adjusting an image or a piece of text, seem like a 10-minute job. In reality, they take up more capacity as they still need to go through the full development processes (at Theodo these include code review, deploying, functional review, validation, etc. to ensure quality and avoid regressions).

Content Management Systems (CMS)

Utilising a CMS means that you can create, manage and modify the content of a website without needing any code changes. A developer has to integrate the CMS using its API, however, once this is done, anyone with the correct permissions can adjust the content. This means that content changes now become much simpler and the client can benefit from up-to-date content, as well as saving developer time over the longer term.

Introducing Contentful

Contentful goes beyond CMS; not only does it allow you to plug adjustable content into your website, it also incorporates additional features to optimise your content.

As a quick introduction, Contentful is a content management platform with a very comprehensible Content Delivery API. It is available in the most popular programming languages (e.g. Javascript, Python), a great help when you are working on a variety of projects utilising different stacks. Overall, some of the best advantages include the ease of setup, clear and practical JavaScript SDK, and Contentful’s intuitive platform.

Before delving into the advantages and disadvantages of Contentful, let’s introduce the technical side: how to integrate it into your code.

Code example

Setting up

The first, and essential, part of using Contentful is to create an account on their platform https://www.contentful.com/. Assuming that this has all been done, we can now turn our eye to integrating it with our code.

If you don’t yet have Contentful installed and are using Node.js, you just need to install the package using:

npm install contentful

Next, you need to initialise the client. To do this, you will need an API key and space ID (some quick steps can be found here). Although optional, you can also specify an environment. We’ll talk about this later, but it is another key advantage of using Contentful.

var client = contentful.createClient({
  space: '<space_id>',
  accessToken: '<access_token>',
	environment: '<environment_name>',
});

That’s all you need! Once Contentful has been set up, you can now use it throughout your project.

Fetching content

There are lots of ways of retrieving content. You can use a specific entry id to retrieve a specific item, however, you can also retrieve groups of content. One of the most useful tools to retrieve content is being able to retrieve by content type. Within the Contentful platform, you can create ‘content types' that define the types of data that you will be adding to your website. For example, you might have a simple type such as colour which will just have a short piece of text (i.e. a title) or you might have a more complicated type such as author which will require a name, a description, an image, etc. To fetch this content type (e.g. fetch all the author types), you need to call:

const authorResponse = await client.getEntries<Author>({
	content_type: 'author',
})

Where Author, in this case, is the type defined by Contentful which needs to be imported. The response can then be used anywhere you need. Within Typescript, the Author type would need to be defined. Although this can be done manually, it is more efficient to use an automatic type generator such as this one.

Optimisation

Contentful can also be used to adjust and optimise your content. A key example is with images. There are many options available (see documentation), however one of the most interesting ones I came across was the ability to change the quality of the image. By reducing the quality of the image, the content can be rendered more quickly on your page, avoiding delays. Once you have retrieved the image URL from Contentful, all you need is to adjust it and specify the quality:

const imageUrlWithAdjustedQuality = `${imageUrl}?fm=jpg&q=${quality}`

With Contentful, the quality is a percentage and can be set to be between 1 and 100.

These are just two examples of what you can do with Contentful. I would highly recommend exploring their documentation and having a look at implementing it yourself too.

Environments

Earlier, I mentioned that we can set the environment when initialising the client. Contentful has a wonderful feature where you can have content in different environments. This means that changes made in Contentful don’t need to directly impact your production website. Contentful does this in a very similar way to Github where you can have different branches (e.g. master, staging), create and destroy branches. The key process to releasing a new branch is as follows:

Create a new branch (copied off of the master branch) where you make all your content changes
Once the changes are complete, point master at this new branch
You can either delete the old master branch or you can keep it in order to revert to an older version if needed

Although this can be done through the terminal, it is highly recommended to use the Contentful platform as it is quite straightforward and can avoid unwelcome surprises.

Key advantages of using Contentful

A key feature that singles out Contentful compared to other CMS is the ability to manipulate images. It does so by not only adjusting the format and size of your image, but also by enabling cropping and adjusting the quality you would like to have - a great optimiser of loading speed.

While CMS is often thought of as being able to integrate content such as images, videos and text, Contentful takes it a step further. While the general page layout is predetermined, its content can be completely set through Contentful. This is because text can be done in markdown, giving more flexibility to what can be displayed.

In the code example, we had a look at fetching all items within a content type. If all the data is useful to you, then this is great. However, you may only want to fetch a specific group of items. For example, if you have a Book content type, you may only want to fetch items with the field genre corresponding to 'Mystery'. This is done using relational queries, e.g. using the API call:

https://cdn.contentful.com/spaces/my-space-id/entries/
    ?content_type=books
    &fields.genre.sys.contentType.sys.id=genre
    &fields.genre.fields.label[all]=Mystery

However, the key benefit of using Contentful compared to other CMS is the simplicity of integration.

Disadvantages and a word of warning

Speed

Fetching all your content from Contentful seems like a dream come true, no more hardcoding text and images, freeing up developer time and making sure that your content is up-to-date. However, this does come with a warning. Fetching all the content whenever a page is loaded can be very slow. The more data you fetch from Contentful, the longer your loading time. However, this does not necessarily need to be an issue depending on which framework you are using. Contentful used alongside NextJS is one of the best solutions. NextJS uses static site generation (SSG), meaning that your content does not need to be refetched whenever you load a page - a significant time-saver. However, this does mean that if you make changes to Contentful, you will need to rebuild the site in order to see these changes. Rather than SSG, you can also look into using Incremental Static Regeneration (ISR) which uses static-generation per page rather than the entire site and would avoid the need to rebuild the entire site when a change is made on Contentful.

Field type

When fetching content types, there are also limitations on the field type we can filter on. One of the types that it is not possible to filter by is Array<Link>. A further note to consider about the Link type is the issue of depth. Links are a great way to model the relationship between content items, however as the depth increases, this can cascade into a circular relationship. By default, you will only access the top level, however, if you do need a higher depth, it is important to understand the possible consequences of expanding the maximum depth.

Other points to consider

Another point to consider is that Contentful cannot be installed locally for local development. To work on your project with your content fetched from Contentful, you will need to have an internet connection. However, as this is also the case for other CMS, it is not a strong disadvantage. One of its key challenges is its cost. At about $489/month for a team, Contentful can be quite costly, especially for smaller projects.

Conclusion

Overall, Contentful is a great way to optimise time for developers and ensure that the client has control of their content. However, you have to be careful about when to use this as projects with a low budget and with small amounts of content might not need such a powerful tool. Another key consideration is whether you are using NextJS on your project. If you are not, you might want to think about switching over to make sure that your content is loaded quickly and efficiently.

More documentation: https://www.contentful.com/developers/docs/references/content-delivery-api/

Logging uWSGI errors and alarms to Sentry

Wed, 02 Feb 2022 00:00:00 GMT

If you already use Sentry to manage your application errors and uWSGI as a WSGI server for your Python project (such as Flask or Django), you are one step away to improve your monitoring setup in a matter of minutes.

It is important to do so: even if you have Sentry configured for your app, if the WSGI layer that is running it fails you won't get any Sentry issue about it.

With uWSGI there are some events that you want to watch for, and triggering a Sentry issue is one easy way of fitting this into your existing error management flow.

Such events include:

when a uWSGI worker is killed because it is serving a request for an excessive long time (the "harakiri" behavior)
when the listen queue of the uWSGI socket is full
when a segfault happens
or any non-applicative log line that you want to monitor

How to integrate uWSGI with Sentry

Unbit (the company behind uWSGI) wrote a plugin exactly for this use case!

Make sure that a libcurl package is installed on the machine that runs uWSGI, for instance:

apt-get install libcurl4-openssl-dev

Install the plugin before running uWSGI:

uwsgi --build-plugin https://github.com/theodo/uwsgi-sentry

NB: this is a fork of the original plugin that makes it work for current versions of uWSGI and Sentry, I will update this article if it is merged into the main repo.

Configure the plugin in your uWSGI INI file:

[uwsgi]
plugin = sentry

; Defines what to do when the "logsentry" alarm is triggered
alarm = logsentry sentry:dsn=https://your@project.sentry.io/dsn,logger=uwsgi.sentry

; Alarm for listen queue full
alarm-backlog = logsentry

; Alarm for segfault
alarm-segfault = logsentry

; Alarm for harakiri (alarm-log uses regexp to match log lines)
alarm-log = logsentry HARAKIRI ON WORKER

You can find more customization options in the Alarm subystem docs.

Conclusion

Deploy this change to your server and bam, you're done!

After this, you could reflect back on what other component of your architecture you are missing alerts on.

Bonus: if you are loving Sentry as we do, here are some tips on how to use Sentry effectively.

React Hooks and Tips to Avoid Useless Component Render Applied on Lists

Mon, 31 Jan 2022 00:00:00 GMT

A few weeks ago, I encountered children list rerender issues on the project I was working on. In this article you will learn :

how I debugged a react performance issue
why virtualization is not always suitable for list rendering issues
what is memoization
how to memoize react components and functions with react hooks and React.memo() to prevent a component from re-rendering.

Overview on the performance issue

I’m working on a platform to purchase orders from suppliers. Each order is composed of clothing items. Here is a typical example of the interface.

Let’s take a very simple example: I am ordering from my supplier Theodo, producing some top quality shirts. I want to order their famous shirts in three colors (white, red and blue). On the left there is a sidebar where you can select a card, here there are three cards, one for each color. On the right there is a form with the information relative to the selected card.

When another card is clicked, the form values on the right is now displaying the information related to the new selected item.

For the new year, a big order has to be done containing more than 600 items. In this situation, the list takes a huge amount of time to load, or doesn’t even load at all. And if I’m lucky enough to have the 600 items displayed inside the sidebar, when another item is clicked on, I’m also waiting... This performance issue is terrible in terms of user experience and needs to be fixed!

First, I need to make sure whether the performance issue actually comes from React. To do so, the user interaction can be recorded with the Google devTools performance tab. When JavaScript is executed it is accounted in scripting. Here, scripting is taking most of the time. I can investigate more on what is happening.

I then used the React profiler to see if this long scripting task comes from long react render times or useless component re-renders. The profiler is included in the React Dev Tools Chrome extension. It records renders, why your components rendered and for how long.

Long story short, you mainly need to understand that a colored bar means that your component rendered, grey bar means it did not.

It did render

It did not render

And guess what? When an item in the list is clicked on, all the items contained in the list re-rendered 🥵 ! When the list is only 3 components long, it is ok to recompute but with 600 components, each click becomes quite expensive.

For example here is the result in the profiler when the second card in the sidebar is clicked on. Basically, everything re-rendered.

Profiler view when an other card is clicked on, everything renders

So the React rendering step needs to be lightened. What options do we got to perform this?

Find a way to reduce the number of items that need to re-render

It can be done thanks to virtualization. Virtualization is a technique to compute in the DOM only the elements located inside the user's viewport. When the user scrolls, elements will be added/deleted in the DOM. A few librairies already implement this for us https://github.com/bvaughn/react-window.
Find a way to avoid useless re-renders

Why virtualization is not the right solution here?

Virtualization is a good option when your rendering phase is taking a long time. But here, it’s the re-renders that take time. If re-renders issues are solved, displaying 600 items only once is not an issue.

Moreover, the average user is making orders of about 50 items without a powerful computer. Virtualization is not very useful for short lists.

Finally, the height of each item card is variable (when a field is filled, the item card will grow), and the number of item is variable (the list can be filtered, items can be created or deletes). Windowing librairies are not well suited for variable items size or number, some custom functions have to be added to make it work. For the user this can lead to a laggy scroll. And re-renders issues won’t be solved.

So let's solve those useless re-renders!

Why do components re-render when another card is clicked on?

The structure is the following: OrderItemsSelection is the parent component, it contains the Form section and the Sidebar. The Sidebar itself has children: the SidebarCards (as many as items).

OrderItemsSelection has a state, selectedItem, and a state setter, setSelectedItem for the selected item id (thanks to a useState hook).

OrderItemsSelection passes selectedItem as a prop for the Form section and the Sidebar.

OrderItemsSelection also passes setSelectedItem setter to the Sidebar, Sidebar passes the setSelectedItem setter to its SidebarCard children to be used on card click.

So when the red shirt card is clicked on, a new state is setted with setSelectedItem, the selected card is now the red shirt card.

As the selected card is a state of the component OrderItemsSelection, it re-renders: a component state change triggers a re-render. Therefore all its children also re-render: the Form section, and the Sidebar. As the Sidebar re-renders, so are its children: the white/red/blue shirts cards.

In "components" tab the value attributed to Hook 1 is displayed (which caused a render)

Component updating schema

But in fact what do we truly need to render in the sidebar? If we take a closer look, the blue shirt card does not need to be rendered as none of its props actually change.

How to prevent re-renders

What is memoization?

Let’s take a look at Wikipedia definition:

In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.

So here, the Sidebar is a parent component of each card component. So we want to cache the blue card component computations because it has unchanged prop values, ie memoize this component. We can find more information about component memoization in the React documentation:

React documentation:

React.memo is a higher order component. If your component renders the same result given the same props, you can wrap it in a call to React.memo for a performance boost in some cases by memoizing the result. This means that React will skip rendering the component, and reuse the last rendered result.

Memoize a React component with React.memo()

Ok so if everything goes as planned I just have to add the memoization to SidebarCard component, and React will magically understand that if props values didn’t change, the component does not need to re-render.

      export default SidebarCard;

Before memoized component

      export default memo(SidebarCard);

After memoized component

But it would be too easy won’t it be?

Expectations

Reality

When we launch the profiler and focus on the blue shirt card:

We can see that components are memoized thanks to (Memo) just after component name, but we can also see that React considered that props did change for the last card: sidebarItems and onSetSelected. What’s interesting is that it recognized that the isSelected boolean did not change. It’s a boolean so the comparison is only made based on the boolean value. With functions, array and object, it’s different. Even if the objects have the same properties, they are considered equal only if it refers to the exact same object.

const Laura1 = { name: "Laura" };
const Laura2 = { name: "Laura" };

Laura1 === Laura2;
// => false

Functions, as well as arrays and objects are stored by reference into memory and not by value. We call these non-primitive data types as opposed to primitive types (string, boolean, number, undefined or null) which are stored by value. Here is a great article to understand all about it.

Memoize a function with useCallback()

This means that a new function onSetSelected was created earlier, which has the same value as the former but not the same reference. They are two different entities even they look alike.

onSetSelected is a function that is called when a card is clicked on to set the new selected item id.

onSetSelected is passed from the sidebar to the card here:

const Sidebar = ({
  onSetSelected, // => received here
  sidebarItems,
  selectedItemId,
  allowMultiSelect = false,
}: SidebarProps): JSX.Element => {
return (
  <div>
    {sidebarItems.map((sidebarItem, index) => (
      <SidebarCard
        sidebarItem={sidebarItem}
        positionIndex={index}
        onSetSelected={onSetSelected} // => passed here
        isSelected={selectedItemsId === sidebarItem.id}
	    />
     ))}
  </div>

onSetSelected is defined in the Sidebar parent:

const OrderItemsSelection: React.FunctionComponent = () => {
  const {
    initialValues,
    submit,
    selectedItemId,
    setSelectedItemId,
  } = useItemsSelection();

  // => onSetSelected definition
  const onSetSelected = (index: number) => {
    setSelectedItemId(index);
  };

  return (
    <Formik initialValues={initialValues} onSubmit={submit}>
      {({ values, validateForm }: FormikProps<ItemsSelectionValues>) => {
        return (
          <>
            <Sidebar
              sidebarItems={values}
              onSetSelected={onSetSelected} // => passed here
              selectedItemId={selectedItemId}
            />
            <ItemsSelectionForm
              initialValues={values.items[selectedItemId]}
              selectedItemId={selectedItemId}
            />
          </>
        );
      }}
    </Formik>
  );
};

So when the OrderItemsSelection component re-renders, a new onSetSelected is generated. The blue shirt card receives a new function (weirdly same looking as the previous one). So it has to re-render.

We want to avoid recomputing the function, as nothing has changed except the re-render in OrderItemsSelection. In short, we need to memoize the function. Fortunately, React has already implemented this for us. A big welcome to the beautiful useCallback hook which will memoize our function!

const onSetSelected = (index: number) => {
      setSelectedItemId(index);
      };

Before without useCallback

    const onSetSelected = 
      useCallback((index: number) => 
      {
        setSelectedItemId(index);
      }
    , []);

With useCallback

Now it’s time to try if our onSetSelected is still a guilty prop.

Profiler with memoized onSetSelected function

Component updating schema

Nice shot, onSetSelected is not guilty anymore, but we still have one targeted prop: the sidebarItem .

Use React.memo() comparison function to tell React when a component should re-render

Same protocol, as done before, where is sidebarItem defined?

const Sidebar = ({
  onSetSelected,
  sidebarItems,
  selectedItemId,
  allowMultiSelect = false,
}: SidebarProps): JSX.Element => {
return (
  <div>
    {sidebarItems.map((sidebarItem, index) => ( // => defined here
      <SidebarCard
        sidebarItem={sidebarItem} // => passed here
        positionIndex={index}
        onSetSelected={onSetSelected}
        isSelected={selectedItemId === sidebarItem.id}
      />
    ))}
  </div>

When Sidebar renders it recomputes the map, and each sidebarItem is recomputed. Since it’s a brand new object and as explained earlier, with a new reference, so it is not equal to the previous object even if it contains the exact same values. So how can we tell our card component to take into account only deep equality of previous and current sidebarItem? Once again, React team has already a solution right in the React.memo doc:

By default it will only shallowly compare complex objects in the props object. If you want control over the comparison, you can also provide a custom comparison function as the second argument.

Nice, the next step is to define a function that can compare the SidebarCard props in a customized way.

Stringify is not suitable for deep comparison

We could have thought to compare the stringifying version of the two objects, but it’s not the best idea as when we have equal properties not in the same order, it would have returned false.


    const Laura1 = { name: "Laura" };
    const Laura2 = { name: "Laura" };
    JSON.stringify(Laura1) === JSON.stringify(Laura2);
    // => true


    const Laura1 = { name: "Laura", age: 25 };
    const Laura2 = { age: 25, name: "Laura" };
    JSON.stringify(Laura1) === JSON.stringify(Laura2);
    // => false

We can use a deep equality. A deep equality is an equality based on what the object owns. Lodash provides isEqual to evaluate deep comparison.

const Laura1 = { name: "Laura", age: 25 };
const Laura2 = { age: 25, name: "Laura" };

JSON.stringify(Laura1) === JSON.stringify(Laura2);
// => false
isEqual(Laura1, Laura2);
// => true

Therefore, the isEqual lodash function is a wiser choice to compare the props.

export default memo(SidebarCard);

Before custom comparison

    
    const compareProps = (
      propsBefore: SidebarCardProps,
      propsAfter: SidebarCardProps
    ) => {
      return isEqual(propsAfter, propsBefore);
    };
    export default memo(SidebarCard, compareProps);

With custom comparison

Drum roll...

🥳 Yeah, got it! So for the 600 items list, when the second card is clicked on, here is the after memoization:

What if we only use memoize custom comparison without useCallback?

The isEqual on the onSetSelected function would have worked, the card wouldn’t have re-rendered. But it was a nice learning in case you have a function which is triggering a re-render.

Is there a difference between useMemo and useCallback?

useMemo is usually used for variable memoization. But you can use a useMemo to memoize a function.

React documentation:

useCallback(fn, deps) is equivalent to useMemo(() => fn, deps).

Why not always use memoization?

Memoizing is not free, it costs memory and the comparison cost. It adds complexity to your code so it will need more efforts to read/to refactor.

Conclusion

Thanks to React.memo() HOC and memoization hooks, useless re-renders were avoided! I hope you've learned a bunch of things!

Go further:

React: Fantastic Hooks and How to Use Them

React-Virtualized: Why, When and How you should use it

Ivan Akulov’s thread on re-renders

Use React.memo() wisely

The Complete Guide to useRef() and Refs in React

Mastering React-admin Resources to Improve Your App Performance

Tue, 18 Jan 2022 00:00:00 GMT

Let me tell you the story of how I improved my react-admin app runtime loading performance from 1 minute to 1 second.

I work at Theodo on a project with many websites. To configure and manage all of them, we had an administration tool (let's call it AdminV1). But AdminV1 was old: it was an Angular project coded with coffee.script and jade, and was not very maintainable. And the product was difficult to use: there were too many menus, UX was bad, it was difficult to onboard new administrators on the application.

The team decided to create a brand new application with modern technologies: AdminV2, with react-admin, a framework for building administration applications.

Everybody was happy: for developers, it was easier to implement new features. With a few lines of code, they could create tables to administrate users, app configurations and privileges. For Product Owners and administrators, the design was more ergonomic and it was easier to manage their applications.

We migrated many features from AdminV1 to AdminV2 and added new administration tools. Sponsors were delighted. But sprint after sprint, AdminV2 became slower to start. After some months, users had to wait more than one minute to access the app. It was painful to develop new features on AdminV2 because each hot reload took 1 minute, and users were frustrated by waiting so long to access the application.

We started to investigate and found that by deleting some Resources in our application, the speed at runtime increased significantly

But what are react-admin Resources ?

"<Resource> components are fundamental building blocks in react-admin apps. They are strings that refer to an entity type.", according to react-admin documentation.

To illustrate, on my project, we need to administrate users of websites. We have to create a resource "/user", associated with components:

UserList (for table view)
UserEdit (to edit an user)
UserCreate (to create a new user).

import UsersList from "components/Users/list";
import UsersEdit from "components/Users/edit";
import UsersCreate from "components/Users/create";

const UserResource = () => (
  <Resource
    key="/user"
    name="/user" // endpoint of the API to call to get users
    create={UsersCreate}
    list={UsersList}
    edit={UsersEdit}
  />
);

React-admin is in charge of storing this information. When the user goes on /user URL, react-admin displays the list, edit or create view and manages all the logic to get, edit and create users.

Sounds like magic, not helping us understand why creating many Resources degrades the performance.

I dived deeper into the react-admin code to understand how the framework creates resources and links them with components.

A <Resource /> component is declared like this:

const Resource = (props: ResourceProps) => {
  const { intent = "route", ...rest } = props;
  return intent === "registration" ? (
    <ResourceRegister {...rest} />
  ) : (
    <ResourceRoutes {...rest} />
  );
};

React-admin Code

Your intent can be either registration or route:

ResourceRegister's purpose is to dispatch an action that saves your Resource into a redux store. Each time you define a <Resource /> component in your app, react-admin will register your Resource's props into the redux store:
- the name, which is the URL of the resource and also the API endpoint to fetch your entity data
- which views are available to manage the entity. For our user entity, available views are list, edit and create

dispatch(
  registerResource({
    name,
    hasList: true,
    hasEdit: true,
    hasShow: false,
    hasCreate: true,
  })
);

React-admin Code

ResourceRoutes is rendered when a user wants to access the URL associated with the Resource. The URL (which is also generally the endpoint of the API) is stored in the Resource's name prop. ResourceRoutes renders all <Route /> components associated with each view defined in the Resource. For our Entity "user", it will be:

const ResourceRoutes = () => (
  <ResourceContextProvider value={name}>
    <Switch>
      <Route
        path={`${basePath}/create`} // Create view
        render={(routeProps) => (
          <WithPermissions component={CreateUser} {...routeProps} />
        )}
      />
      <Route
        path={`${basePath}/:id`} // Edit view
        render={(routeProps) => (
          <WithPermissions component={EditUser} {...routeProps} />
        )}
      />
      <Route
        path={`${basePath}`} // ListView
        render={(routeProps) => (
          <WithPermissions component={ListUser} {...routeProps} />
        )}
      />
    </Switch>
  </ResourceContextProvider>
);

React-admin Code

For example, if an administrator connects to a react-admin app and wants to see a list of users, the framework will load resources like this:

This allows calling the API only when the user goes on the associated Menu and to not call all the endpoints available in the application at runtime.

Ok, so what's going on with AdminV2 ?

In AdminV2, we defined our Resources like this:

const RootComponent = () =>
{
  const resources = getResources()
  return (
    <AdminContext {...props}>
      <AdminUI>
        {resources}
      </AdminUI>
    <AdminContext />
}

GetResources returned all Resources in our application: user Resources and other 600 Resources (yes that's a lot).

So what's happening when a user access AdminV2 ?

First, over 600 <Resources /> components are rendered
Then, each of these <Resources /> dispatches an action to store their data in redux store (the "registration" intent seen above)
React-admin redux reducer gets these 600 actions and for each of them, it stores the data in the redux store
Finally, when the reducer has updated the store with all resources action payload, the home page of AdminV2 is displayed

These 4 steps take more than a minute. Eventually, the user has gone for a coffee, otherwise, he left the app and will never come back again

How to avoid this 1 minute of loading time?

We thought about diverse solutions:

1 - Load only resources to which the user has access

On AdminV2, to access each menu, the user needs to have the associated privilege. If the user has access to one menu, we don't have to load all resources, but only the resources used in the menu.

This solves the problem for regular users with few privileges. But Developers and Product Owners have access to all menus.

2 - Patch react-admin to modify Resource component

We thought of modifying directly the react-admin Resource component, to modify the dispatch action. Instead of dispatching one action for each Resource, we could dispatch one action for multiple resources. This way, we don't have to update the state 600 times.

But this solution is not very maintainable: it implies updating the patch each time we update react-admin.

3 - Lazy load Resources

We don't have to create every Resources at the initialization of the application. An administrator will never access all AdminV2 menus and, even if he wanted to, in most cases, he doesn't have all menus accesses. We need to load Resources only when the user needs them. So we decided to create them when the user accesses the associated url. We modified our RootComponent like this:

const RootComponent = () => {
  const [resources, setResources] = useState([]);
  const location = useLocation();

  useEffect(() => {
    const resourcesToAdd = getResources(location.pathname)
    setResources([...resources, ...resourcesToAdd]);
  }, [location.pathname])

  return (
    <AdminContext {...props}>
      <AdminUI>
        {resources}
      </AdminUI>
    <AdminContext />
  )
}

We modified the getResource function so that it takes the current URL in parameters and returned the Resources associated with the URL.

For example, if an administrator wants to access the user administration menu (URL '/user'), getResources will only return

<Resource name="/user" list={UserList} edit={UserEdit} create={UserCreate} />.

The RootComponent will render the new Resource:

The first intent will be registration and the Resource will be stored in redux state
The second intent will be route. React-admin will create the Routes for user entities and the administrator will access the user management menu

In order not to save resources each time the user goes back on the same URL, we added a filter:

const RootComponent = () => {
  const [resources, setResources] = useState([]);
  const [loadedResources, setLoadedResources] = useRef([]);
  const location = useLocation();

  useEffect(() => {
    const resourcesToAdd = getResources(location.pathname).filter(
      (resource) => !loadedResources.includes(resource.name)
    );
    setResources([...resources, ...resourcesToAdd]);
    setLoadedResources([
      ...loadedResources,
      ...resourcesToAdd.map((resource) => resource.name),
    ]);
  }, [location.pathname]);

  return <AdminUI>{resources}</AdminUI>;
};

Let's sum up

Before, the user had to wait more than one minute to access the app. Then the navigation was quite quick.

Now, the user access AdminV2 in one second. And the navigation stays quite quick!

As developers, when using "black box" framework like react-admin, we do not always try to understand how it really works. By taking the time to explore the source code of these frameworks, by understanding what libraries they use, how they are implemented, we can save our time later and help us detect this kind of performance issue sooner.

That is the conclusion of AdminV2 performance story. And developers developed happily ever after and added a lot of features to AdminV2!

TypeScript: How Type Guards Can Help You Get Rid of 'as'

Wed, 12 Jan 2022 00:00:00 GMT

In TypeScript (TS), the "as" keyword is widely used when manipulating types. However, you should employ it with caution, as it does not provide any guarantee on the real types of your objects and could generate unexpected bugs. In this article, I will go into more detail about “as” and explain how to avoid it by using type guards, which represent a safer alternative.

Before I started working at Theodo Lyon, I had only used Vanilla JavaScript and HTML in my previous web development experiences. So when I discovered TypeScript (TS) earlier this year, it seemed like a miraculous solution for writing safer code! It was when I joined a team working on a web app using React jointly with TypeScript. React is a free and open-source "JavaScript library for building user interfaces" developed by Facebook, while Typescript is a programming language developed as a superset of JavaScript that adds optional static typing to it. In other words, defined variables are assigned a specific and permanent type at creation, hence securing their future use.

A few months ago, we started noticing many obscure bug occurrences with no obvious explanation. Also, the investigation was not satisfactory due to the lack of error logs: usually, the user would simply land on a blank page instead of the desired one.

One of the most common causes was that some objects suddenly became undefined while that was not an option for TS since they had different types. For example, we encountered error messages such as :

However, most of these occurrences would not even leave any error message and could lie undetected for a long time. Hence the investigation was even more painful!

Through our analysis, we found out that most of these errors were linked to the usage of the as keyword in TS.

For example, suppose we have an API used to retrieve books data, and a type Book defined as:

export interface Book {
    id: number;
    author: string;
	publisher?: string;
}

When I fetch the book of id 5:

const book = fetch("baseUrl/get-book?id=5");

the received object will be of type unknown. But since I know the expected return type, I can force TS to use the type Book by using as:

const book = fetch("baseUrl/get-book?id=5") as Book;

But how does "as" work exactly?

This keyword is a Type Assertion used to inform the compiler to consider an object as a different type from the one inferred. The as assertion does not perform any transformation on the data stored in the variable and is removed at compilation. TS even warns us about this in the official documentation:

Furthermore, it only checks that the assertion is feasible. For example, you can not use it to turn a string into a number: "TypeScript only allows type assertions which convert to a more specific or a less specific version of a type." ("TypeScript Handbook")

By digging deeper in the source code, the as does not perform any checks on the object but only serves as a static typing tool:

Screenshot extracted from source code typescript.js

Usually, we used to employ the as keyword in order to handle data coming from external sources. Examples of such sources could be APIs or imported excel files. Since the content was not known in advance, the type could not be inferred, so we had to use as to let the compiler understand the right type.

However, using this keyword in this case creates a false confidence feeling. The purpose of as is neither to protect nor to ensure typing, its main use being informing us about the hypothetical type of the object. For instance, in our previous example, I could fetch a book without knowing that the API has changed its definition of the type Book. Indeed, we could imagine that the type of the id attribute has changed from number to string, when we write:

const book = fetch("baseUrl/get-book?id=5") as Book;

const nextBookId = book.id + 1;

TS thinks that book.id is still a number, which could result in errors.

So, once we figured out the purpose of as, we started by implementing what would later turn out to be a quick fix. We kept the as keyword, but added checks right after the problematic occurrences to make sure that the object’s attributes were not undefined:

const book = fetch("baseUrl/get-book?id=5") as Book;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
if (book.author !== undefined) {
    const authorLastName = book.author.split(" ")[1];
}

This could prevent the problems cited above. However, it was not sustainable in the long-term, because we had to remember to perform the check every time we used an object whose type had been defined with as. Moreover, an "ignore" annotation was needed since the type was clear for our code linter eslint, which considered that the type defined by as was correct.

Yet, some of the bugs persisted, until one day, we discovered type guards through a workshop. Little did we know, at the time, that they could solve most of our problems!

But first, what are type guards?

A type guard is a function that allows you to narrow the type of an object to a more specific one by performing certain checks. Hence, after calling the function, TypeScript understands the new type of the object.

typeof and instanceof are two built-in examples of such type guards:

typeof only returns one of the following: "string", "number", "bigint", "boolean", "symbol", "undefined", "object", "function":
```
typeof 90 === ‘number’

typeof "abc" === ‘string’
```
instanceof checks whether an object is of a specific type, for example:

new Date() instanceof Date === true

However, instanceof does not work with TypeScript interfaces, which are what we usually use.

Since neither operator could solve our issues, we decided to build our own brand new type guards. Let us consider for instance an element object whose type is unknown being actually of type Book. A working function allowing us to verify this assertion would be:

const isBook = (element: unknown): element is Book =>
    Object.prototype.hasOwnProperty.call(element, "author")
    &&
    Object.prototype.hasOwnProperty.call(element, "id");

A custom type guard takes an element in input and returns a type predicate, which in our example would be element is Book. Now, whenever we handle an element that could be a Book, we can use this function to check its type. Also, once the variable verifies the checks, TypeScript interprets its type as a Book allowing us to access its attributes:

const gift: unknown = getGift();

if (isBook(gift)) {
    /*
    * In this if-block the TypeScript compiler actually
    * resolved the unknown type to the type Book
    */

    read(gift);
    // read needs an argument of type Book

    return gift.author;
}

The other advantage of type guards is that we can unit test them, preventing problems:

Here is an example with jest (the JS testing framework we use):

it("should return true if the element is a book", () =>{

    const element1 = { author: "alpha"};

    expect( isBook(element1)).toBe(false);

    const element2 = {"id": 2000};

    expect( isBook(element2)).toBe(false);

    const element3 = { author: "alpha"; "id": 2000};

    expect( isBook(element3)).toBe(true);
})

Still, as we can see above, our type guard is incomplete. An object such as { author: 1904; "id": new Date()} would be wrongly considered to be of type Book.

To avoid this mistake, the function should also check the attributes’ types. Hence, We can modify it as follows:

export const isBook = (element: unknown): element is Book =>
    /*
    * We can not use "in" because element
    * has type "unknown"
    */
    Object.prototype.hasOwnProperty.call(element, "author") &&
    typeof element.author === "string" &&
    Object.prototype.hasOwnProperty.call(element, "id") &&
    typeof element.id === "number";

However, this fix will not be effective since the element type would still be unknown, preventing us from accessing its attributes directly:

No worries, the deliverance could result from 2 options:

either we split the function into 2 type guards, verifying the attributes' existence first, then their types each time we handle types

or, writing up a generic function hasAttributes that could check any attribute’s actual presence, permitting us to scale up the types’ checking. An example of such usage could be:

export const isBook = (element: unknown): element is Book =>
hasAttributes(element, ["author", "id"]) &&
typeof element.author === "string" &&
typeof element.id === "number";

Finally, we can access the attributes because we have checked their existence first.

Wait a second, how can we define `hasAttributes`?

The function takes an object of unknown type and returns a type predicate depending on the attributes given as an input to the function. In other terms, it needs to be a "generic", defined for instance in the following way:

export const hasAttributes = <T extends string>(
    element: unknown,
    attributes: T[]
): element is Record<T, unknown> => {
    if (element === undefined || element === null) {
        return false;
    }
    return attributes.every((attribute) =>
        Object.prototype.hasOwnProperty.call(element, attribute)
    );
};

The type Record<T, unknown> means that the element object in input does possess the wanted attributes but their values are of type unknown.

This is all great, but can we automate it a bit?

It is true that many type guards libraries already exist, like typecheck.macro, but we have not had much luck using them. The main pain point with this library for example was that it makes our unit tests fail, we could not make it work with jest for the present time. Also, some other libraries require us to change all our type declarations to fit theirs in order to use them.

However, since we now know how important type guards are and how much they can ease the developer experience, their automation could be feasible but is at the time being still a challenge.

Conclusion

As we have seen, we have nearly stopped using the as assertion to state objects’ types, since it is very error-prone and rarely necessary. In contrast, type guards have proven to be amazing tools, making our code cleaner, more stable, and more robust. In most cases, they can replace all of your as usages. As a matter of fact, my colleagues and I were not able to come up with any good occurrence which could not be replaced with a type guard, however hard we tried. Indeed, anytime you are faced with a type incoherence (a wrongly typed library, external data etc.) you can resolve it using a type guard.

If you still are not convinced about their enormous potential, try them for yourself. Write your first type guard and get ready to see your developer experience improved!

Why You Should Encrypt Your Disk, and How To Do It on Ubuntu with a Dual Boot

Mon, 10 Jan 2022 00:00:00 GMT

Why use disk encryption?

It goes without saying that your computer can contain sensitive information about you or the organization you are working for, such as:

personal pictures
password or credit card numbers saved in your browser (and so on, access to your drive, important online accounts such as administrative ones, bank...)
emails
administrative documents (come one, we all store scans of our id or our social security number somewhere...)
code or projects
financial or operational spreadsheets

Why is it dangerous?

Stolen identity

Your social security number, your id, or other administrative documents can be used to open bank accounts or credits in your name!

Fraudulent banking transaction

Stolen card numbers are very often sold to groups that will use them to perform online payments on the internet. The point is you can not realize that your number has been stolen until they use it. Then it is too late.

Blackmailing

You or your organization can be the victim of blackmailing. You for your personal pictures, for example and your organization for sensitive information, blueprints or source code.

Cyber-attack

If the stolen machine belongs to a company and contains employee credentials (VPN access for example), they can be used as an entry point on your company network, for a bigger cyber-attack such as ransomware.

Source code leak

If the source code of one of your projects leaks, it makes it easier for hackers to find security breaches in your product or application. It would also make it easier for rival companies to copy your product.

How is it technically possible and how to prevent it?

Your password does not protect your files and data

Even without your password, someone with access to your hard disk can mount it on another operating system (for example by using an Ubuntu live USB key, a very common thing) and access some of the files it contains! It is an easy hacking practice. Nowadays, shady organizations buy second-hand laptops or hard drives to collect sensitive information about their former owner and sell them.

In this video, the French Youtuber Micode shows us how he managed to collect sensitive data and files from second-hand hard disks he bought online:

Also in January 2021, Kaspersky published on their blog an edifying study on data exposed in second-hand devices: **90% of the second-hand devices contain sensitive data!**

"Kaspersky’s Global Research and Analysis Team (GReAT) has examined security in secondhand devices. [...] An overwhelming majority of the devices the researchers examined contained at least some traces of data — mostly personal but some corporate — and more than 16% of the devices gave the researchers access outright. Another 74% gave up the goods when the researchers applied file-carving methods. A bare 11% had been wiped properly."

Sarah Pike on Kaspersky's blog, Jan.29 2021

Encryption to the rescue!

Encryption is a way to prevent that, as an encrypted file can not be understood without the key to decode it.
On Ubuntu, you can encrypt your home folder after installation to protect the files inside. This is a very convenient practice, however it comes with major trade-offs:
- Encryption is handled at the application level. As a consequence, it can be slow, especially when you need to copy or download lots of files (when installing software or project dependencies for example).
- Your computer also contains files outside your home directory that can also provide information on you, such as the installed library or softwares for example. Also, some programs keep information in temporary files or log files. And any information can be temporarily stored in you swap space.
Full disk encryption leverages those two trade-offs as it provides faster kernel-level encryption, and encrypts your whole Ubuntu installation (except the boot partition).

Tutorial part: How to set up full disk encryption on Ubuntu

This article will not cover dual boot set-up, and assume that you either already have a fully-functional dual boot on your device, or your device is ready for installing Ubuntu on dual boot, or you don't need it.

If you want to set up a dual boot, or enable Windows 10 disk encryption alongside Linux disk encryption, I recommend you switch to this longer, more complete tutorial by Mike Kasberg.

(optional) Back up your data if you want to restore it after installation

Disclaimer: you will still have to reinstall your apps, but they should restore their current settings after reinstallation.

(optional): remove all node modules, venv, composer dependencies, ...

First, you might not want to backup folders like node_modules or other project libraries as they can be very long to copy due to the very large amount of file they contain. Moreover, it is easy to reinstall them after your reinstallation.

You can remove them manually or use the following command lines to remove all the folders with a given name in your home directory. Feel free to adapt them to the project you have.
```
#in your home directory
cd ~
#delete all your node modules
find . -name 'node_modules' -type d -prune -exec rm -rf '{}'
#delete all your python venv (supposing you named them venv)
find . -name 'venv' -type d -prune -exec rm -rf '{}'
```
copy your home to an external drive. You can copy your entire home, or select only some folders you want to keep. Here are a few files and folders you might want to keep if you want to keep your apps settings and preferences:
- .ssh
- .config
- Documents
- Downloads
- ...
This list is not exhaustive and might depend on the apps you use

Reinstall Linux with full disk encryption

For this you will need a live CD/usb key of Ubuntu.

This tutorial has been tested on Ubuntu 20.04 LTS.

If you want to keep your dual boot

Unfortunately, the "Install Ubuntu alongside windows 10" option in the Ubuntu installer does not provide full disk encryption. We will need to set it up ourselves. If you don't need a dual boot you can skip to the next section

Select "Try Ubuntu" as you will need to use a few tools to prepare your disk before the installation

Define 2 disk partitions: one that will be encrypted and contain logical volumes for swap and root and one for boot
1. Open GParted
1. Delete your previous Ubuntu partition if you have one (To do so select the partitions and click on the red button in the top bar. Pay attention not to delete any windows partition). Your disk should look like something like (some unused space):
1. Then create 2 new partitions:
  1. A partition for boot:
    - size = 500Mio
    - file-system = ext4
    - label = boot
  2. A partition that will be containing swap volume and your Ubuntu installation:
    - size = <the size you want for your Ubuntu, including swap>
    - file-system = unformated
    - label = root
  At this point, your disk should look like something like this (2 new partitions instead of the unused space)
Note the name of your root partition (here /dev/nvme0n1p6) and boot partition (here /dev/nvme0n1p5). Apply your changes and then you can close GParted!
Encrypt your main partition and split it in two logical volumes for swap and root.

For this, open a terminal and use these commands:
1. get superuser rights: sudo su
2. encrypt your boot+swap partition: cryptsetup luksFormat <your_root_partition> (e.g. in my case: cryptsetup luksFormat /dev/nvme0n1p6 ). Type YES to validate and choose a passphrase (that you will need to unlock your disk at boot time)
3. open the encrypted partition: cryptsetup luksOpen <your_root_partition> ubuntu (in my case: cryptsetup luksOpen /dev/nvme0n1p6 ubuntu)
4. create a physical volume to access this partition: pvcreate /dev/mapper/ubuntu
5. create a volume group to contain your future logical volumes (root and swap) inside this physical volume: vgcreate vgubuntu /dev/mapper/ubuntu
6. Inside this volume group, create 2 logical volumes:
  - one for swap (here I use a size of 4G, feel free to adapt your swap size according to your hardware and needs. Usually swap should be around half your ram): lvcreate -n swap -L 4G vgubuntu
  - one for ubuntu (root), that will use all the remaining space: lvcreate -n root -l +100%FREE vgubuntu
7. Optional: If you want to format your partitions using CLI, you can format them right now. Otherwise, this can be done using the Ubuntu installer:
  - mkswap /dev/mapper/vgubuntu-swap
  - mkfs -t ext4 /dev/mapper/vgubuntu-root
At this point, your terminal should look like this:

Now that all your partitions and logical volumes are ready, you can start your Ubuntu installation (finally)!
1. Open the installer, and follow the installation process until you get to this screen:
Pick "Something else" and Continue
1. You should see all your disk volumes. Here we need to tell Ubuntu which volume to use for which use.
  - Select /dev/mapper/vgubuntu-root, click Edit and pick the following settings (check format if you didn't do it with CLI):
  - Select /dev/mapper/vgubuntu-swap, click Edit and pick the following settings:
  - Select the boot partition you created in step 1-c (/dev/nvme0n1p5 in my case), and pick the following settings:
2. Click install now to start the installation.
Your partition changes should look like this (if you formated them in the installer):

You can finish the installation process.

⚠️ If you want to restore your home folder, you need to create a user with the same name and password as your original one!

⚠️ Do not restart your device at the end of installation! At this point, Ubuntu is installed but it does not know that your disk is encrypted. Thus it is not able to boot yet. We will configure it in the next part.
Setup /etc/crypttab, for Ubuntu to know that then encryption is encrypted and how to unlock.

To do so, click "keep trying" at the end of the installation, and open the terminal. We will mount your new Ubuntu partition and use chroot on it to edit this /etc/crypttab and apply the changes.
1. Get superuser rights: sudo su
2. Get your encrypted partition UUID ( ⚠️ not the PARTUUID), it will be needed to set up the crypttab:
  
  blkid <your_encrypted_partition> (e.g in my case: blkid /dev/nvme0n1p6).
  
  Keep it for later.
3. Mount ubuntu: mount /dev/mapper/vgubuntu-root /target
4. Mount your boot partition: mount <your_boot_partition> /target/boot (i.e. in my case mount /dev/nvme0n1p5 /target/boot
5. use chroot on the /target and mount everything:
```
for n in proc sys dev etc/resolv.conf; do mount --rbind /$n /target/$n; done

chroot /target`

mount -a
```
6. Now, you can edit /etc/crypttab on the new installation (I'll do it with nano for the example):
  
  nano /etc/crypttab
  
  Copy/paste this inside the new crypttab (don't forget to replace your encrypted partition UUID!):
```
ubuntu UUID=<your_encrypted_partition_id> none luks,discard
```
7. Last but not least: apply the modifications of the crypttab:
  
  update-initramfs -k all -c

🥳You're done! Congrats, you can exit your live CD and restart your computer!

If everything worked as expected, you will be asked your passphrase to unlock your disk when you start Ubuntu.

If you don't need a dual boot

If you don't need a dual boot on your computer, you can also erase your entire disk and install Ubuntu on it. This process is much easier. On the Installation Type screen:

select "Erase disk and install Ubuntu".
Before continuing click on the "Advanced Features" button and select "Use LVM" and "Encrypt the new Ubuntu installation for security".

You can go along with the Ubuntu installer then.

(Optional) Restore your data

Log in on your new installation and create a new user (superuser). Log out from the user you want to restore and log in on the new superuser. Then you can copy/paste your home or the files you want to restore to their proper emplacement on the home of the user you are currently not logged in.

⚠️ As said earlier, for the restoration to work properly some files will need both of your users (former and new ones) to have the same name and the same password! You can change your password later.

Once it is done, you can log back on the other user and delete the other superuser if you do not need it.

Conclusion

By following this tutorial, you learned about various fields:

About security

Hardware security is an important part of cyber-security for organizations as well as for individuals. A device can be stolen, lost, or as you will probably throw or sell it at some point, and you want to make sure that the data inside it is not accessible if it happens.
Your password is not required to access some of the files inside your computer if someone has physical access to your hard disk. That can happen while you own it, or even after. To prevent it, you can encrypt your device.

About device encryption

You can use Luks to encrypt a physical volume and then split it in logical volumes for your different needs (swap, root, optionally a dedicated home volume).
Ubuntu can handle an encrypted physical volume if it knows it from /etc/cryptab, and then use its logical volumes as regular partitions.
You can trick Ubuntu installer by unlocking an encrypted partition using cryptsetup luksOpen. Generaly speaking, cryptsetup is a tool to encrypt partitions and unlock them for your OS to use them (including the Ubuntu installer)

About Ubuntu

You can copy a home folder from an Ubuntu installation and restore it on another one if you create a user with the same name and password. Unfortunately, libraries are not backed-up.
GParted is a tool on Ubuntu to manage your physical partitions.
Home folder encryption is not a good practice as the encryption is managed at the application level so it can often be slow. Moreover, you protect only your files. Full disk encryption is handled at the kernel level, hence it is much faster. It also ensures that absolutely no information can be extracted from your device, including your libraries, your swap space...

To go further

If you want to know more about the process of encrypting-decoding data, and when to use file encryption:
- A video explaining How Does Full Disk Encryption Work?
- A video explaining How Does Individual File Encryption Work?
The article on Kaspersky blog about data collection in second-hand device: https://www.kaspersky.com/blog/data-on-used-devices/38610/

Do not hesitate to browse Theodo's blog for more interesting articles or visit Theodo's website to learn more about us !

The React Testing Library Guide I Wish I Had

Tue, 21 Dec 2021 00:00:00 GMT

I would like to share with you a great library I have found, which makes front-end tests fun to write!

With this article, you will be able to write serenely your first tests with React Testing Library. To have a deeper understanding of the library you can use the documentation.

Is it really useful to write front-end tests?

The problem is that as your app grows you cannot be sure that everything still works properly each time you add some new code. Seems quite obvious, right? Then, why do so few people write front-end tests?

When I asked some people about it, they answered: "we don't have enough time", "it's not really useful"...

The first thing you should know is that, yes, at the beginning it seems that tests are not worth the time. But as your app grows it will be more and more difficult to implement them and your app will be more and more unstable. Then, you should implement tests from the beginning of a project: it is easier and it will be timesaver later.

The second thing to know: tests are definitely useful. A quick anecdote to convince you: the first test I have ever written spotted a bug.

Then it's time to learn how to test!

Why use React Testing Library?

Now that you are - I hope - convinced that you should test your front-end code, what library should you use?

My first advice is: don't use snapshots, as this article explains. To sum it up: snapshots are difficult to understand for developers and then it doesn't really test the app the way users use it.

When I implemented front-end tests on my project I chose React Testing Library. React Testing Library allows you to test by simulating human interactions with the UI, while being quite independent of the implementation.

Indeed, the closer the test is to the actual app usage, the more relevant it is.

As a bonus, using React Testing Library neatly can actually improve the accessibility of your app (See an article about accessiblity improvement to understand why).

How to test with React Testing Library?

The principle of React Testing Library is to render a component and then simulate some interaction with it, and then finally compare the result with the expectation.

1. Render a component

The first thing you have to do is to get the component you want to test. It can be a whole page or isolated components. The best thing to do is to find the right balance between testing a page with a lot of components, which can be long because of nested components but is closer to the user experience, and testing components, which is faster but sometimes less relevant.

Anyway, you should never test already tested components and behavior. For example, components from libraries are usually tested and you shouldn't test them again.

The easiest way to get a component is to use directly the render method:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";

test('My first test', async () => {
	render(<MyComponent />);
}

It is also possible to wrap the render function in a provider to mock some features, such as the Redux store or the internationalization:

import * as React from "react";

import {
  RenderResult,
  render as rtlRender,
  RenderOptions as RtlRenderOptions,
} from "@testing-library/react";
import { IntlProvider } from "react-intl";
import { Provider } from "react-redux";
import flattenMessages from "services/i18n/intl";
import frMessages from "translations/fr.json";

import rootSaga from "../redux/sagas-for-tests";

const defaultMessages = flattenMessages(frMessages);

const render = (
  ui: React.ReactElement,
  {
    initialState,
    store = storeCreator(initialState), // custom function to create the store
    messages = defaultMessages,
    ...renderOptions
  }: RenderOptions
): RenderResult => {
  const Wrapper: React.ComponentType = ({ children }) => {
    return (
      <Provider store={store}>
        <IntlProvider messages={messages} locale="fr">
          {children}
        </IntlProvider>
      </Provider>
    );
  };
  return rtlRender(ui, { wrapper: Wrapper, ...renderOptions });
};

With a custom render function like this you can render your component with a mocked state:

import { mockState } from '__fixtures__/state';
import { render } from 'utils/test-utils'; //custom render
	import MyComponent from "../MyComponent";

test('My first test with a mock state', async () => {
	render(<MyComponent />, { initialState: mockState });
}

Now that you can display a component, let us play a bit with it.

2. Simulate user interactions

To interact with the elements of a component you will need a way to access them. You have several ways to do this. You can, for example, query elements by text, label, placeholder, alternative text, or role. Regardless of what attribute you use you have three main methods to get an element.

Generally, I prefer to use getBy, which is the most simple one. But if you need to get the element asynchronously you can use findBy. It can be useful when you wait the results of an asynchronous method for example. If you want to check if an element is not in the DOM, I advise you to use queryBy, which returns "null" if the element was not found.

You should know that all of these methods will throw if more than one element matches, though you can overcome this limitation by using getAllBy, findAllBy, or queryAllBy to get an array.

Have a look at how you would query for a button reading “press me”:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";

test('Get button test', async () => {
	const { getByRole } = render(<MyComponent />);
	const pressMeButton = getByRole("button", { name: "press me" });
}

Once you have your button you can interact with it, by simulating a click on it:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Click on button test', async () => {
	const { getByText } = render(<MyComponent />);
	const pressMeButton = getByText("press me");
	user.click(pressMeButton);
}

Another classical way to interact with an element is when the user types something in an input field. This interaction can also be easily simulated. With this code, for example, you would simulate a user typing "some random text" in an input field labeled "myInputField':

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Fill input test', async () => {
	const { getByText } = render(<MyComponent />);
	const myInputField = getByLabelText("myInputField");
	user.type(myInputField, "some random text");
}

Sometimes you will try to get an element, but you will have an error message saying that it was not found. To help you debug that, React Testing Library has a wonderful method...

3. Debug

It is really easy to debug with React Testing Library. Indeed, it provides a method, debug, to display the entire DOM at a specific moment.

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Debug fill input test', async () => {
	const { getByText, debug } = render(<MyComponent />);
	const myInputField = getByLabelText("myInputField");
	debug(); //displays the DOM with an empty input
	user.type(myInputField, "some random text");
	debug(); //displays the DOM with the input filled with "some random text"
}

4. Examine the result

Now that you are able to render a component and interact with it the way the user would do, it is time to check that everything works as expected.

To do that you just have to use the expect method, for example, to check that an input has the expected text content:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Debug fill input test', async () => {
	const { getByText } = render(<MyComponent />);
	const myInputField = getByLabelText("myInputField");
	user.type(myInputField, "some random text");
	expect(myInputField).toHaveTextContent("some random text");
}

5. Mocks

Finally, you should know that you can mock the different API calls and hooks using Jest to fully test your app. Using mocks enables you to test highly complex functions. Moreover, it enables you to have control over the data.

Conclusion

Testing the front-end with React Testing Library improves the quality of your code itself, it enables you to test the user interactions and above all, it is quite intuitive. Now you no longer have any excuse to avoid testing your front-end!

Disrupting B2B payments & driving purchasing efficiencies for SMBs

Mon, 13 Dec 2021 00:00:00 GMT

SMBs are in need of innovative and connected tools that solve these complex problems. Theodo and Codat partnered to produce a proof of concept application that showcases how fintechs and incumbent banks can help small businesses drive payment efficiencies by leveraging consented data.

What is wrong with B2B payments today?

The traditional ways in which businesses pay bills is over complicated and manual, leaving room for error which increases the cost and burden on SMBs.

To help with growth, finance teams are increasingly expected to perform more strategic roles, yet many of the functions and processes they’re required to manage are archaic and in need of optimisation. Owing to these inefficient practices, the accounts payable team is often left struggling with mounting paperwork, the threat of late payments, data errors, and with no means of providing accurate management reporting. The following statistics outline the extent of this problem:

Poor payment processes cost up to $10 per invoices for SMBs.
60% of businesses admit to late payment fees causing reputational damage and supplier friction.
32% of businesses report duplicate payments for the same invoice.
56% of businesses admit planning is a challenge because they do not have visibility of upcoming bills to pay.

What can you do to help small businesses?

Visibility

Allow businesses to see all of their outstanding bills, their value, and due date in one place to enable them to accurately forecast and schedule upcoming spend.

Security

Reduce invoice fraud and streamline payments validation by using open banking data and other tools to verify a supplier’s bank details before executing large payments.

Automation

Minimise friction between different business systems and make it easier to see what is outstanding and then make the payments in a single interface.

Accuracy

Drive better accuracy with payment amounts and the status of the bill.

Speed

Enable SMBs to focus on their growth by saving them hours of manual reconciliation

What did we build?

Key Features:

Authorisation

Rather than build directly to the accounting platforms, we chose to integrate with Codat. This meant we could rely on a single data model and did not have to write libraries and handlers for complexities such as token refresh and request throttling.

We were able to save considerable engineering time by taking advantage of Codat’s Link product, an out-of-the-box and no-code solution that handles the authorisation process.

Retrieve list of bills

Once the connection is authorised by the user, Codat’s REST API makes it easy to retrieve a list of accounts payable invoices (bills) in a standardised format, meaning once we had built the integration for QuickBooks, it worked seamlessly for other platforms without requiring changes. For each bill we included information such as line item detail and expense category and even a jpg or pdf attachment of the invoice.

Mapping

It’s absolutely critical that the payment is recorded against the correct account in the business’s accounting system - Codat makes this easy by providing a list of accounts available in our API to query.

Reconcile payments

The final step of the puzzle involves reconciling the payment back to the business’s accounting platform. Codat’s API is bi-directional, which enables you to create a payment and reconcile this against outstanding bills. This ensures their accounting platform is always up to date and saves time with manual entry.

How did we achieve this?

Implementing agile practice, the Theodo team started with a proof of concept console application, to ensure the API calls were successful. The Codat Swagger made this process seamless, as it provided the developers with real-time visibility and a simple way of adopting these with Axios.

The team moved on to building a dynamic frontend experience for users using google MUI react components and next.js, with particular focus on transaction consistency, to ensure all data is displayed correctly and latency does not cause any issues.

“The tight feedback loop with the Codat team, combined with a focus on ease of operation, resulted in an ever-improving interface; for example, the team incorporated pagination and loading components to maintain user engagement. The API sandbox set-up meant the team could access the platform and create mock companies, integrating the user experience at every stage of the development process.”

The cost of building an API integration can easily run into the thousands, depending on the number and complexity of the integrations. Codat’s API has a number of advantages that drastically reduce the integration complexity, therefore increasing the speed to market. The first is that only a single integration is needed, a key factor in reducing complexity. Secondly, the API is simple, meaning less work is needed for the system to communicate. The data relationships between systems are also standardised, ensuring clean data transfer. Finally, the precise documentation means that if any changes do need to be made, they can be actioned quickly and efficiently.

The quality of the API offers clients a Portal with deep integrations to a vast number of external platforms, such as Xero and QuickBooks Online (QBO), as well as Desktop (QBD) and complex ERPs (including NetSuite and Intacct). These integrations allow for near real-time data syncs between systems, delivering instantaneous business value for a client.

How can Codat & Theodo help you build better SMB products?

Codat is the universal API for small business data. Codat provides two-way connectivity to over 30 different accounting, banking, and commerce systems used by small businesses, including Xero, Plaid, QuickBooks, Stripe, Oracle NetSuite, Shopify and Sage Intacct. Codat’s integrations are standardised to a single data model, allowing clients to build to Codat once rather than having to sink their own engineering resources into building and maintaining each complex integration themselves.

Theodo partners with businesses of all sizes, helping to increase their market share by building bespoke digital products that users love. Lead-time is critical when reacting to market opportunity, thus pragmatic technical solutions are essential.

Click here to learn more about how to build a B2B payments solution.

SadiqD | Partner @ Theodo

“The market is flooded with solutions that seem to solve every problem in the world. In reality, only a few do what they say on the tin. The more we’ve delivered solutions that centre around Codat, the more opportunities I see with it. For any business looking to help companies make the most from their financial data, Codat is truly a no-brainer. Our engineering team loves it, and the Codat team is so easy to partner with”.

Building Terravision(2021) using Deck.gl and Carto

Tue, 07 Dec 2021 00:00:00 GMT

Adding a detailed and interactive map to a web app can be a time consuming process. Pairing deck.gl and carto with react can make it pain free. Using Carto to manage your data and Deck.gl to render it can make for a great developer and user experience. Resulting in a web app rich with detailed and engaging visualisations.

TLDR: skip to case study section if you're too cool for context...

Intro

At the beginning of the Netflix show The Billion Dollar code, you’re introduced to two German developers, Juri Muller and Carsten Schlute. As the series progresses we learn more about Terravision, an application that the two developers and their team (ART+COM) develop that closely resembles the modern day map software we are all familiar with, Google Earth. The series takes place in the 90s and hence the algorithms and mapping software that we use today has progressed significantly from the first versions of Terravision and Google Earth.

Nowadays, as a web developer, the inclusion of a map in your application can surpass the simple use cases provided by the google maps API. Whether it be needing to display a dynamic view of nearby taxis for your journey home or a heat map showing what people in your city are up to; maps in modern day applications are rich with information and data. Two tools that can be used to build rich map visualisations in web applications are Carto and Deck.gl.

Simply put, Carto is a data warehouse that specialises in data for geospatial analysis with some magic under the hood.

If you’re building an application that requires demographic data detailing the populations of cities in Europe just search through the catalogue, get the URL of the data set and import it using their API. Carto is built with developers in mind and hence they have a great API that pairs really well with the second tool I’ll be talking about, deck gl. If you make a call to their SQL api to obtain the data you’ll see a result containing demographic data mapped to a set of long lat coordinates. This format (GEOJSON) is an industry standard that can be interpreted by most mapping software but can be tedious and cumbersome to generate manually so it’s great to have an easily accessible source.

Note: For readers in the UK another great source of geospatial data is the ONS.

So now we have our maps and the data to enrich them. We need a way to combine them and display the result in our web app. Deck.gl is a software specialising in mapping and geospatial visualisation. It offers a react library designed to create high-performance web-GL based visualisations for large datasets. Hence it’s one of the easiest ways to integrate geospatial visualisations into your applications. The library is really intuitive and highly optimised out of the box. It also contains a bunch of different features that will fit a large number of use cases required by your application.

Case Study

Let’s talk through a real world example. Say you’re an international news company and you want to revamp the user experience of your website. You cover news stories all across the globe but you’ve gotten feedback from users saying finding new, interesting stories can be difficult and your team has proposed a new map interface to display all the current popular stories. As well as displaying the stories on the map you think it would be nice to supplement the articles with contextual information about the countries you’re covering (GDP, HDI, Life Expectancy, etc).

Slight caveat. Unfortunately, Carto is not a free piece of software so for the example I will be using a dataset that I’ve downloaded from Kaggle. Hopefully talking through the full process that I have to follow to integrate a generic dataset into our application and how it differs from using Carto will allow me to highlight the convenience of using Carto in your workflow.

The Data

To begin with you’ll need to source the data for the contextual information. As noted above, a quick search on Kaggle you can find a list of demographic indicators for all countries in the world. Granted this is outdated but this is where something like Carto comes in handy as it provides up to date data in the required format.

Using Deck.gl with next.js and react

Now we have our data, the next task is to set up your project. Deck.gl works with react out of the box so you can use the next project starter to generate a react project and get going from there. Firstly, you’ll need to create a DeckGL component for your page. This is where all of your geospatial visualisations will sit.

<DeckGL initialViewState={INITIAL_VIEW_STATE} controller={true}>
  <StaticMap mapboxApiAccessToken={MAPBOX_KEY} />
</DeckGL>

The above code snippet would display a simple map on the page where you render the DeckGL component. You will see there’s a child component StaticMap. This child handles the map element of our page. Deck.gl can easily integrate with mapbox so all you need to do is create an account and use the public token to add a map to your Deck.gl component.

As you can see the map currently takes the whole of our page but we need to limit it to the hero section of the screen so we can see the articles as well. Adding the following prop to our Deck.gl component allows us to add some styling:

views={[new MapView({ width: '100%', height: '44%' })]}

The resulting page:

So now we have our map displayed on the homepage of our website, the next task is to take the geojson data we downloaded from Kaggle and add it to the map. For this I’ve opted to store the layers in the state which is populated when the page mounts. Using getStaticProps, I read the json file from disk and pass it as a prop to the homepage. This prop is then used to dispatch a redux action that takes the json data as an argument and returns the layers.

Note: The format of the geojson data is not fully inline with what deck.gl expects (one reason why using Carto would be nice here) and hence I have to do some transformation on data load. An alternative solution here would be to process the data once yourself and then have this loaded into the app when the page mounts.

I handle the creation of the layers inside a redux saga. For the geojson data I create a new instance of the PolygonLayer class.

// Create basic polygon layer from geojson data
 const europeLayer = new PolygonLayer({
   id: 'real-poly-layer',
   data: action.payload.geoJson,
   getPolygon: d => (d as { type: string; geometry: { type: string; coordinates: Position[][] } }).geometry.coordinates,
   getFillColor: [160, 160, 180, 200],
   getLineColor: [0, 0, 0],
   getLineWidth: 1,
   lineWidthMinPixels: 1,
   pickable: true,
   stroked: true,
   filled: true,
   wireframe: true,
 })

So we now have an instance of a polygon layer generated from the data read from the disk. The last thing to do now is add it to the Deck.gl component.

layers = { mapLayers };

The result:

As you can see we now have a polygon layer displayed on our map but none of the additional information included in the geojson is visible to the user. Most of deck.gl’s layers can be configured to display tooltips on top of the layer. The pickable prop that I added to the PolygonLayer allows the deck.gl component to access all the additional information and adding a getTooltip prop to your deck allows you to manually configure the structure of your tooltip.

getTooltip={({ object }) => {
if (object) {
    const typedObj = (object as unknown) as {
    properties: { story?: string; scalerank?: number; sovereignt?: string; level?: number; type?: string; geounit?: string; subunit?: string; name?: string; name_long?: string; economy?: string; continent?: string; pop_est?: number; gdp_md_est?: number; income_grp?: string }
    };
    if (typedObj.properties.continent) {
    return `Name: ${typedObj.properties.name_long}\nContinent: ${typedObj.properties.continent}\nGDP: ${typedObj.properties.gdp_md_est}`;
    } else {
    return `City: ${typedObj.properties.name}\nStory: ${typedObj.properties.story}`;
    }
} else {
    return null;
}
}}

Now you can see any specific data that you want when the user hovers over a certain section of the map. Without any need to specify coordinates, boundaries etc. Everything is handled for you.

The last feature we need to add to our map is an icon layer showing where the stories we have on our site come from. In the same saga I mentioned earlier we need to instantiate a new IconLayer class.

// Initial story data that could be pulled in from database
 const pointData = [
   { properties: { name: 'Chavignol', story: '5 Cheese Hacks Only the Pros Know' }, coordinates: [2.844560, 47.397411] },
   { properties: { name: 'Milan', story: '11 Hottest Street Light Trends for 2022' }, coordinates: [9.195585, 45.467905] },
   { properties: { name: 'Kyiv', story: '9 Best Practices for Remote Workers in the Shark Industry' }, coordinates: [30.479160, 50.44044] },
   { properties: { name: 'Zurich', story: '10 Ways Investing in Maple Syrup Can Make You a Millionaire' }, coordinates: [8.519453, 47.385598] },
 ]

 // Icon layer based on the story data above
 const layer = new IconLayer({
   id: 'icon-layer',
   data: pointData,
   pickable: true,
   iconAtlas: 'https://upload.wikimedia.org/wikipedia/commons/e/ed/Map_pin_icon.svg',
   iconMapping: ICON_MAPPING,
   getIcon: d => 'marker',
   sizeScale: 15,
   getPosition: d => (d as { coordinates: Position2D; exits: number }).coordinates,
   getSize: d => 5,
   getColor: d => [50, 168, 82, 255],
 });

The result is a map containing information about both the countries we’ve specified and the stories on our site.

Note: In the real world we would have the icon layer pull from a database so that it’s automatically updated on publishing of a new blog.

Technical Deep Dive

You’ve now seen how easy it is to display rich maps using deck.gl and Carto. I want to take some time to talk about what makes them such interesting pieces of software. In the world of geospatial analysis there are some standard data formats (GeoJSON, Shapefile, KML, CSV, etc) and ArcGis is considered to be at the forefront of the geospatial data handlers. You can think of an ArcGIS server as blob storage with an API for processing and access (although in reality it’s A LOT more complex and feature rich). The ArcGIS server is configured to interpret and process data in a geospatial format and make it easy for developers and analysts to use it where appropriate. Carto can act as both an intermediary or a replacement for ArcGIS. But using it as part of your stack can vastly optimise your load times and improve developer/user experience due to the way it handles your geospatial data.

The Power of the_geom

Let’s talk through another example, you want to create a polygon layer to display on a map of Europe to show the most recent population of each country. To generate this you would need three things:

A map of Europe - https://www.mapbox.com/
The contour data defining the boundaries between countries with a relatively high degree of precision - https://www.kaggle.com/sudhirnl7/human-development-index-hdi
The population data tied to each of the contours for a country - https://www.kaggle.com/sudhirnl7/human-development-index-hdi

So now you have the data, where will you store it? Well for a single country you will likely have millions of data points outlining its boundaries which can amount to ~1GB of data for each country. There are 44 countries in Europe. Now you have to load ~44GB for the user each time they want to see this visualisation. Carto offers a great solution to this. Instead of storing the contour data for a country on a point-by-point basis, it takes the data you’ve scraped from the internet (GEOJSON for example), ingests it and converts it into what they call the the_geom. This compresses millions of data points into one, which can easily be queried, manipulated or converted back into the GeoJSON format all using standard SQL commands. Now you’ve gone from having to load millions of rows of data to only 44.

So we have our data compressed and stored on the Carto servers which can easily be accessed using the Carto API, now we need to figure out how to get it displayed on our map.

More tiles than the kitchen floor

On the homepage deck.gl states it’s a webgl powered framework for large scale datasets. Given how the size of geospatial datasets can quickly grow, having a library that can process and handle large volumes of data on the web is key to building an application with a good user experience. Some of the features I find most interesting are:

Loading Data: Deck.gl is built on top of loaders.gl which is a library the same team built dedicated to handling the loading of large amounts of data. Providing loaders for geospatial, csv, json, vector, image files and more. Leveraging the loaders.gl library means that the load times of large datasets can be kept to a minimum with ease.
Bundle optimisation: Another interesting trait of deck.gl is that it’s built with web developers in mind. Hence features like tree shaking can also be easily configured to keep your bundles as small as possible.
64-bit layers: Given that there are so many use cases for deck.gl the software is built in such a way that it can handle visualisations for many scenarios. The use of emulated 64-bit floating point means that the precision and detail of the visualisations is incredibly high. This means if your data goes down to the centimeter scale then the user will have no issues zooming in to that level.

The End...?

So you've now seen how easy it is to add data rich map visualisations to your web applications. You can see all the code I used to build my demo app here and the link to the actual site TerravisionNews 2021. Any questions feel free to get in touch, PRs/comments welcome.

Comparison of Cloud Run and Lambda to render Blender scenes serverlessly

Tue, 09 Nov 2021 00:00:00 GMT

In a recent post, I explained how to run Blender render jobs on AWS Lambda functions, using Lambda container images. As a follow-up, today I am looking at the same workload running on Cloud Run, the easiest serverless way to run a container on Google Cloud Platform.

Test protocol

vCPUs and RAM

A quick disclaimer is that if you can choose between GCP and AWS, in this case, AWS seems to be a better choice because your Lambda functions can use up to 6 vCPUs whereas Cloud Run is limited to 4. When rendering Blender scenes, increasing CPU power has a huge impact on rendering times. I included a Lambda function with 6 vCPUs in our benchmark and it's about 30% faster than running on 4 vCPU-solutions (be it on GCP or AWS).

The topic of memory is interesting. With Lambda, you cannot independently choose the number of vCPUs and amount of RAM. Based on the amount of RAM you pick, the Lambda will be allocated a certain number of vCPUs. If you pick 10GB of RAM (the largest option available on Lambda) you will get 6 vCPUs. 6.8GB of RAM give you 4 vCPUs, so that's what I went for in order to have the same number of vCPUs on both Cloud Run and Lambda.

With Cloud Run on the other hand, you can independently set vCPU count and RAM amount. The maximum amount of RAM available is 16GB which is significantly more generous than what Lambda offers. Depending on what scene you are willing to render, this can be a crucial point because larger 3D scenes in Blender will require more RAM. So it might happen that you simply cannot render the scene you want on a Lambda with 10GB of RAM. In the use-case that led us to set up this system, I was rendering very small scenes so 10GB were more than enough.

Container image

The container image I am using with Cloud Run is very similar to the one I used on Lambda and described in the previous post. It is also based on the Ubuntu image with Blender that is maintained by the R&D team at the New York Times. The main difference is that I do not install the AWS specific SDK and I run an HTTP server with Gunicorn. The impact of this server on the overall performance of the solution will be minimal since the large majority of the response time is spent rendering the Blender scene.

Region

This recent blog post published by Guillaume Blaquiere shows that not all Google Cloud Platform regions are equal when it comes to Cloud Run performance. I had initially set up our experiment in the us-east4 region and, after reading this, I decided to add a test run in the fastest region. In his article, Guillaume Blaquiere finds out that his code runs 11% faster in the us-west3 region compared to us-east4. I got similar results.

Execution environment

Another variable I took into account is Cloud Run's execution environment. Google gives us the option to either run in a first or second generation execution environment. The second generation is in preview at the moment.

Test scene

For our test, I render a 500-pixel-wide square image of a test scene provided by Mike Pan and available on Blender's Demo Files page. I used Blender's Cycle rendering engine. You can see the output below.

Results

Test settings	Specs	Region	Average duration of 4 function executions (s)
Cloud Run	4 vCPUs, 16GB, first gen	us-east4	79.8
Cloud Run	4 vCPUs, 16GB, second gen	us-east4	81.4
Cloud Run	4 vCPUs, 16GB, first gen	us-west3	77.7
Cloud Run	4 vCPUs, 16GB, second gen	us-west3	71.5
Lambda	4 vCPUs, 6.8GB,	us-east-1	76.1
Lambda	6 vCPUs, 10GB,	us-east-1	51.5

A few observations:

The results with 4 vCPUs are all pretty similar
More vCPUs means noticeably faster render times. That's not breaking news but in our case this really meant Lambda was the way to go.
We can indeed see that Cloud Run does not perform in the same way in all regions. In this case, it performed 14% slower in us-east4 compared to us-west3.
The second generation execution environment of Cloud Run seems to be significantly faster than the first in the us-west3 region but slightly slower in the us-east4 region. Given the fact that this new generation is in preview for now, we can expect things to get better in the future but, for now, test
Using Cloud Run on 4 vCPUs in the us-west3 region is about 6% faster than Lambda with 4 vCPUs in the us-east-1 region.

Next steps

The impact of the region on Cloud Run performance made us wonder if there is a comparable impact on Lambda. So I may update this article with additional information on this.

I would also like to measure the performance impact of using Graviton2 Lambda functions for our task.

It would be interesting to see a serverless solution that lets us use GPUs as this would dramatically speed up rendering jobs. But would the cost of such a solution make it a better solution for our use case? I would have to try!

I will post more article on the topic of serverless Blender in the future. Follow me on Twitter to receive updates!

How to Generate Beautiful PDFs with React and Puppeteer

Wed, 27 Oct 2021 00:00:00 GMT

I recently had to provide a new functionality on my project: the "download as PDF" one. The first thing I wondered was why should we provide this functionality? Doesn't it already exist natively with pretty much all web browsers with a right mouse click / print / "save as pdf" option? Well I tried on my webpage and the result was really disappointing:

I had page breaks in the middle of my data tables, graphs, or any React component which aren't supposed to be split on different pages
I didn't have any title, header, page number except the ugly default ones provided by the browser options
I had many other UX/UI pain points that made the PDF quite unreadable

Example on a Boursorama random page, that looks like:

rendering:

That's when I said "ok this feature may be worth it", but how should I do it? There are many open source libraries that can generate PDFs. But my choice went naturally to the well-known, google-developed library: Puppeteer. According to me, it is the easiest way to deal with PDF generation of Single Page Applications. It may not be so if you don't deal with javascript bundles but with plain HTML/CSS. Indeed, there are easier solutions for this use case like wkhtmltopdf, html-pdf-node or jspdf for example.

In this article, I want to give you a few tips to generate beautiful PDFs of SPAs with Puppeteer. Firstly, I will explain to you how you can create a printable version of your page with React and Puppeteer. Then, I will show you how to use Puppeteer for the generation of your new printable page.

Render a printable version of your page

For this part you don't actually need to have Puppeteer or any other printer service set up. You can make your changes to your code as usual, and then ctrl+P on your page to see what it looks likes:

However, the feedback loop is not as quick as usual.

To adapt your code for printing, you have to bypass the 2 main differences between a web page and a PDF:

The components of a React application are dynamic whereas a PDF is a static file
A PDF has page breaks and a fixed size whereas a webpage is a "one page" app with a variable viewport size

From dynamic webpage to static rendering

Create the "printable version" of your SPA with React. To create the printable version of our page, you will have to add/remove/modify the different components that make up the page.

You basically have 2 solutions for this part:

create a whole new custom design for your page
adapt your current page

If you opt for the second solution (that is less costly), you will have to adapt your existing components. For example, if you have a table with 3 tabs, you will probably want to display the content of all the tabs. Something like displaying the tabs one after the other may do the trick:

Only dynamic:

<Table selectedTabIndex="tab1" />

Dynamic and Static:

const tabNames = ['tab1', 'tab2', 'tab3']

(isPrintable ?
	tabNames.map(tabName => <Table key={tabName} selectedTab={tabName} /> :
	<Table selectedTabIndex='tab1' />
);

In this case, the isPrintable props will determine whether to display the 3 tabs, or just the first one. You may pass this props to every dynamic component of your page, that needs to be adapted for printing.

Deal with page breaks and fixed size with CSS

As you can see with the Boursorama example, your components may be cut off between 2 pages when trying to print your page. It happens because your web browser has no idea where to break page if you don't tell him. This is where the break-inside CSS property steps in. You obviously don't want your previous set of tabs to be cut off in the middle. Neither your graphs or almost any component on your page. Then you would have to adapt the previous code to add this CSS property. It would work with inline-css but you probably don't want to add the style={{ breakInside: 'avoid' }} everywhere in your jsx/tsx files.

You would rather use stylesheets. And instead of adding this property on every CSS class already existing, you'll want to use the media @print option. This will let you custom your webpage for printing only! For example, you may want your text to be a bit bigger or to have a smooth grey color on the printable version, for any esthetic reason or convenience.

We'll just add this in the @media object in your css file:

media @print {
  body: {
    font-size: "16px";
    color: "lightgrey";
  }

  .no-break-inside {
    // apply this class to every component that shouldn't be cut off between to pages of your PDF
    break-inside: "avoid";
  }

  .break-before {
    // apply this class to every component that should always display on next page
    break-before: "always";
  }
}

<MyComponent isPrintable=true className="no-break-inside" />

These few CSS tips should help you improve a lot the rendering of your webpage.

How to use Puppeteer to generate your PDFs

Now, your page is ready for printing. You know it when you pass the isPrintable props to your page, right click + print on your browser, and you are quite comfortable with what you seeing. Here comes the part of printing. You now have a printable version of your webpage, but the users have no idea of it, and even if the ctrl + P on the website, they will see the "dynamic" version of the webpage. To let them generate the PDF version and automate the generation of the latest, you probably want to add a button that will directly generate the PDF server side, and even add some customization. This is what, among other things, Puppeteer is used for.

How Puppeteer works?

Puppeteer is a common and natural way to control Chrome. It provides full access to browser features and, most importantly, can run Chrome in fully headless mode on a remote server [...]

—Dima Bekerman, https://www.imperva.com/blog/headless-chrome-devops-love-it-so-do-hackers-heres-why/


Schema of how Puppeteer works server side

Generation of the React app is done by a web browser. We need the minimal environnement able to execute javascript to render a DOM. Puppeteer will do it by launching a headless chromium. From now on, and since the generation is done on the server, the web browser doesn't need to have a graphical user interface (GUI). Chromium with generate the printable version: the same page the user sees on his web browser but with the isPrintable props activated. Then Puppeteer will execute the pdf function on the page with some custom options that will trigger the printing of the page.

Just add the button with the URL that calls the printer service:

<Button onClick={window.open(downloadUrl, "_blank")}>Download as PDF</Button>

The downloadUrl is actually a GET request on your server that will execute Puppeteer on the server and return content with content-type application/pdf

So what does this Puppeteer code look like?

How to use it?

To be able to actually download the PDF, you just need a few code lines.

The minimal code would then look like:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch(); // launch a browser (chromium by default but you can chose another one)
  const page = await browser.newPage(); // open a page in the browser
  await page.goto("https://printable-version-of-my-wbe-page.com", {
    waitUntil: "networkidle2",
  }); // visit the printable version of your page
  await page.pdf({ format: "a4", path: "./my_file.pdf" }); // generate the PDF 🎉
  await browser.close(); // don't forget to close the browser. Otherwise, it may cause performances issues or the server may even crash..
})();

These are the common steps you'll need to generate the PDF. Depending on your backend, you probably don't want not to download the PDF on the server but to render it on a response object, to send it back to the client (the web browser of the user). You should then adapt the page.pdf() method with const buffer = await page.pdf({ format: 'a4'}); and return this buffer on the _blank page the user opened on his browser, waiting for a response.

Add some options to customize the PDF

You can of course adapt the options your naturally have on your browser, like the paper size, the scale, the margins, etc. with the help of the official documentation: https://github.com/puppeteer/puppeteer/blob/v10.4.0/docs/api.md#pagepdfoptions.

One cool option that I recommend, mainly because the default one provided by Google Chrome is really ugly, is the header or footer template. Just read a HTML file template and pass it through the data you want to display such as the current date, the page number for each page, or even an image/logo:

const footerBase = fs.readFileSync("./footer.html", "utf8");

customFooter = footerBase
  .replace("{{date}}", new Date())
  .replace("{{image_data}}", imageData);

await page.pdf({ format: "a4", footerTemplate: customFooter });

using a html template

<style>
#logo {
  height: 40px;
  content: url("data:image/png;base64,{{image_data}}");
}
</style>

<div id="body">
  <div id="page-number-paragraph">
    <span id="date">{{date}}</span>
    <span>Page</span>
    <span class="pageNumber"/></span>
    <span>/</span>
    <span class="totalPages"></span>
  </div>
  <div id="brand-container">
    <span id="logo"></span>
  </div>
</div>

You now have provided to your PDF a fully customized footer.

There are a lot of other options regarding, the PDF generation, but also for the previous steps of launching the browser, opening a new page, going to the URL, that will let you fully customize your PDF generation on the server.

Conclusion

Finally, by adapting your React/CSS code and using Puppeteer, you can easily provide a fully custom PDF of your page. Moreover, Puppeteer is doing all the stuff server side. Which makes this feature fully transparent, quite fast for the end user, and with the same result for every user on any browser! Puppeteer is really powerful and has a lot of options that make the PDF generation quite easy for the developers, and with a rendering much more custom and beautiful than the default one on users' browsers.

A Quick Intro to Elm for React Developers

Mon, 25 Oct 2021 00:00:00 GMT

Elm is probably my favourite programming language - its opinionated but welcoming design provides a compelling case study in how language design can gently guide developers towards writing maintainable code.

Why would a React developer learn Elm?

Elm is a language designed for creating frontend applications, offering an alternative to JavaScript libraries like React. Both were created circa 2012, Elm as an academic thesis project, and React as an industry-backed library. The underlying technology to both is a virtual DOM and a declarative programming style for creating user interfaces. The main difference between the two is that, while React is simply a library built for JavaScript, Elm is an entire domain-specific language, designed from the ground up for purpose.

This means Elm has some language features which are now considered best practice in JavaScript development, such as immutability, lambda functions and type safety. But Elm goes further than React and TypeScript in many ways, with such a powerful type system that it can boast 'No runtime errors in practice', and a significant speed improvement over its competitors.

As TypeScript gains more popularity, I wonder which other Elm features will move into React. My personal hope is that the famously friendly error messages (see image below) will make their way into the language. Elm is a compiled language, so the compiler is re-run after code changes to generate JavaScript for the browser. While this seems like a hindrance, I find the development loop quite engaging: a kind of 'type-driven-development' where the compiler helps guide your work, and gives the developer greater confidence in their code. This 'live-recompile' is made possible due to Elm's lightning-fast typechecking and compiling, something TypeScript is still sorely lacking.

The history of programming language adoption shows us that what's popular isn't always best, as the trending languages of today are often successful by pure coincidence - or as a result of corporate marketing campaigns. I truly believe the industry is moving towards functional programming: React's creator, Jordan Walke, said himself that ReasonML (a functional language closely related to Elm) is 'the language of React'. He posits that, after the success of React, we should once again turn to functional programming and ask 'What else did we miss?'

An Example Project

To help developers familiar with React get to grips with the syntax, I've written the same application in Elm and React, a counter with an increment and decrement button (I've excluded a bit of boilerplate code and import statements from both to simplify things). The page looks like this:

JavaScript:

const App = () => {
  const [count, setCount] = useState(0);

  const increment = () => setCount(count + 1);
  const decrement = () => setCount(count - 1);

  return (
    <div>
      <button onClick={decrement}>-</button>
      {count}
      <button onClick={increment}>+</button>
    </div>
  );
};

Elm:

type Msg = Increment | Decrement

init = { count = 0 }

update msg model =
  case msg of
    Increment -> { model | count = model.count + 1 }
    Decrement -> { model | count = model.count - 1 }

view model =
  div []
    [ button [ onClick Decrement ] [ text "-" ]
    , text (String.fromInt model.count)
    , button [ onClick Increment ] [ text "+" ]
    ]

As you can see, the two programs are similar in many ways, and quite different in others. If you're struggling with the somewhat alien ML-style syntax, check out this quick syntax guide.

To start, we define our Msg type to be either an Increment or Decrement value - more on this later.

After that, the Elm program is split across 3 self-contained function definitions, init, update and view. This contrasts with React, where everything is contained within one function.

The init function is much like the 2nd line of our React app: it defines the initial state of the Model, essentially the datatype for the entire application's state. Unlike when we use React's Hooks, we are encouraged to store all the state for our entire application in one data structure, for reasons discussed later.

The update function then takes two arguments, the Model and a Message. It then decides how to change the model based on the message received, returning a new copy of the model with the count changed. You can read the code { model | count = model.count + 1 } as 'a copy of model where its count is one greater'. Elm calls this function for us whenever a new message is sent, for example on a button press.

Finally, the view function then takes one argument, the Model, and renders it as HTML. Other than the style, this is mostly the same as the return statement in the React function - except that in Elm our buttons send Messages rather than call lambdas. Elm calls view to re-render the page whenever the Model changes.

What benefits does this style provide?

Most of these differences are a result of the fact that Elm is a purely functional language, meaning we aren't allowed to perform any 'side effects' as we would in React with Hooks. Functional purity gives us important guarantees about our codebase - we can be sure that, given the same inputs, we will always get the same output. In a practical sense, this makes the code easier to read - you know the only values that will change in a function are its parameters, so you don't have to waste time reading the whole surrounding context. This is one reason why Elm files can be longer than JavaScript ones without compromising maintainability, as the smallest unit you have to comprehend is a function, not an entire module.

However, separating out the code into init, update and view has other benefits - the separation between code describing the application logic and how it looks is now enforced by the language.

The guide refers to this structure as 'The Elm Architecture' (see diagram above). The Elm runtime manages our application for us, calling view whenever the model changes and update with every message received. The interactions here will be familiar to users of Redux, which was inspired by Elm, where a Redux 'Action' roughly approximates an Elm Message, and update is akin to a 'Reducer'.

Having a centralised Msg and Model type can boost productivity, as changing these types in a large module will cause the compiler to alert us to all the places that subsequently need updating, freeing us from the task of searching through the codebase manually. Centralising the state into a single Model also has architectural benefits. It allows us to make illegal states un-representable, and forces us to thoroughly describe all the dynamic aspects of our webpage. Modelling the state of our application as a data structure in this way is a powerful tool, that helps us consider what really lies at the core of our application.

A stricter type system can seem like a limitation, as an example, where React lets us simply write {count}, Elm makes us write text (String.fromInt model.count), converting the Int to a String to Html. However, as TypeScript evangelists will tell you, the advantages of type safety often outweigh the additional work, as it increases the predictability of the code and, in Elm's case, eliminates the possibility of a runtime error.

Where are these famous types?

In Elm, types can usually be inferred from the code you write, so you almost never need to write type annotations to reap the rewards of type safety. However, they are often considered good practice as they can make the code more readable and help the compiler show better error messages. So, for the sake of interest, here is the above program with type annotations:

type Msg = Increment | Decrement

type alias Model = { count : Int }

init : Model
init = { count = 0 }

update : Msg -> Model -> Model
update msg model =
  case msg of
    Increment -> { model | count = model.count + 1 }
    Decrement -> { model | count = model.count - 1 }

view : Model -> Html Msg
view model =
  div []
    [ button [ onClick Decrement ] [ text "-" ]
    , text (String.fromInt model.count)
    , button [ onClick Increment ] [ text "+" ]
    ]

First, we give the Model a formal type. We use the syntax type alias here, because we're not introducing a new set of values (as with Msg), simply a shorthand for { count : Int }, a record with an integer field called 'count'. We use this new type to annotate the declaration of init, saying it is just a constant of type Model.

Next, we give update a type annotation, Msg -> Model -> Model. This is essentially saying 'update is a function which takes a Message and a Model, and returns another Model'. This is somewhat obfuscated by the fact Elm functions are 'curried' by default, making the actual meaning more like 'update is a function which takes a Message, and returns a function which takes a Model and returns a Model'.

Finally, we annotate view with the type Model -> Html Msg. This states that the view function takes a model and returns some HTML, specifically HTML with our custom Messages embedded within it. This is a form of parametrised type: the TypeScript equivalent would be Html<Msg>.

What next?

If you're interested in experimenting with Elm, you can neatly integrate Elm components into your personal React projects using @elm-react/component.

For those interested in learning more about the language and its influence on frontend development, I'd recommend one of my favourite talks, 'The life of a file' by Evan Czaplicki, Elm's creator.

Thanks to the compiler's helpful and friendly error messages, the language almost teaches itself (after a brief skim of the guide) - so if you're a 'learn-by-doing' type of developer I recommend elm-live to re-run the compiler every time you change your file. It's an interesting, even somewhat calming way of programming, and while the language may not have seen mainstream success yet, I think it can continue to show us a glimpse of the future of web development.

Migrating from React Redux to React Query

Tue, 19 Oct 2021 00:00:00 GMT

React Redux is an extremely popular package, receiving about 4.5 million weekly downloads. Another popular package, Redux Saga, is a Redux side effect manager — it alone receives over 900k weekly downloads. The combination of React Redux with either Redux Saga or Redux Thunk is one of the most common additions to React projects.

However, over the last year or so, React Query has been gaining popularity quickly as an alternative to React Redux. Last year, React Query had only about 120k weekly downloads, but as of October 2021, React Query downloads are nearly six times more frequent, at nearly 700k downloads per week.

React Query has some distinct advantages over React Redux for storing server state.

A major advantage of React Query is that it is far simpler to write than React Redux. In React Redux, an operation to optimistically update a field requires three actions (request, success, and failure), three reducers (request, success, and failure), one middleware, and a selector to access the data. In React Query however, the equivalent setup is significantly simpler as it only requires one mutation (which consists of four functions) and one query.

Another advantage of React Query is that it works more smoothly with navigation tools built into many IDEs. With React Query, you can use “Go To Definition” to more easily reach your mutation whereas with Redux Saga, you must search the entire codebase for uses of the action.

On any existing project, making the switch to use a new package can be a daunting task. Today, to help make this task more manageable, we will be looking at an example where an application using React Redux with Redux Saga will be migrated to React Query.

Our Initial Project

To start, let's take a look at a basic optimistic Redux setup with three reducers, three actions, a selector, and a saga.

In our example, we have a React website where a user can input their name and their name is displayed on the page. A selector is used on the React page to display the user’s inputted name. When a user inputs their name, a request action is dispatched that causes a reducer and a saga to run. The reducer optimistically updates the store with the new value of the name. Meanwhile, the saga makes a call to the backend to save the user’s name.

If the backend returns an HTTP success, the saga dispatches a success action. The success action causes a reducer to run which updates the value of the name in the store.

If the backend returns an HTTP failure, the saga dispatches a failure action. The failure action causes a reducer to run which changes the value in the store back to its value before the optimistic update.

NameComponent.tsx

import React, {
  ChangeEvent,
  FormEvent,
  FunctionComponent,
  useCallback,
  useEffect,
  useState,
} from "react";
import { useDispatch, useSelector } from "react-redux";
import { selectName, updateNameRequest } from "../redux/slice";

export const NameComponent: FunctionComponent = () => {
  const name = useSelector(selectName);
  const [localName, setLocalName] = useState<string>("");
  const dispatch = useDispatch();

  useEffect(() => {
    if (name) setLocalName(name);
  }, [name]);

  const handleChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
    setLocalName(event.target.value);
  }, []);

  const handleSubmit = useCallback(
    (event: FormEvent) => {
      dispatch(updateNameRequest({ name: localName }));
      event.preventDefault();
    },
    [dispatch, localName]
  );

  return (
    <div>
      <p>{`Your name is ${name ?? "unknown to me"}.`}</p>
      <form onSubmit={handleSubmit}>
        <label>
          {`Enter name: `}
          <input type="text" onChange={handleChange} />
        </label>
        <input type="submit" value="Update Name" />
      </form>
    </div>
  );
};

slice.ts

import { createSlice, PayloadAction } from "@reduxjs/toolkit";
import { RootState } from "..";

export interface NameState {
  name: string | null;
  error: string | null;
}

export const initialState: NameState = {
  name: null,
  error: null,
};

export const nameSlice = createSlice({
  name: "name",
  initialState,
  reducers: {
    updateNameRequest: (state, action: PayloadAction<{ name: string }>) => ({
      ...state,
      ...action.payload,
    }),
    updateNameSuccess: (state, action: PayloadAction<{ name: string }>) => ({
      ...state,
      ...action.payload,
    }),
    updateNameFailure: (
      state,
      action: PayloadAction<{ error: string; name: string }>
    ) => ({
      ...state,
      ...action.payload,
    }),
  },
});

export const selectName = (state: RootState) => state.name;

const { actions, reducer } = nameSlice;

export const { updateNameRequest, updateNameSuccess, updateNameFailure } =
  actions;
export type NameAction =
  | typeof updateNameRequest
  | typeof updateNameSuccess
  | typeof updateNameFailure;
export { reducer as nameReducer };

sagas.ts

import { ActionType } from "typesafe-actions";
import { call, put, select, takeEvery } from "redux-saga/effects";
import {
  selectName,
  updateNameFailure,
  updateNameRequest,
  updateNameSuccess,
} from "./slice";
import { updateName } from "../api/api";

function* updateNameSaga(action: ActionType<typeof updateNameRequest>) {
  const { name } = action.payload;
  const oldName: string = yield select(selectName);

  try {
    yield call(updateName, name);
    yield put(updateNameSuccess(action.payload));
  } catch (error) {
    yield put(
      updateNameFailure({ error: (error as Error).message, name: oldName })
    );
  }
}

export function* nameSagas() {
  yield takeEvery(updateNameRequest, updateNameSaga);
}

store.ts

import { applyMiddleware, createStore } from "@reduxjs/toolkit";
import createSagaMiddleware from "redux-saga";
import { nameSagas } from "../redux/sagas";
import { nameReducer } from "../redux/slice";

const sagaMiddleware = createSagaMiddleware();

export default function configureStore() {
  const store = createStore(nameReducer, applyMiddleware(sagaMiddleware));

  sagaMiddleware.run(nameSagas);
  return { store };
}

Converting the Redux Store to React Query

The first step of our conversion is to create a query to replace the selector and the store. Below is a custom React hook that contains a useQuery to store the user’s name. In this example, getName is a call to the backend to get the name stored in the database.

import { getName } from "../api/api";

export const getNameQuery = () => ({
  queryKey: ["name"],
  queryFn: getName,
});

Converting the Saga to a Mutation

The next step of our conversion is to create an optimistic mutation to replace the saga, reducers, and Redux actions. Below is an optimistic mutation in a custom React hook that optimistically updates the query and invalidates the query after the update.

import { useMutation, useQueryClient } from "react-query";
import { updateName } from "../api/api";

export const useEditName = () => {
  const qc = useQueryClient();
  return useMutation<void, unknown, { newName: string }, string>(
    async ({ newName }) => {
      await updateName(newName);
    },
    {
      onMutate: async ({ newName }) => {
        await qc.cancelQueries("name");

        const currentName: string | undefined = qc.getQueryData("name");
        qc.setQueryData("name", newName);

        return currentName;
      },
      onError: (_error, _variables, context) => {
        if (context) qc.setQueryData("name", context);
      },
      onSettled: async () => {
        qc.invalidateQueries("name");
      },
    }
  );
};

The mutation function in this case calls the backend with updateName to change the name stored in the database.

Before the mutation function, Redux Query executes onMutate. In this case, we cancel the query that we are updating, get the current value for the query, and update the value for the query with the new value. We then return the old value for the query so we can revert if our optimism does not pay off.

If the mutation function throws an error, Redux Query executes onError. All we need to do is set the query to the context parameter if the backend call fails.

Lastly, Redux Query executes onSettled, which simply invalidates the query so Redux Query knows that the data is stale. This forces Redux Query to get the name fresh from the backend.

Final Project

The new query and mutation can be used in the following way in the React component. In the end, we will have a React Query setup with an optimistic mutation. Just two hooks replace the original React Redux and Redux Saga setup.

name.ts

import { useMutation, useQueryClient } from "react-query";
import { getName, updateName } from "../api/api";

export const getNameQuery = () => ({
  queryKey: ["name"],
  queryFn: getName,
});

export const useEditName = () => {
  const qc = useQueryClient();
  return useMutation<void, unknown, { newName: string }, string>(
    async ({ newName }) => {
      await updateName(newName);
    },
    {
      onMutate: async ({ newName }) => {
        await qc.cancelQueries("name");

        const currentName: string | undefined = qc.getQueryData("name");
        qc.setQueryData("name", newName);

        return currentName;
      },
      onError: (_error, _variables, context) => {
        if (context) qc.setQueryData("name", context);
      },
      onSettled: async () => {
        qc.invalidateQueries("name");
      },
    }
  );
};

NameComponent.tsx

import React, {
  ChangeEvent,
  FormEvent,
  FunctionComponent,
  useCallback,
  useEffect,
  useState,
} from "react";
import { useQuery } from "react-query";
import { getNameQuery, useEditName } from "../dataLayer/name";

export const NameComponent: FunctionComponent = () => {
  const editNameMutation = useEditName();
  const { data: name } = useQuery(getNameQuery());
  const [localName, setLocalName] = useState<string>("");

  useEffect(() => {
    if (name) setLocalName(name);
  }, [name]);

  const handleChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
    setLocalName(event.target.value);
  }, []);

  const handleSubmit = useCallback(
    (event: FormEvent) => {
      editNameMutation.mutate({ newName: localName });
      event.preventDefault();
    },
    [editNameMutation, localName]
  );

  return (
    <div>
      <p>{`Your name is ${name ?? "unknown to me"}.`}</p>
      <form onSubmit={handleSubmit}>
        <label>
          {`Enter name: `}
          <input type="text" onChange={handleChange} />
        </label>
        <input type="submit" value="Update Name" />
      </form>
    </div>
  );
};

The last step for this setup is wrapping the application in a QueryClientProvider.

App.tsx

import React from "react";
import { QueryClient, QueryClientProvider } from "react-query";
import "./App.css";
import { NameComponent } from "./components/NameComponent";

const queryClient = new QueryClient();

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <QueryClientProvider client={queryClient}>
          <NameComponent />
        </QueryClientProvider>
      </header>
    </div>
  );
}

export default App;

React Query is quicker to write, easier to navigate, and, now that you know how to use it, an alternative to storing and updating server state with React Redux with Redux Saga or Redux Thunk.

How to Create a Localized Date Pipe in Angular 📅

Mon, 11 Oct 2021 00:00:00 GMT

TLDR:

To translate dates in Angular, we can think about three approaches: setting the app global locale, passing a specific locale to Angular date through arguments, or creating a custom localizedDate pipe. The localizedDate pipe combines Angular date pipe and the lib ngx-translate. This solution prevents us from explicitly passing the locale as an argument in the template while still being able to dynamically change the dates' locale without reloading the app. Those aspects make the localizedDate pipe impure like [ngx-translate]'s translate pipe.

Use a pipe in the template

We use pipes either in a service or a template (the HTML code). In the following example, we are using the template syntax of the slice pipe and the uppercase pipe.

<div>{{ 'Grogu' | slice:1:4 | uppercase }}</div>
<!-- output: <div>ROG</div> -->

This is equivalent to 'Grogu'.slice(1, 4).toUpperCase() in Javascript.

Sadly, there is only a short list of built-in pipes in Angular. Wait a second, could we create our own?

That's fairly easy to do because a pipe is simply a data transformer.

Different ways to translate dates in a template

Angular provides a built-in date pipe. It transforms a date into a string value (eg. 2021-03-21 -> Mar 21, 2021). The date pipe uses the app locale as its default locale (eg. 2021-03-21 -> 21 mars 2021 for a french locale).

To translate dates in a template, we can think about three approaches:

by setting the default app locale
by passing the locale as arguments of the date pipe
by combining ngx-translate and the date pipe in a new custom pipe

Approach 1: Setting the default locale

The date pipe uses the LOCALE_ID to determine the default locale to use. By default, LOCALE_ID is set to en-US. We can change the app locale by LOCALE_ID setting in the AppModule.

@NgModule({
  providers: [
    { provide: LOCALE_ID, useValue: 'fr-FR' }
  ]
})

With the static string value fr-FR, the LOCALE_ID value can not be dynamically changed. Instead of statically setting the LOCALE_ID value, we can provide a service as a value.

@NgModule({
  providers: [
    { provide: LOCALE_ID, useFactory: appLocaleService },
  ]
})

For changes made to appLocaleService to be reflected in our app, we need to reload the Angular app. LOCALE_ID is required in Angular build process. Reloading the app may reset the app state and re-trigger HTTP requests.

Approach 2: Passing the locale as arguments of the `date` pipe

The locale used in the the built-in date pipe can be set through the pipe's arguments.

<div>{{ '2021-03-21' | date:undefined:undefined:'en' }}</div>
<!-- output: <div>Mar 21, 2021</div> -->
<div>{{ '2021-03-21' | date:undefined:undefined:'fr' }}</div>
<!-- output: <div>21 mars 2021</div> -->

Same as the previous approach, the static locale can be replaced by value resolved in a service. That way, we can dynamically change the locale without reloading the entire app.

The repetitive syntax makes this approach frustrating in the syntax.

Approach 3: Using a custom pipe `localizedDate`

The internationalization library ngx-translate provides a translate pipe that can be used as follows.

<div>{{ 'my.traduction.key.for.hello-world' | translate }}</div>
<!-- output (if the current locale is EN): <div>Hello world</div> -->
<!-- output (if the current locale is FR): <div>Bonjour le monde</div> -->

The goal here is to create a localizedDate similar to translate. It will subscribe to any change in the application locale and behave accordingly.

<div>{{ '2021-03-21' | localizedDate }}</div>
<!-- output (if the current locale is EN): <div>Mar 21, 2021</div> -->
<!-- output (if the current locale is FR): <div>21 mars 2021</div> -->

With the localizedDate, we gain in maintainability in the syntaxe and by using a single internationalization library for dates and contents. As others, it has its cost, this approach adds extra computation in the app. More details next in the article.

Our own custom localized date pipe

The idea is the use the built-in date pipe along with ngx-translate. Sounds complicated? Let's figure that out!

@Pipe({
  name: 'localizedDate',
  pure: false
})
export class LocalizedDatePipe implements PipeTransform {

  constructor(private translateService: TranslateService) { }

  transform(value: Date, format = 'mediumDate'): string {
    const datePipe = new DatePipe(this.translateService.currentLang);
    return datePipe.transform(value, format);
  }
}

Yeah, as simple as that. In this implementation, there is:

the declaration of the new pipe with @Pipe (note pure: false, we will come back to that later)
the injection of the app's TranslateService
the transform method of the PipeTransform interface

In the transform method, we are using the service syntax of DatePipe. Hence, we can specify the current locale during instantiation.

Pure vs Impure pipe

Ahmed Ghoul wrote a super article about Pure vs Impure Pipe in Angular. Pure pipes (like the built-in date pipe) are called only if the pipe's inputs changed. On the other hand, impure pipes are called on every change detection cycle.

The localizedDate can not be pure. The main purpose of this pipe is to avoid passing the current locale as input. Therefore, it has to be an impure pipe to listen to the current locale changes triggered by other components.

For the same reason, the translate pipe from ngx-translate is impure as well.

A Stackblitz is worth a thousand words

The source code of the localizedDate in a concrete example can be found here :

The last word

The approach you may want to choose to translate dates depends on your application and needs.

If you have a specific URL per locale and redirect the user when he changes locale, the first approach should work just fine. If you are already using ngx-translate, creating a localizedDate starts to make sense.

To help you with your decision, here is a quick comparison between the three approaches:

	Syntaxe	Reusability	Reload required	Performance
Setting LOCALE_ID	⭐️⭐️⭐️	⭐️⭐️⭐️	YES	⭐️⭐️⭐️
Using date pipe argument	-	⭐️	NO	⭐️⭐️⭐️
Custom localizedDate	⭐️⭐️⭐️	⭐️⭐️⭐️	NO	⭐️

Happy coding 👨🏽‍💻

How I Made My Life Easier by Letting QA Write E2E Tests?

Mon, 20 Sep 2021 00:00:00 GMT

Lately, I have worked as a web developer with a company that deals with logistics. Their business is to supervise the transport of commodities from point A to point B. The application I worked on was intended for the suppliers of this company to track their merchandise. Each persona would connect to the application and perform tasks such as declaring the dispatch or the reception of the products.

The people who worked on the previous projects followed a V-model lifecycle. These projects were delivered with many bugs after a few years. Developers spent a couple more years trying to fix the application. Therefore, the quality of the application is one of their main focuses. We were using an agile methodology and after each sprint, all the application was tested to ensure that no bug or regression appeared. QA (Quality assurance) was responsible for the quality of the application.

Testing the application - how to do it effectively?

The QA was manually testing the application. Running the full set of tests would take half a day, so the tests were performed every other week. It was not frequent enough to catch all the bugs and meanwhile, QA could not focus on their other tasks. It was not satisfying enough for the client and the QA,that is why the developer team suggested implementing end-to-end (e2e) testing.

But this idea was not perfect: in typical e2e testing, developers are writing the e2e tests. While they are implementing those tests, they are not developing features, which is detrimental for the client. On top of that, the QA, which is responsible for the quality of the application, does not understand what the e2e tests are testing.

Our challenge was finding a solution where QA would not have to test the application manually but without making testing the developers’ responsibility.

What if QA could write tests?

If the QA can write the e2e tests, the tests and the quality of the application are directly managed by the QA. The QA ensures thus the high quality of the application. It is pretty fast for the QA, because they write the test once, and it is then run automatically. And no need to wait two weeks now to run the tests anymore, they can be run every day.

Which means developers can fix bugs faster, for they might have created the bug recently and the code is still in their mind.

This was nice and good, but we had one major issue, for our QA did not have a developer background, yet popular e2e test frameworks, such as Cypress, are asking for knowledge in Javascript.

Writing tests without coding (or very little)

A common test framework, called Cucumber, can be used without development skills and both developers and the QA had already used it. Through its language (Gherkin), it allows team members to write tests in a human-readable language.

Cucumber can be then connected to many common testing tools, and it can be connected to Cypress thanks to cypress-cucumber-preprocessor. To develop tests, all you need is a grammar of test sentences, for example, “When I log in”. You still have to code, but your sentences are reusable, so when the grammar is done right, developers are not needed for every test case.

Our process was as follows. First, the QA would decide which test cases were to be tested automatically (critical test cases that are very tedious to run manually). The developers would then translate from Gherkin to Cypress by creating the Gherkin grammar that the QA can use.

You'll find an example of a feature written in Gherkin by the QA and the steps definitions in Cypress (as the developer will code them). You can retrieve this example here.

# Scenario in Cucumber/Gherkin
Feature: Cypress click
    Scenario: Click and navigate
        Given I'm on the "/" page
        When I click on "Querying"
        Then I should have "querying" in the URL

// Translation of the Gherkin steps to Cypres
import { Given, When, Then } from "cypress-cucumber-preprocessor/steps";

const baseUrl = "https://example.cypress.io";
Given("I'm on the {string} page", (url) => {
  cy.visit(baseUrl + url);
});

When("I click on {string}", (label) => {
  cy.get(".home-list").contains(label).click();
});

Then("I should have {string} in the URL", (url) => {
  cy.url().should("include", url);
});

Make our cucumber tests run

Our team was using Xray, a popular plugin that powers up Jira by providing tools to manage tests. As a result, the QA had to write all its test cases in XRay, which allows the QA to write manual tests or cucumber tests. In addition, Xray comes with an API that developers can use to get test cases or send tests results. Each morning, the tests were automatically pulled from XRay, run and then the results were sent to XRay.

Example of a test case in XRay

How was this solution for us?

With this solution, developers do as little code as possible. They only need to translate the Gherkin sentences into Cypress tests, but each sentence is reusable, so the QA can write new tests without the need of development.

The QA has a complete picture of the quality of the application. They decide which test cases should be executed each day and they know immediately if a test fails.

This process is not perfect and it still requires some maintenance from the developers, but no more than for regular automated tests. It even requires less time from developers as they are not responsible for keeping the test cases up-to-date.

And if you want to have your QA writing the e2e tests?

First, train your QA for using Cucumber. It is easy to understand, even with no development skills. Feel free to check the comprehensive Cucumber documentation.

Next, you need to decide where the QA should write their tests. Requiring QA to use git may be excessive. Xray is a good tool as it is designed to be used with Cucumber, but it can be any tool that your QA uses to write its test sets. Ideally, the tool should support Gherkin syntax and have an API to let developers run tests automatically.

Architecture guidelines for large Angular applications

Wed, 15 Sep 2021 00:00:00 GMT

TLDR;

When coding an Angular app for the first time, this is dangerous to keep old habits and reproduce patterns that worked for other apps like React or Vue. Working in a very large (~100 devs) project, I know that old habits can make code of poor quality.

I propose some guidelines inspired from Angular's recommendations and Clean Code practices that will help your team to write maintanable code and ease newcomer's onboarding!

Put the code right where it belongs : use the power of services and pipes to unclog your components.
Code in a reactive way your components with services
Avoid passing inputs/outputs all around thanks to the Dependency Injection System.
Split Services when they become too big. Do the same with modules

Old habits that get you wrong

Those patterns you need to let go of

Angular's intimate relationship with Typescript makes it easy to put data logic everywhere. Therefore it's easy to agree that dumb contains view-printing, and smart contains fetching. It is a sound decision to improve your code readability.

In terms of performance and seperation of duties, Angular proposes an idiomatic way to code. But many of us have been tempted to try famous patterns like Container Components (or smart), or trying to reproduce stateless Components of React and other frameworks.

That being said, Data often plays a central role. Once again, the temptation to try famous patterns like functional components and a Redux data-store is very strong. Indeed, some don't see how to find a robust way to manage their data.

Nevertheless, trying to mimick patterns you learned on other codebases blindly into an Angular application is such a dangerous idea and I want to show you how to make your Angular application maintanable.

Improving readability in consulting companies

I have been working here at Theodo (also in the UK) with teams that encoutered a huge developers turnover. Developers rarely stayed more than one year on the same code base. I even found myself switching teams every 2 months.

Therefore, it is crucial to choose a pattern that will help new developers to quickly switch codebases. That's why I want to introduce you to the right way of thinking your Angular Application Architecture. This approach makes small use of other famous patterns and mainly explains Angular's recommendations.

Guidelines for clean and efficient Angular coding

Use angular-cli

These are the rules you can establish in your team to make you Angular application cleaner. And as a quick rule, I advise you to always use angular-cli when creating new components and services in your app, so you're not tempted to write less.

You are writing presentation code in .ts or .html ? Make a pipe

If the data needs simple mappers, create pipes that you will use in component's template. Indeed those mappers add usually nothing but noise to your code, hide them away !

Don't write functions for data transformation in your component's .ts, and don't write complex functions in your component's html template. Code smells :

You call functions in your template (See why that's a bad idea)
You write a lot of logic in {{ your template }}

Here is an example of a too-intelligent component. It has functions in its template, with logic...

<ul>
  <li *ngFor="let member of team.member">
    {{ isUserFrench(member) ? '🇫🇷' : '🚩' }} Star: {{
    isUserCapableOneOfTheBest(member) }}
  </li>
</ul>

You can put everything in a pipe, a test it seperately !

@Pipe({
  name: 'memberBasicInfoList'
})
export class MemberBasicInfoListPipe implements PipeTransform {
  transform(team: Team): {flag: '🚩' | '🇫🇷', isStar: boolean}[] {
     //calculate flag and isStar
     return {flag, isStar}
  }

Then you have a very readable template !

<ul>
  <li *ngFor="let memberInfo of team | memberBasicInfoList">
    {{ memberInfo.flag }} Star: {{ member.isStar }}
  </li>
</ul>

Make reactive Services that support your components business logic

Services are your intelligent blocks of code. If you wanted to put code in a container component, put it here ! Here are some some rules of thumb :

Wrap all the data fetching in a service, and inject them in the component(s). This service should expose ready to use observables. It should also give methods to send data from components to the service. Same rule applies if a service becomes too big : split it and inject one service into the other service.

Code in a reactive way with observables as inputs and funtions as outputs

Use observables in your components as if they were already containing the data you want, because that is not the role of component to update the data. Then build public methods in your service to notify it of the last events. Your component should contain very minimum logic.

Think of your component as a puppet of the service. It shows the data the service wants. It tells the service when something happens.

Split your services when they become bigger than 100 lines

Put business logic as soon as possible in services and prepare the data inside of them to be displayed. For example a TeamMember.service that relies of TeamsStore.service and for example another User.service (that can come from somewhere else than TeamsModule)

To split your code, detect parts of code that are heavily linked to each others.

When your base module becomes too big and handles many topics : split into smaller modules

Put common business items in a module, for example "TeamsModule". Split them conviniently with Angular routing as soon as possible. When ?

Some parts become more and more independant, and some parts are never displayed together
Your module is too big for a new-comer to understand it

Your parent/children modules can communicate through common services. Learn more about when to split modules here : Angular: Understanding Modules and Services

Let's study an example: Breaking down one component into smaller simpler pieces

We talked about writing reactive services, and breaking down big chuncks of code into smaller meaningful one. This example shows a typical example when you have a very big component that needs to be refactored. I often found myself in this exact situation.

InviteTeamMemberComponent(httpClient: HttpClient, errorService: ErrorService, router: Router)
A. Can call the team-backend and POST a new team member
B. Will show an error message (in global snackbar) if failed
C. Will redirect to Team dashboard if successful
D. Creates FormGroup for team member informations (also stores the data and validates it)
E. Displays the data in the template
F. Keeps track of the loading state of the query

That's a lot of responsibilities, huh ?

Let's create a client that can do http-calls, especially the POST (task A.)

TeamMemberClient(httpClient: HttpClient)
A. Can call the team-backend and POST a new team member

and inject it in another service which has the responsibility to manipulate data (tasks B, C and D).

InviteTeamMemberService(teamMemberClient: TeamMemberClient, errorService: ErrorService, router : Router)
B. Will show an error message (in global snackbar) if failed
C. Will redirect to Team dashboard if successful
D. Creates FormGroup for team member informations (also stores the data and validates it)

that we inject inside the initial lighter component that does the two last UI features (tasks E and F)

InviteTeamMemberComponent(inviteTeamMemberService: InviteTeamMemberService)
E. Displays the data in the template
F. Keeps track of the loading state of the query

Avoid Input/Output and prefer injecting specialized services

You can sometimes avoid Input/Output drilling (more specialized article) with either service or content injection, this will look like a simple advice to Redux advocates, but here is the good news : we don't need redux !

Here is an example of input drilling where a 2 level deep button opens a modale upper in the hierarchy, all managed by a smart ancestor component !

Valuable code is spread among all the components and useless Input/Output mechanism pollutes the intermediary. On top of that, the ancestor component contains the data, and has the responsibility to update it. Here's what you would prefer :

This was an example of refactoring using service. You can also avoid one layer of Input/Output with content projection. In the example above, you could get rid of @Output() edit if you directly inject the button of the action.

What are the advantages of doing so ?

Synchronization within your team : You can still use old patterns, but don't make it a must ! Just communicate on your rules with your team and write coding guidelines that every one of you has read. You'll see tremendous improvement very quickly
Simpler hierarchy : your DOM is not polluted with useless container components. CSS is simpler, and inspecting the application is simpler too ! You can understand it faster ! If other teams use the same coding guidelines, you'll be able to switch teams easily.
Easier onboarding, as a new developer you know where things are by opening a package : services, components and pipes are doing what you expect them to do ! Also, less wrappers means less clicks to get to meaningful code !

AWS Fargate: harness the power of serverless for long-running computational tasks

Tue, 14 Sep 2021 00:00:00 GMT

TLDR: We all know the advantages of serverless computing: pay only for what you use; scale up and down with ease; abstract away the complexities of managing servers. In this article, I illustrate how you can acquire these benefits for long computational tasks which are too large for Lambda or Google Cloud Functions to handle. I share a case study from a recent project and how we used AWS Fargate (along with S3, Lambda, DynamoDB, and more!) to create a robust task architecture.

Recently, a client approached us with a state-of-the-art modelling system which had been designed by industry-leading domain experts. The techniques used in the modelling software meant that, for its use-case, the software could run simulations in an order of magnitude less time than the competing solutions.

Despite being an industry-leading scientific model, the delivery of that solution through software was holding the product back from its full potential. The software interface was a Python GUI and was distributed as source code directly to users. The model could take hours to run and if it failed (often due to invalid input), the modeller would only discover this error on returning to the program after several hours away, only to correct the error and rerun the test case. Often, multiple variations need to be simulated, however the desktop application couldn't execute multiple runs concurrently (as the software is limited by the processing power of the machine it runs on).

The mission

We wanted to bring this software to the market as an industry-leading product in terms of scientific modelling and software engineering. We would leverage cloud services to mitigate some of the disadvantages of the desktop implementation.

Here are some of the main things we wanted to add to the current functionality of the model:

Improved distribution by migrating to the cloud:
- Easier on-boarding (no software installation - just login).
- Improved security (no user access to source code).
- Run the model from any computer.
Increased scalability:
- Execute multiple model runs concurrently to save users' time.
Improved user experience:
- More data validation rules to prevent failed model runs.
- Email notifications for when the model has completed a run.

The challenge

The abstract problem to solve was that we had a 500,000-line black-box program that was tightly coupled to the environment and file system of the machine it was running on. In our case, three issues stand out:

The model code is tightly coupled to the modeller's development environment - it calls os.system in many places to run shell commands on specific libraries (these included dependencies on GNU Fortran - not something you see in your everyday web application!).
The model code is tightly coupled to the modeller's file system - it creates intermediate output files to save data between different steps in a run. This decision was initially made from a scientific standpoint - a modeller might want to check some intermediate output - and had stuck as a design pattern across the software.
The model code has significant complexity, and re-engineering it came with risks to the validity of the results.

These constraints meant that we needed:

To run the model code on a system with the same environment as the modellers' computers.
A way to persist intermediate files in the cloud between different runs.

The strategy

The high-level flow of a modeller executing a model run would go like this:

The modeller goes to the web application from their browser and configures a new run. Parameters that they configure are saved to DynamoDB and input files they upload are saved in S3.
The modeller clicks the "execute" button for a specific task, which triggers (a Lambda to trigger) a task in Fargate. This spins up a Docker image registered in ECR with the correct environment and source code to run the model. The task name is passed in as an environment variable.
The task pulls the correct input files from S3 and the user-defined parameters from DynamoDB.
The model task is then executed as it would be on a modeller's local computer.
The model syncs the resulting output directory to S3 and EventBridge and SES are used to notify the user of the status of the run via email.

The architect of the project drew a handy architecture diagram to illustrate exactly how we achieved these steps and I'll go through the steps that we took along the way.

Dockerize the tasks: Fargate + ECR + VPC

The first step was to create an environment to run the model; Docker was designed to solve this very problem ("well, it worked when it ran on my computer!").

A new container was defined and the required dependencies for the project were resolved (we used Poetry to manage the Python libraries).

We used Docker volumes for the project so that any intermediate files created within the container are synced with our outer file systems to inspect and validate. A docker-compose.yml file was also written to simplify bringing the container up and down and passing in environment variables from a .env file.

There are several discrete tasks that the model can run, each one generating different output files. For our use-case, we would have a single Docker image, which contains the model code for every type of task - the task to be executed would be defined as an environment variable (a container is designed to be brought up with a single purpose, then taken down once the task is complete).

Now that we know the task we want to execute, we can define the functions we want to run for each task. We define a TASK_DEFINITIONS variable which stores handlers for each task: preprocessor, executor (where we will run the main task), postprocessor, and failure (the model can be temperamental to user input, and we need to handle failure explicitly). I will come back to the pre- and post-processors in the next section.

The executor is simply a function which makes a call to the model source code to execute the desired task. This means we can define an env variable - TASK_NAME - and run the model locally by accessing bash from within the container:

docker-compose up -d
docker-compose exec <image-name> bash
python main.py

Note: my local machine doesn't have the correct dependencies to run the model code, but the container that I run on my local machine does.

The container can now be migrated to the cloud so that the model code can be accessed from anywhere in the world without the need to manually distribute the source code.

The image is registered with AWS Elastic Container Registry (ECR) where the container can be stored and accessed (Model registry in our architecture diagram). A Virtual Private Cloud (VPC) is configured to accept requests to trigger an AWS Fargate task.

Fargate is "serverless compute for containers"; this means that our container isn't running in the cloud when we don't need it - AWS handles the provisioning of resources to "spin up" a container from our image at the moment we want to run a task. This means that we are only paying for what we use whilst still having the potential to scale ad infinitum (we can trigger multiple tasks concurrently and AWS will just spin up more instances for us in the cloud). The one drawback of this architecture is that Fargate requires some time to build the container when we request for our task to be run (this was around a minute for our (large) container).

You can see Fargate (Model task) sitting at the centre of our architecture diagram within the VPC in ECS.

We also set up Cloudwatch to create logs for each run to help us and the modellers debug issues with the model.

Syncing input and output directories: S3

Until now, the model works in the cloud so long as the required input files are present in the file system of the Docker image; however, input files are specific to the context of a run, and if we want to get meaningful output then we need to pass in the correct input files at the point that we execute a task.

For each task, a new instance of the image is spun-up, so any intermediate files are lost when the container is pulled down and the context is lost (this is fine, as we want different executions to remain distinct from each other).

We set up an S3 bucket to store the inputs and outputs of tasks, which means that the files which we expect to persist between different tasks can be synced between permanent storage from task to task.

Each task was saving its output to a different directory, so within our TASK_DEFINITIONS, we define an input_dir, from which we would pull the contents from S3 and a number of output_dirs to which we would push the output of the model. These outputs might be downloaded by the user or used by a different task as input files. This syncing of files happens in the preprocess and postprocess of each task.

Data model: DynamoDB

As well as input files, the model also depends on a range of input parameters that the user can define. In the desktop app, the program gets these input parameters by reading and parsing a local file called INPUT_PARAMETERS.txt.

Each model run entity has a corresponding entity within a DynamoDB table (in practice there are multiple types of entity available for the user to manipulate). The database follows the single-table design as described in The DynamoDB Handbook by Alex DeBrie and uses dynamodb-toolbox to model entities. The user can interact with these entities via the web application to set properties of the run.

We created a corresponding manager in the dockerized model code for each type of entity in the database (e.g. RunManager) which can get the entity and also perform simple updates on text fields (set error messages etc.). The entity is retrieved from DynamoDB which then overrides the variables which were previously being retrieved from INPUT_PARAMETERS.txt, which otherwise describes a generic run.

The model is now no-longer dependent on a local file for input of discrete parameters; but how does the user configure those variables to get a final valid output to their desired parameters?

Web Application: Lambda + Next.js

It's now time to plug the main pieces together.

We developed a Next.js frontend to interface with the remote model. This comes with all the advantages of developing UI using technologies that were designed to handle UI (i.e. not Python!). This meant our input methods were more interactive and our outputs more visual, as we were able to leverage existing libraries.

We added validation rules to the inputs (with warning messages) to prevent the user running the model with parameters that don't make sense. We accept input files as upload fields of several forms, which are pre-processed to validate that the values in the files were within the expected range before uploading to S3.

We built our backend REST API using AWS Lambda. As with Fargate, Lambda offers flexible scaling, has a nice integration for the back-office tool we created in Retool, and we didn't mind the infamous cold starts for our use case.

Another advantage of using Lambda was that we can use AWS CloudFormation (infrastructure as code solution) to provision our infrastructure from a serverless.ts file which defines all the resources required for an environment. If we wanted to create a new environment, running sls deploy <env-name> would provision the resources defined in the file instead of needing to interact with the AWS console.

Notifying model outcomes: EventBridge + SES

Until now, a specific model task can be triggered to run in the cloud where it will sync the relevant input files, pull the correct input parameters, run the task, and sync the output files back to S3. Finally, we need to notify the user of the outcome of the model (success + results/ error + logs).

In the model, we defined a RunEventsManager which could send an event via AWS EventBridge when the postprocess or failure handler was completed in the model. We added a Lambda in our backend which was triggered by an EventBridge event (because we were using CloudFormation, we just needed to define the event that triggers the lambda in serverless.ts).

When an event occurs, the Lambda uses AWS SES client to send an email to the user, which notifies them immediately when they can check their results - there's no need to come back to check that the model hasn't failed every 15 minutes anymore!

Summary

Using the described architecture, we had delivered a significant improvement to the user experience in running scientific modelling tasks by bringing the software to the cloud and affording it all the advantages of a serverless implementation.

Rendering Blender scenes in the cloud with AWS Lambda

Sat, 28 Aug 2021 00:00:00 GMT

You can fairly easily leverage AWS Lambda to render scenes with Blender. It can make sense to use Lambda functions if you need to render a large number of assets in little time and if each asset is simple enough to be loaded and rendered in reasonable time (within the maximum lifetime of a Lambda) given the limited resources offered by Lambda functions (maximum 6 vCPUs and 10GB of RAM at the time of writing). If you have to render more complex assets, going with EC2 instances or AWS Thinkbox Deadline is a better solution.

When I came up with the idea of running Blender on Lambda, I naturally checked if someone was doing it somewhere and found out that @awsgeek had done something similar in 2019 so at least I knew it was possible. What we are going to present in this article slightly differs from his approach though.

The architecture

Here is the very simple architecture that allows you to render Blender scenes on AWS Lambda stored on an S3 Bucket. The final renders are stored into the same bucket.

The files you will need to run this "at home" (in the cloud)

To run this on your own AWS account, you will need four files:

Dockerfile describes the container image that each lambda function execution is going to run
serverless.yml describes the infrastructure above
app.py handles the event from the Lambda infrastructure and starts up blender
script.py manipulates Blender from the inside to lad a scene and render it.

File structure

Here is how these files should be arranged:

project directory
+-- Dockerfile
+-- serverless.yml
+-- app
|   +-- app.py
|   +-- script.py

Dockerfile

For the Dockerfile, I started from a great set of Docker images for Blender maintained by the R&D team of the New York Times. We're using the latest image that runs Blender rendering with CPU only (as Lambdas don't have GPU capabilities unfortunately).

FROM nytimes/blender:2.93-cpu-ubuntu18.04

ARG FUNCTION_DIR="/home/app/"
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
RUN pip install boto3
RUN pip install awslambdaric --target ${FUNCTION_DIR}

WORKDIR ${FUNCTION_DIR}

ENTRYPOINT [ "/bin/2.93/python/bin/python3.9", "-m", "awslambdaric" ]

CMD [ "app.handler" ]

Serverless configuration file

The serverless.yml file is fairly simple:

service: blender-on-lambda
frameworkVersion: '2'
configValidationMode: error

custom:
  destinationBucket: blender-on-lambda-bucket

provider:
  name: aws
  region: us-east-1
  memorySize: 10240
  timeout: 900
  ecr:
    images:
      blender-container-image:
        path: ./

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:PutObject"
        - "s3:GetObject"
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - "${self:custom.destinationBucket}"
            - "/*"

functions:
  render:
    image:
      name: blender-container-image
    maximumRetryAttempts: 0

resources:
  Resources:
    S3BucketOutputs:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.destinationBucket}

In this example, I am using the largest available Lambda functions with 10GB or RAM and 6 vCPUs so that the rendering goes as fast as possible.

The Python Lambda handler

The app.py file contains the logic that will be triggered by the Lambda infrastructure and it will receive the event passed as a trigger. In this case, the event must contain the width and height of the image to be rendered.

import json
import os


def handler(event, context):
    os.system(f"blender -b -P script.py -- {event.get('width', 0)} {event.get('height', 0)}")
    return {
        "statusCode": 200,
        "body": json.dumps({"message": 'ok'})
    }

The Python Blender script

And finally, here is the python script that will be executed from inside Blender to do the actual rendering.

from datetime import datetime
import os
import sys

import boto3
import bpy


argv = sys.argv
argv = argv[argv.index("--") + 1:]

s3 = boto3.resource("s3")
BUCKET_NAME = "jrb-material-renderer-bucket"
filename = f"{datetime.now().strftime('%Y_%m_%d-%I:%M:%S_%p')}.png"

s3.Bucket(BUCKET_NAME).download_file("scenes/scene.blend", "/tmp/scene.blend")

bpy.ops.wm.open_mainfile(filepath="/tmp/scene.blend", load_ui=False)
bpy.context.scene.render.filepath = f"/tmp/{filename}"
bpy.context.scene.render.resolution_x = int(argv[0])
bpy.context.scene.render.resolution_y = int(argv[1])
bpy.ops.render.render(write_still = True)

s3.Bucket(BUCKET_NAME).upload_file(f"/tmp/{filename}", f"renders/{filename}")

Cost

In the us-east-1 region of AWS, the largest available lambda will cost you $0.01 per minute. The request charge (the amount charged for every invocation) is probably negligible in comparison. Also, bear in mind that the generous free tier of Lambda might cover your costs! Same for S3.

I hope this can help anyone facing the same need that I did.

If you're interested in offloading your Blender render jobs to the cloud, follow me on Twitter. I'll be posting more content in the coming weeks such as:

cost comparison with using other cloud providers
the cost impact of using GPU-enabled EC2 instances instead of Lambda
leveraging AWS EFS instead of S3 to load the scene into Blender

If there are other topics you'd be interested in, let me know on Twitter! My DMs are open.

Sending Financial Data to Salesforce Without the Hassle Thanks to Codat and Amazon AppFlow

Wed, 11 Aug 2021 00:00:00 GMT

We’ve recently built a secure data pipeline for a customer called Founders First Capital Partners. They needed to automatically load large amounts of financial data into Salesforce, their CRM, in order to make investment decisions. We used Amazon’s managed data integration service — AppFlow — which takes care of creating a secure connection and sending data to Salesforce. To retrieve the data securely from several financial tools, we leveraged Codat. Codat's single API powers robust integrations to over 29 different financial tools including QuickBooks, Stripe, Plaid and Oracle NetSuite. Codat takes care of all the heavy lifting involved in these integrations — authorization, data standardization,ongoing connectivity — leaving us free to focus on how to best use this data.

The architecture

We designed a simple serverless architecture that fit our needs.

Benefits of using these tools

Why would you do this instead of “simply” writing the connection to the APIs?

Less code to write. All you have to do is write code that puts data in an S3 bucket and AppFlow will take care of sending it to Salesforce (or any other supported SaaS tool you choose).
All the usual benefits of serverless and managed services: you only have to focus on the business logic that matters to your use case. AWS ensures your solution is always up and running (with an SLA of 99.9%). There are no software updates or patching to worry about. It scales with your needs.
Enhanced security: for each SaaS that AppFlow connects to, AWS establishes a secure AWS PrivateLink which is a great security guarantee without any additional effort.
The standardization of data provided by Codat allowed us to manipulate data from various sources into one consistent data structure, which is a huge time-saver.

Trade-offs of our solution

AppFlow can sometimes throw error messages that are not very helpful, which can be frustrating. But, just like with other AWS products, their team is open to feedback and continuously improves AppFlow.
The community adoption of AppFlow still seems to be nascent (based on the number of Stack Overflow discussions on the topic), which is a bit of a shame given how helpful this service is. As a result, it can be a struggle to find resources to help you get up and running.

The combination of Codat with Amazon AppFlow allowed us to load large amounts of critical financial data into Salesforce in a very short amount of time, with minimal development effort. I hope you can also save time by using these amazing tools!

Thanks to Nate Ritter, CTO of Founders First Capital Partners, for giving us the opportunity to work on this small piece of serverless sorcery. Founders First Capital Partners are based in San Diego, CA and support diverse founder-led small businesses and those in low to moderate income areas by providing them with funding and acceleration services.

Hasura: GraphQL Without the Baggage.

Wed, 04 Aug 2021 00:00:00 GMT

Spoiler Alert: Star Wars references below. Proceed at your own risk...

The introduction of GraphQL back in 2015 garnered a lot of hype in the development sphere - allowing clients to describe exactly what data they require (preventing over-fetching unneeded information) and unifying multiple resources into a singular request. This substantially improved scalability and reusability of backend endpoints, and makes the development experience all-in-all more pleasant.

So you might be thinking to yourself: "Great. What's the catch?"

Well... Implementing GraphQL into your backend will inadvertently increase complexity - you'll need to dedicate time on defining types, efficient resolvers, mutators, queries, etc. If you're working as a full-stack developer, you might find that using GraphQL is kind of like taking a dollar from one pocket and putting it in another. The added complexity can start to outweigh the benefits, especially on smaller scale projects. On top of that, caching is complex with GraphQL. With no in-built caching capabilities, this often becomes a pain point for developers.

That's not to diminish or underplay the clear benefits of using GraphQL.

However, I want to use this opportunity to touch on some of the less talked about vices of GraphQL, and present Hasura as a possible solution to some of these concerns. Hasura is a GraphQL service that automatically generates GraphQL endpoints from your database schemas, taking out a lot of the complexities and manual work needed. Simply by connecting Hasura to your SQL database, it will generate schemas and resolvers using the existing tables and views. It's quite magical in how quick it is to get up and running...

GraphQL Complexities

Boilerplate & Initial Setup

As you'll usually find with most other large scale dependencies or libraries, the addition of GraphQL will require some level of initial setup and boilerplate, you're adding a new layer to both your frontend and your backend. You're probably going to need to add a GraphQL client, define the types, schemas, error handling, caching on the frontend level. On the backend side, you'll need to define multiple resolvers to fetch different views of the data (get all the data/get a single datapoint with an id), as well as defining mutations for creating new datapoints, editing existing data, and deleting data.

Now you could argue that these would all still be required if you used normal REST endpoints - and you would be right. However, on top of these, you'll need to deal with typing on the backend, caching (you can't rely on native HTTP caching like REST APIs), higher order components, and additional validators to catch some edge-cases that arise from defined schemas.

A solution commonly taken on the backend side, is to define boilerplate abstractions to take the repetition out of having to define basic resolvers/queries for new data classes you add into your application, but again this takes time to setup (plus you're adding more complexity). While this might not be an issue for larger projects that can reap the benefits of using GraphQL, it becomes a massive swaying point away for small projects.

Resolver Hell

Resolvers are the functions that resolve the values for any field in a schema. They can return primitive values, or objects that get further broken down into more primitive types. You'll find that these typically return values from databases or other APIs.

Now there are a number of databases and ORMs that come with GraphQL mappers that will automatically handle resolvers for you when you are defining your data in objects. For example, in a project that I worked on, we use GORM, which handles resolving and schema generation. However, these do come with their overheads, and you would need to define your data models according to these data access tools.

In many cases you'll find that you're dealing with your database directly, and you'll be doing the DB querying yourself. This opens a whole new can of worms to deal with.

A common issue that many face is over-querying the database. Let's demonstrate this with an example:

You are creating an application to manage a large movie cinema company with multiple branches. You have a query that fetches a list of all the cinemas, the screens in each, the movies showing on each screen, and which production companies released them.

Now presume that there are 5 cinemas (branches), with 10 screens each, each playing 5 movies at any given point. This means that we have 1 query to get the cinemas, 5 queries for each cinema getting the screens, and 50 queries for the movies playing on each screen (50 calls to get the details of the movie playing on each screens). That comes down to a total of 56 database calls for this query.

Say you wanted to fetch some basic metadata about each of these movies as well (such as the production companies), that would be another 50 database calls added, which would increase our total to a whopping 106 database calls...

Cinema example above visualised.

This can be mitigated by strategising and coming up with efficient queries and resolvers, but the problem remains that it is very easy to fall into these common patterns that will result in overfetching. It's just the price that comes with allowing the client to specify their data needs and schemas.

There are solutions to deal with this - for example, Dataloader will batch some of these database calls together and cache some of the results. However, these solutions are far from perfect and they have limitations when making certain queries. Not to mention that by adding new dependencies, we start inadvertently increasing complexity of the application.

Performance & Caching

With complex views on the data, comes the cost of query complexity and the number of database calls increasing. It becomes a balancing game... You'll want to use GraphQL's ability to define graph-based traversals to fetch multiple resources (in fact, this is probably the single biggest selling point IMHO), but use it too much (or irresponsibly), and you might find that your queries become slow and start to massively affect performance.

Take the Star Wars example that is shown on the homepage of the GraphQL website:

type Query {
  hero: Character
}

type Character {
  name: String
  friends: [Character]
  homeWorld: Planet
}

type Planet {
  name: String
  climate: String
}

Let's try and visualise it with a flowchart, with the main trio of the original series. If Luke Skywalker is the hero of the original series (which is a false statement, because we all know the Ewoks were the true heroes), he has several friends, and each of these friends have a homeWorld.

GraphQL allows you to traverse the graph of Characters by defining resolvers to fetch the friends. Now this is a fairly simple example, and you'll find that real world data will usually have more complex relations and hence this graph will become larger. This will take a toll on performance, and you may find that certain queries will take unreasonably long to run.

A common approach taken to solve this problem is through caching, and although it won't work well for frequently changing data, this can help reduce the number of queries that need to be made regularly on data that rarely changes.

As an example, Apollo can allow you to define and control caching structures on your web applications, but if you have multiple frontend applications (say a web, Android, and iOS app), you'll probably need to handle caching on each of these levels separately. Again, added complexity that will be introduced by using GraphQL.

However, this options is only viable for data that isn't changing regularly... So in the case where Leia's home planet of Alderaan gets obliterated by the Death Star, or Han Solo and Leia become separated (ergo are no longer considered friends) due to their son falling to the dark side of the force, you may find that caching will result in data inaccuracies. This is where the importance of having well-written and efficient resolvers and database calls comes into play yet again.

And with that, all may seem lost, but from the darkness of caching, boilerplate, and complex resolvers comes the saving grace of Hasura...

Hasura

Hasura is an open source service that allows you to take your relational databases, and automatically generate GraphQL APIs on top of them. Effectively, this removes one of the biggest challenges that backend developers using GraphQL face, which is creating efficient SQL queries and resolvers.

Looking at the challenges posed above, the issue of excessive boilerplate code & high initial costs effectively become neutralised by using a tool like Hasura. I had a Postgres database running on my machine, and it took me a total of 15 minutes to spin up Hasura with Docker. Just by changing the docker-compose, I pointed it to my database, and it loaded up already connected and hosting a GraphQL endpoint with my defined views! Honestly, it felt like some form of vodoo magic! The steps I followed were really quite simple:

Fetch the docker-compose file with curl.
Run docker-compose up -d
Get yourself a cup of coffee while you wait for it to run.
Open the console page at localhost:8080/console
Click on the data tab, and click the "Connect Database" button.
Add a name for your database, the SQL flavour, and lastly the database URL.
Click the connect button.
After it's done connecting, click on the "GraphiQL" tab, and see that your database tables can be queried out of the box!

Now, a common question that may come up about the initial cost of using a technology is how well it integrates with existing codebases. Presume that you're already using a GraphQL endpoint for the backend of your application, and you want to incrementally introduce Hasura to handle the basic queries and subscriptions (yes, it comes prebuilt with subscriptions, and that can be an absolute pain with certain GraphQL servers). You can use a feature called Remote Schemas to point Hasura to your already existing GraphQL endpoint, and it'll automatically combine the schemas together to create a unified GraphQL endpoint that your frontend can use. Additionally, you can use this to delegate more complex business logic in your code to other services (such as a payment API).

The second challenge touched on above was the issue of inefficient and complex resolvers. Without getting into much detail, Hasura compiles down GraphQL queries using abstract syntax trees, validates and checks permissioning at this level, and then constructs the entire query into one raw SQL query. On top of this, they do 2 levels of caching (one at the SQL compilation level, and another at the database level with prepared statements) within their architecture to reduce the amount of times these compilation steps are required. They're building and improving on this architecture and performance with every update. For example, they've introduced JSON aggregations on the database level, effectively removing the need to flatten the results from the database calls in the backend - they claim this increases performance anywhere from 5-8 times! For those who are interested, they have an interesting article on their website talking about their architecture in more detail.

Hasura's Data Wrapper (Source: hasura.io)

Lastly, we touched on the issue of caching & performance. Within the data wrapper layer, there's already 2 levels of caching that will significantly improve performance. Additionally, there is another layer of caching, which is on the query level, that effectively allows you to cache the response being sent back to the client for a defined period of time. You can make a call with the @cached directive to cache the response, supplying an optional ttl argument (how long to cache the response):

query MyCachedQuery @cached(ttl: 120) {
  users {
    id
    name
  }
}

This caching comes working out of the box. In contrast to common caching approaches (such as using Apollo), having this caching done on the server side can be beneficial for unifying the responses on any given platform at any point of time. Imagine you have a cached query for the top players in a leaderboard. If the caches are set at different points on the client, there may be inconsistencies in how this data is displayed. Caching on the server-side circumvents this.

On top of that, Hasura's performance isn't only due to compiled queries, efficient resolvers and caching. There's a ton more work put into improving speed and performance and it is a regularly improved and maintained project that's already heavily utilised in the industry.

So to demonstrate the Star Wars example we saw above, I went ahead and created a database schema to have the following tables:

Character
Planet
friendsWith

I did all of this through the Hasura data admin panel (sort of like a lite version of PGAdmin). Through this data panel, you can choose which tables/views can be tracked by the GraphQL schema, and you can define table relationships and how they manifest in the GraphQL schema.

Switching to the "GraphiQL" tab, we can now see that the explorer now has a bunch of different datapoints:

You can fetch the list of each of the data types by calling the basic data type (for example, Planet), and this comes with pagination and filtering out of the box. Alternatively, if you're looking for some form of aggregation on the data, you can use the _aggregate datapoint, which can run some basic aggregation on the data (such as averaging a field, counting, finding the min/max the field, or even getting the standard deviation). Lastly, you can get a single data entry by using the by_pk variation.

Now, to see this in action, we construct a query asking for the first character in our character table. We ask for the name of the character, their home planet (along with its associated climate), and a list of their friends.

To be able to do this in the matter of a couple minutes is something that I still find magical.

I'll end here with some of the selling points that I think make Hasura really shine:

Automatically generates the schemas and resolvers for your GraphQL endpoints (less pain).
Comes prebuilt with option variables that you can use in your queries (for example, limit and offset, which will allow you to more or less handle lazy loading of datapoints).
Currently works with a number of SQL databases (Postgres, Microsoft SQL, Amazon Aurora, and Google Big Query) with more coming soon.
Easy to integrate with authentication and permissioning platforms, such as Auth0.
Gives you a web UI to manage the GraphQL endpoint, as well as an interface to manage the database and its views itself (think PGAdmin).
Using Remote Schemas, you can hook Hasura up with other GraphQL endpoints and other services to unify your endpoints.
Using Actions, you'll be able to define calls to custom handlers that will execute more complex business logic (which opens the door to serverless by using Lambdas)
You can define event triggers to call actions automatically (when a user signs up, send an email via a Lambda function).
Highly maintained and used by the community, with over 23k stars on Github and 250m+ downloads.

Conclusion

GraphQL provides some fascinating functionality that aims to make the data layer of applications unified and flexible. As with most things, this comes at a cost. The initial setup and boilerplates will mean that you will need to spend time upfront in creating the right infrastructure for using GraphQL. On top of that, the flexibility in schema means that you will need to have additional considerations when it comes to performance, and you will need to come up with efficient queries and resolvers. Lastly, with GraphQL we lose the native HTTP caching capabilities, meaning that caching is not standardized and becomes cumbersome additional work.

Hasura, acting as a service layer between your databases and clients, can automatically generate performant GraphQL queries, mutators, and even subscriptions, with minimal effort on the your end.

Whilst Hasura's solution to automatic generation of schemas and endpoints might not offer the full flexibility that large-scale highly complex applications might need, it opens the door to using GraphQL on smaller scale projects where the startup costs of introducing it would outweigh the benefits. I believe this is where Hasura will excel, and where it will ultimately find its place in the GraphQL market.

Writing a native Ionic plugin for Capacitor in less than 30 minutes

Mon, 12 Jul 2021 00:00:00 GMT

TLDR: When the pandemic first started, I decided to develop a contact-tracing mobile app. I was studying Computer Science at the time and decided to use the same cross-platform framework that we were using on our course - Ionic. I wanted to use Google's Nearby Messages API to share packets of information between iOS and Android devices via Bluetooth, but I couldn't find a plugin for this, so I decided to write my own!

View commits: Android plugin → iOS plugin → Import native plugins → Implementation

Ionic is a cross-platform mobile framework which allows you to develop an app using JavaScript/HTML/CSS and share this single implementation across different native devices. This Ionic project can then be compiled to native source code (Android/iOS) using Ionic's tool, Capacitor.

Using a cross-platform framework like this to develop your mobile app is almost always a good idea, because it means you can maintain a single codebase which runs on both Android and iOS. This means you're not duplicating business logic across two different codebases in two different languages with two different teams (to be avoided).

If you're looking to write some native code in a cross-platform project but you're still deciding which framework to use, check out this Theodo blog post on React Native as well!

Step 1 - Search for existing solutions

Firstly, let's have a quick check that one of the default plugins doesn't do the job for us already.

Secondly, let's search through the community plugins to find the feature we're looking for.

No luck finding an existing plugin? Don't worry - let's move on to step 2!

Step 2 - Start a new app with Ionic + Capacitor

I'm starting a new project from scratch just to show you how it's done.

npm install -g @ionic/cli

ionic start <app-name>
cd <app-name>
ionic serve <-- you can serve your project on localhost

The /src directory of our Ionic project holds the source code for our cross-platform app.

We can use Capacitor to compile this into two native projects:

npm install @capacitor/cli @capacitor/core

npx cap init
npx cap add ios
npx cap add android

When we make changes to our /src directory, we can apply those changes to our native projects by running npx ionic build && npx cap copy.

We now have our project set up and ready to add a custom plugin.

Step 3 - Check the documentation

Capacitor is well-documented and it's worth linking some references here to accompany this guide:

Getting started with Android

Android reference

Getting started with iOS

iOS reference

Step 4 - Define a strategy

There are three main steps to writing a plugin:

(iOS & Android) Write a class in each native project to hold the native code we want to run (this is our plugin).
(iOS & Android) Register these classes inside the bridge of our native projects to expose the methods.
(Javascript - Ionic) Import and call the methods of our registered plugin.

We should avoid duplicating code as much as much as possible (DRY), so all your business logic stays written in JavaScript, keeping the bare minimum written natively.

Your plugins should be narrow in scope (do one specific thing) - make sure to follow this philosophy and give your plugin a specific name ('Plugin' is a bad name!).

We have two main strategies for calling our plugins:

An (asynchronous) method: we call some native method which we can await in our business logic in Ionic.
Event listeners: define a listener in our business logic. Events can be triggered in our native code to pass payloads back to the business logic.

If we are subscribing to a stream of events (expecting our method to resolve with multiple payloads over a longer time period e.g. geolocation/ Bluetooth updates), then event listeners are the appropriate way to handle our native calls.

Step 5 - Write and register the Android plugin

[View the commit]

Open the /android directory of your project in Android Studio.

In /app/src/main/java/../../.. (next to MainActivity.java), we can create a new file (I'm calling mine IonicNativePluginExample.java) and write a class which extends the Capacitor Plugin class:

package /* <package-name> */; // e.g. io.ionic.starter

import com.getcapacitor.JSObject;
import com.getcapacitor.Plugin;
import com.getcapacitor.PluginCall;
import com.getcapacitor.PluginMethod;
import com.getcapacitor.annotation.CapacitorPlugin;

@CapacitorPlugin(name = "IonicNativePluginExample")
public class IonicNativePluginExample extends Plugin {

    @PluginMethod
    public void NativeMethod(PluginCall call){
        JSObject result = new JSObject();
        result.put("message", "Hello Android user!");
        call.resolve(result);
    }

    @PluginMethod
    public void NotifyListeners(PluginCall call){
        JSObject result = new JSObject();
        result.put("message", "Hello Android user!");
        notifyListeners("EVENT_LISTENER_NAME", result);
    }

}

Next, we can register this plugin in MainActivity.java where Capacitor initialises its bridge:

package /* <package-name> */; // e.g. io.ionic.starter

import android.os.Bundle;
import com.getcapacitor.BridgeActivity;

public class MainActivity extends BridgeActivity {
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        registerPlugin(IonicNativePluginExample.class);

    }
}

We can run our project like any other native Android project, by pressing the 'play' button in Android Studio.

If you're looking to develop in Kotlin - Android Studio comes with tooling to convert this Java class to Kotlin (right-click the file you want to convert).

Note: you might need to edit the Gradle distributionUrl to get the project build to succeed - there's an example in my repo here.

Step 6 - Write and register the iOS plugin

[View the commit]

Open the ios/App directory of your project in Xcode.

In /App (next to AppDelegate.swift), we can right-click on the directory to create a new swift file (I'm calling mine IonicNativePluginExample.swift). This will hold our plugin class with our IonicNativePluginExample method.

import Foundation
import Capacitor

@objc(IonicNativePluginExample)
public class IonicNativePluginExample: CAPPlugin {

	@objc func NativeMethod(_ call: CAPPluginCall) {
		call.resolve(["message": "Hello iOS user!"])
	}

	@objc func NotifyListeners(_ call: CAPPluginCall) {
		self.notifyListeners(
			"EVENT_LISTENER_NAME",
			data: ["message": "Hello iOS user!"]
		)
	}

}

Next, we must register our plugin in a new Objective-C file called IonicNativePluginExample.m (same name as our plugin file but with a .m extension). When prompted by Xcode, create a Bridging Header file (which is an empty file called App-Bridging-Header.h), then register the plugin like so:

#import <Foundation/Foundation.h>
#import <Capacitor/Capacitor.h>

CAP_PLUGIN(IonicNativePluginExample, "IonicNativePluginExample",
           CAP_PLUGIN_METHOD(NativeMethod, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(NotifyListeners, CAPPluginReturnPromise);
)

We can run our project like any other native iOS project, by pressing the 'play' button in Xcode.

Step 7 - Import and call our plugin in Ionic

[View the commit]

Finally, we can now access the plugins we have written from our Ionic project. I've created a /plugins directory to export mine from. We import registerPlugin from Capacitor to find the plugin we have registered in Android and iOS:

import { registerPlugin, Plugin } from "@capacitor/core";

// we can take advantage of TypeScript here!
interface NativePluginInterface extends Plugin {
  NativeMethod: () => Promise<Record<"message", string>>;
  NotifyListeners: () => Promise<void>;
};

// it's important that both Android and iOS plugins have the same name
export const IonicNativePluginExample = registerPlugin<NativePluginInterface>(
  "IonicNativePluginExample"
);

Now we can call the methods in our plugin and access the native code:

import { IonicNativePluginExample } from './plugins/IonicNativePluginExample'

...

// add a listener to native events which invokes some callback
IonicNativePluginExample.addListener("EVENT_LISTENER_NAME", ({ message }) =>
    console.log(message);
);

// destructure the methods to call our native code from our non-native app
const { NativeMethod, NotifyListeners } = IonicNativePluginExample;

// native methods are asynchronous
const { message } = await NativeMethod();

// this method will trigger our event listener
NotifyListeners();

Note: to see our changes reflected when we run the project natively, we need to run npx ionic build && npx cap copy.

We now have a cross-platform app set up ready to write a native implementation. We run the same business logic from our Ionic app and get a different implementation for each device we run on.

I made a small edit to our Ionic project to create the simple implementation below - you can view the commit here.

More complex plugins

With the above setup, you should be ready to write more complex native code to suit your needs for a cross-platform Ionic application.

I wrote a native Plugin for Google's Nearby Messages API, which is a publish-subscribe API made by Google which facilitates the transfer of information between internet-connected Android and iOS devices.

Nearby Messages uses a combination of Bluetooth, Bluetooth Low Energy, Wi-Fi, and near-ultrasonic communication between nearby devices to create a unique pairing. This pairing is then used to send small payloads over the internet between nearby devices.

Here's a quick code snippet from the Android plugin:

...

import com.google.android.gms.nearby.Nearby;
import com.google.android.gms.nearby.messages.Message;
import com.google.android.gms.nearby.messages.MessageListener;

...

@NativePlugin()
public class NearbyMessagesPlugin extends Plugin {

		private Message mMessage;
    private MessageListener mMessageListener = new MessageListener() {
        @Override
        public void onFound(Message message) {
            JSObject result = new JSObject();
            result.put("message", new String (message.getContent()));
            notifyListeners("FOUND_MESSAGE", result);
        }

        @Override
        public void onLost(Message message) {
            JSObject result = new JSObject();
            result.put("message",  new String (message.getContent()));
            notifyListeners("LOST_MESSAGE", result);
        }
    };

    @PluginMethod
    public void Subscribe(PluginCall call){
        Nearby.getMessagesClient(getContext()).subscribe(mMessageListener);
    }

    @PluginMethod
    public void Publish(PluginCall call){
        String value = call.getString("message", "-");
        mMessage = new Message(value.getBytes());
        Nearby.getMessagesClient(getContext()).publish(mMessage);
    }
}

Event listeners are appropriate here because although we don't know when/if the phone will pick up a Bluetooth signal, a side-effect can still be triggered in the business logic when it does. Subscribe can be invoked to subscribe to mMessageListener and listen for Bluetooth messages (or we could just call Subscribe in the onCreate of our native code if we wanted to).

Summary

Once you've got a simple implementation, you can move on and write some more complex native code.

If you want a quick-start, you can fork my repository on GitHub, but following along with the article and doing it yourself will really help your understanding.

RisXSS, the missing ESLint rule for React and Vue

Mon, 05 Jul 2021 00:00:00 GMT

XSS attacks are expected to be ranked 3rd in the OWASP Top 10, compared to their 7th rank in 2021 (source). The pace of application creations is increasing and so is the surface of attacks, so making your applications secure is more and more challenging and needed. React and Vue are some of the most used frameworks to build applications and include built-in protection for some types of XSS by escaping user input by default. However, there may be some cases when they need to display user content as raw HTML (a blog post with style for instance).

export const BlogPost = ({ post }) => {
  return (
    <div dangerouslySetInnerHTML={post} />
  );
};

(All the examples in this article use React but are similar with Vue).

A first attempt to prevent XSS attacks

This opens a large hole in the XSS protection. A secure solution is to first sanitize the content with a library that will remove unsafe parts of the content while keeping useful tags and attributes (for styling purpose for instance). The most popular libraries for this are DOMPurify and js-xss.

This solution is not always known though. On a website I worked on, I saw the use of dangerouslySetInnerHTML that was not properly sanitized and introduced a real XSS vulnerability. My first reaction back then was: "How can I make sure this will never happen again?" So I searched the Internet and I happily found ESLint rules designed precisely to prevent the use of dangerouslySetInnerHTML and v-html. I added these rules on my projects and some time later, again, I found another XSS vulnerability.

How did it happen? Easy: some developer wanted to display sanitized content with style, so they used v-html while sanitizing the output first and added a comment to disable the ESLint rule for this line. This is correct in this case. Then, another developer copied-pasted (who does not?) this code in another page, but the content was not sanitized.

When you have no other means than to disable a rule to do something, disabling the rule becomes the norm, and the rule loses its purpose.

Custom ESLint rule to the rescue

This is when I talked with some colleagues and started a custom ESLint rule to warn about insecure HTML only if the HTML is not sanitized. Corentin Normand and Guillaume Klaus implemented the logic (and even wrote a tutorial about it) and it became RisXSS.

RisXSS warns you about insecure use of dangerouslySetInnerHTML and v-html but only if the content is not sanitized. The following snippet will not display a warning:

import { sanitize } from 'dompurify';

export const BlogPost = ({ post }) => {
  return (
    <div dangerouslySetInnerHTML={{ __html: sanitize(post) }} />
  );
};

By offering a smart ESLint rule, RisXSS achieves the best of both worlds:

preventing XSS
allowing developers to display content with style

Now that you understand the need of RisXSS, go check out how to install it on your project on the RisXSS homepage.