Theodo

How to generate Typescript interfaces from your Spring Boot backend

Tue, 09 Aug 2022 00:00:00 GMT

Starting a full stack project with Spring Boot and a modern frontend framework like React in Typescript, you rapidly fall into the issue of defining your interfaces twice:

Once on Spring Boot side where you create your response/request DTO for your controllers,
and again on the frontend where you have to write those interfaces again in Typescript this time.

Not only is this poor developer experience, leading to a loss in productivity, it simply leads to bugs as you can easily forget to upload your interfaces on one side.

Great news! Avoiding this pitfall is actually possible and easy ?! with a quick setup, which is the topic of the following guide, let's go!

How can we generate Typescript interfaces from our Spring Boot backend

Advancement in code generation and support of standards like Swagger/Open API to describe REST APIs allow for effective code generation. Meaning we only need to set up a proper Swagger on our backend api in Spring Boot to benefit from interface generation for all our interfaces.

In practice

We will need to expose a Swagger on Spring Boot application and then to configure our code generation tool to target it.

Setting up SpringDoc

To generate our Swagger we will use SpringDoc as SpringFox is outdated and less maintained. (You can even relatively easily migrate from SpringFox to SpringDoc)

Setting up SpringDoc with Maven is classic, get the latest version, add it to your pom:

<!-- Swagger -->
    <dependency>
      <groupId>org.springdoc</groupId>
      <artifactId>springdoc-openapi-ui</artifactId>
      <version>1.6.9</version>
    </dependency>

If you are using some sort of authentication, you will need to allow unauthenticated access to the Swagger, eg:

// ... rest of your auth
 @Override
  public void configure(WebSecurity web) throws Exception {
    // Allow Swagger to be accessed without authentication
    web
      .ignoring()
      .antMatchers("/api-docs")
      .antMatchers("/api-docs/swagger-config")
      .antMatchers("/swagger-ui.html")
      .antMatchers("/swagger-ui/**");

Which you can limit to dev environment if you are not comfortable allowing your backend users to discover your API endpoints using Swagger, to do so use profiles/environment variables as you can easily disable SpringDoc.

Now start or restart your Spring Boot application, you should be able to get your Swagger (JSON) on http://server:port/context-path/v3/api-docs, for the application created for this guide it's: http://localhost:8080/guide/v3/api-docs.

One additional step is to configure your Swagger to regroup enums usage definition (meaning when you use an enum in two DTO you will expose only one enum definition in Swagger instead of duplicating the definition):

@Configuration
public class OpenApiConfig {
  static {
    // use Reusable Enums for Swagger generation:
    // see https://springdoc.org/#how-can-i-apply-enumasref-true-to-all-enums
    io.swagger.v3.core.jackson.ModelResolver.enumsAsRef = true;
  }

  // ... you can also describe your api bellow
  @Bean
  public OpenAPI openAPI() {
    return new OpenAPI()
      .info(
        new Info()
          .title("My shiny API")
          .description("My shiny API description")
          .version("v0.0.1")
      )
      .externalDocs(
        new ExternalDocumentation()
          .description("Documentation")
          .url("https://my.shiny.api.doc/")
          // can simply be a link to a README
      );
  }

This setup is enough for code generation, however I recommend to dive deeper into SpringDoc, it also offers you an UI http://server:port/context-path/swagger-ui.html which can be an excellent way to document your api for your clients.

Setting up Orval

Orval is a recent code generator that propose code generation up to API client generation with the tool of your choice: fetch, Axios, React Query, SWR. For this guide, I will cover interface generation only, however if you are starting a project, I highly recommend setting up client generation as well.

First install Orval as a dev dependency on your project:

npm install orval --save-dev

then create an orval.config.ts file in your frontend:

import { defineConfig } from "orval";

export default defineConfig({
  evo: {
    output: {
      mode: "tags",
      schemas: "model/api",
      mock: false, // enable/disable test mock generation
      // I recommend enabling this option if you generate an api client
      prettier: true, // recommended if you use prettier
      clean: true, // recreate the whole folder (avoid outdated files)
    },
    input: {
      // use your own Swagger url: http://server:port/context-path/v3/api-docs
      target: "http://localhost:8080/guide/v3/api-docs",
    },
  },
});

Once your backend is running, you can generate your interface running npx orval --config ./orval.config.ts.

I recommend setting up a script in package.json:

{
  ...
  "scripts": {
    ...
    "gen:types": "orval --config ./orval.config.ts"
  }
}

That's it! Now you only need to launch this command to update your frontend interface when you modify your backend, no more duplication 🎉

Going further

You can go a bit further as mentioned in passing in this guide:

SpringDoc provides a great UI to self document your API, and you can customize the Swagger generated for your clients,
Orval has many features including client generation, creating mock for your API, I highly recommend looking into it,
the next step would be to automate client side type generation in CI to test that every pull request has up to date types, you can do so without running your backend as SpringDoc propose a maven plugin to generate the Swagger JSON.

SheetJs: Programmatically generating stylish Excel documents

Thu, 04 Aug 2022 00:00:00 GMT

SheetJs is a javascript library that is used to create and format Excel documents. It is a powerful tool that boasts a consistent weekly download rate of over one million as well as having a dedicated team to battle issues and improve features.

SheetJs was the perfect tool to use on a recent project where we needed to serve data as a readable, formatted Excel document. The data we received was in the form of a JSON object and we were tasked with transforming that data so that it could be easily read and used in calculations. Although the documentation for the methods in SheetJs is extensive, I felt that it is lacking examples of more complete use cases.

xlsx-js-style

At the time of writing this post, SheetJs is by far the most popular library for manipulating and generating Excel documents in Node.js. The community version (listed as xlsx in the npm registry) offers a wide range of functionality for manipulating data but the formatting functionality is locked behind the pro version. The following example uses the xlsx-js-style library, a fork of the SheetJs library that includes more styling options.

Example use case

For the purpose of this blog post, I’ll highlight a simple scenario and step through it piece by piece until we have generated a satisfactory Excel file. Let’s say a client wants us to create an Excel file based on data from fundraising events where people shoot basketballs for charity. Their requirements are as follows:

The Excel sheet is easily readable
The number of individuals in the data can vary
The number of attempts per individual can vary (one attempt or two attempts)
There should be total and average columns

Based on the requirements, let’s say we want the contents of the Excel file to look something like the example below.

In this example, we are populating the file with the amount of basketball shots taken and landed by several people. It contains two headers with some merged cells, a “Total” and “Average” column with a formula, and several other formatting properties.

Project setup

Install the package using the following npm command.

npm install xlsx-js-style

The first thing we should do after importing the package is to create a new workbook and worksheet. The workbook is the Excel file as a whole and the worksheet is a single spreadsheet within the Excel file.

import XLSX from "xlsx-js-style";

const workbook = XLSX.utils.book_new();
const worksheet = XLSX.utils.json_to_sheet([]);

// Logic for manipulating worksheet

XLSX.utils.book_append_sheet(workbook, worksheet, "sheet_name_here");

In the code above, we create a new worksheet and append it to the book at a later time. json_to_sheet() is a method that appends data in the form of an array of json objects to a new worksheet, but in the event that an empty array in provided, it will just return a new empty worksheet.

Use a template

We will be utilizing the XLSX.utils.sheet_add_json() method to append data to an existing worksheet. The data needs to be in the form of an array of objects where the keys are used as headers. There is an optional parameter we can enable to prevent any headers from being created, but for now we will use the default parameters so that the headers will automatically be generated from the keys. For example, the array below will generate the following Excel sheet.

const data = [
  { "Heading 1": 1, "Heading 2": 2, "Heading 3": 3 },
  { "Heading 1": 4, "Heading 2": 5, "Heading 3": 6 },
];

It’s important to note that the order of properties in the first object will determine the order of headers. I recommend using a template to organize the data before adding it to a worksheet. I found that using a template was a simple solution to ensuring that the columns are always consistent in case the data you are using has any optional properties. It can also be useful if you want to have non-unique headers.

Creating a template will require some data pre-processing to determine what properties appear in the data. For example, if the data comes in the form of JSON then you can loop through the data and maintain a template object that records what properties have been seen.

const getTemplate = () => {
  const rawData = fetchRawData();
  // Let's say the raw data contains 3 properties
  // [{ property1: 12 },
  //  { property1: 14, property2: 1 },
  //  { property3: 42 }]
  const dataTemplate = {};
  rawData.forEach((data) => {
    Object.keys(data).forEach((propertyKey) => {
      // If property doesn't exist in template, add it with an empty value
      if (!dataTemplate[propertyKey]) dataTemplate[propertyKey] = 0; // or ''
    });
  }
  // dataTemplate = { property1: 0, property2: 0, property3: 0 }
}

However, in the fundraiser example we can just create the template manually since it is so simple.

Populating the spreadsheet

After we create the template, we can use it as a base whenever we encounter a new entity in the data.

import XLSX from "xlsx-js-style";

const template = getTemplate(); // A function like the one above that produces a template from the raw data
// Template example: { name: '', shotsTaken1: 0, shotsTaken2: 0, shotsTakenTotal: 0, shotsLanded1: 0, shotsLanded2: 0, shotsLandedTotal: 0, average: 0 }
const stringifiedTemplate = JSON.stringify(template);
const rawData = fetchRawData();
// [{ name: 'Jane Doe', shots_taken: { attempt1: 10, attempt2: 12 }, shots_landed: { attempt1: 2, attempt2: 3 } }, ...]
const excelData = rawData.map((data) => {
  const entity = JSON.parse(template);
  entity.name = data.name;
  entity.shotsTaken1 = data.shots_taken.attempt1;
  entity.shotsTaken2 = data.shots_taken?.attempt2 ?? 0;
  entity.shotsLanded1 = data.shots_landed.attempt1;
  entity.shotsLanded2 = data.shots_landed?.attempt2 ?? 0;
  return entity;
});

XLSX.utils.sheet_add_json(worksheet, excelData);

We parse the template for each person we encounter in the data and then populate the existing fields with the information from the raw data. Alternatively, if the raw data comes in chunks and you need to access a person's template multiple times, you can set excelData to an object and attach each template to a unique string ID. Then you can grab the values of excelData using Object.values() and append to the worksheet. In the example below, the data for shots taken and the data for shots landed come in as two different batches of data.

const excelData = {};

rawShotsTakenData.forEach((data) => {
  const { id } = data;
  if (!excelData[id]) {
    excelData[id] = JSON.parse(template);
  }
  excelData[id].name = data.name;
  excelData[id].shotsTaken1 = data.shots_taken.attempt1;
  excelData[id].shotsTaken2 = data.shots_taken?.attempt2 ?? 0;
});

rawShotsLandedData.forEach((data) => {
  const { id } = data;
  excelData[id].shotsLanded1 = data.shots_landed.attempt1;
  excelData[id].shotsLanded2 = data.shots_landed?.attempt2 ?? 0;
});

XLSX.utils.sheet_add_json(worksheet, Object.values(excelData));

Running one of the snippets above should result in a sheet similar to the picture below.

The information is in the layout we want, but the headers are not in the state we want them. We want non-unique headers, but it is impossible to have non-unique keys in a javascript object, so what can we do? Well, since we created a template, we can now ensure that the order of properties will be the same for every person and create a custom header array to add to the worksheet. Since the custom header array will be an array of strings, we will be using the XLSX.utils.sheet_add_aoa() method to add it to an existing worksheet.

While we are at it, we can also add the secondary header that specifies which attempts are for shots taken and shots landed. The only important thing to note is that we have to leave space for all the columns we will be merging. For example in the customMergeHeaders variable below, the zero index is a placeholder to skip column 1, the first index is the name of the first merged header, and the second and third are placeholders so that the next header will be in the right spot. Make sure you add the headers to the worksheet first before the data.

const customMergeHeaders = [
  "",
  "# of Basketball Shots Taken",
  "",
  "",
  "# of Basketball Shots Landed",
  "",
  "",
  "",
];
const customHeader = [
  "Name",
  "Attempt 1",
  "Attempt 2",
  "Total",
  "Attempt 1",
  "Attempt 2",
  "Total",
  "Average",
];
XLSX.utils.sheet_add_aoa(worksheet, [customMergeHeaders, customHeader]);
XLSX.utils.sheet_add_json(worksheet, excelData, {
  skipHeader: true,
  origin: -1,
});

The skipHeader property is included in the XLSX.utils.sheet_add_json() method call so that a header row is not generated. The origin: -1 property is there to specify that we want to append the data at the end of the worksheet (otherwise it will just starting overwriting cells from A1). All the data and headers should be in the right place after this step is complete.

Formatting

This is where we start using the styling options provided by xlsx-js-style. Up until this point, we have been using functionality provided to us by the community version of SheetJs.

Once all the data is in place, we can modify the cell styles to make the worksheet presentable and user-friendly. The first style modification we are going to make is merging the header cells. This is done by providing an array of range objects to the worksheet. A range object consists of an s and e property which represents the group of cells to merge. Cells are described as a set of coordinates where c stands for the column and r stands for the row. For the example above, we will telling the worksheet to merge two groupings of cells.

worksheet["!merges"] = [
  { s: { c: 1, r: 0 }, e: { c: 3, r: 0 } },
  { s: { c: 4, r: 0 }, e: { c: 6, r: 0 } },
];

The merged headers we want should span over the three columns of data for the “shots taken” and “shots landed” categories which are cells B1 (c1,r0) to D1 (c3,r0) and E1 (c4,r0) to G1 (c6,r0).

Next, we can iterate through all the existing cells and update the styles we want for them. We can get the range of columns and rows from decoding the special worksheet property !ref. Like the coordinates above, range should have a s and e property. By getting the coordinates of the last cell in our worksheet, we can get the length of the columns and rows.

const range = XLSX.utils.decode_range(worksheet["!ref"] ?? "");
const rowCount = range.e.r;
const columnCount = range.e.c;

As we iterate through all the cells, we can check for specific columns or rows where we want to add different formatting. In the example code below, we target the first two rows and first column so that we can bold the names and headers. Each cell contains an object that holds information like the value and styling. In this case, we want to modify the s property to include styles for bolding the value and aligning it in the center. The rest of the cells are just assigned center alignment.

If there are certain values that you want to assign a special style, you can check the v property to check that cell's value. As a side note, if a cell was never assigned a value, trying to access its style property may result in an undefined error.

for (let row = 0; row <= rowCount; row++) {
  for (let col = 0; col <= columnCount; col++) {
    const cellRef = XLSX.utils.encode_cell({ r: row, c: col });
    // Add center alignment to every cell
    worksheet[cellRef].s = {
      alignment: { horizontal: "center" },
    };
    if (row === 0 || row === 1 || col === 0) {
      // Format headers and names
      worksheet[cellRef].s = {
        ...worksheet[cellRef].s;
        font: { bold: true },
      };
    }
  }
}

Lastly, we have to add the appropriate formulas for the total and average columns. Formulas are applied in the same way as styles except in this case we modify the f property. We are targeting the third and sixth column for the totals and the seventh column for the average. The code below can be placed right before the else block from above.

else if (col === 3 || col === 6) {
  // Add formulas for totals
  const valueOneCellRef = XLSX.utils.encode_cell({r: row, c: col - 2});
  const valueTwoCellRef = XLSX.utils.encode_cell({r: row, c: col - 1});
  worksheet[cellRef].f = `${valueOneCellRef}+${valueTwoCellRef}`;
} else if (col === 7) {
  // Add formulas for average
  const totalShotsCellRef = XLSX.utils.encode_cell({r: row, c: 3});
  const totalLandedCellRef = XLSX.utils.encode_cell({r: row, c: 6});
  worksheet[cellRef] = {
    f: `${totalLandedCellRef}/${totalShotsCellRef}`,
    s: {
      ...worksheet[cellRef].s;
      numFmt: '0.00%',
    },
  };
}

Exporting

The very last step to the process is exporting the Excel file. This can be done using the XLSX.write() function to create a buffer and using the buffer to upload or write the file somewhere.

XLSX.utils.book_append_sheet(workbook, worksheet, "sheet_name_here");
const excelBuffer = XLSX.write(workbook, {
  type: "buffer",
  cellStyles: true,
});

fs.writeFileSync("fundraising_data.xlsx", excelBuffer);

Conclusion

SheetJs is a great tool for generating Excel sheets and, while daunting at first, a proper strategy can simplify the process. Breaking the process up into creating a template, ingesting the raw data, appending to an excel worksheet, and formatting the cells has helped me outline a clear plan of action for dealing with more complicated data structures. I hope that the simple strategies outlined in this blog post can help you get started or inspire you to create interesting solutions. Thanks for reading!

Creating an API Gateway to SQS direct connection using CloudFormation

Thu, 04 Aug 2022 00:00:00 GMT

A common way of enqueuing messages to an AWS Simple Queue Service (SQS) is by sending a POST request to an endpoint hosted by an API Gateway. Using the AWS console, you can easily set up this connection between the API Gateway and SQS. However, we can also create this connection in CloudFormation and enjoy the benefits of Infrastructure-as-Code (IaC).

One of the main benefits of IaC here would be that you can automatically deploy changes and iterations of a stack utilizing these two services together. This would make it easier to avoid drift and human error in manual deployments.

In our example, we will be writing our CloudFormation configuration in JSON.

First, we create the API Gateway resource itself. We will be using a RestAPI here, which is specified under the Type field.

"APIGateway": {
    "Properties": {
        "Description": "API Endpoint to receive JSON payloads and queue in SQS",
        "Name": "CreativelyNamedAPIGateway"
    },
    "Type": "AWS::ApiGateway::RestApi"
},

Creating the queue is simple enough in CloudFormation. You will need to configure some of the parameters such as the maximum message size, message retention period, and the amount of time a call/action to recieve messages will wait for messages to arrive.

We also need to specify a policy for our queue. Here is a basic template of SQS and its policy that should fit most use cases, but feel free to customize it to your needs.

"DestinationQueue": {
    "Properties": {
        "DelaySeconds": 0,
        "MaximumMessageSize": 262144, // maximum size of message in bytes.
        "MessageRetentionPeriod": 1209600, // time SQS will retain a message.
        "QueueName": {
            "Ref": "queueName" // this queueName was defined under the Parameters section in our CF file. Feel free to change it.
        },
        "ReceiveMessageWaitTimeSeconds": 10, // time SQS will wait for message to arrive when RecieveMessage has been called.
        "VisibilityTimeout": 30 // period of time in seconds in which a received message is made unavailable to other consumers while it is being processed.
    },
    "Type": "AWS::SQS::Queue"
},

"PolicySQS": {
    "Properties": {
        "PolicyDocument": {
            "Statement": [
                {
                    "Action": "SQS:*",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "111111111111" // your AWS account ID goes here
                    }
                    "Resource": {
                        "Fn::GetAtt": ["DestinationQueue", "Arn"]
                    },
                    "Sid": "Sid1517269801413"
                }
            ],
            "Version": "2012-10-17"
        },
        "Queues": [
            {
            "Ref": "DestinationQueue"
            }
        ]
    },
    "Type": "AWS::SQS::QueuePolicy"
},

After creating the queue, we need to create an IAM role to allow for the API Gateway to send messages to SQS. Since the API gateway is only concerned with sending messages to SQS, that’s the only permission we need to worry about.

"APIGatewayRole": {
    "Properties": {
        "AssumeRolePolicyDocument": {
            "Statement": [
                {
                    "Action": ["sts:AssumeRole"],
                    "Effect": "Allow",
                    "Principal": {
                        "Service": ["apigateway.amazonaws.com"]
                    }
                }
            ],
            "Version": "2012-10-17"
        },
        "Path": "/",
        "Policies": [
            {
                "PolicyDocument": {
                    "Statement": [
                        {
                            "Action": "sqs:SendMessage",
                            "Effect": "Allow",
                            "Resource": {
                                "Fn::GetAtt": ["DestinationQueue", "Arn"]
                            }
                        },
                        {
                            "Action": [
                            "logs:CreateLogGroup",
                            "logs:CreateLogStream",
                            "logs:PutLogEvents"
                            ],
                            "Effect": "Allow",
                            "Resource": "*"
                        }
                    ],
                    "Version": "2012-10-17"
                },
                "PolicyName": "api-gateway-sqs-send-msg-policy"
            }
        ],
        "RoleName": "api-gateway-sqs-send-msg-role"
    },
    "Type": "AWS::IAM::Role"
},

Next, we add our endpoints to the API Gateway. We will create an endpoint called ‘v1’, and then nest another endpoint under it called ‘enqueue’. These are endpoints that are specific to the queuing functionality of our API Gateway. When we call the API gateway, the URL we use would look something like this: https://<yourAPIurl>/v1/enqueue.

"v1Resource": {
    "Properties": {
        "ParentId": {
            "Fn::GetAtt": ["APIGateway", "RootResourceId"]
        },
        "PathPart": "v1",
        "RestApiId": {
            "Ref": "APIGateway"
        }
    },
    "Type": "AWS::ApiGateway::Resource"
},
"enqueueResource": {
    "Properties": {
        "ParentId": {
            "Ref": "v1Resource"
        },
        "PathPart": "enqueue",
        "RestApiId": {
            "Ref": "APIGateway"
        }
    },
    "Type": "AWS::ApiGateway::Resource"
},

Once that’s done, we need to configure POST and OPTIONS methods for the API Gateway to send messages through to the SQS. For the headers, you will need to specify the Content-Type to be application/x-www-form-urlencoded so that the message sent through the API is in a format that is accepted by SQS. The example below shows the specific parameters that were set for the POST method.

To resolve any potential CORS errors, we will need to include an OPTIONS method configuration under our gateway. These are the only essential methods for our configuration.

"OptionsMethod": {
    "Type": "AWS::ApiGateway::Method",
    "Properties": {
        "AuthorizationType": "NONE",
        "ResourceId": {
            "Ref": "enqueueResource"
        },
        "RestApiId": {
            "Ref": "APIGateway"
        },
        "HttpMethod": "OPTIONS",
        "Integration": {
            "IntegrationResponses": [
                {
                    "StatusCode": 200,
                    "ResponseParameters": {
                        "method.response.header.Access-Control-Allow-Headers": "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'",
                        "method.response.header.Access-Control-Allow-Methods": "'POST,OPTIONS'",
                        "method.response.header.Access-Control-Allow-Origin": "'*'"
                    },
                    "ResponseTemplates": {
                        "application/json": ""
                    }
                }
            ],
            "PassthroughBehavior": "WHEN_NO_MATCH",
            "RequestTemplates": {
                "application/json": "{\"statusCode\": 200}"
            },
            "Type": "MOCK"
        },
        "MethodResponses": [
            {
                "StatusCode": 200,
                "ResponseModels": {
                    "application/json": "Empty"
                },
                "ResponseParameters": {
                    "method.response.header.Access-Control-Allow-Headers": false,
                    "method.response.header.Access-Control-Allow-Methods": false,
                    "method.response.header.Access-Control-Allow-Origin": false
                }
            }
        ]
    }
},
"PostMethod": {
    "Properties": {
        "AuthorizationType": "NONE",
        "HttpMethod": "POST",
        "Integration": {
            "Credentials": {
                "Fn::GetAtt": ["APIGatewayRole", "Arn"]
            },
            "IntegrationHttpMethod": "POST",
            "IntegrationResponses": [
                {
                    "StatusCode": "200",
                    "ResponseParameters": {
                        "method.response.header.Access-Control-Allow-Origin": "'*'"
                    }
                }
            ],
            "PassthroughBehavior": "NEVER",
            "RequestParameters": {
                "integration.request.header.Content-Type": "'application/x-www-form-urlencoded'"
            },
            "RequestTemplates": {
                "application/json": "Action=SendMessage&MessageBody=$input.body"
            },
            "Type": "AWS",
            "Uri": {
                "Fn::Join": [
                    "",
                    [
                        "arn:aws:apigateway:",
                        {
                            "Ref": "AWS::Region"
                        },
                        ":sqs:path/",
                        {
                            "Ref": "AWS::AccountId"
                        },
                        "/",
                        {
                            "Ref": "queueName"
                        }
                    ]
                ]
            }
        },
        "MethodResponses": [
            {
                "ResponseModels": {
                    "application/json": "Empty"
                },
                "StatusCode": "200",
                "ResponseParameters": {
                    "method.response.header.Access-Control-Allow-Origin": false
                }
            }
        ],
        "ResourceId": {
            "Ref": "enqueueResource"
        },
        "RestApiId": {
            "Ref": "APIGateway"
        }
    },
    "Type": "AWS::ApiGateway::Method"
},

Finally we create the deployment and stage resources so that we can deploy our API to a stage, making it ready for invocation.

"prodDeployment": {
    "DependsOn": "PostMethod",
    "Properties": {
        "RestApiId": {
            "Ref": "APIGateway"
        }
    },
    "Type": "AWS::ApiGateway::Deployment"
},
"prodStage": {
    "Properties": {
        "DeploymentId": {
            "Ref": "prodDeployment"
        },
        "RestApiId": {
            "Ref": "APIGateway"
        },
        "StageName": "Prod"
    },
    "Type": "AWS::ApiGateway::Stage"
},

If you wish to create a custom domain for your API Gateway, you will need to add some additional resources. You will need a total of 3 resources to set up the custom domain: the API Custom Domain resource, a route 53 resource, and an API mapping resource that maps your custom domain to the stage that your API gateway was deployed on.

"ApiGWCustomDomain": {
    "Type": "AWS::ApiGateway::DomainName",
    "Properties": {
        "DomainName": "custom-domain-name.com",
        "CertificateArn": { "Ref": "SSLCertificate" },
        "EndpointConfiguration": {
            "Types": ["EDGE"]
        },
        "SecurityPolicy": "TLS_1_2"
    }
},

"DNS": {
    "Type": "AWS::Route53::RecordSet",
    "Properties": {
        "HostedZoneName": "custom-domain-name.com.",
        "Name": "custom-domain-name.com.",
        "Type": "A",
        "AliasTarget": {
            "HostedZoneId": {
                "Fn::GetAtt": ["ApiGWCustomDomain", "DistributionHostedZoneId"]
            },
            "DNSName": {
                "Fn::GetAtt": ["ApiGWCustomDomain", "DistributionDomainName"]
            }
        }
    }
},

"APIMapping": {
    "Type": "AWS::ApiGateway::BasePathMapping",
    "DependsOn": ["ApiGWCustomDomain", "DNS"],
    "Properties": {
        "DomainName": "custom-domain-name.com",
        "RestApiId": {
            "Ref": "APIGateway"
        },
        "Stage": "Prod"
    }
},

Once this is all in place you can create your CloudFormation stack by copying and pasting this JSON template in the appropriate template field when you create a stack on the AWS console. Once your stack is deployed, you can use the API Gateway’s URL to start sending content to SQS via a post request. In our example, we used the Axios HTTP client to send our POST request.

axios.post(`https://custom-domain-name.com/v1/enqueue`, queueMessage, {
  headers,
});

Now you should have the tools you need to configure an API Gateway to SQS stack through CloudFormation. All the templates above are just starting points. Through CloudFormation, you can easily tweak the templates to meet your needs and redeploy the entire stack in each iteration of testing. Perhaps in the future you will consider using CloudFormation to configure more complex stacks in your apps and projects.

Using Amazon Kinesis Data Firehose to generate business insights

Wed, 03 Aug 2022 00:00:00 GMT

This article will focus on using Amazon Kinesis Data Firehose to route Lambda destination logs to S3 and connecting to QuickSight (in order to analyze product performance from a business perspective).

Event-driven architecture is a popular application design approach which uses events to trigger and communicate between decoupled services. An event can be any change in state or an update: a button press, credit card swipe, item being placed in a cart, et cetera. We used AWS Lambda and EventBridge to build out our event-driven architecture. AWS gives us the ability to route the execution results of each Lambda invocation to other AWS services — this is the concept of Lambda destinations. Lambda destinations allow you to route asynchronous function results as an execution record to a destination resource without writing additional code. An execution record contains information about the request and response (in JSON format).

Source: Amazon

As these events trigger Lambda executions, it is common to want to track performance of these Lambdas, answering questions like:

What percentage of new user form submissions failed during our onboarding in October?
Did we maintain the threshold of successful credit card swipes during the holiday period?

The answers to these sorts of questions can provide valuable business insights. A development team can use these metrics in a more practical approach towards product development. They can point to concrete data to support their decision making process. It provides higher product visibility, which usually helps the development cycle.

Project Setup

This implementation utilizes the Serverless framework: an open source software that builds, compiles, and packages code for serverless deployment, and then deploys the package to the cloud.

How can we set up a system to answer performance questions like these?

The first step in analysis is to gather and store data — it makes sense to add an S3 bucket as the Lambda destination. The problem here is that it is only possible to route Lambdas to one of four locations:

another Lambda function
SNS
SQS
EventBridge

This begs the question — How can we route these Lambda results to S3? Answer: Kinesis Data Firehose — a tool used to stream data into data buckets (in our case S3) and convert data into required formats for analysis without the need for building processing pipelines (read more here).

Architecture

We configure each Lambda with EventBridge destinations on both the success and failure cases. These success/failure events are used as triggers to the sendEventToFirehose Lambda (responsible for sending the event to Kinesis Data Firehose). Once the data is successfully streamed into the S3 bucket, we configure a QuickSight instance to use this S3 bucket as a data source (to generate an analytics dashboard).

Implementation

In this implementation, we assign a pattern event type to each Lambda. The pattern acts as a schema, helping AWS filter out the relevant events to be forwarded to the function. Each Lambda function also has destinations for the success and failure cases, in the form of event bus identifiers or ARNs. We abstract out the event bus names/ARNs, as this serverless.yml is shared between multiple developer environments (each with its own stage/event bus).

exampleLambda:
  handler: path/to/handler
  events:
    - eventBridge:
      eventBus: ${self:custom.eventBus}
      pattern:
        source:
          - Event Source (e.g. Lambda)
        detail-type:
          - Event Detail Type (e.g. Example Lambda)
  destinations:
    onSuccess: ${self:custom.eventBusArn}
    onFailure: ${self:custom.eventBusArn}
  environment:
    EVENT_BUS_NAME: ${self:custom.eventBus}

The sendEventToFirehose Lambda (configuration below) is triggered by events with detail types corresponding to successes/failures of the Lambdas like the one above. The environment key is used to define any environment variables used in the function (in this case, the name of the Kinesis Data Firehose delivery stream — a property of the CloudFormation stack used to host the Kinesis Data Firehose instance). We do not assign any destinations for this Lambda, as it does not represent any business logic we are looking to analyze.

## sendEventToFirehose Lambda configuration
 sendEventToFirehose:
    handler: path/to/sendEventToFirehose.handler
    events:
      - eventBridge:
          eventBus: ${self:custom.eventBus}
          pattern:
            source:
              - Lambda
            detail-type:
              - Lambda Function Invocation Result - Success
              - Lambda Function Invocation Result - Failure
    environment:
      DELIVERY_STREAM_NAME: ${cf:firehose-${sls:stage}.DeliveryStreamName}

The next order of business is to implement this newly configured Lambda function. In order to route the event to Kinesis Data Firehose, we create a PutRecordCommand which is to be sent by the Firehose client. The input for this command includes:

DeliveryStreamName: the name of the stream to put the data record into
Record.Data: the data blob to put into the record, which is base64-encoded when the blob is serialized.

import { EventBridgeHandler } from 'aws-lambda';
import {
  FirehoseClient,
  PutRecordCommand,
  PutRecordCommandInput,
} from '@aws-sdk/client-firehose';

const client = new FirehoseClient({ region: 'us-east-1' });

export const sendEventToFirehose: EventBridgeHandler<
  | 'Lambda Function Invocation Result - Success'
  | 'Lambda Function Invocation Result - Failure',
  { eventName: string },
  void
> = async (event) => {
  const input: PutRecordCommandInput = {
    DeliveryStreamName: process.env.DELIVERY_STREAM_NAME,
    Record: { Data: Buffer.from(`${JSON.stringify(event)}\n`) },
  };
  const command = new PutRecordCommand(input);
  await client.send(command);
};

Firehose CloudFormation Stack

We deploy the Firehose setup on its own CloudFormation stack, separate from the serverless flow it is tracking (.cform below). We assign StageName as a parameter. We then define all the resources that this CloudFormation Stack will be deploying:

S3 bucket to which the events are routed
CloudWatch Log Group to allow us to inspect logs
Firehose Role — This IAM role will allow resources inside of our AWS account to access the Firehose we created. It will also allow the Firehose stream to use our S3 bucket and CloudWatch.
Firehose delivery stream (which configures Firehose to the S3 and CloudWatch instances)

In order to connect this newly formed CloudFormation stack to our Serverless setup, we declare the name of the Firehose delivery stream as an output (which we referenced earlier — as an environment variable in our sendEventToFirehose Lambda 😲 💡 🧠 ).

##  firehoseStack.cform

Parameters:
  StageName:
    Type: String

Resources:
  EventsBucket:
    Type: AWS::S3::Bucket
  EventsCloudWatchLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/kinesisfirehose/EventsFirehose-${StageName}
  EventsCloudWatchLogStream:
    Type: AWS::Logs::LogStream
    Properties:
      LogGroupName: !Ref EventsCloudWatchLogGroup
      LogStreamName: !Sub EventsFirehoseLogStream-${StageName}
  EventsFirehoseRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub EventsFirehoseRole-${StageName}
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - firehose.amazonaws.com
              Action: sts:AssumeRole
              Condition:
                StringEquals:
                  sts:ExternalId: !Ref 'AWS::AccountId'
        Policies:
          - PolicyName: EventsFirehoseRolePolicy
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action: s3:*
                  Resource:
                    - !GetAtt EventsBucket.Arn
                    - !Sub ${EventsBucket.Arn}/*
                - Effect: Allow
                  Action: logs:PutLogEvents
                  Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/kinesisfirehose/EventsFirehose-*
  EventsFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: !Sub EventsFirehose-${StageName}
      S3DestinationConfiguration:
        CloudWatchLoggingOptions:
          Enabled: True
          LogGroupName: !Ref EventsCloudWatchLogGroup
          LogStreamName: !Ref EventsCloudWatchLogStream
        BucketARN: !GetAtt EventsBucket.Arn
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 5
        CompressionFormat: UNCOMPRESSED
        RoleARN:
          Fn::GetAtt: [EventsFirehoseRole, Arn]

Outputs:
  DeliveryStreamName:
    Description: Events Firehose Delivery Stream Name
    Value: !Ref EventsFirehose
    Export:
      Name: !Sub DeliveryStreamName-${StageName}

Connecting to QuickSight

Connecting this Firehose stream to QuickSight is as easy as following the steps here that AWS provides. JSON manifest files are used to specify files in Amazon S3 to import into Amazon QuickSight. In our case, we use the fields in:

fileLocations: to specify the files to import
- URIPrefixes: an array that lists URI prefixes for S3 buckets/folders
globalUploadSettings(optional): to specify import settings for those files (i.e. text qualifiers). Uses default values if not specified.
- format: format of files to be imported
- textqualifier: character used to signify where text begins and ends

// manifest.json
{
    "fileLocations": [
        {
            "URIPrefixes": [
                "s3://<bucket name>/"
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON",
        "textqualifier":"\""
    }
}

Final Output

Here are some visual examples of the output that QuickSight provides as a result. QuickSight can break down the data into groups, allowing for a variety of visuals similar to those below (performance by date, performance by function, and more). The QuickSight dashboard offers a whole assortment of views, filters, and metrics to play around with in order to craft the most insightful data representation.

Example QuickSight Output

Source: Amazon

Conclusion

Congratulations, the setup is complete! You can build a custom analytics dashboard and present your findings to other business constituents! Now you can use these metrics to drive product direction:

What should we iterate on?
Where should we focus our attention?

The choice is yours.

Simplify your full-stack applications with XState

Wed, 06 Jul 2022 00:00:00 GMT

XState is a popular state management library for JavaScript and TypeScript using finite state machines (FSMs) and state-charts. The general use of FSMs has been widely popularised over the last few years by React and Redux, but state machines have held a key role in the core of computer science long before the days of complex interfaces on the modern web. The same concept which informed the building blocks of the Turing machine, are today used to manage your navigation between Twitter threads. Finite state machines’ sustained, battle tested, popularity stems from the fact that they follow intuitive patterns which come naturally to humans; when you’re thinking about how you’ll “cook” your ready meal at home, you likely don’t picture a series of if statements, but instead might see it as a series of steps and transitions between them.

Today we use FSMs in day-to-day software development to add structure to seemingly abstract flows of logic. When you think of all possible permutations of a user’s path through the Twitter UI, they are seemingly infinite. But drawing them out in a state-transition diagram demonstrates that the logic is in fact quite simple. To software developers structure and simplicity are bliss, not (only) because we’re lazy, but because simple and robust code is easier to maintain and test.

Why XState

XState is just one of many modern state management libraries used by JavaScript and TypeScript developers. Some of the more popular ones are Redux, MobX and Recoil, with XState being the new kid on the block.

With well-established competition, it’s natural to wonder how XState fits in the market.

XState could be described as the only true established state-machine in the JavaScript ecosystem, since it adheres to the W3C specification. This means that when we write state machines with the XState API, these are being converted to a standardised XML notation under the hood. This XML could then theoretically be imported into another (standard compliant) state management framework, making it completely language agnostic.

The visualiser is, in my opinion, XState’s most powerful distinguishing feature. This tool leverages Finite Automata’s user-friendliness and readability by automatically generating the state diagrams for us. Even better, the visualiser enables us to define the state machines with a drag and drop interface! Remember, these features aren’t (only) because developers are lazy, but are really there to provide an intuitive bridge between developers and people without a technical background. UI design tools, such as Figma’s user flows or more traditional sequence diagrams, look almost exactly the same, with edges connecting nodes in a relatively simple diagram. This enables non-technical stakeholders to easily read and validate your work, or even design it themselves.

Using XState in the Backend

Although their use is predominantly discussed for frontend use cases, state machines can also provide backend developers with a robust platform to handle cases where an entity (which might be stored in a database or simply passed though an API) might find itself in multiple different states.

These applications of XState differ from their UI counterparts by the fact that the state machines are not the source of truth for the state - this typically being the database. XState can take properties of the entity and infer its state; which means that as long as we assume that the entity only changes via the state machine, it is still safe and deterministic (more on this later). Note that this means our servers are still stateless, and we still reap all the benefits of this fact such as horizontal scaling.

Let’s take a simple example of a food delivery app and see how it might handle an order on its way from a restaurant’s kitchen right to your door following the this state diagram:

1. Defining the machine, states and their transitions

import { createMachine } from "xstate";

const deliveryMachine = createMachine({
  id: "OrderDelivery",
  initial: "Placed",
  states: {
    Placed: {
      on: {
        Prepare: "Preparing", // Implicit transition declaration
        Cancel: {
          target: "Cancelled", // Explicit transition declaration
        },
      },
    },
    Preparing: {
      on: {
        "Send out": {
          target: "Out for Delivery",
        },
        Cancel: {
          target: "Cancelled",
        },
      },
    },
    "Out for Delivery": {
      on: {
        Deliver: {
          target: "Delivered",
        },
      },
    },
    Delivered: {
      type: "final",
    },
    Cancelled: {
      type: "final",
    },
  },
});

After installing and importing the library, we can start by defining our states in the JSON format. Here we have 3 types of states: the initial one (defined by this declaration before the list of states); regular states (for now these are just empty objects); and the final states. XState supports one other history type which we won’t cover here but they provide a way to return to a previous state of a region (nested or compound states) when we transition back to it.

Now, just having a set of states isn’t much help to us if we can’t get out of the initial sate. This is where we’d use transitions, which are nothing more than a set of objects in the on property of a state that name the event and point the interpreter to the target state after the transition has taken place.

Reading the documentation you might see two ways of declaring the transition, both are highlighted in the example above and are equivalent.

2. Actions and Effects

Until now our state machine is really only useful as a process visualisation tool. To complete it, stuff needs to happen in different states. For example, anytime we transition to the next state, we fire an update to the customer’s app telling them their pizza is one step closer to them.

These actions (also called fire-and-forget effects) can be invoked either at the entry to a state (no matter the transition to enter it), the exit of one (no matter the transition to exit it), or on a specific transition object. It's up to you which implementation you use, but the order in which actions are run should not matter (that is, you should always ensure actions will not cause side-effects that will affect each other). XState will always execute them in the following order: exit, transition, then entry.

For example, sending the notification to the client, we might do it in one of two ways:

"Preparing": {
	"on": {
		"Send out": {
			"Target": "Out for Delivery",
			"actions": () => {
				console.log("Your Pizza is almost there!");
				// Trigger API events and updates in here
			}
		}
	}
},
"Out for delivery": {
	"entry": () => {
		console.log("Your Pizza is almost there!");
		// Trigger API events and updates in here instead
	},
	"on": {
		"Deliver": "Delivered"
	}
}

Should we for some reason need to trigger an asynchronous event during at transition (for example to call and external API, or even invoking a second machine), XState provides Invoked Effects. These are used where the outcome of an asynchronous side effect can change the state of our machine, (e.g. if we cannot successfully send a notification to an orange-clad delivery partner, then we shouldn’t transition to out for delivery). Invoked effects are declared inside the state as follows:

"Preparing": {
  "invoke": {
    "id": "callDriver",
    "src": async () => {
      // Dispatch service to book a delivery driver
    },
    "onError": {
      "target": "Cancelled"
    }
  }
}

In short, actions are used when we don’t care about the result of them, like ephemeral updates/logs or firing off events to external services. Effects are everything else, where they take a long time and the result matters to us, such as API calls.

3. Context

A simple extension to a pure finite state machine is the extended state. This model allows the finite automata to track variables on top of the state it’s in. In XState, this is called the context and it is simply an object we define when the machine is created.

But when we pair XState’s context with Typescript, we get a very simple, yet extremely powerful extension to state machines: typestates. As the name implies, typestates link the type of an object to a state value; in other words, we can infer the state an object is in with its type. Practically this is done with XState’s 3rd generic type on createMachine where we define an interface with a state value and the corresponding context type:

type OrderTypeState =
  | {
      value: 'Placed';
      context: OrderPlacedContextType;
    }
  | {
      value: 'Preparing';
      context: OrderPeparingContextType
    }
  | {
      value: 'Out for Delivery';
      context: OrderDeliveringContextType
    }
  | {
      value: 'Delivered';
      context: OrderDeliveredContextType
    }
  | {
      value: 'Cancelled';
      context: OrderCancelledContextType
    };

const deliveryMachine = createMachine<OrderState, OrderEvent, OrderTypeState>({
  "id": "OrderDelivery",
  "initial": "Placed",
  "context": {
    customerAddress: ...,
    customerPhone: ...,
    deliveryDriver: undefined,
    cook: undefined,
    ...restOfDatabaseOrderEntry
  },
  "states": { ... }
});

With this, we can now infer the state of our machine given a context (this was fetched from storage), and initiate a service (instance of the machine), providing the initial context and state. This means that in the current session we have a fresh instance of our machine in the right state on which we can make deterministic transitions.

import { interpret } from "xstate";

const runTransition = (
  context: OrderMachineContext,
  event: OrderMachineEvent
) => {
  const initialState = // infer initial state from context

  // Create an instance of the machine
  const service = interpret(deliveryMachine.withContext(context));

  service.start(initialState.value); // Set the start state

  service.send(event);

  service.stop();

  if (service.state.changed) {
    // Return updated context to store in database
    return service.state.context;
  }
  throw Error("Failed to run transition");
};

But there’s more!

We’ve seen how XState stands apart from it’s competitors and how even backend developers can wield its powers, but the library offers so much more that I didn’t have the chance to cover. I'll leave you with some cool features of XState which I didn't get round to mentioning in this article:

Guards: These make transition checks even more robust by adding additional constraints and logic to them. Rather than only checking whether the state is right for the transition, we also ensure that the extended state (context) is correct. This can be a huge help with things like checking right user permissions.
The actor model: What’s more fun than one state machine? Many state machines! The actor model is a design pattern where we try to break down complex machines into multiple agents, each of which is responsible for a very specific task. These agents communicate by exchanging messages and spawning other actors. XState has a huge amount of documentation around this and is unrivalled in how well it handles multi-actor systems.
Hierarchical and Parallel states: These are somewhat of a halfway point to the actor model, but on a smaller scale. These are particularly useful to avoid state explosion since we avoid having to define all the states at a top level when they can only be accessed as children of a particular state.
Testing: In typical dev fashion, I’ve put testing in the nice to have section. But XState offers some nifty tools out of the box for mocking side-effects and testing pure logic, all of which is well worth a read in the docs.

Creating a Controlled Form with React Hook Forms

Fri, 24 Jun 2022 00:00:00 GMT

React Hook Forms is a form library for React applications to build forms with easy to configure validation through the process of calling hooks to build form state and context. React Hook Forms serve as an alternative to another popular form library, Formik. The use cases for React Hook Forms is how easy it is to handle event handlers such as onSubmit, onChange, onBlur etc. In addition, it is a really lightweight package with zero dependencies, and can have easy integration with component libraries.

In my opinion, it is an easy library to work with due to the documentation being easy to navigate as well as the flexibility of form control that is provided - a developer can develop basic forms with default html input fields, or they can develop complex forms with programatic behaviour that uses both custom, in-built and external components.

Creating a form in React without a form library

Let’s walk through creating a registration form for a site using React and TypeScript. In this example, we are going to create a Controlled Form, meaning that we handle data directly using React rather than having the data handled implicitly by React.

First let’s initialise our file to create the form.

export const BetaMaleForm = (): JSX.Element => {
  return <></>;
};

Now let’s add our input fields required for the registration form. In this example, we will have a username, password, and email fields, with a submit button to post the form.

export const BetaMaleForm = (): JSX.Element => {
  return (
    <form onSubmit={handleSubmit}>
      <Label>
        Username:
        <input type="text" name="username" />
      </Label>
      <Label>
        Password:
        <input type="password" name="password" />
      </Label>
      <Label>
        Email:
        <input type="email" name="fullName" />
      </Label>
      <input type="submit" value="Submit">
    </form>
  );
};

Then, we will need to implement some storage to hold the values each field will contain.

export const BetaMaleForm = (): JSX.Element => {
  const [username, setUsername] = useState("");
  const [password, setPassword] = useState("");
  const [email, setEmail] = useState("");

  return (
    <form onSubmit={handleSubmit}>
      <Label>
        Username:
        <input type="text" name="username" value={username} />
      </Label>
      <Label>
        Password:
        <input type="password" name="password" value={password} />
      </Label>
      <Label>
        Email:
        <input type="email" name="fullName" value={email} />
      </Label>
      <input type="submit" value="Submit">
    </form>
  );
};

Finally, when the user types, we need to be able to take their updates and apply them to the field value by capturing change events. This is where the controlled component concept comes in.

export const BetaMaleForm = (): JSX.Element => {
  const [username, setUsername] = useState("");
  const [password, setPassword] = useState("");
  const [email, setEmail] = useState("");

  const handleSubmit = () => {
    alert(`Username: ${username}, Password: ${password}, Email: ${email}`);
  };

  const handleUsernameChange = (e: ChangeEvent) => setUsername(e.target.value);

  const handlePasswordChange = (e: ChangeEvent) => setPassword(e.target.value);

  const handleEmailChange = (e: ChangeEvent) => setEmail(e.target.value);

  return (
    <form onSubmit={handleSubmit}>
      <Label>
        Username:
        <input
          type="text"
          name="username"
          onChange={handleUsernameChange}
          value={username}
        />
      </Label>
      <Label>
        Password:
        <input
          type="password"
          name="password"
          onChange={handlePasswordChange}
          value={password}
        />
      </Label>
      <Label>
        Email:
        <input
          type="email"
          name="fullName"
          onChange={handleEmailChange}
          value={email}
        />
      </Label>
      <input type="submit" value="Submit">
    </form>
  );
};

We’re done! We’ve created a registration form using React using the in-built hooks to handle storage and capturing change. However, imagine that over time when you need to manage this file you will need to add more fields to the file. This would mean more useState hooks, and then more onChange handlers to write. This would increase the length of our component file and would contribute to unreadability.

A small disclaimer - the reason why there is a lot of code is because of how we transform the form into a controlled component. If you would rather create an uncontrolled component then you would not need to focus on adding handlers for input changes or submissions. It is only when we want React to hold the single source of truth for the form values do we use controlled components.

A risk of this component file is that the more fields that need to be added to the form, the more handlers and state that need to be created - this will only get worse if a form needs advanced behaviours such as key press handlers for each field, predictive text functionality, and functionality to focus the next field.

Using React Hook Forms

Now we have seen how to build a form using React’s in built hooks, let’s do the same thing but using the React Hook Form library. I’ll take you through this incrementally.

The first function that needs to be called to initialise our form is the useForm hook. This hook’s main purpose is to set up the form management and state that will be shared between all fields linked to the form.

The useForm hook will return useful properties to help us handle form behaviour. This object is what we call methods which contains several useful functions the developer can use to initialise the form management.

const methods = useForm();

Let’s go through the hooks provided by the form and rebuild the form we’ve created from the start. First we’ll add the useForm hook to the component, and remove all the state and change handlers from the previous example:

export const AbsoluteChadForm = (): JSX.Element => {
  const methods = useForm();

  return (
    <form onSubmit={handleSubmit}>
      <Label>
        Username:
        <input type="text" name="username" />
      </Label>
      <Label>
        Password:
        <input type="text" name="password" />
      </Label>
      <Label>
        Email:
        <input type="text" name="fullName" />
      </Label>
      <input type="submit" value="Submit">
    </form>
  );
};

Delving deeper into the methods objects, this will contain several functions which give control over form behaviour. The methods I’d like to showcase are:

register - registers a field to the form defined by useForm
setValue - programatically sets a input field value by accessing the value’s key name
getValues - returns all non-nullish values entered in the form
handleSubmit - handles submit by taking in callbacks that define how submit is handled, and how errors are handled

Developers will need to call useForm per form they want to create e.g. if there are 5 forms on a single page and all the logic is written on a single file, then useForm will be called 5 times to handle each form accordingly.

The hook itself can take in an object as a parameter with a set of options that be passed to configure custom form behaviour. Useful options include:

Setting the mode of the form - the mode determines when form validation will occur e.g if mode = 'onChange' then when a connect input field triggers a change event, form validation will occur
Setting default values of the text field

Let’s add some configuration to the form by defining explicitly the submission mode and the default values for each of the inputs:

export const AbsoluteChadForm = (): JSX.Element => {
  const methods = useForm({
    mode: "onSubmit",
    defaultValues: { username: "", password: "", email: "" },
  });

  return (
    <form onSubmit={handleSubmit}>
      <Label>
        Username:
        <input type="text" name="username" />
      </Label>
      <Label>
        Password:
        <input type="text" name="password" />
      </Label>
      <Label>
        Email:
        <input type="text" name="fullName" />
      </Label>
      <input type="submit" value="Submit">
    </form>
  );
};

The most important method is the register method because it allows a developer to connect an input component to the form defined by useForm(). This is because register returns 4 important attributes:

const { onChange, onBlur, ref, name } = register("fullName");

Similar to when we had the onChange handler in our form without the library, this onChange function will handle any keyboard events that are fired when focused on the input field. The ref and onBlur will be used to manage when the input is focused or not. Finally, name is the necessary attribute required to pass values to a HTML form.

This register function can be implemented into the form by deconstructing the object returned by the function and passing them as props to the input fields we have:

export const AbsoluteChadForm = (): JSX.Element => {
  const methods = useForm({
    mode: "onSubmit",
    defaultValues: {
      fullName: "",
      email: "",
      password: "",
    },
  });

  const onSubmit = () => {
    console.log(methods.getValues());
  };

  return (
    <form onSubmit={handleSubmit(onSubmit)}>
      <input {...methods.register("fullName")} type="text" name="fullName" />
      <input {...methods.register("email")} type="email" name="email" />
      <input
        {...methods.register("password")}
        type="password"
        name="password"
      />
      <input type="submit" value="Submit" />
    </form>
  );
};

Comparing the library functionality against a form using in-built React hooks, a clear advantage of using this library is the minimal amount of lines required for a component file. This is because change and submit handlers can be removed, and useState hooks are not needed. Developers can simply “hook” inputs into the form using the register method (while passing it a reference to the form values that the input field connects to aka name).

By allowing developers to simply “hook” into the input fields defined in the form via useForm and named references, then you can easily set default values/placeholders without having to manage them for each input manually - the maintenance for user experience is reduced.

Adding validation with React Hook Forms

The beauty of the register function is that it can be used to define validation rules for the input field addressed. For this case, let’s say we want to have validation rules for the password field that are as follows:

minimum 8 characters
has an uppercase letter
has a special character

Validation can be defined by passing an object with the require rules to the register function as a parameter.

<input
  {...register("fullName", { required: true, minLength: 8, pattern: "" })}
  type="text"
  name="fullName"
/>

When the user clicks the submit button, due to the mode being set to ‘submit’, validation will be performed on the submit button click, and this involves checking the validation rules defined in the register function.

Information on adding user-defined functions for validation rules can be seen in the documentation for register. React Hook Forms also has the ability to use Yup validation in addition to the existing validation methods.

Delving into the advanced topics

Using React Hook Forms with component libraries

One great use of React Hook Forms that was useful for my projects has been its integration with component libraries such as Material UI or Chakra UI. Should a development team want to create an MVP with a component library while having easy connection to the React hook form library, then Controller is a lifesaver.

Controller is a wrapper component that can be used to wrap components and propagate react-hook-form attributes and behaviours down to the components

export const ControlledForm = (): JSX.Element => {
  const methods = useForm({
    mode: "onSubmit",
    defaultValues: {
      fullName: "",
      email: "",
      password: "",
    },
  });

  const onSubmit = () => {
    console.log(methods.getValues());
  };

  return (
    <form onSubmit={handleSubmit(onSubmit)}>
      <input {...methods.register("fullName")} type="text" name="fullName" />
      <Controller
        control={methods.control}
        name="email"
        render={({
          field: { onChange, onBlur, value, name, ref },,
          formState,
        }) => (
          <TextField
            type="text"
            variant="outlined"
            onChange={onChange}
            onBlur={onBlur}
            name={name}
            value={value}
          />
        )}
      />
      <input
        {...methods.register("password")}
        type="password"
        name="password"
      />
      <input type="submit" value="Submit" />
    </form>
  );
};

Although this method is easy to read, as well as being easy to implement, to integrate external components into your application, a cleaner solution can be to use the useController hook function instead.

import { TextField } from "@mui/material";
import { useController } from "react-hook-form";

interface CustomInputProps {
  name: string;
  control: Object;
}

export const CustomInput = ({
  name,
  control,
}: CustomInputProps): JSX.Element => {
  const {
    field: { onChange, onBlur, name, value, ref },
  } = useController({
    name,
    control,
    rules: { required: true },
  });

  return (
    <TextField
      type="text"
      variant="outlined"
      onChange={onChange}
      onBlur={onBlur}
      name={name}
      value={value}
    />
  );
};

This will allow developers to keep their code cleaner by adding a hook in the component file rather than having a wrapper on the page file. Apart from the difference being one is a wrapper and the other is a hook, they both have the same behaviour.

The key variables for controlling a component is the control variable. This is an object returned by the register function with the purpose of registering components to the form created. Accessing the contents inside this variable is not recommended by react-hook-forms.

Complex Input Behaviour

I’ve mentioned previously that the benefit of react hook forms is that it abstracts handling event listeners on the library side for updating values in input fields and storing them accordingly, but the library is not limited to this abstraction as it allows you to manually set when to trigger event handlers on input fields to update the form values at any time you require.

Recalling the register function, rather than deconstructing the function inside an input element, you can define register as so:

const { onChange } = register("firstName")

...

<input onChange={onChange} type='text' name='fullName' />

If you deconstruct the props before passing it to the input field to connect to the form, you can manually set each individual property that is provided by register. The drawback if this approach for a basic form is that you could omit important event handlers provided by the function and you do not get the appropriate behaviour for your form. However, if you need business logic to be run before calling the onChange function then you can so, and then call onChange with the new formatted input value passed as a parameter.

Complex Form Components + Reusability

A common issue I have faced in the projects I have worked with is the scalability of forms to accommodate both new form fields, as well as utilising complex form behaviour. To tackle the issue of scalability you need to consider two approaches to building form field components:

Creating React Form Field Components that can be reused
Creating Field Components with completely different behaviours entirely.

You might ask why you cannot have both approaches in a codebase. You can absolutely do that but in my opinion it makes the codebase look messy. So I recommend trying to split these approaches in two, using some of the underused hooks of React Hook Forms.

Reusable Components

React Hook Forms has an article dedicated to creating a “Smart Form Component”, which involves creating a wrapper component with the useForm hook called inside, and passing down the form methods to the components, whether it is a handpicked selection of methods or all of them. The result is that you have a wrapper component that injects form methods into the child components inside the wrapper:

// From the React Hook Form guide

import React from "react";
import { useForm } from "react-hook-form";

interface FormProps {
	defaultValues: { firstName: string, email: string, password: string },
	children: JSX.Element[],
	onSubmit: () => void,
}

export const Form = ({ defaultValues, children, onSubmit }: FormProps): JSX.Element {
  const methods = useForm({ defaultValues });
  const { handleSubmit } = methods;

  return (
    <form onSubmit={handleSubmit(onSubmit)}>
      {React.Children.map(children, child => {
        return child.props.name
          ? React.createElement(child.type, {
              ...{
                ...child.props,
                register: methods.register,
                key: child.props.name
              }
            })
          : child;
       })}
    </form>
  );
}

import React from "react";
import { Form, Input, Select } from "./Components";

export const SmartForm = (): JSX.Element => {
  const onSubmit = (data) => console.log(data);

  return (
    <Form onSubmit={onSubmit}>
      <input type="text" name="fullName" />
      <input type="email" name="email" />
      <input type="password" name="password" />
      <input type="submit" value="Submit" />
    </Form>
  );
};

Components requiring complex behaviour

If it is the case that a form component will require complex behaviour that should not be reused for a different component, then it is a good idea to understand how to gain access to the form methods and state required. One good hook to use is useFormContextwhich acts very similar to React’s useContext. This hook allows you to fetch the form methods provided by useForm without having to call the hook again (because if you do, then a new form is initialised).

export const AbsoluteChadForm = (): JSX.Element => {
  const methods = useForm({
    mode: "onSubmit",
    defaultValues: {
      fullName: "",
      email: "",
      password: "",
    },
  });

  const onSubmit = () => {
    console.log(methods.getValues());
  };

  return (
    <FormProvider {...methods}>
      <form onSubmit={handleSubmit(onSubmit)}>
        <input {...register("fullName")} type="text" name="fullName" />
        <input {...register("email")} type="email" name="email" />
        <input {...register("password")} type="password" name="password" />
        <ComplexBehavingComponent name="complexField" />
        <input type="submit" value="Submit" />
      </form>
    </FormProvider>
  );
};

interface ComplexBehavingComponentProps {
	name: string
}

export const ComplexBehavingComponent = ({name}: ComplexBehavingComponentProps): JSX.Element => {
	const { register, setValue, getValues } = useFormContext()

	const onChange = (e: ChangeEvent) => {
		if (e.target.value === 'hello world') setValue(name, 'software dev')
	}

	return (
		<input {...register(name, { onChange: onChange }) type='text' name={name} />
	)
}

The benefits of this hook is that instead of having to prop drill to lower level components, or if you want to connect components lower in the DOM tree to the Form component at the top level, this is possible using the FormProvider wrapper at the level where form data will be sent to children and grand-child props, and then accessed with the useFormContext hook.

Summary

Overall, we have looked at using the basics of React Hook Forms, the advantages it provides as a developer compared with using React’s existing hooks, and have provided examples on how to take advantage of React Hook Form’s utilities for complex form behaviour. Hopefully, this article has inspired you to use the library and make your React forms incredibly effective. For more information on React Hook Forms, see the website for access to the API documentation.

4 ways to build a custom Strapi Admin Panel

Thu, 16 Jun 2022 00:00:00 GMT

Few months ago, I started a project as a software engineer in which we were using Strapi V3.

For those who are not familiar with Strapi, it is an open-source, headless CMS (Content Management System) which allows users to create, manage and expose data thanks to a built-in Admin Panel.

On my project, we quickly began to customize the Strapi’s Admin Panel by directly overriding the files in our project: we just had to reproduce the path to the specific file we wanted to modify.

You can think of it as if you wanted to eject a specific file which needs to be modified. For example, if I want to change the color of a the “Publish/Unpublish” button from blue to green, I have to:

Find where the button’s color is handled in the Github repository of Strapi V3. It is in the file : packages/strapi-plugin-content-manager/admin/src/containers/EditView/Header/index.js. (This is the file I want to eject)
Create the folder relative to the package in the extensions folder, /extensions/content-manager/
Recreate the path = create the index.js file: extensions/content-manager/admin/src/containers/EditView/Header/index.js
Copy paste the old file (from the repository or the node modules)
Replace "primary" by "success" on line 13

Then, when building the Strapi Admin Panel, Strapi would know that this file had been modified and would use this one instead of the old one. And so, the Publish/Unpublish button is green.

Then Strapi V4 was live ! And in Strapi V4, this functionnality is not available anymore.

So I quickly wondered if it would still be possible to customize our Strapi’s interface. In this article, I am going to present you four ways to customize your Strapi Admin Panel, starting with the easiest one: the use of an already existing plugin. Then the configuration options technique, followed by the injection file method, in order to finish with the more complex one: the creation of your own plugin.

To illustrate, I am taking the example of a media: the writers need a CMS to write, edit and publish their articles and they chose Strapi V4.

Existing plugin

The first and easiest thing to do when you want to customize something in your Strapi’s interface is to try and find a plugin that does what you want.

To find plugins you can look into:

In my case, the editors are not huge fans of the markdown syntax used in the RichText Object of Strapi V4, they would appreciate something more realistic to see directly the result without having to use the preview mode.

I found the plugin "strapi-plugin-editorjs" which is an EditorJS plugin for Strapi to replace the RichText field offered by Strapi. The documentation is very clear on how to install and use the plugin.

The only thing to do in this case is:

yarn add strapi-plugin-react-editorjs

And after this, the RichText object in the Strapi Admin Panel uses the EditorJS one:

What if the plugin I want does not exist ?

Configuration options

First, you can very easily modify the Strapi Admin Panel thanks to the configuration options: this is used for the translations, the logo, the favicon, the available locales, the theme, the tutorials and even the notification (see the documentation here).

I am going to quickly show you how to use it with the translations file and how to change all the wordings of your Strapi’s interface.

You have to find the translation you want to change: for example if I want to change the title of the homepage from ”Welcome 👋” to “Hi Chloé!”:

💡 To easily find the translation file in the github folder: you can replace the .com by .dev in the Github URL which opens a VisualStudio tab in your internet tab. And then, find directly the translation to change thanks to the old message 💡

// src/admin/app.js

export default {
  config: {
    translations: {
      en: {
        "app.components.HomePage.welcome.again": "Hi Chloé!",
      },
    },
  },
  bootstrap() {},
};

You will see:

💡Use yarn develop —watch-admin in order to see directly the modifications you are applying without having to relaunch your server. 💡

Injection zone

It is also possible to customize the Admin Panel using the injection zones API without having to generate a plugin with injectContentManagerComponent.

I am going to take a simple example to show you easily how to add a custom component in your Strapi’s interface: I want to add a button on the Strapi’s interface which is printing “Hello World”

First create the /extensions folder in src/admin and then you can create your component:

// src/admin/extensions/components/HelloWorldButton/index.js

import React from "react";
import { Button } from "@strapi/design-system/Button";
import Up from "@strapi/icons/Up";

const HelloWorldButton = () => {
  return (
    <Button
      variant="secondary"
      startIcon={<Up />}
      onClick={() => alert("Hello World")}
    >
      Hello World
    </Button>
  );
};

export default HelloWorldButton;

And then use the injectContentManagerComponent in the app.js file to inject your new component directly into your app:

// src/admin/app.js

import HelloWorldButton from "./extensions/components/HelloWorldButton";

export default {
  bootstrap(app) {
    app.injectContentManagerComponent("listView", "actions", {
      name: "HelloWorldButton",
      Component: HelloWorldButton,
    });
  },
};

Which is doing that:

You can choose where to put your component thanks to the two first parameters of the injectContentManagerComponent:

This example of injection may seem a little useless so I want to present you a use case in which I used it.

In my case, I have a homepage which highlights some articles. These articles are chosen by my editors thanks to a relation with my “Article” collection.

I only want to see in my homepage articles that are published and not the ones that are drafted. Unfortunately, the selector used to do the relation with the “Article” collection allows users to choose between all the articles (even the drafted ones). My writers think that it can be dangerous because even if there is a little point of color indicating that it is not yet published, it is possible that one day someone makes a mistake and puts on the homepage a draft article.

In my project, I have created a component that uses the native selector of Strapi but I have added a check in order to allow the user to select only the published articles.

NB: You can also use this method to inject a component of a specific plugin into another one by using the method injectComponent() which works the same way as injectContentManagerComponent() (documentation here)

New plugin ?

If what you want to do is more complex than translations or add components into the Strapi’s interface, you will probably have to create your own plugin. I recommand you read this article or see this video which are basically tutos on how to create your own plugin.

This solution is more complex and will take much more time than the two other solutions. If you can it is better to use the injection zone in order to save some time.

Conclusion

Even if it was globally easier and quicker to customize the Strapi Admin Panel using Strapi V3, it was not very stable. This method can be seen as a file ejection: it is basically to take some code from Strapi and to rewrite it in your own codebase which creates a strong relation between an extern library (Strapi) and your own code. At each new version of Strapi if a file changed you had to change also your code and find where to do it and how to do it. It was not very maintainable and it could break a lot of things if you had done a lot of modifications.

In Strapi V4, there is still a connection between Strapi and your codebase if, for example, you are reusing Strapi's component (as I did in my injection example). But the connection is much smaller and your modifications are more stable. The choice of Strapi to remove the ejection method seems pragmatic in order to maintain your app more easily.

Increasing user acquisition by 15% through experiments on the onboarding funnel

Wed, 15 Jun 2022 00:00:00 GMT

A summary of my findings while helping a business increase their user acquisition by performing modifications and experiments to their onboarding funnel. A increase of ~15% was seen when synchronous email verification was removed from the flow, this however did come at the cost of a few risks to the business.

TLDR:

Statistics from the onboarding funnel of a Healthcare app I was working on showed that 15% of users that made it to an email verification screen, dropped off. To combat this problem the team brainstormed a wide range of solutions to make the process more streamlined. The solution we decided to go with was to remove the ‘synchronous’ email verification, and add in the ability for users to verify their emails after signing up. This increased the overall onboarding success percentage by ~15%, with no negative impact for users with unverified emails.

The Problem

While developing an app for a healthcare company we noticed a high percentage of users (15%) were being lost on our ‘verify your email’ screen of the onboarding process. This healthcare company was in the start-up phase of its lifecycle and looking to progress into its series A funding round, during this stage it was quite critical for the business to show proof of adoption of its intellectual property. This was mainly measured by two metrics, the first and most of important was the amount of users on the app (user acquisition), and the second was if the users were interacting with the app (user activity/ user retention). Since user acquisition was then the main focus for the business, it was in our interest to make the onboarding funnel to get users onto the app as painless as possible. This ~15% drop off then became a huge focus as it meant 3 out of every 20 users who tried to onboard onto app left and never came back at the email verification stage.

As can be seen from the onboarding funnel breakdown above, there are two clear drop-offs, one right after the first screen if the users are not interested in the app, and one after the verification screen.

The hypothesis was that this screen actively dragged users attention away from the app, causing them to lose focus or interest and simply not complete the onboarding journey. A few case studies were done on ways we could improve this process, ideas such as an onboarding completion stepper, clearer wording, identity federation signup and others were explored. However, we found that the most pragmatic solution was to simply remove email verification from the onboarding process. This was due to the fact that doing this showed the biggest amount of potential gain (~15% more user acquisition) for a medium sized amount of work, whereas all of the other improvements would’ve seen minor gains at best.

Execution (Tech Implementation)

The tech stack for this project consisted of a React-Native frontend with a AWS serverless backend, and our identity service provider was AWS’s Cognito service. The biggest challenge that came with our method was surprisingly not actually the infrastructure required to do an email verification asynchronously. It was how to do it so that none of the production users would experience any downtime or broken accounts.

There are two main functions that Cognito provides that are important to this particular userflow, these functions are SignIn and SignUp.

SignUp allows users to start their account creation within the Cognito service, while a user is in this state they are not regarded as confirmed and cannot be authorised to use the protected lambdas via Appsync.

The SignIn function then completes the Cognito onboarding by requiring the user to confirm with either their phone number or email that they are indeed that user.

Infrastructure:

There are two possible ways that AWS allows you to have a user signup in your cognito userpool and not be ‘verified’:

1.) Configure your Cognito infrastructure to have the autoVerifyAttributes flag set to which attribute you want to verify, either 'email' or 'phone_number', this will auto-verify this attribute as soon as the user 'signs up'

2.) Create a OnCognitoPreSignUp trigger which invokes a lambda when a signUp call is triggered, this lambda will run before all of the actual signup logic. In this lambda you can set the user to be verified. This will stop the user needing to be ‘verified’ before proceeding with sign-in.

App Version Considerations:

In this project users could be on a variety of app versions since we restricted the amount of ‘codepush’ changes for app store releases. Due to the fact we had all the apps pointing to a single backend, we needed to keep this in mind as backend and frontend releases were not simultaneous on this project. This is a huge consideration for this kind of project and can very easily lead to broken app versions and downtime for users not on the latest version (something we were trying to avoid).

For this project method 2 was chosen as it allowed us to incrementally do the work to update the app behind the scenes using feature flags (handled by Firebase). Approach 2 also gave us more control on what each version of the app should be seeing, where as approach 1 would have repercussions on all app versions. Due to this decision we were able to do continuous releases of the app, and then trigger the switch from synchronous verification to asynchronous verification without any users having a broken experience.

Changes to the user flow

The frontend changes required for transitioning users from a verified to unverified flow were quite straightforward. Essentially the email verification screen just needed to be hidden and the signup and signin Cognito commands needed to be called from the password screen, as we wanted the user to be signed in for the second half of the onboarding process to access our protected lambdas. These changes were all controlled by feature flags from the frontends perspective.

Standard User Flow:

It is important to note in this flow that the user is actually ‘signed in’ and authenticated with AWS partway through the onboarding flow, this allows the user to access lambdas which return business specific options for the user to select.

Auto-verified User Flow:

All of the magic here happened when Cognitos Signup function was called from the frontend, in the latest version of the app we called this function with clientMetadata, sending a Confirm attribute if the user should be automatically confirmed. This lambda is called before any internal Cognito sign-up logic is run, as can be seen by its invocation being before the sign-up step in the flow. The invocation of this lambda is fairly straightforward, if the feature flag is switched on and the frontend is sending a ‘autoConfirm’ flag, then the lambda verifies the users. This then allows the user to call Cognito’s Sign-in function and become a verified user which allows them to call our protected lambdas.

Risk analysis

The risks of not having a user verify their email during the onboarding can be quite severe if taken advantage of, however most can be mitigated. All of the risk related to this stems from the possibility of the user mistyping their email, if a user gets their email wrong and logs into the app there are a variety of concerns for the business. These ‘unverified’ users could:

use features that may send emails to them or ask for a password reset, if their email is unverified this can lead to another person (who may not even be aware of the application) getting emails about this user while completely unaware.
- This can be mitigated by ensuring no marketing emails or sensitive application features can be used by ‘unverified’ users.
a larger risk is if the ‘unverified’ users mistypes their email such that it lands in the inbox of a person that intends to do harm, this person would be able to verify their email for them if the verification process is done via a ‘magic link’.
If this attacker is then able to verify their email, they could request a password reset and then completely have access to the users account.
- To mitigate for this you could either disable password resets or use user information or a code to verify their email.

However, as dire as these risks seem they are quite large edge cases, that in reality, probably wont happen at all, due to this the increased risk was acceptable for the goal of trying to get a smoother user onboarding into the platform. One way the business did choose to decrease the risk even further was by making the user double check the email they typed in was correct through use of a prompt.

Risk Reflection

Even though the risk was mitigated at the time, and it was the most pragmatic solution to take, I am still unsettled by any increasing security risk. To further reduce the risk of a breach or a phoney verification there are some actions that can be taken, at a further time cost for the user. such as:

Instead of using a magic link that authenticates anyone who has received an email with it, other methods such as a link that asks for a verification code found within the users account can be used.
Use a link that asks for a crucial bit of information that user has already entered on the app to add a bit of extra security around this email verification process.

Result

Looking at the comparison between conversion rates of the app version before the tech implementation and after (seen below). We can see there is about a 15% improvement on the overall funnel.

Overall the experiment run by this business to decrease the amount of users lost by the onboarding flow was a great success with a 15% increase of signups and no security breaches reported.

How to upgrade your PyTorch model to run on AWS Sagemaker

Mon, 30 May 2022 00:00:00 GMT

AWS Sagemaker makes it easy to train and deploy a machine learning model in the cloud. Sagemaker is an AWS service built on top of the Elastic Container Service (ECS) and S3. When you provide your training script and dataset, Sagemaker uploads them to S3, trains your model, and saves your trained model to S3 to be downloaded or deployed directly using Sagemaker Endpoints.

Training your model on Sagemaker gives you access to a range of powerful machines, and the ability to distribute your training across as many as you like! Proceed with caution, though; if you use Sagemaker's power to the max, you may end up paying a lot for it

In this article, we'll quickly go through the steps to upgrade your normal PyTorch training script, into a Sagemaker-compatible script, that can be distributed over multiple GPUs.

Upgrade your training script

To start with we've got our normal PyTorch training file, with a model and a training function that saves the model at the end. You can see the one I'm using here. We could run this in a notebook, or a python terminal. We only need to make a few changes to make this script run on AWS.

Top-level code environment

Firstly, our train function needs to be run within the top-level code environment. We do this by putting the code within a if __name__ == '_main__' block. When we run the python file from the terminal, the variable __name__ gets set to "__main__". This means that we can run

python -m training_script.py

And it will run the block contained within if __name__ == '__main__', but if we import some functions from our file like this

from training_script import my_funcmy_func()

Then the block will not be run, because here training_script.__name__ = "training_script".

We care about this because if we use our trained model in a SageMaker Endpoint, Sagemaker will import our model from our training script file, and we don't want our training script to run every time! Take a look at the Python docs here to learn about the Python top-level code environment.

if __name__ == "__main__":
    train()

When Sagemaker runs our endpoint, it is passed the training hyperparameters as arguments. The standard way to parse these is to use the argparse library. We also have access to loads of environment variables containing information about the EC2 instance we are running on, such as how many GPU cores are available, or where to save the trained model.

Our training script will be run on a Docker container on AWS. Sagemaker will expect our data to be in a specific place when it runs our training script, and it provides that path in the SM_CHANNEL_TRAINING environment variable. We can actually specify any SM_CHANNEL_??? and put different data in each, such as SM_CHANNEL_VALIDATION or SM_CHANNEL_TEST.

When our training job is finished, Sagemaker will upload our trained model to S3. To do this, it expects it to be in the directory specified by the SM_MODEL_DIR environment variable.

if __name__ == "__main__":
    parser = argparse.ArgumentParser() # hyperparameters:
    parser.add*argument("--batch-size",type=int,default=64)
    parser.add_argument("--test-batch-size", type=int, default=1000)
    parser.add_argument("--epochs", type=int, default=10)
    parser.add_argument("--lr", type=float, default=0.01)
    parser.add_argument("--momentum", type=float, default=0.5)
    # directories to save the model and get the training data:
    parser.add_argument("--model-dir", type=str, default=os.environ["SM_MODEL_DIR"])
    parser.add_argument("--data-dir", type=str, default=os.environ["SM_CHANNEL_TRAINING"])
    *, args = parser.parse_args()
    train(args)

We can then update our train function to use these hyperparameters, such as args.batch_size. We also need to update our dataset loaders to get the data from the directory args.data_dir, and our saving function to save to args.model_dir.

We could stop here if we wanted, as this script is ready to be used on AWS! That being said, we're not taking advantage of a big feature of Sagemaker: distributed training.

Distributed Training

One of the powers of the cloud is horizontal scaling; the ability to increase the number of machines that are running your code. Sagemaker's built-in models are built to take advantage of this, and you can specify the number of instances you want to run your training script on. We can build this into our PyTorch model, and speed up training!

We need to use the pytorch.distributed package to parallelise our training loop. It's surprisingly simple to get your model running on multiple instances at the same time using PyTorch's DistributedDataParallel.

Some new imports:

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
import json

And we need to add some new arguments in our top level environment:

parser.add_argument("--hosts", type=list, default=json.loads(os.environ["SM_HOSTS"]))
parser.add_argument("--current-host", type=str, default=os.environ["SM_CURRENT_HOST"])
parser.add_argument("--num-gpus", type=int, default=os.environ["SM_NUM_GPUS"])
parser.add_argument("--backend",type=str,default=None)

hosts is an array of the names of all the instances that our training script is running on
current_host is the name of the host that is currently running the code
num_gpus is the number of gpu's on the instance Backend is the PyTorch distributed backend that we want to use (see here )

Inside the train function, we set the device to "cuda" if we have some gpu's available:

use_cuda = args.num_gpus > 0
device = torch.device("cuda" if use_cuda else "cpu")

We'll also want to put the data onto the GPU if there's one available using data, target = data.to(device) ,target.to(device) in the test and train loops.

Then we need to start a process group on each of our machines, we tell PyTorch how many instances there are (world_size) and where the current process sits in that list (host_rank) so that it can link them up during training and average gradients between machines.

world_size = len(args.hosts)
os.environ["WORLD_SIZE"] = str(world_size)
host_rank = args.hosts.index(args.current_host)
os.environ["RANK"] = str(host_rank)
dist.init_process_group(backend=args.backend, rank=host_rank, world_size=world_size)

We then need to wrap the model in DistributedDataParallel and set it to the device

model = Net().to(device)
model = DDP(model)

We need to split the training data across our different instances. To do this we use a DistributedSampler. The sampler splits the dataset for us across the different instances.

train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_set, num_replicas=dist.get_world_size(), rank=dist.get_rank()
)
train_loader = torch.utils.data.DataLoader(
    train_set, batch_size=args.batch_size, sampler=train_sampler, shuffle=False
)

Now we've got a distributed model!

Logging

When you're training on your own computer or in an environment you can access directly, you can use python's built in print function to keep tabs on your model in the terminal. On Sagemaker, print statements will print to the docker container's stdout and you'll never see it! To get logging working on Sagemaker we just need to add:

import logging
import sys
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))

We can now use logging.info(...) everywhere we would normally use print(…), and we'll see it in the Jupyter notebook!

Model Loading

Now our simple training script is ready to run on Sagemaker! We could start straight away, and our trained model would be saved to model_dir/model.pth. However we'd have to go to the S3 bucket we saved it in and retrieve it manually. In order to deploy our model with Sagemaker ndpoints, we need to provide a function to load that model from storage.

def model_fn(model_dir):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = torch.nn.DataParallel(Net())
    with open(os.path.join(model_dir, "model.pth"), "rb") as f:
    model.load_state_dict(torch.load(f))
    return model.to(device)

This function receives the model_dir, and we create a Net() object and load in our saved hyperparameters.

Now we are ready to go! You can see the completed code, with all these changes here.

Conclusion

You've just upgraded your training script to work on AWS Sagemaker, and take advantage of its distributed machine learning power! Training your PyTorch model on AWS is great if you need a lot of power. You've now got an AWS compatible training script, that can run distributed across many instances. There's a little more work to be done to start your training job, which we can do using a Sagemaker Notebook instance.

_{^{Rocket illustrations by Storyset}}

Embedding videos in a React Native app

Fri, 20 May 2022 00:00:00 GMT

Web Illustrations by Storyset

For the last few months, I have been working on a React Native multimedia app. We already supported reading articles and listening to podcasts, but the problem started when we began to implement the video player. The app crashed, video player didn’t render as expected, and we couldn’t integrate it with the rest of the application!

In this article, I’ll describe how we overcame these difficulties and embedded videos inside a React Native mobile app using react-native-video.

react-native-video is the most popular video library for React Native. The other libraries like expo-video-player or expo-av require Expo modules to run, and since we weren’t using Expo in our project, we didn’t want to spend time setting up our environment. Little did we know that using react-native-video wouldn’t be a walk in the park...

Installation:

To replicate our app, let’s create a new react native app (npx react-native init MultimediaApp), install the react-native-video (yarn add react-native-video) and update App.tsx to add a simple video component:

import Video from 'react-native-video';

const videoStyle = {
    position: 'absolute',
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
    height: 250,
  };

const viewStyle = {height: 250};

...
<Section title="Step One">The example video:</Section>
<View style={viewStyle}>
  <Video
    source={{
      uri: 'https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4',
    }}
    style={videoStyle}
    controls={true}
    resizeMode="cover"
    hideShutterView={true}
    paused={true}
  />
</View>

Now, let’s try to run the app!

IOS:

It should all work fine! In our case, the app ran successfully with no errors. The video player looked like this:

Android

The app doesn’t run

Unfortunately, things weren’t that smooth with Android. The app didn't even run, and I got the following error:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
   > Could not find com.yqritc:android-scalablevideoview:1.0.4.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/com/yqritc/android-scalablevideoview/1.0.4/android-scalablevideoview-1.0.4.pom

...

Required by:
         project :app > project :react-native-video

Luckily, it turned out it’s quite a common issue with an easy fix. It happens because jcenter() was removed from React Native v0.65+, but react-native-video still uses it. To fix it, we need to declare the jcenter() to our build, inside android/build.gradle:

allprojects {
    repositories {
			(...)
			jcenter()
		}
}

Video player doesn’t render as expected, and the app may crash

I was relieved to find that the app was no longer crashing. However, I noticed that the video player still didn’t render as expected! It stopped playing whilst off-screen, and when I scrolled, the video control panel moved while the video stayed in place. Sometimes when I scrolled too fast the app crashed.

The app didn't return any errors, so it was really hard to debug. The issues on the react-native-video GitHub page helped me to find an answer.

This buggy rendering happens because the react-native-video doesn’t use the ExoPlayer by default. To switch the app to use ExoPlayer, we need to create a react-native.config file on the project root level and add the following:

module.exports = {
  dependencies: {
    "react-native-video": {
      platforms: {
        android: {
          sourceDir: "../node_modules/react-native-video/android-exoplayer",
        },
      },
    },
  },
};

For React Native 0.60 and above, the package is linked to the project automatically, but, as official library documentation suggests, if you have trouble, try linking the react-native-videomanually by following the library documentation.

Phew! That was quite a journey! Our app should now support videos for both IOS and Android!

Let’s talk about one more case - what if the app supports both video and audio?

The app doesn’t work with react-native-track-player

This was the hardest part to debug. Again, no useful errors were printed, the app compiled, but it crashed immediately after opening.

Initially, I didn’t know it was an issue with react-native-track-player, so I wasn’t sure what to look for when finding solutions. To find the root cause, I created a simple app and followed all the steps above to install react-native-video to ensure that it is not just a react-native-video problem anymore. When I started reading more about the packages we’re using, it turned out react-native-video and react-native-track-player are using the same package dependency: android-exoplayer.

This package is being installed in both places and because react-native-video and react-native-track-player use different, incompatible versions of android-exoplayer, the app crashes immediately after opening.

To fix it, we need to unify the version of android-exoplayer manually. The easiest solution is to use patch-package, following the steps described here.

Conclusions

Using react-native-video can be tricky, especially for Android, but I hope this article will make your development process much easier!

If you’d like to see a detailed review of react-native-video, I’m currently writing an article that compares it with expo-av and expo-video-player, so stay tuned! 😊

Writing Quality Code Using the Chain of Responsibility Pattern

Fri, 29 Apr 2022 00:00:00 GMT

At the beginning of my software engineering internship, I had little experience with writing clean code. The features were functional but there was hardly any underlying organization. I was lacking guidelines on what classes to create and how they should interact with each other. That’s when my mentor introduced me to design patterns. These are solutions for creating flexible and maintainable modules.

In the upcoming sections, I will showcase the use of the Chain of Responsibility pattern and how it helped us write scalable and easily testable code on a production-level API.

A design problem

On a project for a renewable-energy provider, I was working on a consumption monitoring app. One goal was to retrieve the user’s activities over a time period. An activity is a carbon footprint calculated based on energy consumption. To compute the activities, we had to fetch and process the consumption in the following priority order:

The real consumption (meter reading), retrieved from an external API
An estimate of the household's annual consumption, retrieved from an external API
An estimate calculated from the user's behavioral data (thereafter labeled macro estimate).

The returned activities are the result of the first of these methods that returns a non-empty array.

A trivial solution

While designing a solution for the presented situation, we have to take into account the maintainability of the code, the simplicity of adding and removing data sources as well as testing our service. An initial solution may consist of cramming everything in one class.

namespace Component\Activity\Provider;

class ActivityProvider {

    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate): array
    {
        if (count(retrieveMeterActivities($startDate, $endDate)) > 0) 
            return retrieveMeterActivities($startDate, $endDate);

        if (count(retrieveEstimatedActivities($startDate, $endDate)) > 0) 
            return retrieveEstimatedActivities($startDate, $endDate);

        if (count(retrieveMacroActivities($startDate, $endDate)) > 0) 
            return retrieveMacroActivities($startDate, $endDate);

        return [];
    }
    
    private function retrieveMeterActivities(\DateTime $startDate, \DateTime $endDate)
    {
        $data = $this->fetchMeterData($startDate, $endDate);
        return $this->processMeterData($data);
    }

    private function fetchMeterData(\DateTime $startDate, \DateTime $endDate)
    {
        /* Fetch data */
    }

    private function processMeterData(array $data)
    {
        /* Process raw meter data */ 
    }

    private function retrieveEstimatedActivities(\DateTime $startDate, \DateTime $endDate)
    {
        $data = $this->fetchHouseholdEstimatedData($startDate, $endDate);
        return $this->processHouseholdEstimatedData($data);
    }

    private function fetchHouseholdEstimatedData(\DateTime $startDate, \DateTime $endDate) 
    {
        /* Fetch data */ 
    }

    private function processHouseholdEstimatedData(array $data)
    {
        /* Process raw estimated data */
    }

    private function retrieveMacroActivities(\DateTime $startDate, \DateTime $endDate)
    {
        $data = $this->fetchMacroEstimatedData($startDate, $endDate);
        return $this->processMacroEstimatedData($data);
    }

    private function fetchMacroEstimatedData(\DateTime $startDate, \DateTime $endDate) 
    {
        /* Fetch data */
    }

    private function processMacroEstimatedData(array $data)
    {
        /* Process raw macro estimated data */
    }
}

The outcome is a service that is hard to follow and maintain. It is also more complex to test the different ways of retrieving the activities.

The Chain of Responsibility approach

The Chain of Responsibility pattern consists of having multiple services (called handlers) and running through them in a determined order to handle a request. Each service decides in runtime to either handle the request or pass it to the next handler. The CoR pattern can be used for instance in multi-stage validation or in managing multiple data providers.

In our case, each data provider (meter reading, household consumption estimate, macro estimate) represents a "handler". A request is handled if the service returns an array of activities.

To promote loose coupling, the 3 data providers will implement an interface that exposes a retrieveActivities function.

namespace Component\Activity\Provider;

interface ChainedActivityProviderInterface {
    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate);
}

Each of the three services encapsulates the fetching and processing logic that match their respective data sources.

namespace Component\Activity\Provider;

class MeterReadingActivityProvider implements ChainedActivityProviderInterface {

    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate): array
    {
        $data = $this->fetchData($startDate, $endDate);
        return $this->processData($data);
    }

    private function fetchData(\DateTime $startDate, \DateTime $endDate)
    {
        /* Process Data */
    }
    
    private function processData(array $data)
    {
        /* Process Data */
    }
}

namespace Component\Activity\Provider;

class HouseholdEstimateActivityProvider implements ChainedActivityProviderInterface {
    // Same structure as MeterReadingActivityProvider
}

namespace Component\Activity\Provider;

class MacroEstimatedElectricityActivityProvider implements ChainedActivityProviderInterface {
    // Same structure as MeterReadingActivityProvider
}

Lastly, to link the services, we can use a “main” activity provider in which they are injected and ran through in the injected order. As soon as one of the handlers returns a valid response (a non-empty array in this case), we return its value. Note that the main activity provider handles the case where none of the chained providers returns a valid response.

namespace Component\Activity\Provider;

class ActivityProvider {
    /**
     * @var ChainedActivityProviderInterface[]
     */
    private array $chainedActivityProviders;

    /**
     * @param ChainedActivityProviderInterface[] $chainedActivityProviders
     */
    public function __construct(array $chainedActivityProviders)
    {
        $this->chainedActivityProviders = $chainedActivityProviders;
    }

    /**
     * @return Activity[]
     */
    public function retrieveActivities(\DateTime $startDate, \DateTime $endDate): array
    {
        foreach ($this->chainedActivityProviders as $chainedActivityProvider) {
            $activities = $chainedActivityProvider->retrieveActivities($startDate, $endDate);
            if (!empty($activities)) {
                return $activities;
            }
        }

        return [];
    }
}

Finally, we need to inject the chained activity providers. In Symfony, one neat way of doing it is using the serices configuration file. This makes reordering as well as adding and removing the services an easy task.

# config/services.yaml
services:
    # ...

Component\Activity\Provider\ActivityProvider:
    arguments:
        $chainedActivityProviders: [
            '@Component\Activity\Provider\MeterReadingActivityProvider',
            '@Component\Activity\Provider\HouseholdEstimateActivityProvider',
            '@Component\Activity\Provider\MacroEstimatedElectricityActivityProvider'
        ]

With this setup in place, we call the retrieveActivities function of Component\Activity\Provider\ActivityProvider to retrieve a user's activities.

The trade-offs of the CoR pattern

Implementing the Chain of Responsibility pattern offered many advantages:

Testing the main activity provider: We only need to test the behavior of the chain. Given a mocked chain of activity providers, the function should return the first valid result.
Implementing the open/closed principle: the principle states that the behavior of classes should be changed through extension and not modification. This comes into play when we add, remove, or reorder the activity providers. We can easily change the behavior of the main activity provider by changing the configuration file.

However, the usage of the CoR pattern comes with a few drawbacks, namely:

Being unaware that the CoR pattern is implemented makes the code harder to understand and debug
Depending on the used implementation, it is possible to have no handler to fall back to (if none of the defined handlers can handle the request). Always check for the fallback strategy.

Conclusion

In the long run, the Chain of Responsibility pattern helps organize and write modular code. However, it also comes with an initial cost higher than the trivial solution. They become interesting when the implemented system is prone to expansion. Indeed, setting up the chain is the most expensive part, while adding new chained elements keeps a constant cost over time.

The implementation using injections is one of many ways to utilize this pattern. For implementations in other languages, refer to refactoring-guru.

Generally, design patterns should be perceived as blueprints that offer ready-to-use solutions to common problems. Since the human brain is built to work with patterns, understanding these solutions makes the code base more intelligible and easier to work with. However, one should be aware of the layer of complexity they can add due to their initial cost.

How to externalize large libraries from your application bundle?

Wed, 27 Apr 2022 00:00:00 GMT

During the past months, I spent a lot of time trying to improve the frontend developer experience of about 15 developers on a large e-commerce website. The objectives included the reduction of our bundle size and CI run times. During this time, the development team introduced 3D assets, using the BabylonJS library to provide users with a more immersive experience regarding the articles displayed on the website. Unsurprisingly, this new feature increased our bundle size by 8 MB, representing more than 40% of our application size of the time. This in turn significantly increased our build times locally but also on the CI runners.

To mitigate these consequences, the approved solution was to serve the Babylonjs library from a dedicated CDN excluding it from the bundle for each build.

I am writing this article to share my learnings on how we managed to keep this dependency and more generally any large app dependency while reducing the consequences on the build size and time to a minimum.

TL;DR

Adding a large library increases bundle size and building times locally and on the CI/CD environments.
Many solutions as lazy solution exist to separate large dependencies for client, but it does not solves the issues for building processes.
To shorten building process, we externalized a large library to remove it from my application bundle.
To do so, the library has been externalized in a script, and dynamically loaded the library logic from it, without modifying any import from my application
Finally I measured the performance gain through a minimal example that you can find at the end of the article.

What are the impact(s) of adding large libraries as dependencies?

As a general rule, the bigger the package is, the greater the impact will be on a project's developer experience.

Download time

First, big dependencies impact installation times. When a dependency is added to a project, the designated packages and all its sub-dependencies are added to the node_modules directory. The time to compute dependencies, download the packages and sometimes build them will increase proportionally to the size of the new package. Yet, fast Internet access and cache have made these delays short, and exporting a whole package as an external dependency would not worth it if it was only to shorten these download times.

Build size

Another concern could be the bundle size increase related to the added dependency. However, modern bundling techniques such as lazy loading, code splitting and tree shaking already solve this issue by bundling only necessary code and splitting it into chunks. Moreover, moving packages into CDN would hide the problem because the user browser would still require to fetch the library. It is just not counted in the bundle served by the server.

Build time

Finally, the biggest issue with large dependencies is the build time increase. Building the application happens very frequently in the development workflow: building locally in watch mode when adding or modifying code or remotely with CI/CD. Globally, all the library code will be parsed and built into the bundle, so the larger the package is, the longer it will take to be included in the bundle. Externalizing the dependency on a CDN allows developers to retain the package functionality without having to include it at every build step.

For instance, on my project, the frontend was managed by Next.JS and we regularly analyzed our bundle size with @next/bundle-analyzer. Just after installing BabylonJS and developing the 3D feature, a new chunk appeared, weighting more than 8 MB, which represented around 40% of total bundle size while increasing our CI build times by 30 seconds.

How to do it!

There are three main steps to externalize a large dependency into a CDN:

Configure the application bundler to load a global variable as an import
Find or build a script to globally scope the library logics
Dynamic loading of component once the external script is loaded

Bundler configuration

The purpose of a bundler is to take all the dependencies of a project and to split them all in a little number of files containing the module logic in order to improve the loading speed of the built application. This article is only going to focus on Webpack because it is the most used currently, but the method used here can be used on other bundlers as well.

In order to bundle modules, Webpack does multiple operations:

It recursively resolves all module imports from the application entrypoint
It packs the associated code in a JavaScript asset and divides it into multiple chunks

To externalize a dependency, Webpack needs to know which dependency we are talking about so that it may resolve the dependency imports differently. One way of doing this is to map the webpack dependency imports to a global window.myDependency variable. This is the goal of the next part of the article.

Webpack has to be configured to not pack the dependency to externalize and to link all its uses to window.myDependency. A single configuration option allows to do that: externals (link to Webpack documentation). This parameter takes an array of string or function returning a string. During the building, when Webpack reads an import, it tests if the dependency location is included in the external array (or function callback is called). If the matching element is a string externalElement, Webpack will replace the dependency location by reading directly into the variable global.externalElement. If the matching element is a function, it will follow the same process with the returned value.

Here we want to replace the Babylon import @babylonjs/core by BABYLON due to the way they will be exposed in the CDN's script (explained in the next part). The Webpack configuration looks like the following:

module.exports = (config, { isServer = false } = {}) => {
  config.externals = config.externals || [];
  config.externals.push(({ request }, callback) => {
    if (request.match("^@babylonjs/core")) {
      return callback(null, "BABYLON");
    }
    callback();
  });

  return config;
};

Find or create the new library script and export it into a CDN

A dependency can not be externalized directly: the associated script has to have some modification. Indeed, the classic NPM or Yarn packages contain modules which are not strictly speaking scripts. The main purpose of modules is to expose logic (function, constants, etc.) that will be imported from other files. As explained in the previous part, the variable specified by the Webpack external option has to be accessible in the global scope. The dependency script must therefore be set globally when running in the browser.

Most of the time, dependencies already provide usable scripts from an already existing CDN. This was the case for BabylonJS for example which simplifies the process a lot. If your dependency does not already provide a CDN solution, you have to build the library yourself. The simplest way to do this is to follow this Webpack documentation. That allows to assign to window or global variables the elements exposed by the library. The result script should be the equivalent of one used with CDN.

Once the necessary script is build, it can be easily hosted on a CDN like UNPKG or JS Delivr. Some libraries made this task even easier by providing these scripts through their own CDN. BabylonJS was one of these. For the ones who built their own files you can host them on a private CDN.

Conditionnal loading for components using the external script

If you followed the previous steps, you should have a fetchable script exposing one variable containing all the library logic and a Webpack configuration bounding the imports of the library to the global variable. The last part is to inject the script in the code. One approach would be to inject the script globally in the application. This would create performance issues for parts of the app that do not require the dependency. The other solution is to load the script dynamically. In this case, the library logic is not available once the application is running which means the application needs to be notified before actually being able to use the dependency.

There are multiple ways to load a script conditionally and attach a callback once loaded. The simplest is by manipulating directly the DOM: adding a script with the src pointing to the script URL and onload calling the callback function. In our case, we were working with Next.js, and since v12, this wonderful library provides a <Script /> component (documentation link) doing all this work! I used it this method to display my 3D asset (contained in the <Asset3d /> component):

import { useState } from "react";
import Script from "next/script";
import dynamic from "next/dynamic";

const Asset3d = dynamic(() => import("./Asset3d"));

const Asset3dContainer = () => {
  const [isBabylonScriptLoaded, setIsBabylonScriptLoaded] = useState(false);

  return (
    <div className="asset-3d">
      <Script
        id="babylonjs-core"
        onLoad={() => setIsBabylonScriptLoaded(true)}
        src="https://unpkg.com/babylonjs@4.2.0/babylon.js"
      />
      {isBabylonScriptLoaded && <Asset3d />}
    </div>
  );
};

export default Asset3dContainer;

Multiple elements have to be taken in account:

In my component, BabylonJS requires the external script to be fetch to have all the logic loaded. I used a state updated through <Script />'s onLoad callback
Once the script is loaded, BabylonJS logic is available in window.BABYLON so <Asset3d /> can be dynamically loaded

After these 3 steps, my 3D asset was finally displayed on my page! A check shows that BabylonJS chunk disappeared from my bundle!

With a wrapper as the one I showed above for your library, you should be able to display your component with all the library logic integrated while the latter is not considered as a dependency! Congratulations!

Limits

Yet, not every dependency has to be moved into a CDN. There are three limits on externalizing a dependency from an application:

Even if a CDN package is build, it has no other optimization as tree-shaking.
Moreover, CDN are maybe fast, but they require an Internet access during development time.
If your project is using TypeScript, then every import from the removed library in not typed. You may either ignore these issues locally but reduce type coverage, or you reinstall this dependency as a devDependency. In the second case, this would still mean downloading the dependency, but it won't be included in the final bundle or in the build process (except with Next.js, but type checking can be disabled during the build if already done previously).

Thus, it is a cost-benefit trade-off between development time and use frequency of the package on one hand and building time on the other hand. In my case, the exported package was BabylonJS because 3D was only used on a few pages, but it also could be React-pdf if it is rarely used in the application, or any other large package not often used.

Conclusion

On my project, based on Github Actions data, I reduced the CI building time processes from around 30 seconds. To have better metrics, I created a minimal Next.js project which externalize BabylonJS. In the commit history, the same 3D asset is displayed, first with BabylonJS as dependency, then without it. You can look at it to see a functional example.

To test the performance of this change, I ran eight times a Next.js build command and timed it with the time command. Median building time decreases by 30 seconds, from 64.4 to 33.295 seconds! This time reduction is almost identical to the one without BabylonJS at all.

Another benefit of shorter CI runs are lower costs. According to the current Github action pricing (0.008€ per second) and around 1000 builds per month, a reduction of 30 seconds on each build saves 240€ per month on the Github action bill.

To conclude, externalizing a dependency is a complex process to setup, but it reduce all building process, by just removing the time used to build the dependency. It is worthless for small libraries, but useful for large ones, even more if they are used only on a little pages of your web application. I hope you enjoyed this article and use it to speed your CI/CD!

Working with a Third-party Provider, Three Lessons I Learned to Reduce the Lead Time

Fri, 08 Apr 2022 00:00:00 GMT

For two years, I worked as a web developer with an investment bank. One of their key aims is to digitalize their finance system in order to speed up the procedure.

At this moment the company was developing a more stable third-party API to provide the core data and which could manage the required performance for all of its consumers.

Unfortunately, the new third-party service ran into a slew of issues. One of them was stability, and it's one of the most annoying, especially since the entire app depends on it.

The more reliant you are on a third-party service, the less control you have over your app. This suggests an increase in lead time when developing a feature or fixing an issue that is dependent on a third party.

This page is about what the team tried and what we were successful at. But also where we failed to save time with our provider in order to deliver a defined quality to the consumer.

From how we started :

From the start of the project, we had a third-party owner in our team. His objective was to prepare :

the migration of our core data,
the connection to the third-party provider APIs,
and ensure that everything worked flawlessly once completed.

However, he left the project for a new adventure before we obtain the agreement for the migration. I volunteered to replace him. Everything was pretty well-prepared and documented, and I only needed to follow the process.

But, the first weeks we plugged into it, we checked almost all the problems that you can ever have with a provider :

✔️ Availability

✔️ Data integrity

✔️ Breaking change in business rules

✔️ Endpoint performance

Every day, we accumulated more and more problems, and we didn’t know when we will see the end. The time spent asking for investigation or fix increased drastically.

From these issues, I learned how to shorten the time to make a request done and make issues disappear.

My First Lesson: Become aware of your problems before treating them

With experience, we reduce the investigation duration. However, switching context caused me to be less efficient in delivering the features.

So I approached my IT lead and asked him, "What can I do?" His first answer was, “Do you realize how much time you spent on the third party problems?”

Based on this question, I set a board where I record the request frequency, the time spent, the author, and the topic. The result was, in 20 days, I spent 255 min to treats 22 requests.

This board also sheds light on two typical causes :

Poor communication
Lack of autonomy

What I learned here is: that referencing all the out-of-context time, gives you a much clearer idea of how much time you spent on it, thinking about the issues, and will push to set up some actions.

So let’s tackle the causes now!

Second lesson: Making a request needs communication and negotiation

Knowing how to communicate the needs to the provider is a thing I learned and still learning in this project.

Making an efficient request is not only about how clear you ask it. It's also about the information you provide to your interlocutor to convince them to prioritize your request. For that the information should be relevant to help him evaluate the work needed and to commit faster.

Identifying the interlocutor

To save time, asking for a request from the right person avoid doing some back and forth to relay information.

So, identifying the positions of the provider team is the first thing I did.

My next action was to define with the provider an owner for each case/field. Since what I learned was, that putting ownership in only one person will «empower» and lead him to put more effort to solve your request.

An example was, I sent a request to the whole provider team to investigate why we couldn’t create any company. The answer I obtained was, “it’s not us it’s another team”.

I then asked a developer who treats most of the time with them, the owner of the authentication process.

The time to find the right owner took us 2 hours because he was not a part of the provider team. The fix, once we found the owner, took 10 minutes to update the API key.

Know each other :

It happened pretty often that when it comes to explaining problems after investigations from our side or the provider side, it’s hard to understand because :

we don’t know the architecture of each other systems,
each team uses their abbreviations,
we work differently.

A good idea from one of our lead tech was to clarify all these points:

Share the architecture of the application,
Ask each time the definition of an abbreviation that we don’t understand, but also we use the less abbreviation possible, and only the ones that everyone knows.
Ask how they do something if it takes longer than we expect.

Knowing their architecture, and how they work will make us understand the problematics, help us communicate more efficiently, and finally unlock situations faster.

The importance of building a clear request :

What we learnt then was: it’s crucial to prepare your request.

This mean that the request has :

a clear why
accessible materials
a measurable goal

With a clear request, you will make your provider understand better your needs, you’ll have less work to do, and it will be harder for them to refuse it facing facts.

A clear request is also a way to respect your provider. Advice from our sensei to make an even better request is to formally ask your provider to refuse the request if it’s not perfect.

This is the best way to learn how to create the perfect interface for your specific relationship.

It is essential that to respect the provider the same way you respect your customers.

The importance of the explaining the impact

What I often did before was in a hurry, ask them to investigate a problem, but without explaining to them the impact

What could happen is that they don’t answer. They are busy and let you request again, to be sure that it’s important, or they would take their time.

Here is a typical example I could do :

– We have a problem, the private individual creation endpoint is on timeout, could you investigate on it?

In this example, the provider doesn’t know the emergency level and will put the minimum effort to answer this.

After several failures, my lead tech told me to learn the standard of the company to do a good request.

The previous request became then :

– We have an emergency problem right now, 10000 customers cannot create their account on the production environment, because the private individual creation endpoint doesn’t work since 14h. Could you investigate, please?

Here you have a clear impact. It helps your provider to understand the emergency of your request and to prioritize it.

Make it faster : Bring the material

To reduce the time to convince our provider to help us, or to achieve a fix, I needed to prepare my talk and material. This help the provider to localize and take decision faster.

There are plenty of tools that help us to gather all important information to prepare our requests to the provider. Here are some that we use :

Rest API client to test your provider APIs (Postman / RestClient):
- What we do if their endpoint doesn’t work is to send them a screenshot of our Postman call, with the environment, URL, headers, body, response code, and response body visible. Your provider could extract more data from your screenshot than you imagine and can guide you.
Monitoring tools (Kibana, Datadog, ...):
- These tools allow us to visualize and analyze our application logs through different aspects like histograms, charts, ... We will see how essential it is below.
Sharing workspace for sharing documentation or joint action (Notion:
- What we do with this tool is a sharing board where we track the current or future problems that could impact our provider or us. We put significant information that helps us to track the progress and to define the priority to treat them.
Draw schema (Excalidraw, whimsical) :
- Explain problems, architecture or workflow with a drawing allow a better understanding. It makes easier to localize the problem, and to evaluate the time to fix it. Good advice that I learned was to draw with your customer/provider during a meeting help to synchronise your understanding of a problem.
Alerting tools (Sentry + Slack):
- To save time, and not be constantly checking our monitoring tools, we set a notification alert for each time something goes wrong from our provider server. This will help us to react faster, and to mention the problem to our provider. With this process, it happened that we detected the problem earlier than our provider.

Establish good quality criteria/KPI

One day, the provider, announce to us that they are preparing for critical data migration.

The risks if it is not done correctly can have impacts on :

data integrity ⇒ if you provide the wrong data, you lose the trust of the customer
performance of their SQL queries/ endpoint response time ⇒ a slow app (3 second to answer) can decrease 53% of your customer
and the availability ⇒ If you have international customers, your apps need to be available 7/7 and 24/24, and you must inform your customer when there is any maintenance

At this time, I was not confident to ask our provider for a guarantee that the process will work without any negative impacts.

So I asked for help from our “sensei”. He made me realize that I was not confident because we didn’t have any good criteria to which the third party is committed.

But what are good criteria?

Good criteria should be:

clear to make it easy to remember to constantly have it in mind
measurable to determine if it is reached, or if you will reach it soon
accessible with checkpoints to be able to check it at any time
realistic to be reachable

To define our criteria, our first step was to collect all problems that we encountered and sort them to have a map of the actual situation.

From this information, we could define a clear plan on what to measure.

Monitoring, monitoring, always monitoring ... and master your tools

In the beginning, we were monitoring the liveness of our provider endpoint by coding a scheduler that calls all the endpoints every 5 min and registers the endpoint state in our database. That was not a great idea.

The result was poorly shareable for everyone, and we needed to layout the data to make it understandable, which took way too much time. We even deactivated it, because it created too many logs.

Another bad idea was to monitor all the routes, by calling them several times to make an aggregate of the response time.

We got the results, but it was manual and painful to do it frequently.

We then asked our sensei for help again, and he made us realize, that we already have all the tools that we needed, but we didn’t know how to master them.

To avoid all these pains and to save some time, we spent 2 hours studying our monitoring tools.

We were then able to automatize the monitoring by defining the duration of a single event in our code (a span), and using our monitoring tools (Kibana) to display these events in a form of a dashboard.

After getting the performance monitoring data in an accessible form, we came to our sponsor with some checkable numbers to tell them that we were not confident about the migration if they could guarantee us that 95% of their endpoints response time are below 2sec.

This made them aware that they were not ready with some facts.

The impact was big. It results in our provider's PM coming to us to make an official communication that they had to delay the data migration, this in front of our sponsor.

Use monitoring to measure the KPI and to negotiate with your provider

Here is another example of how monitoring helps us to convince our provider, with minimum time.

Every time we encountered a data integrity problem we asked our provider to modify the data, but nothing will stop that if it happens again.

We needed to find the root cause, and some long-term solution to prevent it from happening again.

How? By gathering the material, we visually showed them:

how often does the problem occur
how much time both of us spent
how much time we could save.

With this information, we reconciled our interests (not having bugs anymore) with their interests (not having to waste time on hotfix frequently) and proposed a “win-win” situation.

We then convinced them to work together to find the root cause and fix it by adding a business rule in their code.

Using visual management helped us to have much more impact to negotiate with our provider to investigate the root cause instead of fixing the problem temporarily. We then never heard about the problem again until now.

Now commit!!

We saw how to do a great request, but still, something is missing.

The last point is to make your provider commit to it. A good commitment, has a confirmation message, with a check date and an identified owner.

So you know when to check and know who to talk with when the request is still not resolved.

Go further

Go drink a beer ... or a tea with the provider team :

A thing that we didn’t achieve was to invite the provider team to a bar.

This is not a joke. We could figure it out that they are probably not so different from you.

This could also make us aware of their working environment, and how they work, and so better interact together.

Third lesson : Share your tools and methodologies

Another important lesson, that I learned during these years was how to avoid overwhelm when I was in front of tons of requests from others teams or our providers.

What I learned was to delegate the request but not anyhow.

I aimed to make everyone a copy of myself when it comes to the method and knowledge of my topic.

Teaching to make everyone autonomous will release your stress, gain time, and last but not least, put your footprint on the methodology. So even if you leave, people will work as you do.

In this paragraph, I will tell you how I made things work as I was present, and not to be overwhelmed with my problems.

Make your team autonomous

Help your team to identify the problem

5 different teams, depending on us to get the provider data.

In the beginning, we got contacted multiple times a day for data reading or writing problems.

However, the provider data could be recomputed with many business rules set by others teams through different layers.

To overcome this situation, we helped everyone to identify the origin of the problem. For that, we set a status error (424) with a message containing the problem, when the problem comes from our provider.

Since that, the contact frequency reduced drastically and the teams became much more compliant.

Help your team pay attention to the provider problem

Previously I showed you that alerting tools can be great to gather material and be notified if something goes wrong.

But there is also a drawback that we experienced by not mastering it.

What we didn’t do well, was logging our error. The alerting tools caught all the error logs, which some of them were not, and sent too many false positive notifications.

The consequence was that we became immune from it, and we didn’t pay attention to the notifications.

So it is important to set up well your alerting tools and your log to make everyone use it.

Monitor your requests

In my first lesson, I learned to monitor all the requests, and the result was in 20 days I spent 255 min to threatening 22 requests.

So what did I do then?

I sorted all the requests and realized that most of them are easy to solve.

What they required is to know in which endpoint they could find the specific information they needed.

The problem was no one was familiar with these endpoints, and there is no easy way to get the information.

To solve this, I made a Q and A, postman collection, and documentation relative to the most frequent requests, to make everyone autonomous.

The 255 minutes dropped to 10, with 2 requests. Finally, I had some time to drink a coffee!

So teaching the team is time-consuming, but it’s a great investment in the long term.

If you work 6 months on the project, I let you count how much time you gain, and what you can do with that?

Make your provider autonomous in front of problems

A typical problem that we encountered was the increasing response time each time they did a deployment, for different reasons.

This leads us to always check the response time of all of their endpoints, after each deployment. It took us 30 min each time.

To avoid this, we shared our monitoring response time tools with them with the endpoints that we used the most.

It cost us 30 min to create an access login and to teach them how to use it.

With that, we are pretty confident, that they will be more aware of the impact of their deployments and fix it if necessary.

And this would make them also more reactive since they are committed to the response time that they need to deliver.

The result is that our provider, check frequently the dashboard, and contact us by themselves to tell us that they improved the response times of some endpoints which were already less than we expected.

Don’t miss any requests : track it with a co-board:

When it comes to new version deployment or data migration, to follow critical request resolution, we created an exclusive channel during important events.

But it would be counterproductive to create many channels for smaller requests, like data/logs analysis.

What we tried was to implement a light Kanban board (we used Notion for that) to track progress, and don’t forget any of them. This board could be filled by the problem owner team.

It is also useful to track the weekly/monthly meeting with the provider if you couldn’t attempt it.

To be honest, our first try doesn’t work. Our provider told us that they will have a hundred consumers and couldn’t maintain a board like that, if they have to do it for every one of them.

What we are planning to do is to maintain it at the beginning by ourselves, to track the requests that we needed to ask again, and the time spent. This will give us some material to determine if we should convince harder our provider to use it.

Invest into long term by teaching

There are two ways to put your footprint on a project:

you work a lot, it’s worked, hurra everyone will congrats you 5 min. However, you lost your hair and the problem should be fixed well, because the day you disappear, I let you imagine the rest of the story.
you are pretty smart (or have a good sensei), you teach everyone little by little a routine about how you do to deal with the provider. The day you disappear, everything is still working as you were here. You succeed to put your footprint on this ownership. People will work as you teach them. So you can enjoy your vacation.

Conclusion

What I can retain from all of these experiences was, that there is always a way to make things done by becoming proactive.

It’s not easy, and it’s time-consuming. But, you should not think about losing time, but investing time for the long term.

To sum up what I've learnt over this project:

Communicate clearly your request to the right interlocutor with a why, materials to only post it once with respect, and expect them a commitment.
Monitoring the most sensible “points” and making them accessible, would make you gain a bunch of time to prepare your request, and convince your provider to prioritize it.
Before looking for new tools or coding some new ones for monitoring, look at what you already have, and master it, it can be better than what you expect.
Make everyone autonomous to decentralize your task. This avoids you from switching contexts, and put your footprint on the methodology. The project will still work even if you disappear ... at the coffee shop.

... And last but not least don’t forget the drink with your provider, it’s important to drink!

Serverless, DynamoDB, Streams, Filter Patterns and You

Wed, 06 Apr 2022 00:00:00 GMT

This article will cover filter patterns in the serverless framework as they apply to DynamoDB streams, but the principles apply to filter patterns on other event types as well (kinesis, event bridge, etc...)

DynamoDB Streams provide near real time information on all data modifications that occur in your database at no extra cost (No paying for event messages or any in between computations). We can then use these streams to trigger Lambdas without having to setup any extra infrastructure! Read more about DynamoDB Streams here.

This sounds great but quickly the problem becomes that you have a Lambda triggered on every single data modification for your DynamoDB table wasting the saved costs by spinning up a Lambda whose only job is to recognize it has no job to do and then stop processing. The answer to this problem is to set up filter patterns. Filter patterns allow you to execute a listener to an event only under certain circumstances removing the data validation part of our Lambda and rely on our filter pattern to only invoke the Lambda when necessary. Plus stream filter patterns come with no extra costs making them the perfect solution.

Before we dive in

Filter patterns work on any event structure but our examples are specific to DynamoDB streams which have an event structure like the following:

{
  "eventName": "INSERT" | "MODIFY" | "REMOVE",
  "dynamodb": {
    "Keys": {...},
    "NewImage": {...},
    "OldImage": {...}
  }
}

The expamples below use the serverless-lift plugin to create our database instances. This is why the stream arn's look so nice they are a direct reference to a database created in the serverless file. Read more about serverless lift and DynamoDB here!

How to implement filter patterns in your project

Lets say you have a series of jobs making updates to a single record and you have a lambda that is going to send an email to your users when the jobs finish processing. To denote when a job is finished there is a jobState column on your record. You could have a serverless lambda function setup like this:

emailUsersWhenJobComplete:
  handler: src/emailUsersWhenJobComplete.handler
  events:
    - stream:
      type: dynamodb
      arn: ${construct:batchJobs.tableStreamArn}
      filterPatterns:
        - eventName: [INSERT, MODIFY]
          dynamodb:
            NewImage:
              jobState:
                S: [DONE]

Let's break the filter pattern down.

First the filter pattern says to only pay attention to INSERT or MODIFY events meaning any remove events are automatically ignored. The next few lines of the filter pattern drill down on the event structure to get to the attribute that we need. Then it says that we expect the value of jobState to be DONE. Meaning any inserts or updates to our table records that happen before the job is done are ignored by our pattern and our lambda doesn't even spin up.

This is good, but what about updates that happen to our record after it hits the done state. Those will trigger our lambda to run as well causing us to send emails we don't want to send. So we only want it to process when the jobState wasn't DONE before the modification. We can accomplish that with filter patterns as well. Our serverless lambda function will now look like this:

emailUsersWhenJobComplete:
  handler: src/emailUsersWhenJobComplete.handler
  events:
    - stream:
      type: dynamodb
      arn: ${construct:batchJobs.tableStreamArn}
      filterPatterns:
        - eventName: [INSERT]
          dynamodb:
            NewImage:
              jobState:
                S: [DONE]
        - eventName: [MODIFY]
          dynamodb:
            NewImage:
              jobState:
                S: [DONE]
            OldImage:
              jobState:
                S:
                  - anything-but: [DONE]

This updates the filter pattern we had before and splits out the insert and the modify into seperate patterns. AWS applies filter patterns in a way that ensures the lambda will be triggered if the stream event matches any of the provided patterns.

The MODIFY event has also been updated to reference OldImage. This is the state the database record was in before the modify event was emitted. By doing this, we have now specified that if an update happens and our record ends up with a jobState of DONE we still only want to trigger our lambda and send an email if the jobState before this update was not DONE, solving our earlier problem. Users will now only get notified when the job finishes and not if any updates are made after the job finishes.

What else can filter patterns do? Quite a lot. Lets say we have a shopping platform and we want to send our users a coupon if they make an order over $100.00. We could have a serverless lambda setup like this:

sendCoupon:
  handler: src/sendCoupon.handler
  events:
    - stream:
      type: dynamodb
      arn: ${construct:orders.tableStreamArn}
      filterPatterns:
        - eventName: [INSERT]
          dynamodb:
            NewImage:
              total_amount_cents:
                N:
                  - numeric: [ >, 10000]

Or if we need to run extra structural checks when a user adds a pipe with a radius greater than 5 cm, a length greater than 10 m, and any PVC based material type in a 3d modeling system:

structureCheck:
  handler: src/structureCheck.handler
    events:
      - stream:
        type: dynamodb
        arn: ${construct:structures.tableStreamArn}
        filterPatterns:
          - eventName: [INSERT]
            dynamodb:
              NewImage:
                type:
                  S: [piping]
                radius_cm:
                  N:
                    - numeric: [>, 5]
                length_m:
                  N:
                    - numeric: [>, 10]
                material:
                  S:
                    - prefix: [PVC]

As you can see pairing DynamoDB streams with serverless filter patterns lets you perform in-depth event logic without having to spin up any extra infrastructure or invoke any extra functions. Next time you need to publish an event after a database write in your serverless project why not try out a filter pattern. You just might save yourself some time and money.

Links

Images from unsplash

Information on DynamoDB stream event structures and testing events

Full list of filter pattern rules and raw json filter pattern examples

Information on the Serverless lift plugin

Understanding Codat Webhooks

Fri, 01 Apr 2022 00:00:00 GMT

What is a webhook?

As developers, we often find ourselves wanting to see exactly when an operation in an external API finishes or when a particular event happens within an external service. It would make sense to poll that API regularly to see if something has changed, checking in on all our operations until they’re complete. This polling can be taxing and really annoying to manage–which is the problem webhooks solve.

Let’s look at one simple example of using a webhook: Github and Slack. When we create a new pull request in Github, if you’ve integrated Slack, you’ll automatically get sent a Slack message if you’re assigned as a reviewer on that pull request. This is a perfect example of utilizing a webhook–instead of constantly polling Github to look for new pull requests, Slack simply listens for the Github webhook to call its API, then parses that request to send messages to the reviewers–no polling required.

Webhooks let us specify endpoints for our application that an external service will call when a particular event occurs. Whenever we are using a webhook, we need to specify to the external provider what type of event to notify us of and what API endpoint in our application we want this webhook to call. Once configured , we’ll get a request from the external service whenever that event occurs.

Why does Codat use webhooks?

Codat interfaces with a number of different accounting, banking, and eCommerce platforms. Each of these platforms have different rate limiting restrictions, and all these services have the potential to go down at any time. To deal with this, Codat’s API works asynchronously, so requests to Codat are acknowledged and given an operation key that can be used to get the status of that API call. So, when we start an operation in Codat, we don’t know immediately when that operation will complete. It might be a few seconds until the operation actually finishes–this is where we benefit from having a webhook to notify us that it’s finished.

One of Codat’s most important webhooks is the “Push Operation Status Changed” event. When we get a call of this type, it tells us that the operation we started to save data to Codat has completed. (Or, if it encountered a problem, it will give us a detailed description of what went wrong.) The webhook is what allows Codat to asynchronously carry out our requests without making us poll over and over to see the status of those requests.

What does this look like in practice? Let’s say our application has a table called “Expenses” and we want to sync this data as a directCost object in Codat. Additionally, we probably want to know when an expense has been successfully pushed. Using a webhook, our flow would be very simple: Create the directCost object based on a row in our Expenses table. Mark that expense as having a status of “Push in progress” and save the push operation key. When the push operation is complete, we’ll get a Codat webhook callback, and we’ll find our expense based on the push operation key we saved and update based on the push operation status: If it succeeded, we’ll mark our expense with “Push Complete”. If it failed, we’ll mark it with “Push Failed”. And that’s it–since we’re using a webhook, we never need to do any polling to get the status of our expense as it’s syncing.

Securing Codat Webhooks

If you’re using authentication for your backend, you’ll probably have to set up an exception for the Codat webhook endpoint. Codat offers two forms of authentication–basic auth with a set username and password and bearer authentication with a static token. Either way, you will have to check against these static credentials in your backend to make sure the webhook endpoint is available to Codat but not to the outside world.

Multiple push operations

Codat’s data types rely a lot on each other, so it’s not uncommon to be in a situation where you need to manage multiple sequential push operations.

For example, let’s say you again wanted to push a directCost, but as part of that directCost, you wanted a reference to a new Account object that did not exist yet in Codat. To do so, you would have to go through an exchange like this:

Managing the dependencies between different objects that you want to push to Codat will require special backend logic.

Managing connection status

The first step after creating a company in Codat is commonly to add a connection to an accounting platform–and webhooks can notify us when that connection is established.

The “Company data connection status changed” status is what notifies us when we successfully connect to an accounting platform like Quickbooks or Xero, and gives us the ID for that new connection. Many of the calls we will want to make to Codat will take in this connection ID in addition to the company ID, so storing this in the backend is a great idea–and getting it in a webhook makes that easy.

This can also be helpful when we disconnect an application.If the user disconnects Codat from their accounting platform, getting notified by the webhook will be crucial, otherwise our application won’t know that the connection has been invalidated and we’ll get errors when we try to push data.

It’s important to note that the Codat connection status changed webhook will not trigger automatically when Codat is disconnected on the accounting platform side. Only when Codat next attempts a sync will it realize the connection is severed and update the connection status for that platform.

Summary

It is crucial to understand webhooks when building applications with Codat. If you’re planning on using Codat to integrate with accounting, eCommerce, or banking platforms, you should be sure to include webhooks in your architecture’s design.

How to beautify java code reliably

Wed, 09 Mar 2022 00:00:00 GMT

I recently had to set up code formatting on a spring boot java application. Auto code formatting is important to avoiding useless diffs in source files, reducing noise in code review, allowing reviewers to focus on what matters.

Ideally we want:

Automatic formatting on file save in IDE's (vscode, IntelliJ)
Capacity to run in CI

I wanted to integrate well with vscode and after trying some solution like:

java autoformat, which I couldn't run in CI and that was not really portable between IDEs
maven auto formatting plugin (didn't run on save easily in IDEs)

I ended up, from a friend suggestion, looking into a java prettier plugin. Prettier allowed for an easy integration with most Web IDE, can be configured to format on save for vscode and is trivial to run in CI.

Installing prettier for java

Let's start with the setup in command line:

Install nodejs: (I highly recommend using a node version manager like fnm) and to install a recent node version (current long term support is 16+)
Add a package.json file at the root of your project and add prettier and the java plugin:

{
  "name": "api",
  "version": "0.0.1",
  "private": true,
  "devDependencies": {
    "prettier": "2.5.1",
    "prettier-plugin-java": "1.6.1"
  },
  "engines": {
    "node": "16.x"
  },
  "scripts": {
    "format": "prettier --write \"{,**/}*.{java,yml,yaml,md,json}\"",
    "check-format": "prettier --check \"{,**/}*.{java,yml,yaml,md,json}\"",
    "lint": "npm run check-format"
  }
}

Install the required node_module: npm install

💡 Don't forget to add node_modules to your .gitignore.

💡 I highly recommend adding a .prettierignore that list autogenerated files to avoid formatting those, eg:

node_modules
target

You can now format java code running npm run format 🚀

💡 Do not forget to add node as a dependency for your project installation and to run npm ci in your project installation script for new developers.

We now want to improve developer experience by integrating with IDEs, running the formatter in CLI being impractical.

Monorepo

If you are using a monorepo, I highly recommend setting up prettier in the root of your monorepo instead of setting up prettier in each sub directory.

Integrating with IDEs

vscode

To set up prettier in vscode: add the prettier for java extension to recommended vscode extensions for the project, to do so, you can simply add the id of the extension to: .vscode/extensions.json:

{
  "recommendations": [
	"pivotal.vscode-boot-dev-pack",
	"vscjava.vscode-java-pack",
	"redhat.vscode-xml",
	"sonarsource.sonarlint-vscode",
	"gabrielbb.vscode-lombok",
	"dbaeumer.vscode-eslint",
+	"esbenp.prettier-vscode", // the classic prettier extension
	"shengchen.vscode-checkstyle",
  ]
}

then set up this extension as the default linter for java file in .vscode/settings.json :

{
// rest of your configuration
// automatic format on save, I recommend the save on focus lost option as well
+	"editor.formatOnSave": true,
+	"[java]": {
+		"editor.defaultFormatter": "esbenp.prettier-vscode"
+	}
}

Already we can easily format java code on save:

I highly recommend checking in the .vscode folder in your versioning tool (most likely git) to share this configuration with all developers.

IntelliJ

To integrate with prettier java formatter:

First install the File Watchers IntelliJ plugin
Then configure the plugin Tools/File Watchers:

Name: Prettier-java
File type: Java
Program:  full path to `.bin/prettier` eg: `node_modules/.bin/prettier`
Arguments: --write $FilePathRelativeToProjectRoot$
Output paths to refresh: $FilePathRelativeToProjectRoot
Auto-save edited files to trigger the watcher: check it
Trigger the watcher on external changes: check it

prettier documentation can be found here for IntelliJ integration

Integrate code formatting check in CI

Now we only need to integrate this check in CI, I'm using gitlab for this example, but the principles can be transposed to any other CI.

First we need to add a new job:

stages:
  - build
  - test
  - deploy

# ... rest of the file

# beautify backend files with prettier
lint-back:
  image: node:lts-alpine # We use a node image to run prettier
  stage: test # must match an existing stage
  needs: [] # none required optimization that explain to gitlab ci that this test has no dependency and can start as soon as possible
  script:
    - npm ci
    - npm run lint
# ... rest of the file

now just push your change, and you are all set:

Choosing the best storage solution in GCP

Wed, 09 Mar 2022 00:00:00 GMT

Are you new to GCP, looking to make sense of all the storage offerings GCP provides or simply hoping to learn something new? You're in the right place!

In this blog we're going to compare the key differences between GCP's numerous storage solutions. We will start by taking a look at a handy decision tree which gives you an overview of the various options. We will then categorise the available storage solutions into blob, relational, NoSQL, file, block and other storage. For each option, we will delve a bit deeper into the specifics and we will highlight the best suited use cases as well as flag any potential downsides. Let's get started!

Decision tree

As promised, we're starting with a decision tree to help you choose the best storage tool for your use case. Don't worry if you're not sure what all this means - we'll take a look at each tool below.

Now let's look at each of the options listed above in a bit more detail.

Blob storage

Blob storage allows you to store unstructured object data. This means that we're looking at data which often consists of items (e.g. media files) which don't need to sit within a file structure.

Cloud Storage

We're starting with one of the most commonly used storage options - Cloud Storage. Cloud Storage is GCP's blob storage solution and it is often used alongside other tools such as Compute Engine (service for creating and running virtual machines), App Engine (platform for building highly scalable applications using serverless) and BigQuery (data warehousing solution - more info below). In terms of its structure, it uses buckets to group items.

Benefits

Buckets are really powerful organisation tools, which allow you to control lots of properties, including lifecycle policies (e.g. delete object 2 years after creation) and object versioning (by default, object versioning is switched off, so if updated, the object will be overwritten).
You can choose to either host your bucket in a single region, or it can be dual-regional or multi-regional. The dual-regional option in particular isn't seen across many storage options. It can be a great way of saving on costs (in comparison to multi-regional buckets) whilst bringing your data closer to your users and making sure it is available even in the event of a regional outage.
You can choose a suitable storage class based on how often you need to access your data. The four storage classes available are standard (access regularly), nearline (access once a month), coldline (access once a year) and archive (access less often than once a year), with the cost of storage decreasing with each subsequent class.

Downsides

Cloud Storage cannot mimic file structure of nested folders. Each item is accessed directly via its own url.
Coldline and archive classes carry an associated retrieval cost, which will be charged when you access your data. This means that the costs can sneak up on you, especially if you’re not keeping track of how often you’re accessing objects in your long-term storage buckets.

In terms of common use cases, Cloud Storage is an excellent option for archiving data for audits and for storing data for disaster recovery. It is also great for minimising latency for users who download large data objects since you can choose dual-regional or multi-regional storage to bring your data closer to your users.

Relational storage

Unlike with blob storage, relational databases organise data into tables which can be linked to each other. They are the industry standard and GCP provides a couple of options for storing relational data within the cloud.

Cloud SQL

Cloud SQL is the obvious choice when setting up a relational database in GCP. It provides a fully managed service which now allows you to create instances of MySQL, PostgreSQL as well as Microsoft SQL Server.

Benefits

Easy to migrate to from on-premise relational databases, either using MySQL dump or CSV file imports.
Variety of methods for connecting to your database, including access via private IP and Cloud SQL proxy.
Automatic data replication with failover.
Automated storage capacity management.

Downsides

Storage capacity is limited to 30 TB, which may catch some users out.
Cloud SQL is great for vertical scaling, but if you're interested in horizontal scalability, you'd best look elsewhere.

Cloud SQL is an excellent option if you're looking to replicate your on-premise database within the cloud, or if you'd simply like to stick with a database you know already. This is exactly what happened with AutoTrader, a digital automotive marketplace. They were looking to move their on-premise infrastructure to the cloud, and they chose to move their existing Oracle database into Cloud SQL.

Cloud Spanner

GCP’s alternative to Cloud SQL is Cloud Spanner. Cloud Spanner provides a relational database structure with the added advantage of horizontal scaling.

Benefits

If you're concerned that Cloud SQL doesn't provide enough capacity for your needs, Cloud Spanner is definitely worth a consideration since it scales to PB capacity.
Support for Google Standard SQL as well as PostgreSQL.
It provides strong consistency, meaning that once a user makes a change to the data, the update will be consistent for all users immediately.
It automatically replicates data across multiple zones.

Downsides

Whilst you can create backups on demand, there is no in-built option to generate them automatically.
Cloud Spanner isn't MySQL compliant.

Since Cloud Spanner offers strong consistency globally, it is well suited to financial applications. A fun real-life example is Pokemon Go, which uses Cloud Spanner due to the aforementioned feature.

NoSQL storage

By comparison to relational storage, NoSQL databases store data in non-tabular formats. This can include document storage, key-value pairs or wide-column databases. GCP offers two main NoSQL databases, which we will explore below.

Cloud Bigtable

Cloud Bigtable consists of sparsely populated columns, where rows are indexed by a single key. It is possible to group columns into column families to capture which columns are related. Each row-column intersection can contain multiple cells at different timestamps, and this allows Cloud Bigtable to record a history of data updates.

Benefits

It can comfortably handle PB capacity.
It offers very low latency, consistently hitting targets of sub-10ms.
Easy integration with big data tools, including Hadoop and BigQuery.
Clusters can be resized with no downtime.

Downsides

The smallest possible cluster consists of 3 nodes and provides 30,000 operations per second.
Cloud Bigtable doesn't scale to zero, so you pay for nodes regardless of whether they are being used.

Cloud Bigtable works well for operational and analytical use cases, and is often used with IoT applications. A real-life example is Dow Jones, a news content and business information provider, who uses Cloud Bigtable to store their data before processing it for analytics.

Firestore (previously Datastore)

Previously known as Datastore, Firestore is the new generation of this NoSQL document database. Unlike Cloud Bigtable, it uses JSON to store data.

Benefits

Integration with Firebase - GCP's app development offering.
Firestore offers strong consistency. This is a typical feature of relational databases.
Multi-regional replication is available.
It provides ACID transactions, allowing you to be confident that there are no transactional discrepancies within your data.

Downsides

It only scales up to TB capacity.

Firestore is an excellent choice for mobile and web applications at global scale. Thanks to its integration with Firebase, setting up Firestore can be a seamless experience. An example of Firestore being used in the real life is The New York Times, who use Firestore to support their real-time rich text editing tool.

File and block storage

We are now moving on from nonSQL storage to file storage, by which we mean storage which allows us to replicate the commonly used file structure of nested folders. The best way of thinking about it is that it mimics the file system of a computer. Block storage, by comparison, stores data as separate pieces, where each piece can be uniquely identified. Block storage is very performant since it allows for data to be accessed via multiple paths.

Filestore

Filestore (not to be confused with Firestore) is GCP's fully managed network attached storage (NAS) option. It is often used alongside instances of Compute Engine and Kubernetes Engine. Filestore also offers four tiers (basic HDD, basic SSD, high scale SSD and enterprise) to allow you to pick the best solution for your needs.

Benefits

Filestore offers high performance - throughput of 25 GB/s and 920,000 operations per second.
It scales up to 100 TB capacity.
Capacity automatically scales up and down to minimise cost.

Downsides

It is rarely used without an associated Compute Engine or Kubernetes Engine instance.

Filestore is often used for file sharing as well as high performance computing, e.g. genomics processing.

Persistent disk

We are now delving into the world of storage solutions for Compute Engine. GCP provides a variety of options within the persistent disk offering. As expected, at the the highest level of categorisation, you can choose between SSD and HDD.

Benefits

Automatic encryption using GCP-supplied keys or your own.
Durable storage - there is very low risk of loss of data, with the risk lower for regional persistent disks.
There is no charge for operations.
The disk types include standard, balanced, extreme or SSD, so you have a lot of freedom to choose the best suited alternative.

Downsides

Its usage is primarily restricted to Compute Engine.
A persistent disk can also be used with Kubernetes Engine as storage for persistent volumes.
Persistent disks can be zonal or regional, so they aren't well suited for global applications.

The most common use case for persistent disks is, unsurprisingly, attaching them to Compute Engine instances.

Local ephemeral disk

The final file storage type we'll cover is local ephemeral disk. The main difference between local ephemeral disks and persistent disks hides in the name - persistent disks retain data even if they are not attached to a virtual machine and the data will be kept until the disk is deleted. By comparison, the data stored on local disks is lost as soon as the virtual machine it is attached to is stopped or deleted.

Benefits

Local disks provide higher throughput and lower latency than persistent disks since they are physically attached to the server.
It is possible to attach up to 9 TB per instance using a maximum of 24 local disks.

Downsides

Local disks are only available as SSD, so HDD local disks don't exist.
You can't attach local disks to certain virtual machine types.
Whilst you can be confident that your disk is always encrypted using GCP-supplied keys, you can't supply your own encryption keys.

Local disks are best suited if you need fast scratch disk or cache.

Other storage types

In this final section we introduce the remaining two storage solutions which don't fit well into any of the categories mentioned above.

Memorystore

Memorystore is GCP's fully managed service for Redis and Memcached. If you are interested in using either Redis or Memcached but don't want the hassle of managing them, Memorystore is the storage option for you.

Benefits

Memorystore provides sub millisecond latency for lightning fast data access.
It is easy to migrate from an existing Redis or Memcached instance using the lift-and-shift functionality.
Memorystore natively integrates with GCP's IAM service for robust security.

Downsides

Memorystore's capacity is limited to 300 GB.
If cost is a significant factor, then you should consider hosting your own Redis or Memcached instance.

BigQuery

GCP's data warehouse solution, BigQuery can handle data storage as well as analysis. It isn't the most common storage solution, and is often used alongside other storage options for data analysis only.

Benefits

BigQuery is great for data analysis - all your data can be in one place so you don't have to worry about data ingestion.
It provides many specialised services, including BigQuery ML for machine learning and BigQuery BI Engine for interactive data analysis.
You can also stream data directly into BigQuery from other storage solutions.

Downsides

Cost is relatively high, so it may not be the most efficient long-term solution.
It's not recommended to use BigQuery for data storage unless you want to use its analysis capabilities as well.

One of GCP’s customers using BigQuery is The Home Depot, who use this tool to keep track of their stock across 50,000+ items and 2,000+ locations.

Conclusion

GCP provides a massive variety of data storage tools for all sorts of use cases. Selecting the best option can be difficult, since there are so many tools to choose from. My best advice is to start from your data type, this will often significantly narrow down the available possibilities. Afterwards, considering the location of your data (e.g. zonal or global), required consistency and capacity will help you decide. Make sure to consider cost as well - you can use GCP's pricing calculator to get a sense of how expensive different solutions would be.

I hope that you're this leaving this blog post with a clearer idea of the different storage options GCP provides. Remember that if none of the tools listed here fits your use case, you can always build your own solution on Compute Engine. Best of luck with your cloud journey!

Useful links

Intro to storage options

Cloud SQL for AutoTrader

Cloud Spanner for Pokemon Go

Cloud Bigtable for Dow Jones

Firestore for The New York Times

BigQuery for The Home Depot

Pricing calculator

Images from Unsplash

How AnotherDay and Theodo built a big data geospatial analytics platform

Wed, 02 Mar 2022 00:00:00 GMT

AnotherDay’s Cascade platform provides businesses with the ultimate risk and intelligence management platform. With risk assessment at the heart of so many international standards, many business suffer from competing standards for tracking and mitigating risk. Often, large spreadsheets, unwieldy slide decks and or folders with hundreds of sensitive intelligence files become impossible to manage with challenges such as version control and centralisation. Cascade provides both a single centralised platform to collate intelligence and assets and also a powerful analytics platform to make sense of this data and to extract effective risk mitigation strategies.

Today, an often used strategy for asset risk quantification is to run accumulations. This is where a radius is selected and for each of the businesses assets, the total value of all the other assets lying within that radius from the asset is calculated. A business can then look for where most value has accumulated, and therefore where they have incurred the most financial risk. This form of analysis is often used in areas such as terrorism modelling.

The Cascade platform needed the ability to run these sorts of accumulations across large datasets, in excess of 1 million assets. As these modelling runs were potentially long-running it was also desirable that these tasks could be run asynchronously from a queue.

AnotherDay worked with Theodo to build this functionality into the Cascade platform. As experts in React, Django and AWS the task was to make this end to end functionality production ready in 5 weeks.

The modelling flow begins by uploading the assets into the Cascade platform. The data was provided in CSV format which was uploaded to S3 via the frontend. This triggered a lambda function which performed validation and cleaning of the data before leveraging the aws_s3 extension for RDS PostgreSQL to transfer the data directly into RDS from S3.

Asset upload flow

To store the geospatial data in RDS, Theodo used the PostGIS extension for PostgreSQL. A second RDS instance was provisioned alongside the existing database for Cascade in order to protect the main database from the expensive modelling queries.

To trigger an accumulation modelling run, a user creates an accumulation entity in the frontend specifying parameters such as radius. Saving this entity triggers a JSON message sent to an SQS Queue with the model parameters and run details.

Celery (an asynchronous task runner) was provisioned in Fargate, providing a worker pool that subscribes to the SQS queue. The model task is picked up by a celery worker which then runs the model SQL against the PostGIS database. A naive implementation of the model scales as N² with an analysis of 20,000 assets taking 15 minutes. Leveraging the r-Tree index provided by PostGIS provides blazing fast performance, lowering the model runtime to 15 seconds.

Running an accumulation model

The top 50 results (in terms of accumulated value) are saved to the main database for display in the frontend. Theodo used the Mapbox API to build an interactive results map, displaying the numbered results and the accumulation radius. For the rest of the results the aws_s3 extension is used again to output a CSV to S3 which allows the entire result set to be downloaded.

Focusing on system design tailored to the business domain, and leveraging AWS & open source tools, sets AnotherDay up to create a new standard for risk management. Theodo ensured best practices were followed to create a state-of-the-art platform with many exciting avenues to explore around integrating live intelligence data into the modelling process.

Architecting a Modern Monorepo with NX and Turborepo

Tue, 01 Mar 2022 00:00:00 GMT

What is a Monorepo?

A monorepo is a consolidated repository containing the source code of multiple projects, which are commonly managed by independent teams and also often share common packages.

Not just Code Collocation

Recently a common software development practice is to have a full stack JavaScript application, this leads to many developer teams believing they have a Monorepo "Because all of my code is in the same repository".

A monorepo is far more than just code collocation, we'll address the benefits, drawbacks and showcase two industry leading Smart Build Systems.

Benefits of a Monorepo

There are a number of important benefits to using a Monorepo (with a Smart Build System).

Consistency - It's possible to share UI modules, documentation and other such shared packages
Visibility - Everyone in the organisation can see all the code, improving cross-team collaborations and communication
Build and Development Caching - Some tools like NX and Turborepo allow for efficient rebuilds to allow rebuilding only changed packages
Single source of truth - As all changes to the repo are atomic, there won't be a situation where one team is working with outdated legacy code
A single CI/CD pipeline - There is no need for multiple pipelines in each application as we can handle it in one centralized place
Deduplication of node modules - Node modules are the computational equivalent of a black hole already, so any deduplication or help here is always appreciated

Downsides of a Monorepo

There aren't many downsides to using a Monorepo, but there are some that are worth mentioning:

Slower IDE Performance - For large Monorepos it's possible to have a slower IDE performance due to the increased number of files, this can slow down imports, intellisense and other IDE features
Learning Curve - Some tools to manage a Monorepo add technical complexity, especially during the setup

How do I make a Monorepo?

Making a Monorepo for a fullstack JavaScript code base is easier than ever, the easiest solution with no additional dependencies is NPM Workspaces, and the other two solutions we will cover are NX and TurboRepo.

Workspaces - Keep It Simple Stupid

Workspace is a generic term that refers to a set of features in the NPM cli which adds support for managing multiple packages from the root of the repository, NPM workspaces allow you to deduplicate node modules and run commands against multiple projects at the same time.

Below is an example structure of a repository using NPM workspaces

├── package.json
└── packages
    ├── package-a
    │   └── package.json
    └── package-b
        └── package.json

# ./package.json
{
  ...
  "workspaces": ["./packages/*"]
}

Workspaces provides two basic features, as for the most part Workspaces are just code collocation with the ability to run projects from one terminal in conjunction.

Running commands against multiple packages from the top level repository, e.g. npm run test --workspaces
Deduplication of node modules

Ideal for 💡

Small projects
You don't need a smart build system
You don't want to add additional dependencies

NX - Advanced and Powerful

NX is one of the most advanced build systems available and offers some incredibly powerful features to take your codebase to the next level.

Some of the most acclaimed features of NX are as follows:

Integrations and plugins for many platforms (including a VSCode Plugin)
Rich plugin ecosystem 🚀
Ability to build and test only what you need to
Automatic project dependency management
Vastly faster builds with NX Cloud
Distributed task execution
Code generation tools

Create a Next.js NX project

Create a new NX project, with a Next.js application

npx create-nx-workspace@latest

The NX CLI will then ask you which template you would like to use to create this new application, for our case we will choose "Next.js".

Create a shared React library

One common usage of NX is to create a shared library which can be used by multiple apps. Shared libraries are created in the libs directory by default.

nx g @nrwl/react:lib ui-shared

Running this command generates a new ui-shared library, which can be used by multiple apps. The beauty of this pattern is that if you change ui-shared, it will rebuild other apps that use it, whereas, if you make a change in next-app NX won't rebuild ui-shared.

Here you can see the process of importing our new shared library, and using it in our Next.js app.

Affected Builds

When you run nx test next-app, you are telling Nx to run the next-app:test task plus all the tasks it depends on.

For a small project this is okay, however for a large project you really only want to run the tests for the files and changes you've made to the app. To handle this, NX has the affected command. nx affected --target=next-app will figure out which tasks to run, and only run those tasks.

To visualize these "affected libraries", use the nx affected:graph command, in this example below we've added a second-shared lib which is also used in next-app.

Now we're going to make a change to second-shared, as this is used by next-app, and next-app-e2e you should expect that they should both be re-built.

As epected you can see that the next-app and next-app-e2e projects are affected, but the ui-shared lib is not.

Incremental Builds

Once you've got an established NX Monorepo, you will likely have many applications and libraries. It is common that you will only be editing a small subset of these applications at once, Incremental builds allows NX to only rebuild a small subset of the libraries, which results in much better performance.

This then works closely with NX Cloud. With NX Cloud you and your team mates share the same cache, allowing you all to benefit from improved performance.

The above chart is from NX and highlights the performance differences between normal builds, and incremental builds (both cold and warm).

Running the Nx incremental build having already cached results from a previous run or from some other coworker that executed the build before. In a real world scenario, we expect always some kind of cached results either of the entire workspace or part of it. This is where the teams really get the value and speed improvements.

Ideal for 💡

Medium to massive projects
You want all of the fancy modern features you can get
You don't mind dealing with some complicated looking nx.json files and tuning them to your liking
You want a battle hardened, community supported platform with industry backing

Turborepo - The New Kid on the Block

Turborepo is a recently open sourced project acquired by Vercel, the company which brought us Next.js.

Turborepo's goal is to take what's great about other build systems such as Lerna, and NX, whilst shipping it in a small simple package, which works hard to stay out of your way.

Setup

Setting up Turborepo is as easy as it gets, just run npx create-turbo@latest.

This sets up an example project, with a web and docs apps, and a shared ui package. Unlike NX there is a lot less boilerplate and this resembles a typical project structure you'd see with a "normal" Monorepo, this makes Turborepo much more approachable to a monorepo newcomer.

Affected

Turborepo has its own flavour of affected from NX, both work very similarly, however Turborepo doesn't produce an interactive graph like NX does, Turborepo has a basic graphviz image export.

Other Features

Turborepo has a few other features:

Cloud caching
Parallel execution
Profile in the browser

Ideal for 💡

Vercel users, Remote Caching works great
Small to medium projects
You don't mind losing some features of NX

Feature Comparison

Below is a feature comparison for Workspaces, NX, and Turborepo.

Feature	Workspaces	NX	Turborepo
Node Module Deduplication	✅	✅	✅
Run Commands Across Projects	✅	✅	✅
Shared Libraries	✅	✅	✅
Profile in Browser	❌	❌	✅
Dependency Graph	❌	✅	🚧 (non interactive)
Incremental Builds	❌	✅	✅
Cloud Caching	❌	✅	✅
Code Generation	❌	✅	❌
Distributed Task Execution	❌	✅	❌

Conclusion

Hopefully you now appreciate the importance of a Smart Build System within a monorepo. As much as I'd recommend giving both NX and Turborepo a shot, if you don't have time to experiment with both, I recommend trying NX.

Turborepo has most of the features that NX has, but NX does that and more, in a more performant package. Given the current state of Turborepo due to it's recent inception, I'd rather have a NX monorepo that has been battle hardened which you know won't let you down.

However, with Turborepo's recent backing of Vercel, I expect NX to face some strong competition in 2022, and as you all know, competition brings innovation.

References and Reading Materials

NX Documentation
Turborepo Docs
NX and Turborepo comparison (Disclaimer, it's from NX)
monorepo.tools Fantastic Resource! 🔥

Bringing the client closer to their product using Contentful

Tue, 15 Feb 2022 00:00:00 GMT

You need to make sure that your website content keeps up-to-date with changing designs and client requirements. The trade-off? This often impacts developer time. Small changes, like adjusting an image or a piece of text, seem like a 10-minute job. In reality, they take up more capacity as they still need to go through the full development processes (at Theodo these include code review, deploying, functional review, validation, etc. to ensure quality and avoid regressions).

Content Management Systems (CMS)

Utilising a CMS means that you can create, manage and modify the content of a website without needing any code changes. A developer has to integrate the CMS using its API, however, once this is done, anyone with the correct permissions can adjust the content. This means that content changes now become much simpler and the client can benefit from up-to-date content, as well as saving developer time over the longer term.

Introducing Contentful

Contentful goes beyond CMS; not only does it allow you to plug adjustable content into your website, it also incorporates additional features to optimise your content.

As a quick introduction, Contentful is a content management platform with a very comprehensible Content Delivery API. It is available in the most popular programming languages (e.g. Javascript, Python), a great help when you are working on a variety of projects utilising different stacks. Overall, some of the best advantages include the ease of setup, clear and practical JavaScript SDK, and Contentful’s intuitive platform.

Before delving into the advantages and disadvantages of Contentful, let’s introduce the technical side: how to integrate it into your code.

Code example

Setting up

The first, and essential, part of using Contentful is to create an account on their platform https://www.contentful.com/. Assuming that this has all been done, we can now turn our eye to integrating it with our code.

If you don’t yet have Contentful installed and are using Node.js, you just need to install the package using:

npm install contentful

Next, you need to initialise the client. To do this, you will need an API key and space ID (some quick steps can be found here). Although optional, you can also specify an environment. We’ll talk about this later, but it is another key advantage of using Contentful.

var client = contentful.createClient({
  space: '<space_id>',
  accessToken: '<access_token>',
	environment: '<environment_name>',
});

That’s all you need! Once Contentful has been set up, you can now use it throughout your project.

Fetching content

There are lots of ways of retrieving content. You can use a specific entry id to retrieve a specific item, however, you can also retrieve groups of content. One of the most useful tools to retrieve content is being able to retrieve by content type. Within the Contentful platform, you can create ‘content types' that define the types of data that you will be adding to your website. For example, you might have a simple type such as colour which will just have a short piece of text (i.e. a title) or you might have a more complicated type such as author which will require a name, a description, an image, etc. To fetch this content type (e.g. fetch all the author types), you need to call:

const authorResponse = await client.getEntries<Author>({
	content_type: 'author',
})

Where Author, in this case, is the type defined by Contentful which needs to be imported. The response can then be used anywhere you need. Within Typescript, the Author type would need to be defined. Although this can be done manually, it is more efficient to use an automatic type generator such as this one.

Optimisation

Contentful can also be used to adjust and optimise your content. A key example is with images. There are many options available (see documentation), however one of the most interesting ones I came across was the ability to change the quality of the image. By reducing the quality of the image, the content can be rendered more quickly on your page, avoiding delays. Once you have retrieved the image URL from Contentful, all you need is to adjust it and specify the quality:

const imageUrlWithAdjustedQuality = `${imageUrl}?fm=jpg&q=${quality}`

With Contentful, the quality is a percentage and can be set to be between 1 and 100.

These are just two examples of what you can do with Contentful. I would highly recommend exploring their documentation and having a look at implementing it yourself too.

Environments

Earlier, I mentioned that we can set the environment when initialising the client. Contentful has a wonderful feature where you can have content in different environments. This means that changes made in Contentful don’t need to directly impact your production website. Contentful does this in a very similar way to Github where you can have different branches (e.g. master, staging), create and destroy branches. The key process to releasing a new branch is as follows:

Create a new branch (copied off of the master branch) where you make all your content changes
Once the changes are complete, point master at this new branch
You can either delete the old master branch or you can keep it in order to revert to an older version if needed

Although this can be done through the terminal, it is highly recommended to use the Contentful platform as it is quite straightforward and can avoid unwelcome surprises.

Key advantages of using Contentful

A key feature that singles out Contentful compared to other CMS is the ability to manipulate images. It does so by not only adjusting the format and size of your image, but also by enabling cropping and adjusting the quality you would like to have - a great optimiser of loading speed.

While CMS is often thought of as being able to integrate content such as images, videos and text, Contentful takes it a step further. While the general page layout is predetermined, its content can be completely set through Contentful. This is because text can be done in markdown, giving more flexibility to what can be displayed.

In the code example, we had a look at fetching all items within a content type. If all the data is useful to you, then this is great. However, you may only want to fetch a specific group of items. For example, if you have a Book content type, you may only want to fetch items with the field genre corresponding to 'Mystery'. This is done using relational queries, e.g. using the API call:

https://cdn.contentful.com/spaces/my-space-id/entries/
    ?content_type=books
    &fields.genre.sys.contentType.sys.id=genre
    &fields.genre.fields.label[all]=Mystery

However, the key benefit of using Contentful compared to other CMS is the simplicity of integration.

Disadvantages and a word of warning

Speed

Fetching all your content from Contentful seems like a dream come true, no more hardcoding text and images, freeing up developer time and making sure that your content is up-to-date. However, this does come with a warning. Fetching all the content whenever a page is loaded can be very slow. The more data you fetch from Contentful, the longer your loading time. However, this does not necessarily need to be an issue depending on which framework you are using. Contentful used alongside NextJS is one of the best solutions. NextJS uses static site generation (SSG), meaning that your content does not need to be refetched whenever you load a page - a significant time-saver. However, this does mean that if you make changes to Contentful, you will need to rebuild the site in order to see these changes. Rather than SSG, you can also look into using Incremental Static Regeneration (ISR) which uses static-generation per page rather than the entire site and would avoid the need to rebuild the entire site when a change is made on Contentful.

Field type

When fetching content types, there are also limitations on the field type we can filter on. One of the types that it is not possible to filter by is Array<Link>. A further note to consider about the Link type is the issue of depth. Links are a great way to model the relationship between content items, however as the depth increases, this can cascade into a circular relationship. By default, you will only access the top level, however, if you do need a higher depth, it is important to understand the possible consequences of expanding the maximum depth.

Other points to consider

Another point to consider is that Contentful cannot be installed locally for local development. To work on your project with your content fetched from Contentful, you will need to have an internet connection. However, as this is also the case for other CMS, it is not a strong disadvantage. One of its key challenges is its cost. At about $489/month for a team, Contentful can be quite costly, especially for smaller projects.

Conclusion

Overall, Contentful is a great way to optimise time for developers and ensure that the client has control of their content. However, you have to be careful about when to use this as projects with a low budget and with small amounts of content might not need such a powerful tool. Another key consideration is whether you are using NextJS on your project. If you are not, you might want to think about switching over to make sure that your content is loaded quickly and efficiently.

More documentation: https://www.contentful.com/developers/docs/references/content-delivery-api/

Logging uWSGI errors and alarms to Sentry

Wed, 02 Feb 2022 00:00:00 GMT

If you already use Sentry to manage your application errors and uWSGI as a WSGI server for your Python project (such as Flask or Django), you are one step away to improve your monitoring setup in a matter of minutes.

It is important to do so: even if you have Sentry configured for your app, if the WSGI layer that is running it fails you won't get any Sentry issue about it.

With uWSGI there are some events that you want to watch for, and triggering a Sentry issue is one easy way of fitting this into your existing error management flow.

Such events include:

when a uWSGI worker is killed because it is serving a request for an excessive long time (the "harakiri" behavior)
when the listen queue of the uWSGI socket is full
when a segfault happens
or any non-applicative log line that you want to monitor

How to integrate uWSGI with Sentry

Unbit (the company behind uWSGI) wrote a plugin exactly for this use case!

Make sure that a libcurl package is installed on the machine that runs uWSGI, for instance:

apt-get install libcurl4-openssl-dev

Install the plugin before running uWSGI:

uwsgi --build-plugin https://github.com/theodo/uwsgi-sentry

NB: this is a fork of the original plugin that makes it work for current versions of uWSGI and Sentry, I will update this article if it is merged into the main repo.

Configure the plugin in your uWSGI INI file:

[uwsgi]
plugin = sentry

; Defines what to do when the "logsentry" alarm is triggered
alarm = logsentry sentry:dsn=https://your@project.sentry.io/dsn,logger=uwsgi.sentry

; Alarm for listen queue full
alarm-backlog = logsentry

; Alarm for segfault
alarm-segfault = logsentry

; Alarm for harakiri (alarm-log uses regexp to match log lines)
alarm-log = logsentry HARAKIRI ON WORKER

You can find more customization options in the Alarm subystem docs.

Conclusion

Deploy this change to your server and bam, you're done!

After this, you could reflect back on what other component of your architecture you are missing alerts on.

Bonus: if you are loving Sentry as we do, here are some tips on how to use Sentry effectively.

React Hooks and Tips to Avoid Useless Component Render Applied on Lists

Mon, 31 Jan 2022 00:00:00 GMT

A few weeks ago, I encountered children list rerender issues on the project I was working on. In this article you will learn :

how I debugged a react performance issue
why virtualization is not always suitable for list rendering issues
what is memoization
how to memoize react components and functions with react hooks and React.memo() to prevent a component from re-rendering.

Overview on the performance issue

I’m working on a platform to purchase orders from suppliers. Each order is composed of clothing items. Here is a typical example of the interface.

Let’s take a very simple example: I am ordering from my supplier Theodo, producing some top quality shirts. I want to order their famous shirts in three colors (white, red and blue). On the left there is a sidebar where you can select a card, here there are three cards, one for each color. On the right there is a form with the information relative to the selected card.

When another card is clicked, the form values on the right is now displaying the information related to the new selected item.

For the new year, a big order has to be done containing more than 600 items. In this situation, the list takes a huge amount of time to load, or doesn’t even load at all. And if I’m lucky enough to have the 600 items displayed inside the sidebar, when another item is clicked on, I’m also waiting... This performance issue is terrible in terms of user experience and needs to be fixed!

First, I need to make sure whether the performance issue actually comes from React. To do so, the user interaction can be recorded with the Google devTools performance tab. When JavaScript is executed it is accounted in scripting. Here, scripting is taking most of the time. I can investigate more on what is happening.

I then used the React profiler to see if this long scripting task comes from long react render times or useless component re-renders. The profiler is included in the React Dev Tools Chrome extension. It records renders, why your components rendered and for how long.

Long story short, you mainly need to understand that a colored bar means that your component rendered, grey bar means it did not.

It did render

It did not render

And guess what? When an item in the list is clicked on, all the items contained in the list re-rendered 🥵 ! When the list is only 3 components long, it is ok to recompute but with 600 components, each click becomes quite expensive.

For example here is the result in the profiler when the second card in the sidebar is clicked on. Basically, everything re-rendered.

Profiler view when an other card is clicked on, everything renders

So the React rendering step needs to be lightened. What options do we got to perform this?

Find a way to reduce the number of items that need to re-render

It can be done thanks to virtualization. Virtualization is a technique to compute in the DOM only the elements located inside the user's viewport. When the user scrolls, elements will be added/deleted in the DOM. A few librairies already implement this for us https://github.com/bvaughn/react-window.
Find a way to avoid useless re-renders

Why virtualization is not the right solution here?

Virtualization is a good option when your rendering phase is taking a long time. But here, it’s the re-renders that take time. If re-renders issues are solved, displaying 600 items only once is not an issue.

Moreover, the average user is making orders of about 50 items without a powerful computer. Virtualization is not very useful for short lists.

Finally, the height of each item card is variable (when a field is filled, the item card will grow), and the number of item is variable (the list can be filtered, items can be created or deletes). Windowing librairies are not well suited for variable items size or number, some custom functions have to be added to make it work. For the user this can lead to a laggy scroll. And re-renders issues won’t be solved.

So let's solve those useless re-renders!

Why do components re-render when another card is clicked on?

The structure is the following: OrderItemsSelection is the parent component, it contains the Form section and the Sidebar. The Sidebar itself has children: the SidebarCards (as many as items).

OrderItemsSelection has a state, selectedItem, and a state setter, setSelectedItem for the selected item id (thanks to a useState hook).

OrderItemsSelection passes selectedItem as a prop for the Form section and the Sidebar.

OrderItemsSelection also passes setSelectedItem setter to the Sidebar, Sidebar passes the setSelectedItem setter to its SidebarCard children to be used on card click.

So when the red shirt card is clicked on, a new state is setted with setSelectedItem, the selected card is now the red shirt card.

As the selected card is a state of the component OrderItemsSelection, it re-renders: a component state change triggers a re-render. Therefore all its children also re-render: the Form section, and the Sidebar. As the Sidebar re-renders, so are its children: the white/red/blue shirts cards.

In "components" tab the value attributed to Hook 1 is displayed (which caused a render)

Component updating schema

But in fact what do we truly need to render in the sidebar? If we take a closer look, the blue shirt card does not need to be rendered as none of its props actually change.

How to prevent re-renders

What is memoization?

Let’s take a look at Wikipedia definition:

In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.

So here, the Sidebar is a parent component of each card component. So we want to cache the blue card component computations because it has unchanged prop values, ie memoize this component. We can find more information about component memoization in the React documentation:

React documentation:

React.memo is a higher order component. If your component renders the same result given the same props, you can wrap it in a call to React.memo for a performance boost in some cases by memoizing the result. This means that React will skip rendering the component, and reuse the last rendered result.

Memoize a React component with React.memo()

Ok so if everything goes as planned I just have to add the memoization to SidebarCard component, and React will magically understand that if props values didn’t change, the component does not need to re-render.

      export default SidebarCard;

Before memoized component

      export default memo(SidebarCard);

After memoized component

But it would be too easy won’t it be?

Expectations

Reality

When we launch the profiler and focus on the blue shirt card:

We can see that components are memoized thanks to (Memo) just after component name, but we can also see that React considered that props did change for the last card: sidebarItems and onSetSelected. What’s interesting is that it recognized that the isSelected boolean did not change. It’s a boolean so the comparison is only made based on the boolean value. With functions, array and object, it’s different. Even if the objects have the same properties, they are considered equal only if it refers to the exact same object.

const Laura1 = { name: "Laura" };
const Laura2 = { name: "Laura" };

Laura1 === Laura2;
// => false

Functions, as well as arrays and objects are stored by reference into memory and not by value. We call these non-primitive data types as opposed to primitive types (string, boolean, number, undefined or null) which are stored by value. Here is a great article to understand all about it.

Memoize a function with useCallback()

This means that a new function onSetSelected was created earlier, which has the same value as the former but not the same reference. They are two different entities even they look alike.

onSetSelected is a function that is called when a card is clicked on to set the new selected item id.

onSetSelected is passed from the sidebar to the card here:

const Sidebar = ({
  onSetSelected, // => received here
  sidebarItems,
  selectedItemId,
  allowMultiSelect = false,
}: SidebarProps): JSX.Element => {
return (
  <div>
    {sidebarItems.map((sidebarItem, index) => (
      <SidebarCard
        sidebarItem={sidebarItem}
        positionIndex={index}
        onSetSelected={onSetSelected} // => passed here
        isSelected={selectedItemsId === sidebarItem.id}
	    />
     ))}
  </div>

onSetSelected is defined in the Sidebar parent:

const OrderItemsSelection: React.FunctionComponent = () => {
  const {
    initialValues,
    submit,
    selectedItemId,
    setSelectedItemId,
  } = useItemsSelection();

  // => onSetSelected definition
  const onSetSelected = (index: number) => {
    setSelectedItemId(index);
  };

  return (
    <Formik initialValues={initialValues} onSubmit={submit}>
      {({ values, validateForm }: FormikProps<ItemsSelectionValues>) => {
        return (
          <>
            <Sidebar
              sidebarItems={values}
              onSetSelected={onSetSelected} // => passed here
              selectedItemId={selectedItemId}
            />
            <ItemsSelectionForm
              initialValues={values.items[selectedItemId]}
              selectedItemId={selectedItemId}
            />
          </>
        );
      }}
    </Formik>
  );
};

So when the OrderItemsSelection component re-renders, a new onSetSelected is generated. The blue shirt card receives a new function (weirdly same looking as the previous one). So it has to re-render.

We want to avoid recomputing the function, as nothing has changed except the re-render in OrderItemsSelection. In short, we need to memoize the function. Fortunately, React has already implemented this for us. A big welcome to the beautiful useCallback hook which will memoize our function!

const onSetSelected = (index: number) => {
      setSelectedItemId(index);
      };

Before without useCallback

    const onSetSelected = 
      useCallback((index: number) => 
      {
        setSelectedItemId(index);
      }
    , []);

With useCallback

Now it’s time to try if our onSetSelected is still a guilty prop.

Profiler with memoized onSetSelected function

Component updating schema

Nice shot, onSetSelected is not guilty anymore, but we still have one targeted prop: the sidebarItem .

Use React.memo() comparison function to tell React when a component should re-render

Same protocol, as done before, where is sidebarItem defined?

const Sidebar = ({
  onSetSelected,
  sidebarItems,
  selectedItemId,
  allowMultiSelect = false,
}: SidebarProps): JSX.Element => {
return (
  <div>
    {sidebarItems.map((sidebarItem, index) => ( // => defined here
      <SidebarCard
        sidebarItem={sidebarItem} // => passed here
        positionIndex={index}
        onSetSelected={onSetSelected}
        isSelected={selectedItemId === sidebarItem.id}
      />
    ))}
  </div>

When Sidebar renders it recomputes the map, and each sidebarItem is recomputed. Since it’s a brand new object and as explained earlier, with a new reference, so it is not equal to the previous object even if it contains the exact same values. So how can we tell our card component to take into account only deep equality of previous and current sidebarItem? Once again, React team has already a solution right in the React.memo doc:

By default it will only shallowly compare complex objects in the props object. If you want control over the comparison, you can also provide a custom comparison function as the second argument.

Nice, the next step is to define a function that can compare the SidebarCard props in a customized way.

Stringify is not suitable for deep comparison

We could have thought to compare the stringifying version of the two objects, but it’s not the best idea as when we have equal properties not in the same order, it would have returned false.


    const Laura1 = { name: "Laura" };
    const Laura2 = { name: "Laura" };
    JSON.stringify(Laura1) === JSON.stringify(Laura2);
    // => true


    const Laura1 = { name: "Laura", age: 25 };
    const Laura2 = { age: 25, name: "Laura" };
    JSON.stringify(Laura1) === JSON.stringify(Laura2);
    // => false

We can use a deep equality. A deep equality is an equality based on what the object owns. Lodash provides isEqual to evaluate deep comparison.

const Laura1 = { name: "Laura", age: 25 };
const Laura2 = { age: 25, name: "Laura" };

JSON.stringify(Laura1) === JSON.stringify(Laura2);
// => false
isEqual(Laura1, Laura2);
// => true

Therefore, the isEqual lodash function is a wiser choice to compare the props.

export default memo(SidebarCard);

Before custom comparison

    
    const compareProps = (
      propsBefore: SidebarCardProps,
      propsAfter: SidebarCardProps
    ) => {
      return isEqual(propsAfter, propsBefore);
    };
    export default memo(SidebarCard, compareProps);

With custom comparison

Drum roll...

🥳 Yeah, got it! So for the 600 items list, when the second card is clicked on, here is the after memoization:

What if we only use memoize custom comparison without useCallback?

The isEqual on the onSetSelected function would have worked, the card wouldn’t have re-rendered. But it was a nice learning in case you have a function which is triggering a re-render.

Is there a difference between useMemo and useCallback?

useMemo is usually used for variable memoization. But you can use a useMemo to memoize a function.

React documentation:

useCallback(fn, deps) is equivalent to useMemo(() => fn, deps).

Why not always use memoization?

Memoizing is not free, it costs memory and the comparison cost. It adds complexity to your code so it will need more efforts to read/to refactor.

Conclusion

Thanks to React.memo() HOC and memoization hooks, useless re-renders were avoided! I hope you've learned a bunch of things!

Go further:

React: Fantastic Hooks and How to Use Them

React-Virtualized: Why, When and How you should use it

Ivan Akulov’s thread on re-renders

Use React.memo() wisely

The Complete Guide to useRef() and Refs in React

Mastering React-admin Resources to Improve Your App Performance

Tue, 18 Jan 2022 00:00:00 GMT

Let me tell you the story of how I improved my react-admin app runtime loading performance from 1 minute to 1 second.

I work at Theodo on a project with many websites. To configure and manage all of them, we had an administration tool (let's call it AdminV1). But AdminV1 was old: it was an Angular project coded with coffee.script and jade, and was not very maintainable. And the product was difficult to use: there were too many menus, UX was bad, it was difficult to onboard new administrators on the application.

The team decided to create a brand new application with modern technologies: AdminV2, with react-admin, a framework for building administration applications.

Everybody was happy: for developers, it was easier to implement new features. With a few lines of code, they could create tables to administrate users, app configurations and privileges. For Product Owners and administrators, the design was more ergonomic and it was easier to manage their applications.

We migrated many features from AdminV1 to AdminV2 and added new administration tools. Sponsors were delighted. But sprint after sprint, AdminV2 became slower to start. After some months, users had to wait more than one minute to access the app. It was painful to develop new features on AdminV2 because each hot reload took 1 minute, and users were frustrated by waiting so long to access the application.

We started to investigate and found that by deleting some Resources in our application, the speed at runtime increased significantly

But what are react-admin Resources ?

"<Resource> components are fundamental building blocks in react-admin apps. They are strings that refer to an entity type.", according to react-admin documentation.

To illustrate, on my project, we need to administrate users of websites. We have to create a resource "/user", associated with components:

UserList (for table view)
UserEdit (to edit an user)
UserCreate (to create a new user).

import UsersList from "components/Users/list";
import UsersEdit from "components/Users/edit";
import UsersCreate from "components/Users/create";

const UserResource = () => (
  <Resource
    key="/user"
    name="/user" // endpoint of the API to call to get users
    create={UsersCreate}
    list={UsersList}
    edit={UsersEdit}
  />
);

React-admin is in charge of storing this information. When the user goes on /user URL, react-admin displays the list, edit or create view and manages all the logic to get, edit and create users.

Sounds like magic, not helping us understand why creating many Resources degrades the performance.

I dived deeper into the react-admin code to understand how the framework creates resources and links them with components.

A <Resource /> component is declared like this:

const Resource = (props: ResourceProps) => {
  const { intent = "route", ...rest } = props;
  return intent === "registration" ? (
    <ResourceRegister {...rest} />
  ) : (
    <ResourceRoutes {...rest} />
  );
};

React-admin Code

Your intent can be either registration or route:

ResourceRegister's purpose is to dispatch an action that saves your Resource into a redux store. Each time you define a <Resource /> component in your app, react-admin will register your Resource's props into the redux store:
- the name, which is the URL of the resource and also the API endpoint to fetch your entity data
- which views are available to manage the entity. For our user entity, available views are list, edit and create

dispatch(
  registerResource({
    name,
    hasList: true,
    hasEdit: true,
    hasShow: false,
    hasCreate: true,
  })
);

React-admin Code

ResourceRoutes is rendered when a user wants to access the URL associated with the Resource. The URL (which is also generally the endpoint of the API) is stored in the Resource's name prop. ResourceRoutes renders all <Route /> components associated with each view defined in the Resource. For our Entity "user", it will be:

const ResourceRoutes = () => (
  <ResourceContextProvider value={name}>
    <Switch>
      <Route
        path={`${basePath}/create`} // Create view
        render={(routeProps) => (
          <WithPermissions component={CreateUser} {...routeProps} />
        )}
      />
      <Route
        path={`${basePath}/:id`} // Edit view
        render={(routeProps) => (
          <WithPermissions component={EditUser} {...routeProps} />
        )}
      />
      <Route
        path={`${basePath}`} // ListView
        render={(routeProps) => (
          <WithPermissions component={ListUser} {...routeProps} />
        )}
      />
    </Switch>
  </ResourceContextProvider>
);

React-admin Code

For example, if an administrator connects to a react-admin app and wants to see a list of users, the framework will load resources like this:

This allows calling the API only when the user goes on the associated Menu and to not call all the endpoints available in the application at runtime.

Ok, so what's going on with AdminV2 ?

In AdminV2, we defined our Resources like this:

const RootComponent = () =>
{
  const resources = getResources()
  return (
    <AdminContext {...props}>
      <AdminUI>
        {resources}
      </AdminUI>
    <AdminContext />
}

GetResources returned all Resources in our application: user Resources and other 600 Resources (yes that's a lot).

So what's happening when a user access AdminV2 ?

First, over 600 <Resources /> components are rendered
Then, each of these <Resources /> dispatches an action to store their data in redux store (the "registration" intent seen above)
React-admin redux reducer gets these 600 actions and for each of them, it stores the data in the redux store
Finally, when the reducer has updated the store with all resources action payload, the home page of AdminV2 is displayed

These 4 steps take more than a minute. Eventually, the user has gone for a coffee, otherwise, he left the app and will never come back again

How to avoid this 1 minute of loading time?

We thought about diverse solutions:

1 - Load only resources to which the user has access

On AdminV2, to access each menu, the user needs to have the associated privilege. If the user has access to one menu, we don't have to load all resources, but only the resources used in the menu.

This solves the problem for regular users with few privileges. But Developers and Product Owners have access to all menus.

2 - Patch react-admin to modify Resource component

We thought of modifying directly the react-admin Resource component, to modify the dispatch action. Instead of dispatching one action for each Resource, we could dispatch one action for multiple resources. This way, we don't have to update the state 600 times.

But this solution is not very maintainable: it implies updating the patch each time we update react-admin.

3 - Lazy load Resources

We don't have to create every Resources at the initialization of the application. An administrator will never access all AdminV2 menus and, even if he wanted to, in most cases, he doesn't have all menus accesses. We need to load Resources only when the user needs them. So we decided to create them when the user accesses the associated url. We modified our RootComponent like this:

const RootComponent = () => {
  const [resources, setResources] = useState([]);
  const location = useLocation();

  useEffect(() => {
    const resourcesToAdd = getResources(location.pathname)
    setResources([...resources, ...resourcesToAdd]);
  }, [location.pathname])

  return (
    <AdminContext {...props}>
      <AdminUI>
        {resources}
      </AdminUI>
    <AdminContext />
  )
}

We modified the getResource function so that it takes the current URL in parameters and returned the Resources associated with the URL.

For example, if an administrator wants to access the user administration menu (URL '/user'), getResources will only return

<Resource name="/user" list={UserList} edit={UserEdit} create={UserCreate} />.

The RootComponent will render the new Resource:

The first intent will be registration and the Resource will be stored in redux state
The second intent will be route. React-admin will create the Routes for user entities and the administrator will access the user management menu

In order not to save resources each time the user goes back on the same URL, we added a filter:

const RootComponent = () => {
  const [resources, setResources] = useState([]);
  const [loadedResources, setLoadedResources] = useRef([]);
  const location = useLocation();

  useEffect(() => {
    const resourcesToAdd = getResources(location.pathname).filter(
      (resource) => !loadedResources.includes(resource.name)
    );
    setResources([...resources, ...resourcesToAdd]);
    setLoadedResources([
      ...loadedResources,
      ...resourcesToAdd.map((resource) => resource.name),
    ]);
  }, [location.pathname]);

  return <AdminUI>{resources}</AdminUI>;
};

Let's sum up

Before, the user had to wait more than one minute to access the app. Then the navigation was quite quick.

Now, the user access AdminV2 in one second. And the navigation stays quite quick!

As developers, when using "black box" framework like react-admin, we do not always try to understand how it really works. By taking the time to explore the source code of these frameworks, by understanding what libraries they use, how they are implemented, we can save our time later and help us detect this kind of performance issue sooner.

That is the conclusion of AdminV2 performance story. And developers developed happily ever after and added a lot of features to AdminV2!

TypeScript: How Type Guards Can Help You Get Rid of 'as'

Wed, 12 Jan 2022 00:00:00 GMT

In TypeScript (TS), the "as" keyword is widely used when manipulating types. However, you should employ it with caution, as it does not provide any guarantee on the real types of your objects and could generate unexpected bugs. In this article, I will go into more detail about “as” and explain how to avoid it by using type guards, which represent a safer alternative.

Before I started working at Theodo Lyon, I had only used Vanilla JavaScript and HTML in my previous web development experiences. So when I discovered TypeScript (TS) earlier this year, it seemed like a miraculous solution for writing safer code! It was when I joined a team working on a web app using React jointly with TypeScript. React is a free and open-source "JavaScript library for building user interfaces" developed by Facebook, while Typescript is a programming language developed as a superset of JavaScript that adds optional static typing to it. In other words, defined variables are assigned a specific and permanent type at creation, hence securing their future use.

A few months ago, we started noticing many obscure bug occurrences with no obvious explanation. Also, the investigation was not satisfactory due to the lack of error logs: usually, the user would simply land on a blank page instead of the desired one.

One of the most common causes was that some objects suddenly became undefined while that was not an option for TS since they had different types. For example, we encountered error messages such as :

However, most of these occurrences would not even leave any error message and could lie undetected for a long time. Hence the investigation was even more painful!

Through our analysis, we found out that most of these errors were linked to the usage of the as keyword in TS.

For example, suppose we have an API used to retrieve books data, and a type Book defined as:

export interface Book {
    id: number;
    author: string;
	publisher?: string;
}

When I fetch the book of id 5:

const book = fetch("baseUrl/get-book?id=5");

the received object will be of type unknown. But since I know the expected return type, I can force TS to use the type Book by using as:

const book = fetch("baseUrl/get-book?id=5") as Book;

But how does "as" work exactly?

This keyword is a Type Assertion used to inform the compiler to consider an object as a different type from the one inferred. The as assertion does not perform any transformation on the data stored in the variable and is removed at compilation. TS even warns us about this in the official documentation:

Furthermore, it only checks that the assertion is feasible. For example, you can not use it to turn a string into a number: "TypeScript only allows type assertions which convert to a more specific or a less specific version of a type." ("TypeScript Handbook")

By digging deeper in the source code, the as does not perform any checks on the object but only serves as a static typing tool:

Screenshot extracted from source code typescript.js

Usually, we used to employ the as keyword in order to handle data coming from external sources. Examples of such sources could be APIs or imported excel files. Since the content was not known in advance, the type could not be inferred, so we had to use as to let the compiler understand the right type.

However, using this keyword in this case creates a false confidence feeling. The purpose of as is neither to protect nor to ensure typing, its main use being informing us about the hypothetical type of the object. For instance, in our previous example, I could fetch a book without knowing that the API has changed its definition of the type Book. Indeed, we could imagine that the type of the id attribute has changed from number to string, when we write:

const book = fetch("baseUrl/get-book?id=5") as Book;

const nextBookId = book.id + 1;

TS thinks that book.id is still a number, which could result in errors.

So, once we figured out the purpose of as, we started by implementing what would later turn out to be a quick fix. We kept the as keyword, but added checks right after the problematic occurrences to make sure that the object’s attributes were not undefined:

const book = fetch("baseUrl/get-book?id=5") as Book;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
if (book.author !== undefined) {
    const authorLastName = book.author.split(" ")[1];
}

This could prevent the problems cited above. However, it was not sustainable in the long-term, because we had to remember to perform the check every time we used an object whose type had been defined with as. Moreover, an "ignore" annotation was needed since the type was clear for our code linter eslint, which considered that the type defined by as was correct.

Yet, some of the bugs persisted, until one day, we discovered type guards through a workshop. Little did we know, at the time, that they could solve most of our problems!

But first, what are type guards?

A type guard is a function that allows you to narrow the type of an object to a more specific one by performing certain checks. Hence, after calling the function, TypeScript understands the new type of the object.

typeof and instanceof are two built-in examples of such type guards:

typeof only returns one of the following: "string", "number", "bigint", "boolean", "symbol", "undefined", "object", "function":
```
typeof 90 === ‘number’

typeof "abc" === ‘string’
```
instanceof checks whether an object is of a specific type, for example:

new Date() instanceof Date === true

However, instanceof does not work with TypeScript interfaces, which are what we usually use.

Since neither operator could solve our issues, we decided to build our own brand new type guards. Let us consider for instance an element object whose type is unknown being actually of type Book. A working function allowing us to verify this assertion would be:

const isBook = (element: unknown): element is Book =>
    Object.prototype.hasOwnProperty.call(element, "author")
    &&
    Object.prototype.hasOwnProperty.call(element, "id");

A custom type guard takes an element in input and returns a type predicate, which in our example would be element is Book. Now, whenever we handle an element that could be a Book, we can use this function to check its type. Also, once the variable verifies the checks, TypeScript interprets its type as a Book allowing us to access its attributes:

const gift: unknown = getGift();

if (isBook(gift)) {
    /*
    * In this if-block the TypeScript compiler actually
    * resolved the unknown type to the type Book
    */

    read(gift);
    // read needs an argument of type Book

    return gift.author;
}

The other advantage of type guards is that we can unit test them, preventing problems:

Here is an example with jest (the JS testing framework we use):

it("should return true if the element is a book", () =>{

    const element1 = { author: "alpha"};

    expect( isBook(element1)).toBe(false);

    const element2 = {"id": 2000};

    expect( isBook(element2)).toBe(false);

    const element3 = { author: "alpha"; "id": 2000};

    expect( isBook(element3)).toBe(true);
})

Still, as we can see above, our type guard is incomplete. An object such as { author: 1904; "id": new Date()} would be wrongly considered to be of type Book.

To avoid this mistake, the function should also check the attributes’ types. Hence, We can modify it as follows:

export const isBook = (element: unknown): element is Book =>
    /*
    * We can not use "in" because element
    * has type "unknown"
    */
    Object.prototype.hasOwnProperty.call(element, "author") &&
    typeof element.author === "string" &&
    Object.prototype.hasOwnProperty.call(element, "id") &&
    typeof element.id === "number";

However, this fix will not be effective since the element type would still be unknown, preventing us from accessing its attributes directly:

No worries, the deliverance could result from 2 options:

either we split the function into 2 type guards, verifying the attributes' existence first, then their types each time we handle types

or, writing up a generic function hasAttributes that could check any attribute’s actual presence, permitting us to scale up the types’ checking. An example of such usage could be:

export const isBook = (element: unknown): element is Book =>
hasAttributes(element, ["author", "id"]) &&
typeof element.author === "string" &&
typeof element.id === "number";

Finally, we can access the attributes because we have checked their existence first.

Wait a second, how can we define `hasAttributes`?

The function takes an object of unknown type and returns a type predicate depending on the attributes given as an input to the function. In other terms, it needs to be a "generic", defined for instance in the following way:

export const hasAttributes = <T extends string>(
    element: unknown,
    attributes: T[]
): element is Record<T, unknown> => {
    if (element === undefined || element === null) {
        return false;
    }
    return attributes.every((attribute) =>
        Object.prototype.hasOwnProperty.call(element, attribute)
    );
};

The type Record<T, unknown> means that the element object in input does possess the wanted attributes but their values are of type unknown.

This is all great, but can we automate it a bit?

It is true that many type guards libraries already exist, like typecheck.macro, but we have not had much luck using them. The main pain point with this library for example was that it makes our unit tests fail, we could not make it work with jest for the present time. Also, some other libraries require us to change all our type declarations to fit theirs in order to use them.

However, since we now know how important type guards are and how much they can ease the developer experience, their automation could be feasible but is at the time being still a challenge.

Conclusion

As we have seen, we have nearly stopped using the as assertion to state objects’ types, since it is very error-prone and rarely necessary. In contrast, type guards have proven to be amazing tools, making our code cleaner, more stable, and more robust. In most cases, they can replace all of your as usages. As a matter of fact, my colleagues and I were not able to come up with any good occurrence which could not be replaced with a type guard, however hard we tried. Indeed, anytime you are faced with a type incoherence (a wrongly typed library, external data etc.) you can resolve it using a type guard.

If you still are not convinced about their enormous potential, try them for yourself. Write your first type guard and get ready to see your developer experience improved!

Why You Should Encrypt Your Disk, and How To Do It on Ubuntu with a Dual Boot

Mon, 10 Jan 2022 00:00:00 GMT

Why use disk encryption?

It goes without saying that your computer can contain sensitive information about you or the organization you are working for, such as:

personal pictures
password or credit card numbers saved in your browser (and so on, access to your drive, important online accounts such as administrative ones, bank...)
emails
administrative documents (come one, we all store scans of our id or our social security number somewhere...)
code or projects
financial or operational spreadsheets

Why is it dangerous?

Stolen identity

Your social security number, your id, or other administrative documents can be used to open bank accounts or credits in your name!

Fraudulent banking transaction

Stolen card numbers are very often sold to groups that will use them to perform online payments on the internet. The point is you can not realize that your number has been stolen until they use it. Then it is too late.

Blackmailing

You or your organization can be the victim of blackmailing. You for your personal pictures, for example and your organization for sensitive information, blueprints or source code.

Cyber-attack

If the stolen machine belongs to a company and contains employee credentials (VPN access for example), they can be used as an entry point on your company network, for a bigger cyber-attack such as ransomware.

Source code leak

If the source code of one of your projects leaks, it makes it easier for hackers to find security breaches in your product or application. It would also make it easier for rival companies to copy your product.

How is it technically possible and how to prevent it?

Your password does not protect your files and data

Even without your password, someone with access to your hard disk can mount it on another operating system (for example by using an Ubuntu live USB key, a very common thing) and access some of the files it contains! It is an easy hacking practice. Nowadays, shady organizations buy second-hand laptops or hard drives to collect sensitive information about their former owner and sell them.

In this video, the French Youtuber Micode shows us how he managed to collect sensitive data and files from second-hand hard disks he bought online:

Also in January 2021, Kaspersky published on their blog an edifying study on data exposed in second-hand devices: **90% of the second-hand devices contain sensitive data!**

"Kaspersky’s Global Research and Analysis Team (GReAT) has examined security in secondhand devices. [...] An overwhelming majority of the devices the researchers examined contained at least some traces of data — mostly personal but some corporate — and more than 16% of the devices gave the researchers access outright. Another 74% gave up the goods when the researchers applied file-carving methods. A bare 11% had been wiped properly."

Sarah Pike on Kaspersky's blog, Jan.29 2021

Encryption to the rescue!

Encryption is a way to prevent that, as an encrypted file can not be understood without the key to decode it.
On Ubuntu, you can encrypt your home folder after installation to protect the files inside. This is a very convenient practice, however it comes with major trade-offs:
- Encryption is handled at the application level. As a consequence, it can be slow, especially when you need to copy or download lots of files (when installing software or project dependencies for example).
- Your computer also contains files outside your home directory that can also provide information on you, such as the installed library or softwares for example. Also, some programs keep information in temporary files or log files. And any information can be temporarily stored in you swap space.
Full disk encryption leverages those two trade-offs as it provides faster kernel-level encryption, and encrypts your whole Ubuntu installation (except the boot partition).

Tutorial part: How to set up full disk encryption on Ubuntu

This article will not cover dual boot set-up, and assume that you either already have a fully-functional dual boot on your device, or your device is ready for installing Ubuntu on dual boot, or you don't need it.

If you want to set up a dual boot, or enable Windows 10 disk encryption alongside Linux disk encryption, I recommend you switch to this longer, more complete tutorial by Mike Kasberg.

(optional) Back up your data if you want to restore it after installation

Disclaimer: you will still have to reinstall your apps, but they should restore their current settings after reinstallation.

(optional): remove all node modules, venv, composer dependencies, ...

First, you might not want to backup folders like node_modules or other project libraries as they can be very long to copy due to the very large amount of file they contain. Moreover, it is easy to reinstall them after your reinstallation.

You can remove them manually or use the following command lines to remove all the folders with a given name in your home directory. Feel free to adapt them to the project you have.
```
#in your home directory
cd ~
#delete all your node modules
find . -name 'node_modules' -type d -prune -exec rm -rf '{}'
#delete all your python venv (supposing you named them venv)
find . -name 'venv' -type d -prune -exec rm -rf '{}'
```
copy your home to an external drive. You can copy your entire home, or select only some folders you want to keep. Here are a few files and folders you might want to keep if you want to keep your apps settings and preferences:
- .ssh
- .config
- Documents
- Downloads
- ...
This list is not exhaustive and might depend on the apps you use

Reinstall Linux with full disk encryption

For this you will need a live CD/usb key of Ubuntu.

This tutorial has been tested on Ubuntu 20.04 LTS.

If you want to keep your dual boot

Unfortunately, the "Install Ubuntu alongside windows 10" option in the Ubuntu installer does not provide full disk encryption. We will need to set it up ourselves. If you don't need a dual boot you can skip to the next section

Select "Try Ubuntu" as you will need to use a few tools to prepare your disk before the installation

Define 2 disk partitions: one that will be encrypted and contain logical volumes for swap and root and one for boot
1. Open GParted
1. Delete your previous Ubuntu partition if you have one (To do so select the partitions and click on the red button in the top bar. Pay attention not to delete any windows partition). Your disk should look like something like (some unused space):
1. Then create 2 new partitions:
  1. A partition for boot:
    - size = 500Mio
    - file-system = ext4
    - label = boot
  2. A partition that will be containing swap volume and your Ubuntu installation:
    - size = <the size you want for your Ubuntu, including swap>
    - file-system = unformated
    - label = root
  At this point, your disk should look like something like this (2 new partitions instead of the unused space)
Note the name of your root partition (here /dev/nvme0n1p6) and boot partition (here /dev/nvme0n1p5). Apply your changes and then you can close GParted!
Encrypt your main partition and split it in two logical volumes for swap and root.

For this, open a terminal and use these commands:
1. get superuser rights: sudo su
2. encrypt your boot+swap partition: cryptsetup luksFormat <your_root_partition> (e.g. in my case: cryptsetup luksFormat /dev/nvme0n1p6 ). Type YES to validate and choose a passphrase (that you will need to unlock your disk at boot time)
3. open the encrypted partition: cryptsetup luksOpen <your_root_partition> ubuntu (in my case: cryptsetup luksOpen /dev/nvme0n1p6 ubuntu)
4. create a physical volume to access this partition: pvcreate /dev/mapper/ubuntu
5. create a volume group to contain your future logical volumes (root and swap) inside this physical volume: vgcreate vgubuntu /dev/mapper/ubuntu
6. Inside this volume group, create 2 logical volumes:
  - one for swap (here I use a size of 4G, feel free to adapt your swap size according to your hardware and needs. Usually swap should be around half your ram): lvcreate -n swap -L 4G vgubuntu
  - one for ubuntu (root), that will use all the remaining space: lvcreate -n root -l +100%FREE vgubuntu
7. Optional: If you want to format your partitions using CLI, you can format them right now. Otherwise, this can be done using the Ubuntu installer:
  - mkswap /dev/mapper/vgubuntu-swap
  - mkfs -t ext4 /dev/mapper/vgubuntu-root
At this point, your terminal should look like this:

Now that all your partitions and logical volumes are ready, you can start your Ubuntu installation (finally)!
1. Open the installer, and follow the installation process until you get to this screen:
Pick "Something else" and Continue
1. You should see all your disk volumes. Here we need to tell Ubuntu which volume to use for which use.
  - Select /dev/mapper/vgubuntu-root, click Edit and pick the following settings (check format if you didn't do it with CLI):
  - Select /dev/mapper/vgubuntu-swap, click Edit and pick the following settings:
  - Select the boot partition you created in step 1-c (/dev/nvme0n1p5 in my case), and pick the following settings:
2. Click install now to start the installation.
Your partition changes should look like this (if you formated them in the installer):

You can finish the installation process.

⚠️ If you want to restore your home folder, you need to create a user with the same name and password as your original one!

⚠️ Do not restart your device at the end of installation! At this point, Ubuntu is installed but it does not know that your disk is encrypted. Thus it is not able to boot yet. We will configure it in the next part.
Setup /etc/crypttab, for Ubuntu to know that then encryption is encrypted and how to unlock.

To do so, click "keep trying" at the end of the installation, and open the terminal. We will mount your new Ubuntu partition and use chroot on it to edit this /etc/crypttab and apply the changes.
1. Get superuser rights: sudo su
2. Get your encrypted partition UUID ( ⚠️ not the PARTUUID), it will be needed to set up the crypttab:
  
  blkid <your_encrypted_partition> (e.g in my case: blkid /dev/nvme0n1p6).
  
  Keep it for later.
3. Mount ubuntu: mount /dev/mapper/vgubuntu-root /target
4. Mount your boot partition: mount <your_boot_partition> /target/boot (i.e. in my case mount /dev/nvme0n1p5 /target/boot
5. use chroot on the /target and mount everything:
```
for n in proc sys dev etc/resolv.conf; do mount --rbind /$n /target/$n; done

chroot /target`

mount -a
```
6. Now, you can edit /etc/crypttab on the new installation (I'll do it with nano for the example):
  
  nano /etc/crypttab
  
  Copy/paste this inside the new crypttab (don't forget to replace your encrypted partition UUID!):
```
ubuntu UUID=<your_encrypted_partition_id> none luks,discard
```
7. Last but not least: apply the modifications of the crypttab:
  
  update-initramfs -k all -c

🥳You're done! Congrats, you can exit your live CD and restart your computer!

If everything worked as expected, you will be asked your passphrase to unlock your disk when you start Ubuntu.

If you don't need a dual boot

If you don't need a dual boot on your computer, you can also erase your entire disk and install Ubuntu on it. This process is much easier. On the Installation Type screen:

select "Erase disk and install Ubuntu".
Before continuing click on the "Advanced Features" button and select "Use LVM" and "Encrypt the new Ubuntu installation for security".

You can go along with the Ubuntu installer then.

(Optional) Restore your data

Log in on your new installation and create a new user (superuser). Log out from the user you want to restore and log in on the new superuser. Then you can copy/paste your home or the files you want to restore to their proper emplacement on the home of the user you are currently not logged in.

⚠️ As said earlier, for the restoration to work properly some files will need both of your users (former and new ones) to have the same name and the same password! You can change your password later.

Once it is done, you can log back on the other user and delete the other superuser if you do not need it.

Conclusion

By following this tutorial, you learned about various fields:

About security

Hardware security is an important part of cyber-security for organizations as well as for individuals. A device can be stolen, lost, or as you will probably throw or sell it at some point, and you want to make sure that the data inside it is not accessible if it happens.
Your password is not required to access some of the files inside your computer if someone has physical access to your hard disk. That can happen while you own it, or even after. To prevent it, you can encrypt your device.

About device encryption

You can use Luks to encrypt a physical volume and then split it in logical volumes for your different needs (swap, root, optionally a dedicated home volume).
Ubuntu can handle an encrypted physical volume if it knows it from /etc/cryptab, and then use its logical volumes as regular partitions.
You can trick Ubuntu installer by unlocking an encrypted partition using cryptsetup luksOpen. Generaly speaking, cryptsetup is a tool to encrypt partitions and unlock them for your OS to use them (including the Ubuntu installer)

About Ubuntu

You can copy a home folder from an Ubuntu installation and restore it on another one if you create a user with the same name and password. Unfortunately, libraries are not backed-up.
GParted is a tool on Ubuntu to manage your physical partitions.
Home folder encryption is not a good practice as the encryption is managed at the application level so it can often be slow. Moreover, you protect only your files. Full disk encryption is handled at the kernel level, hence it is much faster. It also ensures that absolutely no information can be extracted from your device, including your libraries, your swap space...

To go further

If you want to know more about the process of encrypting-decoding data, and when to use file encryption:
- A video explaining How Does Full Disk Encryption Work?
- A video explaining How Does Individual File Encryption Work?
The article on Kaspersky blog about data collection in second-hand device: https://www.kaspersky.com/blog/data-on-used-devices/38610/

Do not hesitate to browse Theodo's blog for more interesting articles or visit Theodo's website to learn more about us !

The React Testing Library Guide I Wish I Had

Tue, 21 Dec 2021 00:00:00 GMT

I would like to share with you a great library I have found, which makes front-end tests fun to write!

With this article, you will be able to write serenely your first tests with React Testing Library. To have a deeper understanding of the library you can use the documentation.

Is it really useful to write front-end tests?

The problem is that as your app grows you cannot be sure that everything still works properly each time you add some new code. Seems quite obvious, right? Then, why do so few people write front-end tests?

When I asked some people about it, they answered: "we don't have enough time", "it's not really useful"...

The first thing you should know is that, yes, at the beginning it seems that tests are not worth the time. But as your app grows it will be more and more difficult to implement them and your app will be more and more unstable. Then, you should implement tests from the beginning of a project: it is easier and it will be timesaver later.

The second thing to know: tests are definitely useful. A quick anecdote to convince you: the first test I have ever written spotted a bug.

Then it's time to learn how to test!

Why use React Testing Library?

Now that you are - I hope - convinced that you should test your front-end code, what library should you use?

My first advice is: don't use snapshots, as this article explains. To sum it up: snapshots are difficult to understand for developers and then it doesn't really test the app the way users use it.

When I implemented front-end tests on my project I chose React Testing Library. React Testing Library allows you to test by simulating human interactions with the UI, while being quite independent of the implementation.

Indeed, the closer the test is to the actual app usage, the more relevant it is.

As a bonus, using React Testing Library neatly can actually improve the accessibility of your app (See an article about accessiblity improvement to understand why).

How to test with React Testing Library?

The principle of React Testing Library is to render a component and then simulate some interaction with it, and then finally compare the result with the expectation.

1. Render a component

The first thing you have to do is to get the component you want to test. It can be a whole page or isolated components. The best thing to do is to find the right balance between testing a page with a lot of components, which can be long because of nested components but is closer to the user experience, and testing components, which is faster but sometimes less relevant.

Anyway, you should never test already tested components and behavior. For example, components from libraries are usually tested and you shouldn't test them again.

The easiest way to get a component is to use directly the render method:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";

test('My first test', async () => {
	render(<MyComponent />);
}

It is also possible to wrap the render function in a provider to mock some features, such as the Redux store or the internationalization:

import * as React from "react";

import {
  RenderResult,
  render as rtlRender,
  RenderOptions as RtlRenderOptions,
} from "@testing-library/react";
import { IntlProvider } from "react-intl";
import { Provider } from "react-redux";
import flattenMessages from "services/i18n/intl";
import frMessages from "translations/fr.json";

import rootSaga from "../redux/sagas-for-tests";

const defaultMessages = flattenMessages(frMessages);

const render = (
  ui: React.ReactElement,
  {
    initialState,
    store = storeCreator(initialState), // custom function to create the store
    messages = defaultMessages,
    ...renderOptions
  }: RenderOptions
): RenderResult => {
  const Wrapper: React.ComponentType = ({ children }) => {
    return (
      <Provider store={store}>
        <IntlProvider messages={messages} locale="fr">
          {children}
        </IntlProvider>
      </Provider>
    );
  };
  return rtlRender(ui, { wrapper: Wrapper, ...renderOptions });
};

With a custom render function like this you can render your component with a mocked state:

import { mockState } from '__fixtures__/state';
import { render } from 'utils/test-utils'; //custom render
	import MyComponent from "../MyComponent";

test('My first test with a mock state', async () => {
	render(<MyComponent />, { initialState: mockState });
}

Now that you can display a component, let us play a bit with it.

2. Simulate user interactions

To interact with the elements of a component you will need a way to access them. You have several ways to do this. You can, for example, query elements by text, label, placeholder, alternative text, or role. Regardless of what attribute you use you have three main methods to get an element.

Generally, I prefer to use getBy, which is the most simple one. But if you need to get the element asynchronously you can use findBy. It can be useful when you wait the results of an asynchronous method for example. If you want to check if an element is not in the DOM, I advise you to use queryBy, which returns "null" if the element was not found.

You should know that all of these methods will throw if more than one element matches, though you can overcome this limitation by using getAllBy, findAllBy, or queryAllBy to get an array.

Have a look at how you would query for a button reading “press me”:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";

test('Get button test', async () => {
	const { getByRole } = render(<MyComponent />);
	const pressMeButton = getByRole("button", { name: "press me" });
}

Once you have your button you can interact with it, by simulating a click on it:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Click on button test', async () => {
	const { getByText } = render(<MyComponent />);
	const pressMeButton = getByText("press me");
	user.click(pressMeButton);
}

Another classical way to interact with an element is when the user types something in an input field. This interaction can also be easily simulated. With this code, for example, you would simulate a user typing "some random text" in an input field labeled "myInputField':

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Fill input test', async () => {
	const { getByText } = render(<MyComponent />);
	const myInputField = getByLabelText("myInputField");
	user.type(myInputField, "some random text");
}

Sometimes you will try to get an element, but you will have an error message saying that it was not found. To help you debug that, React Testing Library has a wonderful method...

3. Debug

It is really easy to debug with React Testing Library. Indeed, it provides a method, debug, to display the entire DOM at a specific moment.

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Debug fill input test', async () => {
	const { getByText, debug } = render(<MyComponent />);
	const myInputField = getByLabelText("myInputField");
	debug(); //displays the DOM with an empty input
	user.type(myInputField, "some random text");
	debug(); //displays the DOM with the input filled with "some random text"
}

4. Examine the result

Now that you are able to render a component and interact with it the way the user would do, it is time to check that everything works as expected.

To do that you just have to use the expect method, for example, to check that an input has the expected text content:

import { render } from "@testing-library/react";
import MyComponent from "../MyComponent";
import user from "@testing-library/user-event";

test('Debug fill input test', async () => {
	const { getByText } = render(<MyComponent />);
	const myInputField = getByLabelText("myInputField");
	user.type(myInputField, "some random text");
	expect(myInputField).toHaveTextContent("some random text");
}

5. Mocks

Finally, you should know that you can mock the different API calls and hooks using Jest to fully test your app. Using mocks enables you to test highly complex functions. Moreover, it enables you to have control over the data.

Conclusion

Testing the front-end with React Testing Library improves the quality of your code itself, it enables you to test the user interactions and above all, it is quite intuitive. Now you no longer have any excuse to avoid testing your front-end!